Granny's Programming Pearls
"inside of every large program is a small program struggling to get out"
JavaRanch.com/granny.jsp
The moose likes Java in General and the fly likes JTable challenge Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Android Security Essentials Live Lessons this week in the Android forum!
JavaRanch » Java Forums » Java » Java in General
Bookmark "JTable challenge" Watch "JTable challenge" New topic
Author

JTable challenge

Titus Campbell
Greenhorn

Joined: Sep 17, 2002
Posts: 10
Ok... here is what I need.
I want to build a browse screen for a log file generated by cash registers. Jtable containing some summary information... when a row is selected, detailed information is presented in an opposite panel.
Problem starts with the fact the this log file is huge... I don�t want to store all of this information in memory... my java app would probably choke.
Second problem... this log file has been stored as raw binary data in which a record my contain fields with different formats. Records also have varying lengths... impossible to do anything with data stored like this. So, I parse the data into �Record� objects and write these out to an intermediate file via ObjectOutputStream.
I started with this..
class RecordTableModel extends AbstractTableModel {
private ObjectInputStream data;

...
public Object getValueAt(int rowIndex, int columnIndex) {
//read record# rowIndex form intermediate file
//return appropriate field from Record object
}
...
}
Problem is, I can�t figure out how to effeciently read random objects from the intermediate file. No way to seek to a given record within the file because Records have varying lengths.
Any ideas...
Ernest Friedman-Hill
author and iconoclast
Marshal

Joined: Jul 08, 2003
Posts: 24183
    
  34

Write an index file containing the offset of each object in the main file.


[Jess in Action][AskingGoodQuestions]
Titus Campbell
Greenhorn

Joined: Sep 17, 2002
Posts: 10
Ok... looks like what I am after is a file-based implementation of a hashmap. Surely someone has already written this... I found one on the net but looks like it created a new file for every object stored... can't have that.
I could write one myself if I could figure out how to do two things...
-get the current file pointer position from an ObjectOutputStream object
-seek to a given offset from an ObjectInputStream
So far I haven't been able to figure this out...
Does anybody know of an existing file-based hashmap implementation OR a way to perform 'seekFilePointer' and 'getCurrentFilePointer' operations from an Object(Input/Output)Stream object..
Thanks
Jim Yingst
Wanderer
Sheriff

Joined: Jan 30, 2000
Posts: 18671
Hmmm, using serialized objects here makes this problem rather hairy, IMO, as OOS and OIS aren't really designed for the sort of info you need here. Lesse though...
I think that in order to get random access to serialized objects like this, you may well need to open and close an ObjectOutputStream for each separate object written. This generates header info for each separate object, whihc will be expected if you try to read an object from an arbitrary file position. The problem is that you will need to close eachof these streams (of GC will close them for you) - and by default the close() will also close the underlying FileOutputStream (or whatever you're using). Ugh. You can create a custom FilterOutputStream class which overrides close() to do nothing, so that you can insulate the inner FileOutputStream from the OOS's close() method. And the same for reading from an input stream. Or, you may find it's useful to develop your own record storage format, so that you don't have to deal with OOS/OIS and their expectations about headers and need to close. Or maybe you won't actually have to use separate OOS/OIS for each object, but don't be surprised if you do.
Maybe it's best to use ByteArrayOutputStream to create an intermediate byte[] for each object. Then you can insulte the file from each oos.close(), since the oos.close() just closes the baos. Then you write the byte[] to the file, with no close().
To find the offset of each object, I can think of a couple ways. OOS doesn't have the info you need, but you can wrap it around another stream object that does. One way is using NIO classes to get a link between an InputStream and the position of the file being written to:

This hasn't been tested at all, and doubtless contains some errors. But hopefully the idea is good for something...
Alternately (if you can't use JDK 1.4), you can create a custom FilterOutputStream which counts how many bytes have been written, and use that to track file position.


"I'm not back." - Bill Harding, Twister
Jim Yingst
Wanderer
Sheriff

Joined: Jan 30, 2000
Posts: 18671
Another option: create a separate file for every 100 objects (or make the 100 a configurable parameter). File data-0 contains records 0-99, file data-1 contains 100-199, etc. Now when the table needs to display row 3737, it's easy to look up file data-37, and reading 100 objects to get one isn't [i]too bid a deal). But in practice, the table is going to try to look up a bunch of consecutive entries, for example looking up rows 0-33 in succession. So use a HashMap of SoftReferences to create a cache, and store each group of records you read as a List of 100 objects. Then getValueAt() checks the cache first, and only reads the appropriate data file if that data isn't in the cache. The SoftReferences allow CG to prune the cache as needed.
Yet another option: use a database to take care of all the indexing, and store each serialized object as a BLOB. If you've already got a DB available, this is pretty easy. You can also use something like mckoi or hsql to create a relatively simple DB which can be easily packaged as part of an application, making it easy to install on new machines.
Titus Campbell
Greenhorn

Joined: Sep 17, 2002
Posts: 10
Thanks for the reply Jim...
I'm probably going to go in the direction you've mentioned and write my own serialization and deserialization process and use RandomAccessFile to write the records out... I can get the file offset from RandomAccessFile easily..
Check this code out though... Amazingly it works, although for some reason I don�t really trust it. I only wrote it out of desperation.

[ December 10, 2003: Message edited by: Titus Campbell ]
Titus Campbell
Greenhorn

Joined: Sep 17, 2002
Posts: 10
Hey Jim
DB is unavailable and unfortunatly... the format of the log file cannot be changed. The software module that logs is a 3rd party app which we don't even have the source code to. Better yet, the company that wrote it doesn't even exist anymore. Hardware and Software constraints are a pain!!!
Thanks for the ideas... I'll post the outcome.
Jim Yingst
Wanderer
Sheriff

Joined: Jan 30, 2000
Posts: 18671
Check this code out though... Amazingly it works, although for some reason I don�t really trust it. I only wrote it out of desperation.
The question I'd have is whether you can do a random-access read from a file like this. When you open the ObjectInputStream it will read the required header info from the beginning of the file - then you want to skip to a later point in the file and read from there. You can use methods like skip()/seek()/position() for this ( on the underlying stream/channel) - but you can't control whether the ObjectInputStream has done some sort of lookahead, reading part of the next object (which you don't care about) into a buffer of some sort (even just one or two chars), waiting until you call readObject() to read the rest of that object. When you try to the jump to another part of the file, the OIS may get very confused. Even if the OIS doesn't get confused, I think you're relying on a particular implementation of OIS, which does not have guaranteed behavior. I don't know about OIS really, but I've had problems in the past when I have nested InputStreams and I try to skip() an underlying stream - because the outer InputStreams sometimes do readahead when you're not expecting it; there's no rule against this. With output streams it's simpler - if you call flush() you guarantee that everything written so far is written to all underlying streams. And you know they can't "write ahead" because they wouldn't know what to write yet. But input streams are not so predictable in general...
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: JTable challenge
 
Similar Threads
Populate a JTable
Problem with JTable
JTable editing
JTable issues
Populate a JTable