Reading in db file

[James Clinton]

I'm currently playing around with the RandomAccessFile (RAF) class and api (which I have little experience of until now) and am having some interesting *unexpected* results I could do with some help on.

I've sucessfully managed to read in the header details of the file which are cookie [257], each rec len [159] and num fields [7].

  
RandomAccessFile file = new RandomAccessFile(FILE_NAME,"rw");
file.seek(0);
int cookie = file.readInt();
int eachRecLen = file.readInt();
short numFieldsInEachRecord = file.readShort();
System.out.println("BLAH..");

However, when I read the next 159 bytes to read the Field Names I seem to get too much information back as it spilts on to the first record.

  
byte [] buf = new byte[159];
file.readFully(buf);
String s = new String(buf);
System.out.println(s);

OUTPUT
------
♦name location @ ♦size ♦ smoking ☺ ♦rate ♦date
♣owner Palace smallville

please help.

[Anton Golovin]

Hi, James! You may wish to use the following code:

  
byte [] buf = new byte[159];
file.readFully(buf);
String s = new String(buf, "US-ASCII");
System.out.println(s);

to remove those extra characters. The reason the extra characters are there is that in some situations, new String(byte[]) constructor fails to properly convert bytes into the right characters; I suspect that this is cause for some automatic failures, because it seems this wrong conversion depends on the language configuration of each individual computer - it works ok on one and does not work on another. On mine it does not work, and I think it is becuase the language of the OS (US-EN) and the preferred program language (RU) make the constructor fail. Spacifying the character set in the constructor removes this problem.

[James Clinton]

Hi Anton,

unfortunately this didnt make any difference - the output was exactly the same. anymore ideas?

[Anton Golovin]

 // reads a record in thread-safe or unsafe manner
    private String[] readRecord(int recNo, boolean threadSafe) throws
            EOFException, IOException {
 
        if (reader == null) { // checks if a valid read stream is avail.
            throw new IOException("Can not read from data file.");
        }
        // calculates starting read position
        final long startReadPosition = this.position(recNo);
        // stores value of a record's textual fields' length
        final int recordLength = header.getRecordLength();
 
        int flag = 0; // will store record's flag value
        // will store record's fields as a byte array
        final byte[] temp = new byte[recordLength];
 
        if (threadSafe == THREAD_SAFE) { // needs thread-safe data access
 
            // locks the writer for thread safety
            synchronized (writer) {
                reader.seek(startReadPosition); // positions reader's file pointer
                flag = reader.readByte(); // reads flag
                // returns null for deleted records
                if (flag == header.getDeletedFlag()) {
                    return null;
                }
 
                // will store record as bytes
                reader.readFully(temp); // reads in record as bytes
            }
        }
 
        // does not care about thread safety
        if (threadSafe == THREAD_UNSAFE) {
 
            reader.seek(startReadPosition); // positions reader's file pointer
            flag = reader.readByte(); // reads flag
            // returns null for deleted records
            if (flag == header.getDeletedFlag()) {
                return null;
            }
 
            // will store record as bytes
            reader.readFully(temp); // reads in record as bytes
        }
 
        // converts record as bytes into record as a String
        // by using the schema-defined charset
        final String tempRecord = new String(temp, Schema.CHARSET);
        // stores record values - data and metadata
        final String[] convertedRecord = header.convertRecord(tempRecord);
        // log the read:
        // DataUtilities.log("IOStream: Read record # " + recNo);
        return convertedRecord;
    }

This is my method. If it does not work, another method may be to convert each individual byte into the corresponding character and to put the characters one by one into a StringBuffer... very cumbersome, though.

[James Clinton]

Technically, there is nothing different between our code..sooo frustrating.

Regarding your final comment, are you suggesting that I read in each byte 1 by 1 and do some conversion until 159 bytes (i.e a record) has been read? If so I dont see how this would make a difference to the result.

thanks.

[Anton Golovin]

Well, here's what it is:

Before reading a record, the file pointer must be positioned at its start. Before that can be done, though, one should read in the datafile header - the field names, etc... After the header is read, you will know the offset for the first record, and you will be able to read it. Here's the code:

 
 // private constructor for the class
    private Schema(String dataFile) throws FileNotFoundException, IOException {
 
        try { // sets up schema-reading stream
            File dbFile = new File(dataFile);
            schemaReader = new RandomAccessFile(dbFile, "r");
            schemaReader.seek(0);
        } catch (FileNotFoundException fnfEx) { // data file missing
            throw new FileNotFoundException("Can not find file " +
                                            dataFile + ".");
        } catch (IOException ioEx) { // read insuccessful
            throw new IOException("Can not read file " + dataFile + ".");
        }
 
        // reads data file identifier
        int fileCookie = schemaReader.readInt();
        // makes sure the file passed in is a data file
        if (this.fileCookie != fileCookie) {
            throw new FileNotFoundException(
                    "Selected file not a data file or data is corrupted.");
        }
 
        // sets the passed in file as the data file
        this.dataFile = dataFile;
 
        try { // reads number of fields
            this.numberFields = schemaReader.readShort();
        } catch (IOException ioEx) { // handles io errors
            throw new IOException(
                    "Can not determine number of fields in data file.");
        }
 
        try { // reads schema
 
            fieldNameLengths = new byte[this.numberFields];
            fieldNames = new String[this.numberFields];
            fieldLengths = new byte[this.numberFields];
 
            for (int i = 0; i < this.numberFields; i++) {
                // reads length of a field name
                fieldNameLengths[i] = schemaReader.readByte();
                // array holds byte representation of a field name
                byte[] temp = new byte[fieldNameLengths[i]];
                schemaReader.readFully(temp); // reads field-name bytes
                // converts bytes to a textual field name
                try {
                    fieldNames[i] = new String(temp, CHARSET);
                } catch (UnsupportedEncodingException unsuppEncEx) {
                    throw new IOException(
                            "Can not convert byte data to text using " +
                            CHARSET + " charset.");
                }
                // reads lengths of data fields (columns)
                fieldLengths[i] = schemaReader.readByte();
                // calculates length of textual fields in a record
                this.recordLength += fieldLengths[i];
            }
 
        } catch (IOException ioEx) { // handles io errors
            throw new IOException("Failed to read schema from data file " +
                                  dataFile + ". " + ioEx.getMessage());
        }
 
        try { // calculates length of schema section of data file
            this.headerLength = (int) schemaReader.getFilePointer();
        } catch (IOException ioEx) { // handles io errors
            throw new IOException("Can not determine length of schema.");
        }
 
        // calculates convenience variable: total length of one record
        this.totalRecordLength = this.flagLength + this.recordLength;
 
    } // end constructor
 

[ October 13, 2004: Message edited by: Anton Golovin ]
[ October 13, 2004: Message edited by: Anton Golovin ]

[James Clinton]

Ahh, I see my mistake - I was treating the field headers as if they were in the data section but they're ofcourse in the scheme section - (as per the spec, doh!).

Thanks for your help, goodluck with your result.

[Anton Golovin]

Originally posted by James Clinton:
Ahh, I see my mistake - I was treating the field headers as if they were in the data section but they're ofcourse in the scheme section - (as per the spec, doh!).

Thanks for your help, goodluck with your result.

You are most welcome, James. Let me likewise wish you success in your undertaking. May I ask you a question? Are you reading your records into a cache?
[ October 13, 2004: Message edited by: Anton Golovin ]

[James Clinton]

Are you reading your records into a cache?

I haven't made a decision about cache-ing yet, but I have considered it. My thinking was -

Build a cached model of the flat file and referred to this for all non-writes until a write was performed. Then I was either going to update the cache and the model at the same time to keep then in sync OR flush the cache and rebuilt it.

What do you think?

[Nit Kad]

I am just going to start my assignement and I was thinking about caching the db file or not. There is definite advantage if the data is small. But do we have to consider that it might increase beyond a point of caching in a object?

[James Clinton]

I would think so yes, otherwise it'll go down as a limiation in your design.

How have people implemented this in the past - is there a pattern to follow, recommedations and no-go areas we should be aware of?

[Anton Golovin]

I cache, using an ArraList to read in records sequentially at startup; records are in the form of String[] arrays, and deleted records are entered as nulls.

I write in the choices.txt file about caching. It makes searches very quick. The only requirement, in my view, that caching imposes is that before a cache is updated, the data file must be updated.

[Clivant Yeo]

I implemented my project using memory caching. I was about to finish when to my dismay, my server program will crash at about 70000 records after the second client had locked into the server(I tested it when I had nothing better to do). I am reimplementing my project to the non-caching mode, as there are the GUI and the network connections (Sockets for my case) which needs memory more than my records (they are stored in the database file after all), plus the memory that the network connections will consume is quite arbitrary, it's better to keep as much for them as possible. Just my two cents.

Regards,
Clivant

[James Clinton]

Thanks for that Clivant.

I think I'm going to do the same as it probably wont earn any extra marks.