Database file reader

[Roberto Perillo]

Howdy y'all, what y'all doin'?!

Guys, here's a simple tool I created that reads the .db file that is sent along with the SCJD assignments we currently have. This will particularly be helpful to people in the beginning of the development, which is when we have lots of doubts, like, "where should I start from", or "I think my .db file is corrupted", etc.

In order to correctly work, it is likely that a few changes will be required, and they should be done according to the database description, which can be found in the instructions.html file. For instance, some databases have 2 bytes storing each record flag; in this case, it would be necessary to change the value of the RECORD_FLAG_BYTES field to 2, as well as the VALID field.

Note that this code only reads the database file, that's why FileInputStream is being used. Also, I'd really really appreciate if you guys could test it and let me know if you run into any issue.

/*
 * @(#)DBFileReader.java    1.0 05/13/2009
 * 
 * Candidate: Roberto Perillo
 * Prometric ID: Your Prometric ID here 
 * Candidate ID: Your candidate ID here 
 * 
 * Sun Certified Developer for Java 2 Platform, Standard Edition Programming
 * Assignment (CX-310-252A)
 * 
 * This class is part of the Programming Assignment of the Sun Certified
 * Developer for Java 2 Platform, Standard Edition certification program, must
 * not be used out of this context and must be used exclusively by Oracle and
 * Sun Microsystems, Inc.
 */
package suncertify.db;

import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;

/**
 * <p>
 * The <code>DBFileReader</code> reads the .db file that is provided to each
 * SCJD candidate. Essentially, it displays on the console the content of the
 * file, starting by the "magic cookie", total overall length (in bytes) of each
 * record and the number of fields in each record. Finally, it displays the
 * content of each record, in the format "Field name: value". It also informs
 * the status of each record, that is, if it is either valid or deleted. Note
 * that the content is displayed exactly as it appears in the .db file, which
 * means that no transformations are done (such as formatting the date).
 * </p>
 * <p>
 * This class was first made with the information provided in the
 * <code>instructions.html</code> provided in the URLyBird 1.3.1 assignment,
 * therefore, it is likely that it will require a few changes in order to
 * correctly read the content of the .db files provided in other assignments.
 * </p>
 * 
 * @author Roberto Perillo
 * @version 1.0 05/13/2009
 */
public class DBFileReader {

    /** The bytes that store the "magic cookie" value */
    private static final int MAGIC_COOKIE_BYTES = 4;

    /** The bytes that store the total overall length of each record */
    private static final int RECORD_LENGTH_BYTES = 4;

    /** The bytes that store the number of fields in each record */
    private static final int NUMBER_OF_FIELDS_BYTES = 2;

    /** The bytes that store the length of each field name */
    private static final int FIELD_NAME_BYTES = 2;

    /** The bytes that store the fields length */
    private static final int FIELD_LENGTH_BYTES = 2;

    /** The bytes that store the flag of each record */
    private static final int RECORD_FLAG_BYTES = 1;

    /** The value that identifies a record as being valid */
    private static final int VALID = 0;

    /** The .db file location */
    private static final String DATABASE_LOCATION = "C:\\db-1x1.db";

    /** The character encoding of the file */
    private static final String ENCODING = "US-ASCII";

    /**
     * Reads the .db file. The values are read according to the information
     * provided in the class-level fields (such as MAGIC_COOKIE_BYTES) and to
     * values read at runtime (such as field name length). Each relevant value
     * is displayed on the console in the form "Field: value". For instance:
     * 
     * <pre>
     * Magic cookie: 888
     * Record length: 888
     * Number of fields in each record: 8
     * </pre>
     * 
     * The content of each record is displayed until EOF is reached.
     * 
     * @param args
     *            The arguments provided in the command line when this
     *            application is run.
     */
    public static void main(final String [] args) {
        try {
            final InputStream database = new FileInputStream(DATABASE_LOCATION);

            final byte [] magicCookieByteArray = new byte[MAGIC_COOKIE_BYTES];
            final byte [] recordLengthByteArray = new byte[RECORD_LENGTH_BYTES];
            final byte [] numberOfFieldsByteArray = new byte[NUMBER_OF_FIELDS_BYTES];

            database.read(magicCookieByteArray);
            database.read(recordLengthByteArray);
            database.read(numberOfFieldsByteArray);

            final int magicCookie = getValue(magicCookieByteArray);
            final int recordLength = getValue(recordLengthByteArray);
            final int numberOfFields = getValue(numberOfFieldsByteArray);

            String message = "Magic cookie: " + magicCookie;
            System.out.println(message);

            message = "Record length: " + recordLength;
            System.out.println(message);

            message = "Number of fields in each record: " + numberOfFields;
            System.out.println(message);
            System.out.println();

            final String [] fieldNames = new String[numberOfFields];
            final int [] fieldLengths = new int[numberOfFields];

            /*
             * At this point, the information of each field is provided,
             * therefore, the main point of this "for" loop is to get the name
             * and length of each field
             */
            for (int i = 0; i < numberOfFields; i++) {
                final byte nameLengthByteArray[] = new byte[FIELD_NAME_BYTES];
                database.read(nameLengthByteArray);
                final int nameLength = getValue(nameLengthByteArray);

                final byte [] fieldNameByteArray = new byte[nameLength];
                database.read(fieldNameByteArray);
                fieldNames[i] = new String(fieldNameByteArray, ENCODING);

                final byte [] fieldLength = new byte[FIELD_LENGTH_BYTES];
                database.read(fieldLength);
                fieldLengths[i] = getValue(fieldLength);
            }

            int record = 1;

            /*
             * The actual content of each record is read here. The file is read
             * until EOF is reached.
             */
            while (true) {
                final byte [] flagByteArray = new byte[RECORD_FLAG_BYTES];
                final int eof = database.read(flagByteArray);

                if (eof == -1) {
                    break;
                }

                final int flag = getValue(flagByteArray);

                message = "*********** RECORD " + record + " ***********";
                System.out.println(message);

                for (int i = 0; i < numberOfFields; i++) {
                    final byte [] buffer = new byte[fieldLengths[i]];
                    database.read(buffer);
                    message = fieldNames[i] + ": "
                            + new String(buffer, ENCODING);
                    System.out.println(message);
                }

                if (flag == VALID) {
                    message = "Status: valid record";
                } else {
                    message = "Status: deleted record";
                }

                System.out.println(message);
                System.out.println();
                record++;
            }

            database.close();

        } catch (final FileNotFoundException exception) {
            final String message = "The given file does not exist.";
            throw new RuntimeException(message, exception);
        } catch (final IOException exception) {
            final String message = "The following error occurred while trying "
                    + "to read the database file: " + exception.getMessage();
            throw new RuntimeException(message, exception);
        }
    }

    /**
     * Converts to <code>int</code> the content of a given <code>byte</code>
     * array.
     * 
     * @param byteArray
     *            The <code>byte</code> array that contains the number to be
     *            converted.
     * @return An <code>int</code> that represents the content of the
     *         <code>byte</code> array, provided as argument.
     */
    private static int getValue(final byte [] byteArray) {
        int value = 0;
        final int byteArrayLength = byteArray.length;

        for (int i = 0; i < byteArrayLength; i++) {
            final int shift = (byteArrayLength - 1 - i) * 8;
            value += (byteArray[i] & 0x000000FF) << shift;
        }

        return value;
    }
}

[Rodrigo R. Branas]

Great tool!! It´s very useful for those who are performing the project and need to discover the size of each field and so on!!

Congratulations!!

I just did some ajusts like que size of some fields and everything becomes perfect!!

[Roberto Perillo]

Fala Rodrigo!!!

I'm glad you liked it! That's the idea. Since there are a few .db formats, you end up having to make some adjustments in order to work correctly. But just changing the number of bytes of the class-level fields should be enough. Did you have to have to change something else?

[Rodrigo R. Branas]

I just changed the size of number of fields from 4 to 2 and the size of number of each field from 2 to 1.

Everything else worked perfectly...

Thanks

[Yusuf Ziyad]

Thanks a lot, I just got my assignment (URLyBird 1.3.1) and tried your program and it worked great, I just had to change DATABASE_LOCATION and comment out package suncertify.db;

[Roberto Perillo]

Howdy, Mr. Zew. Welcome to JavaRanch!

That's the spirit! It shouldn't be necessary to change many things to use this tool. That's the idea!
Also, please take a look at an important administrative private message I just sent you.

[owolabi legz]

Simply put, this tool is a life saver!

I was so confused about the database file I got from Sun for my SCJD assignment (URLyBird 1.1.2). I just downloaded the assignment a week ago and I have been trying unsuccessfully to make sense of the .db file I received. Until I ran across your tool, that is.

Please do what you have to do to make this tool popular on the ranch. I am sure other SCJD candidates will find it extremely useful.

Thank you.

[Roberto Perillo]

Howdy, Owolabi!

this tool is a life saver!

Ok, I like this description!

When we download the project, we usually have lots of doubts, and it usually takes sometime to understand everything that has to be done. The idea of this tool is to offer a "kick off" to people that is starting with the development. With it, you can have an understanding of how the database is organized, and then you can start writing your own code!

[J Flew]

Thanks Roberto

This tool was a great help.

[Ulises Pulido]

This guy is great, follow his posts he has created many tools that will help you on your assignment. Thanks Roberto

[Grary Stimon]

I'm worried that I'm missing some simple approach to accessing B&S 2.1.1 .db file records...

Roberto's tool is the closest thing I've found to accessing the .db file provided with my B&S 2.1.1 materials. Somehow, though, I think it should be easier to skip right to the records and read data than Roberto's tool makes it appear. Maybe I feel this way because the Camerlengo, Monkhouse SCJD book's sample project has no code reckoning with RandomAccessFile file seeking.

Can anyone comment on how I should locate the first record in my B&S .db file, e.g., using

seek()

?

Thanks,

G

[Roberto Perillo]

Howdy, Ulises!

Thanks for the words, partner! I feel really happy when I see a "passed" post here, and even more happy when I'm able to help with that!

Grary, my friend, this tool is just for you to have an idea of how the reading of the .db file can be done. It simply reads the .db file and displays its content on the console. I believe that, in your case, you'll need to have something a little bit more related to your particular implementation (such as getting the data of a given record, according to its number, etc). For this, you'll have to go through some thinking and implement your solution. For instance, in my implementation of the Data class, you can go directly to the location of a given record in the .db file:

database.seek(getFilePointerOffset(recordNumber));

Where the getFilePointerOffset method gets the offset of a particular record. For that, I have a formula that finds out what is the offset in the .db file of a given record (I use information, like, header length, record length, etc).

This tool is just for you to have an idea of how this can be done, so you can start thinking about your particular implementation.

[vaibhav mishra]

great tool, I am jsut starting off with some help from Andrew's book, there's a lots of stuff , and information about database in the instructions file seem cryptic to me, but this tool helped, now I get a clear picture on what's in that db file

[Bob Wheeler]

Thanks Roberto for the tool. I had just to change it here, change it there and it worked. For instance I had to change the RECORD_LENGTH_BYTES to zero.

cheers
Bob
Ps. working on Early Birdy Stuff (real name: URLyBird)

[Binu Johny]

Why is byteArray[i] & 0x000000FF necessary ?

[Nicolas Zozol]

Thanks a lot. I'll just have to understand all the byte/int stuff !

[Roberto Perillo]

Howdy, Nicolas! Welcome to JavaRanch!

Glad to know this tool was helpful!!!

[Nicolas Zozol]

I used it like a booster. But I'm in love with Enums, so I'm changing quite everything :

 
/**
 * Fields used in the database
 * @author Nicolas Zozol
 */
enum DBField {

    name(32), location(64), specialities(64), size(6), rate(8), owner(8);
    int length;

    private DBField(int length) {
        this.length = length;
    }

    public int getLength() {
        return length;
    }    
}
   

in update() method :

  
//record is a string that concatanes every fields with correct size
 String record = "";
        DBField[] values = DBField.values();
        for (int i = 0; i < values.length; i++) {
            DBField field = values[i];
            record += fillWithBlanks(data[i], field.getLength());
        }
(...)
  

But starting with your work is so helpful !

[Naveen Narayanan]

Thanks Roberto for presenting this nicely readable.
It saved a lot of hard-coded numbers from my assignment.

Probably recordLength is a field missing from the db file I have.
Second 4 –byte from the file start is “4 byte numeric, offset to start of record zero”
And I feel its file pointer to data-section-start.

possibly recordLength is to be build from adding schema-description-section fieldLengths.

Am I right sir ?

[Roberto Perillo]

Howdy, champion!

Thanks Roberto for presenting this nicely readable.

You are very welcome, champion!

Now, I'm not sure if I got your point... could you please elaborate it a little bit better?
Which one is your assignment?

[Naveen Narayanan]

Thanks roberto

line no. 99,100 of DBFileReader.java

 
database.read(magicCookieByteArray);
database.read(recordLengthByteArray);   

First 4 byte magic-cookie , next 4 byte record-length , right ?
My assignment says a Data File Format

Start of file
4 byte numeric, magic cookie value identifies this as a data file
4 byte numeric, offset to start of record zero

So, is that "start of record zero" pointing to Data section. (offset into file equal to "offset to start of record zero" value)
I feel probably my db format doesn't have a recordLength value directly given

[Roberto Perillo]

Interesting... I had never seen this description...

Well, I think you don't really have the record length info in your .db file. But that shouldn't be a problem. The only think is that you have to make sure that these 4 bytes related to the "offset to start of record zero" are considered when you read your database, so it is read correctly.

When you run the tool, are the records displayed correctly? Which one is your assignment?

[Naveen Narayanan]

Yes Roberto,

Well, I think you don't really have the record length info in your .db file

correct, I feel its manageable by reading field by field.

When you run the tool, are the records displayed correctly? Which one is your assignment?

Exactly, I got a clean display of all the records when I run your DBFileReader , with only couple of changes like,

 RECORD_FLAG_BYTES =2; 

My assignment is URLyBird 1.2.2
Interestingly recordLength & "offset to start of record zero" are having same value by coincidence

[Kedarnath Bhagwat]

Hi Robert ,

I saw one post about using seek .
I don't see this in your tool .

Also I don't see FileInputStream having this method .

So will I need to use RandomAccessFile instead of FileInputStream .

Please help me !

[Roel De Nijs]

Hi Kedarnath,

This tool is created for everybody to read the complete content of your database file, so an InputStream will be more than enough. For the real assignment, you will jump directly to the position of a record and then you will use the RandomAccessFile and its seek-method.

Kind regards,
Roel

[Kedarnath Bhagwat]

Thanks Robert !
I am using your code .
Only change that I made is I am using RandomAccessFile .
I am able to fetch all the data .

I am getting the following error .
Exception in thread "main" java.lang.RuntimeException: The given file does not exist.
at scjd.DBFileReader.main(DBFileReader.java:184)
Caused by: java.io.FileNotFoundException: C:\db-1x3.db (Access is denied)

Robert, what changes I will need to make ?
I want to record the location for each record so that I will be able to use the seek command.

[Roel De Nijs]

Hi Kedarnath,

Make sure your database file exists in the specified DATABASE_LOCATION and is accessible and readible.

Kind regards,
Roel

[Kedarnath Bhagwat]

Thanks !
I can read the file with Parameter "r" .
The problem is when I pass "rw"

It says Access Denied .

[Cristiano Sganzerla]

Naveen Narayanan wrote:Thanks roberto

line no. 99,100 of DBFileReader.java

 
database.read(magicCookieByteArray);
database.read(recordLengthByteArray);   

First 4 byte magic-cookie , next 4 byte record-length , right ?
My assignment says a Data File Format

Start of file
4 byte numeric, magic cookie value identifies this as a data file
4 byte numeric, offset to start of record zero

So, is that "start of record zero" pointing to Data section. (offset into file equal to "offset to start of record zero" value)
I feel probably my db format doesn't have a recordLength value directly given

I didnt get the recordlenght. I got the offset too.

Offset means the number of bytes that need to be jumped, if you want to acess the first record.

Its redundant, because, if you put a variable that stores the number of bytes read until the first record, it should be the same as the content of these bytes.

the same as record lenght, that is the sum of the flag bytes with the sum of each field size.

both are redundant.

Roberto, i compared my tool with yours, and was almost the same. With few changes.

And, ahhh, i used the method

private static int byteArrayToInt(final byte [] byteArray) {  
        return new BigInteger(byteArray).intValue();  
} 

These shift operators, grrrrrrr

Long time ago since my first cert
I never used in my job and forgot almost everthing about this section





By the way

Great tool !!!

[Michael Hilgert]

Hi,
I'm now studying for SCJD and found this nice thread.
Can I create the database for myself (ASCII-File) or do I need a special file-format (database)?

Thanks,
Michael

[Roel De Nijs]

You need an assignment that you have to purchase and it will come with a database file which you must use. Different assignment versions will (or can) have different different database files.

[Michael Hilgert]

My question meant, that I want to try out standard code before getting the assignment.
Can I create my own file to test something or is it a special database-format?

[Roel De Nijs]

No special file format, just use a FileoutputStream or a RandomAccessFile to create your database file.

[Michael Hilgert]

ok, thank you.

[Olu Shiyan]

@Roberto, your DBReader is awesome - talk about plug n play code. My assignment is URLyBird 1.1.1 and I didnt need to change the size contants you have in your code ; it just worked and greatly familiarized me with the db file. Thanks.

[Roberto Perillo]

I'm very glad to hear that, Olu! Our goal here is to help people get the SCJD certification, and the idea of this tool is to give people a first step in the development. I'm glad it worked well, and remember that you can always count on us to solve your doubts!

[Olu Shiyan]

Yup! This thread about the find() implementation in Data class and the filtering needed for the GUI has been very helpful as well.


Cheers.

[Andy Gertjejansen]

wonderful!!

[Roberto Perillo]

Good to know you liked it, Andy! And welcome to JavaRanch!

[Greg Funston]

Awesome tool Roberto. It has made my life easier already. I mistakenly put my file in the wrong place and fumbled around for a few minutes but I just changed the path and the package and voila, I had access. Now to evaluate the results and carry on. It would have taken me a while to develop my own reader so this is a great help.

Thanks,
Greg Funston SCJP