Database file reader

[Roberto Perillo]

Howdy, Greg.

Alright, good news! I'm glad this was helpful! The idea is just to get you started, so the goal was achieved!

When you are done with the development of the Data class, you might also want to take a look at the Tests for the Data Class/Locking Mechanism tool. It will help you test your Data class and mainly your locking mechanism. I think it will also be helpful!

[Greg Funston]

I appreciate the code Robert. I am sure it will come in handy.

Cheers,
Greg Funston SCJP

[Venkat Chadaram]

Excellent! Tool. Downloaded SCJD exam - Couldnt get my head around the db file. This tool worked against my db file without any changes. Thanks a ton. Had I got this tool in 2007 I would have had all Sun Developer exam certifications. This is the only Developer exam I don't have yet.

[Roberto Perillo]

Alright, Venkat! I'm very glad this was helpful to you!

[Alexandra Zhang]

Thank you very much Robert.

I just finished reading your paper and trying to start with the implementation following your implementation steps this afternoon. I'm working on URLyBird 1.1.2.

And your DBFileReader works so well to me. The only change I made are the constant values of RECORD_FLAG_BYTES, and DATABASE_LOCATION. From this class I learned both implementations and the coding style you've showed.

I'm quiet a novice in Java. Hopefully I would learn and make a great deal of improvements after finishing this assignment.

[hmontiel montiel]

Hi guys,

I think the tool is great but I was not able to use it because apparently has a fixed column name length, and for the URLyBird 1.1.3 version the column name are flexible defined on one on the columns.

This is the definition of the 1.1.3 .db file:

Start of file
4 byte numeric, magic cookie value. Identifies this as a data file
2 byte numeric, number of fields in each record

Schema description section.
Repeated for each field in a record:
1 byte numeric, length in bytes of field name
n bytes (defined by previous entry), field name
1 byte numeric, field length in bytes
end of repeating block

Data section.
Repeat to end of file:
1 byte flag. 00 implies valid record, 0xFF implies deleted record
Record containing fields in order specified in schema section, no separators between fields, each field fixed length at maximum specified in schema information

End of file

All numeric values are stored in the header information use the formats of the DataInputStream and DataOutputStream classes. All text values, and all fields (which are text only), contain only 8 bit characters, null terminated if less than the maximum length for the field. The character encoding is 8 bit US ASCII.

Database schema
The database that URLyBird uses contains the following fields: Field descriptive name Database field name Field length Detailed description
Hotel Name name 64 The name of the hotel this vacancy record relates to
City location 64 The location of this hotel
Maximum occupancy of this room size 4 The maximum number of people permitted in this room, not including infants
Is the room smoking or non-smoking smoking 1 Flag indicating if smoking is permitted. Valid values are "Y" indicating a smoking room, and "N" indicating a non-smoking room
Price per night rate 8 Charge per night for the room. This field includes the currency symbol
Date available date 10 The single night to which this record relates, format is yyyy/mm/dd.
Customer holding this record owner 8 The id value (an 8 digit number) of the customer who has booked this. Note that for this application, you should assume that customers and CSRs know their customer ids. The system you are writing does not interact with these numbers, rather it simply records them. If this field is all blanks, the record is available for sale.

I created the following printer (please forgive the terrible OO design and coding standards) I did it in 35 minutes to visualize the .db file so I didn't spent too much time in making it reusable, and it probably will only work on the 1.1.3 version of URLyBird.

 
import java.io.EOFException;
import java.io.File;
import java.io.RandomAccessFile;


public class ReadNPrint {

	public static void main(String args[]) throws Exception{
		
		RandomAccessFile raf = new RandomAccessFile(new File("db-1x3.db"),"r");
		int magicCookie = raf.readInt();
		int fields = raf.readShort();
		System.out.println("Cookie:"+magicCookie);
		System.out.println("Num of Fields:"+fields);
		for(int i=0; i<fields;i++)
		{
			System.out.println("---- Column Description ----");
			int nameSize = raf.readUnsignedByte();
			System.out.println("Desc Size:"+nameSize);
			byte[] buff=new byte[nameSize];
			raf.read(buff);
			System.out.println(new String(buff,"US-ASCII"));
			System.out.println("Value Size:"+raf.readUnsignedByte());
			System.out.println("---- Column Ends ----");
		}
		System.out.println("++++ Records ++++");
		try{
			while(true){
				byte valid = raf.readByte();
				System.out.print(valid+" - ");
				byte buff[] = new byte[159];
				raf.read(buff);
				String s =  new String (buff,"US-ASCII");
				System.out.println(s);
				
			}
		}catch (EOFException exc){}
		raf.close();
	}
	
}

[Roberto Perillo]

Howdy, Hmontiel. Welcome to JavaRanch!

Well champ, what happens if you set the FIELD_NAME_BYTES variable to 1?

[hmontiel montiel]

Roberto Perillo wrote:Howdy, Hmontiel. Welcome to JavaRanch!

Well champ, what happens if you set the FIELD_NAME_BYTES variable to 1?

Hi Roberto,

Thanks for the welcome, my bad, didn't quite got the comments on the settings after carefully reading them.
It required of adjustments, this is how the settings work for 1.1.3 URLyBird version (bold the changes):

/** The bytes that store the "magic cookie" value */  
    private static final int MAGIC_COOKIE_BYTES = 4;  
  
[b]    /** The bytes that store the total overall length of each record */  
    private static final int RECORD_LENGTH_BYTES = 0;  
  
    /** The bytes that store the number of fields in each record */  
    private static final int NUMBER_OF_FIELDS_BYTES = 2;  
  
    /** The bytes that store the length of each field name */  
    private static final int FIELD_NAME_BYTES = 1;  
  
    /** The bytes that store the fields length */  
    private static final int FIELD_LENGTH_BYTES = 1;  
  
    /** The bytes that store the flag of each record */  
    private static final int RECORD_FLAG_BYTES = 1;  [/b]
  
    /** The value that identifies a record as being valid */  
    private static final int VALID = 0;  

Works great!
By the way excellent document "Demystifying the OCMJD Certification".

-Hector

[Alexandru Dragoi]

Related to this code:

    for (int i = 0; i < numberOfFields; i++) {  
        final byte [] buffer = new byte[fieldLengths[i]];  
        database.read(buffer);  
        message = fieldNames[i] + ": "  
            + new String(buffer, ENCODING);  
        System.out.println(message);  
    }  

What can you use to for filling up unused bytes for each field up to a specified length? Can you use blancks, or can you use something like this:

    private static void writeFieldValue(final OutputStream destination, final String source, int fieldLength) throws IOException {
        final byte[] bytes = new byte[fieldLength];
        final byte[] sourceBytes = source.getBytes(ENCODING);
        System.arraycopy(sourceBytes, 0, bytes, 0, source.length());
        destination.write(bytes);
    }

and when reading, you can do something like this:

new String(buffer, ENCODING).replaceAll("\u0000", "")

[Roel De Nijs]

I used something like this:

byte[] emptyRecord = new byte[lengthRecord];
Arrays.fill(emptyRecord, (byte) 32);

[Paweł Baczyński]

I used spaces (\u0020). Why? Because the db file provided to me already used this approach. And as I noted in choices.txt I would not expect any field beginning with spaces I just justified using of String.trim() when reading data ;).

[Alexandru Dragoi]

Pawel Pawlowicz wrote:I used spaces (\u0020). Why? Because the db file provided to me already used this approach. And as I noted in choices.txt I would not expect any field beginning with spaces I just justified using of String.trim() when reading data ;).

Ohh...so I guess it depends on the database file you receive with your assignment. You can read it with the tool provided by Roberto and then you can figure out what they used to fill out the unused bytes...

[Roel De Nijs]

Alexandru Dragoi wrote:Ohh...so I guess it depends on the database file you receive with your assignment.

There is in the instructions a statement that says that the file is also used by another application. So you could use that as an argument to keep the same padding character (as you don't know if the other application can handle another padding char).

[Alexandru Dragoi]

For me it seems that the bytes that store the total overall length of each record (RECORD_LENGTH_BYTES) are redundant information.

The candidate can calculate the length of each field by reading the value stored in FIELD_LENGTH_BYTES .
Then he can make the calculation:
record_length = field_length1 + field_length2 + ... + field_lengthn + record_flag_length,
where record_flag_length is 1 byte.

What is your opinion about this?

[Roel De Nijs]

Alexandru Dragoi wrote:What is your opinion about this?

Indeed, the record length could be calculated, making the record length in the database file redundant. But why would you calculate it yourself if it's provided in the file? Although it's a fairly easy calculation, it's extra code. And we all know: the more lines of code you have, the more you have to maintain and the more likely to have bugs.

Just for completeness: I believe the length of the record flag is assignment dependent (it's 1 or 2 bytes).

[Roberto Perillo]

100% agreed!

[Gautham Kumar]

Hi Roberto,

I have completed SCJP, SCWCD, SCBDC and now i am attempting OCMJD. I am always grateful to this forum which helped to achieve these certifications.

Today i downloaded Bodgit and Scarper assignment. I downloaded your paper and was reading it for the last few days.

It is excellent piece of work. You should attempt to write a book "cracking OCMJD assignment" with the help of some experts from this forum.

I also tried your tool to read the db file and it was seemless after modifying "RECORD_FLAG_BYTES = 2" as Roel mentioned it is either 1 or 2 depending on assignment.

I could display the db file content instantly after running your tool by reading my db file.

one question, You have declared "VALID = 0;" but in my assignment says "00 implies valid record, 0x8000 implies deleted record".

Do i need to worry about this field?

Also any suggestions or tips on where to go from here? Can i proceed with implementing the server? Please let me know

[Rehan Zahoor]

0x8000 has something to do with hexadecimal/ octal representation of an integer literal. Do as the requirements specs says. It is important. And mention this in your documentation very clearly.

[Gautham Kumar]

Hi Roberto,

I am trying to understand your tool to read DB file. I could get it working for Bogdit and Scraper exam. This took is really great.

Easpecially the following line the getValue method :

value += (byteArray[i] & 0x000000FF) << shift;

0x000000FF = 255, what exactly are you trying to do in above statement? Are you performing leftshift and reading the file?

Please explain if possible.

[Roel De Nijs]

Gautham Kumar wrote:Easpecially the following line the getValue method :

value += (byteArray[i] & 0x000000FF) << shift;

0x000000FF = 255, what exactly are you trying to do in above statement? Are you performing leftshift and reading the file?

Please explain if possible.

My good buddy Roberto provided a database reader tool which will work for every possible data file. The only thing you have to do is make some changes to the constants depending on the instructions of your assignment. So this tool can be used by everybody (as you experienced yourself). But that comes with a price: the weird looking getValue method.

So for example: the magic cookie is a number (e.g. 257), but that number is written as a byte-array. How many bytes? That's mentioned in your instructions. But for this example we assume the same value as the MAGIC_COOKIE_BYTES constant in the DBFileReader class (4). And that's exactly what the getValue-method does: convert a byte-array to its corresponding int value. The int value is easier to read, also much easier to compare with another int value (otherwise you would need to compare byte arrays). Converting a byte array to an int-value requires an algorithm. The getValue method implements that algorithm (which requires some shifting).

The first class I wrote was also a database file reader. But this reader was only useful for my assignment (that's why I didn't have shared it). My reader used the RandomAccessFile to access the database file. And instead of reading byte arrays, I could use methods like readShort and readInt instead of reading a byte array and then converting this array to an integer. So my code doesn't use any shifting, but if you have a look at the javadoc of these methods you'll see these methods use shifting too

Hope it helps!

[Roberto Perillo]

Neat!

[Gautham Kumar]

Thanks, Roel.

Much Appreciated for the detailed explaination.

I can create my own databasereader and implement all methods of sun interface and use this tool(class) s utility class to read data from the DB file.

Is this an accpetable approach?

Please let me know,

[Roel De Nijs]

Gautham Kumar wrote:I can create my own databasereader and implement all methods of sun interface and use this tool(class) s utility class to read data from the DB file.

Is this an accpetable approach?

That would be violating a must requirement (which will result in failure): all submitted code must be your own code...

[Gautham Kumar]

Thanks, Roel.

I will implement my own version of DBFileReader to read the database file and take some guidance from Roberto's tool.

[Roel De Nijs]

Gautham Kumar wrote:I will implement my own version of DBFileReader to read the database file and take some guidance from Roberto's tool.

Happy coding!