Data file format from new assignment

[Michael Dreese]

Hi JavaRanchers,
I have a few questions on the data file format from the new assignment. The following is the data file format:

Data file Format
The format of data in the database file is as follows:
Start of file
4 byte numeric, magic cookie value identifies this as a data file
4 byte numeric, offset to start of record zero
2 byte numeric, number of fields in each record
Schema description section.
Repeated for each field in a record:
2 byte numeric, length in bytes of field name
n bytes (defined by previous entry), field name
2 byte numeric, field length in bytes
end of repeating block
Data section. (offset into file equal to "offset to start of record zero" value)
Repeat to end of file:
2 byte flag. 00 implies valid record, 0x8000 implies deleted record
Record containing fields in order specified in schema section, no separators between fields, each field fixed length at maximum specified in schema information
End of file
All numeric values are stored in the header information use the formats of the DataInputStream and DataOutputStream classes. All text values, and all fields (which are text only), contain only 8 bit characters, null terminated if less than the maximum length for the field. The character encoding is 8 bit US ASCII.
Database schema
The database that Bodgitt and Scarper uses contains the following fields: Field descriptive name Database field name Field length Detailed description
Subcontractor Name name 32 The name of the subcontractor this record relates to.
City location 64 The locality in which this contractor works
Types of work performed specialties 64 Comma separated list of types of work this contractor can perform.
Number of staff in organization size 6 The number of workers available when this record is booked
Hourly charge rate 8 Charge per hour for the subcontractor. This field includes the currency symbol
Customer holding this record owner 8 The id value (an 8 digit number) of the customer who has booked this. Note that for this application, you should assume that customers and CSRs know their customer ids. The system you are writing does not interact with these numbers, rather it simply records them. If this field is all blanks, the record is available for sale.

What is the magic cookie value? Can I define it by myself? or I just use the one from the given db file?
According to the data file format, 2 byte flag 00 implies valid record, but the given db-2x2.db file uses 0001 instead. Am I getting the idea of "2 byte flag 00" wrong? Can somebody clarify it for me?
Also, I need some explanation for the following sentence - "null terminated if less than the maximum length for the field. The character encoding is 8 bit US ASCII."
What is "null terminated"? Does that mean I have to append NULL if less than the max. length for the field?
Do I have to do any setting for 8 bit US ASCII?
I'd appreciate any helps.
Thanks in advance,
Michael
p.s. for those of you just want to read the records in the db file, you can do "more db.db > out" on the command line. You will get a plain text file to show all the records, not the binary data.

[Sun qinlong]

hi frieng:
Today I got the assignment . Also I have some confuse about the .db file .So I think I can't help you ,but we can talk about this question each other .
ok?
keep in tuch !!!

[Michael Dreese]

Originally posted by Sun qinlong:
we can talk about this question each other .
ok?
keep in tuch !!!

Sure. We can help each other.

[John Smith]

Types of work performed specialties 64 Comma separated list of types of work this contractor can perform.
Number of staff in organization size 6 The number of workers available when this record is booked

It's not about flight reservation anymore?
Eugene.

[Ta Ri Ki Sun]

Originally posted by Eugene Kononov:

It's not about flight reservation anymore?
Eugene.

nope, mines Ho(tel) room reservation, and it appears there are others as well

[Michael Dreese]

Eugene:
I think it is the new one. It comes with a version number. Mine is version 2.1.2.
Cheers,
Michael

[Ta Ri Ki Sun]

Originally posted by Michael Dreese:
Eugene:
I think it is the new one. It comes with a version number. Mine is version 2.1.2.
Cheers,
Michael

bah, mines a few back then "Version 1.3.1"

[Michael Dreese]

Originally posted by Ta Ri Ki Sun:


bah, mines a few back then "Version 1.3.1"

Dose yours come with the data file format?

[Ta Ri Ki Sun]

Originally posted by Michael Dreese:

Dose yours come with the data file format?

yep, it didn't last week but this mornings download included it

[Mark Spritzler]

Yes, Eugene, no more flight bookings.
One is Booking Hotel Rooms, as we got from the post above. I think they should also have one for booking a contractor or other home improvement company, that was the Beta exam.
Mark

[Michael Dreese]

Originally posted by Mark Spritzler:
I think they should also have one for booking a contractor or other home improvement company, that was the Beta exam.
Mark

Yes, Mark. My assignment is about booking the home improvement contractors.
Cheers,
Michael

[Thomas Kijftenbelt]

check; mine too.
TK

[Mark Spritzler]

And do you have to create your own actual data to? That I found touhg to come up with unique names for contractors. Well except for the one I work for.
Mark

[Michael Dreese]

Mine come with a data file; however, I have trouble in understanding the magic cookie and record validation flag.
I already sent email to ask Sun about the flag problem because the provide data file is not the same as what the data file format described.
Also, only one interface is provided and it is for data access. It used some kind of cookie mechanism for lock/unlcok methodes.

[Andrew Monkhouse]

What is the magic cookie value? Can I define it by myself? or I just use the one from the given db file?

If you were given one in a sample db file, then I would use that one.
The magic number in the airline assignment it was 0xC0C0BABE - Is it the same?

According to the data file format, 2 byte flag 00 implies valid record, but the given db-2x2.db file uses 0001 instead. Am I getting the idea of "2 byte flag 00" wrong? Can somebody clarify it for me?

Sorry, cant help with this one - in the airline assignment it was 0 for valid, 1 for deleted.

Also, I need some explanation for the following sentence - "null terminated if less than the maximum length for the field. The character encoding is 8 bit US ASCII."
What is "null terminated"? Does that mean I have to append NULL if less than the max. length for the field?

Refer to the following:

Record containing fields in order specified in schema section, no separators between fields, each field fixed length at maximum specified in schema information

You have to create fixed length fields - to do this, they are specifying what the pad character is. So if you need to put the number 123 into a 4 character field, it would be: 0x31323300.
This caused confusion for some people in the previous assignment, as they would read in the data, expecting to get normal strings, and get the string with nulls on the end (simple solution: trim()).

Do I have to do any setting for 8 bit US ASCII?

Normally you would be working with 16 bit chars, so you should be doing some conversion between the unicode representation and the 8 bit ascii representation.
Good luck
Andrew
[ April 05, 2003: Message edited by: Andrew Monkhouse ]

[Barry Gaunt]

My UBB tags aren't working for some reason - sorry for the hard to read answer

You have to use "square brackets []" not parentheses "()".

[Andrew Monkhouse]

Thanks - I hate it when I do something silly like that.
Fixed the original post.

[Michael Dreese]

Andrew, thanks for your answers.

The magic number in the airline assignment it was 0xC0C0BABE - Is it the same?

It is not the same. In my db file, the magic number is 0x00000202.
Cheers,
Michael

[Barry Gaunt]

Michael Dreese said:

According to the data file format, 2 byte flag 00 implies valid record, but the given db-2x2.db file uses 0001 instead. Am I getting the idea of "2 byte flag 00" wrong? Can somebody clarify it for me?

I can confirm that there is 0x0001 in the supplied db-2x2.db file. The assignment text says that it should be a 2 byte flag, 00 (valid) or 0x8000 (deleted). Should I contact Sun about this discrepancy?
I downloaded yesterday (2003/4/10).
Also I can confirm that the "magic cookie" is 0x00000202

[Ta Ri Ki Sun]

quote:
--------------------------------------------------------------------------------
Do I have to do any setting for 8 bit US ASCII?
--------------------------------------------------------------------------------
Normally you would be working with 16 bit chars, so you should be doing some conversion between the unicode representation and the 8 bit ascii representation.

afaik US ASCII is in fact 7 bit and not 8 bit.

[Michael Dreese]

Originally posted by Barry Gaunt:
I can confirm that there is 0x0001 in the supplied db-2x2.db file. The assignment text says that it should be a 2 byte flag, 00 (valid) or 0x8000 (deleted). Should I contact Sun about this discrepancy?

You should contact sun as soon as possible. They made a mistake on the data file. Once you contact them, they will send you a correct data file for your assignment. I got mine in a week (they work pretty slow). Don't need to wait for the data file, just go ahead to use 0x0001 first for your code. You can change it later on.
Cheers,
Michael
[ April 11, 2003: Message edited by: Michael Dreese ]

[Werner Joerling]

Termination
In my db.db-file the fields are not terminated with a special character. They are simply filled with blanks. Is that another mistake in the db-file? (see the 00/01-flag-discussion)
Encoding
In a first try (on my DE-locale-system) I wrote to the db.db-file via a mapped ByteBuffer and didn't care about charsets and encodings.

RandomAccessFile file = new RandomAccessFile(db.db, "rw");
FileChannel channel = file.getChannel();
ByteBuffer buffer = channel.map(...);
String name = "Tilly";
buffer.put(name.getBytes());

It seems working fine. (Still with special characters beyond the 128 base characters.
Each character becomes one byte in the file. Maybe the instructions want to confuse here. Do I think wrong here?

Originally posted by Michael Dreese:

Also, I need some explanation for the following sentence - "null terminated if less than the maximum length for the field. The character encoding is 8 bit US ASCII."
What is "null terminated"? Does that mean I have to append NULL if less than the max. length for the field?
Do I have to do any setting for 8 bit US ASCII?

[Michael Dreese]

Originally posted by Werner Joerling:
Termination
In my db.db-file the fields are not terminated with a special character. They are simply filled with blanks. Is that another mistake in the db-file? (see the 00/01-flag-discussion)

I think there is no need special character for the termination of each field in the record. I think they just want each record with the same length. Also, null (0x00) is used for filling up unused bytes for each field.

It seems working fine. (Still with special characters beyond the 128 base characters.
Each character becomes one byte in the file. Maybe the instructions want to confuse here. Do I think wrong here?

I don't quite get what you mean here. Would you mind to explain a bit more to me?
Cheers,
Michael

[Werner Joerling]

What I mean is that as long as I only use characters from the 8-bit-US-ASCII charset there is nothing more to do than putting the Strings as byte-arrays to the file. In this case I do not need any conversions betweeen 8- and 16-bit-characters.

Originally posted by Michael Dreese:

I don't quite get what you mean here. Would you mind to explain a bit more to me?

[ April 12, 2003: Message edited by: Werner Joerling ]

[Michael Dreese]

Originally posted by Werner Joerling:
What I mean is that as long as I only use characters from the 8-bit-US-ASCII charset there is nothing more to do than putting the Strings as byte-arrays to the file. In this case I do not need any conversions betweeen 8- and 16-bit-characters.

The default encoding for String is 16 bits character, so I think you should change this setting for your String object first before you can put it into a byte array.
Cheers,
Michael
[ April 12, 2003: Message edited by: Michael Dreese ]
[ April 12, 2003: Message edited by: Michael Dreese ]

[Erik Manor]

I have noticed the same problem with my assignment.
1. My instructions say

i byte flag. 00 implies valid record, 0xFF implies deleted record.

00 is octal for 0 (1 byte), 0xFF is hex for 255. Since byte is signed 255 does not fit in a byte without a cast. With a cast 255 is -1.
2. My instructions also specify that any field where the value does not fill the length pf the field to use null termination. However my data file does not use null place holders but rather space (ascii/byte value 32).
I was just simply going to use 0 and -1 as the flag for the deleted/valid record flag and use space values to fill in remaining byte values in fields instead of null since that is what the data file does. I then was going to note this "along with many other things about this assignment" in the choices.txt file. My fear is that either the instructions are wrong or the db file is wrong and I will be failed automatically if the grader does not realize this.
Also the reason for keeping the file format is because of some app the company runs against it for reporting. Does anyone know if the grader actually does have an app they run against the file to validate the format. If so I need to make sure I have the correct db file/instructions.

[Vitaly Zhuravlyov]

Originally posted by Erik Manor:

00 is octal for 0 (1 byte), 0xFF is hex for 255. Since byte is signed 255 does not fit in a byte without a cast. With a cast 255 is -1.

You can use readUnsignedByte(). In this case there is no problem with 0xFF.
Also make your code independent on space or null trailing in field values.