File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Developer Certification (SCJD/OCMJD) and the fly likes NX:[URLyBird]How to use the specified Data File Format? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Certification » Developer Certification (SCJD/OCMJD)
Bookmark "NX:[URLyBird]How to use the specified Data File Format?" Watch "NX:[URLyBird]How to use the specified Data File Format?" New topic
Author

NX:[URLyBird]How to use the specified Data File Format?

Richard Jackson
Ranch Hand

Joined: Jun 25, 2003
Posts: 128
Hello ,every professionals:
In my URLyBird assignment(ver 1.2.3),there is a section about "Data File Format" and "Database Scheme".
--------------------------------------------------

Data file Format
The format of data in the database file is as follows:
Start of file
4 byte numeric, magic cookie value. Identifies this as a data file
2 byte numeric, number of fields in each record
Schema description section.
Repeated for each field in a record:
1 byte numeric, length in bytes of field name
n bytes (defined by previous entry), field name
1 byte numeric, field length in bytes
end of repeating block
Data section.
Repeat to end of file:
1 byte flag. 00 implies valid record, 0xFF implies deleted record
Record containing fields in order specified in schema section, no separators between fields, each field fixed length at maximum specified in schema information
End of file
All numeric values are stored in the header information use the formats of the DataInputStream and DataOutputStream classes. All text values, and all fields (which are text only), contain only 8 bit characters, null terminated if less than the maximum length for the field. The character encoding is 8 bit US ASCII.
Database schema
The database that URLyBird uses contains the following fields: Field descriptive name Database field name Field length Detailed description
Hotel Name name 64 The name of the hotel this vacancy record relates to
City location 64 The location of this hotel
Maximum occupancy of this room size 4 The maximum number of people permitted in this room, not including infants
Is the room smoking or non-smoking smoking 1 Flag indicating if smoking is permitted. Valid values are "Y" indicating a smoking room, and "N" indicating a non-smoking room
Price per night rate 8 Charge per night for the room. This field includes the currency symbol
Date available date 10 The single night to which this record relates, format is yyyy/mm/dd.
Customer holding this record owner 8 The id value (an 8 digit number) of the customer who has booked this. Note that for this application, you should assume that customers and CSRs know their customer ids. The system you are writing does not interact with these numbers, rather it simply records them. If this field is all blanks, the record is available for sale.
-------------------------------------------------
Please tell me how to implement the data access function according to the specified statements.
Thanks a lot
Regards,
Jackson


Regards, Richard
Ta Ri Ki Sun
Ranch Hand

Joined: Mar 26, 2002
Posts: 442
Please tell me how to implement the data access function according to the specified statements.

Sorry Richard, but you'll have to be alot more specific for us to help you, your post as it stands asks for a solution to dozens of problems, so I can only suggest you break them up as you think about it, because there is no single solution
have fun
TQ
Richard Jackson
Ranch Hand

Joined: Jun 25, 2003
Posts: 128
Hello,every one.I'm glad to see the reply.
I've two questions waiting for the reply:
1)If I define the required header info in the seperated header file (e.g called Header.java),is it correct or not as follows?
*************************
int magicCookie; //4 bytes
short numField; //2 bytes
byte fieldnameLen; //1 byte
String fieldName; //n bytes
byte fieldLen; //1 byte
byte valid=00;
byte deleted=0xFF; // maybe a error occurs,because " byte 0xFF" is invalid
// please give me help about defining the 'deleted' variable
**************************
2)Need I do like this to put all numeric value aboved into the header file?
Thanks in advance.
Regards,
Richard.
Andrew Monkhouse
author and jackaroo
Marshal Commander

Joined: Mar 28, 2003
Posts: 11422
    
  85

Hi Richard
I am not clear on what you are trying to achieve with this header file, especially since header files are not really a Java concept.
My first thought was that you were trying to set up some constants that could be accessed in other classes. So you would have a constant MAGIC_COOKIE which contains the magic cookie you expect to find. However you only have one "fieldName", and there are multiple fields in the database, so that doesn't make sense.
My second thought was that you were attempting to create a Value Object for the database. However then you would need more than one Value Object: There is only one cookie, and one "number of fields" in each database; There is more than one "field name" / "field length" pair; There is more than one record in the database. So if you were trying to generate Value Objects you would presumably have a Value Object for a record, another ValueObject for the schema (possibly containing multiple Value Objects for each "field name" / "field length" pair).
Are you just asking about whether you have your types correct for the data you are trying to read?
Regards, Andrew


The Sun Certified Java Developer Exam with J2SE 5: paper version from Amazon, PDF from Apress, Online reference: Books 24x7 Personal blog
Philippe Maquet
Bartender

Joined: Jun 02, 2003
Posts: 1872
Hi Andrew and Richard,
At first reading, I understood your post as Andrew did and agreed with him. But after a second reading, I finally noticed that your Header file has the java extension.

1)If I define the required header info in the seperated header file (e.g called Header.java),is it correct or not as follows?
*************************
int magicCookie; //4 bytes
short numField; //2 bytes
byte fieldnameLen; //1 byte
String fieldName; //n bytes
byte fieldLen; //1 byte
byte valid=00;
byte deleted=0xFF; // maybe a error occurs,because " byte 0xFF" is invalid

It's OK, except for the sequence :
byte fieldnameLen; //1 byte
String fieldName; //n bytes
byte fieldLen; //1 byte

While reading the header part of the file, you should construct an array (or ArrayList) of field objects.
Cheers,
Phil.
Richard Jackson
Ranch Hand

Joined: Jun 25, 2003
Posts: 128
Hi,thanks for the replys from Andrew and Philip.
Actually,the questions I posted is only about the data file format.Because I am confused to construct the database according to the specified statements.After looking a reference book,I decide to try it.If I can create a header file object(in a seperated Header.java file),I can invoke it wherever I need .Above all the instructions.html file require you to do like this.
So if you were trying to generate Value Objects you would presumably have a Value Object for a record, another ValueObject for the schema (possibly containing multiple Value Objects for each "field name" / "field length" pair).

Andrew,your statement make me clear a little.But I don't still understand that statements including the instructions.html.
How can I organize the data efficiently into a database?
Look forward to seeing your new idea!
Regards,
Richard.
Andrew Monkhouse
author and jackaroo
Marshal Commander

Joined: Mar 28, 2003
Posts: 11422
    
  85

Hi Richard
How can I organize the data efficiently into a database?

It is in a database - you just have to work with it
Perhaps you should start by just writing a simple program that will enable you to view (just using System.out.println() statements) the schema that is in the database?
So you will have to:
  • open the file
  • read the cookie
  • read the number of fields
  • for each of the fields specified, store the following:
  • read the field name
  • read the size of the field

  • And just print each bit out as you read it.
    Once you have gone that far, you should be able to extend that concept to reading in records. Then hopefully you will have some ideas about how to work with it.
    Regards, Andrew
    Richard Jackson
    Ranch Hand

    Joined: Jun 25, 2003
    Posts: 128
    Hi,Andrew!Thanks for your help.
    I've still confused the following state:
    1 byte flag. 00 implies valid record, 0xFF implies deleted record

    If I define the two variables as follows,
    byte validFlag=00;
    byte deletedFlag=0xFF;
    the program will occur errors because of incorrect data type.
    Please tell me 1)How to define them?
    2)Is it correct to write the second one as 'int deletedFlag=0xFF;'?
    -------------------------------------------------------------------------
    Wait for your reply.
    Regards,Richard.
    Jim Yingst
    Wanderer
    Sheriff

    Joined: Jan 30, 2000
    Posts: 18671
    2)Is it correct to write the second one as 'int deletedFlag=0xFF;'?
    Mmmm, I suppose it depends how you use it, but probably not. In Java, 0xFF means 255, and a byte has range -128 to 127 - that's why you can't just say
    byte b = 0xFF;
    But you can say
    byte b = (byte) 0xFF;
    or equivalently
    byte b = -1;


    "I'm not back." - Bill Harding, Twister
    Richard Jackson
    Ranch Hand

    Joined: Jun 25, 2003
    Posts: 128
    Hi,Jim! Thank you very much.
    I understand that.
    Richard Jackson
    Ranch Hand

    Joined: Jun 25, 2003
    Posts: 128
    Hi,all.
    I do a exercise to test reading the data from the specified .db file.
    The sample program code shows as follows,

    Who can tell me the errors in the code?
    The code can compile and run successfully but there are 7 groups of data about fieldname and fieldnameLen occcuring in the result.
    I don't understand the reason.
    And I want to know the correct way to read all records.Thanks in advance.
    Regards,
    Richard
    [Andrew: added line breaks to make code more readable]
    [ August 26, 2003: Message edited by: Andrew Monkhouse ]
    Andrew Monkhouse
    author and jackaroo
    Marshal Commander

    Joined: Mar 28, 2003
    Posts: 11422
        
      85

    Hi Richard,

    You indicate in your comment that this is the magic cookie value, but then place it in a variable named recordSize? This will only cause you confusion later.

    OK You now know the number of fields. The schema tells you that following this is a repeated section, so you are going to have to do a loop here to work through that repeated section.

    This will be the size of the field you want to read in.

    There should be no need to use the seek() method at all in this sample application. Later, when clients are reading and writing specific records, you will need to seek to those record locations, but for now you can just read everything consecutively.

    This is out of sequence. The schema tells you that the order is name-length, name, field-length.

    Read the API for readUTF() - you will find that you cannot use it. readUTF() expects to be able to read the size of the data it is reading before reading the data itself. You have already explicitly read the field name length, so readUTF() cannot read it. Even without that issue, you will find that readUTF expects to find the size in a specific format, which your schema does not support. You will have to use a different method.
    Regards, Andrew
    [ August 09, 2003: Message edited by: Andrew Monkhouse ]
    Richard Jackson
    Ranch Hand

    Joined: Jun 25, 2003
    Posts: 128
    In these days,I study some points about stream.
    According the parts of instructions file:
    All numeric values are stored in the header information use the formats of the DataInputStream and DataOutputStream classes.

    I just try to do the pieces,please tell me whether I can continue the thought or not.
    The following code is involved about access the numeric values.

    Is it correct? I don't know the next step...
    I am waiting for your fine solutions.
    Regards,
    Richard
    Richard Jackson
    Ranch Hand

    Joined: Jun 25, 2003
    Posts: 128
    Hi,Andrew,
    I am sorry and I just fogot to correct the previous code so that someone maybe confused.Because the code that I just posted a moment ago is different from that of old.

    Like this,am I able to get the correct magicCookie value?
    For the next step,how can I do to use it?
    Anxious to get your help.
    Regards,
    Richard.
    Andrew Monkhouse
    author and jackaroo
    Marshal Commander

    Joined: Mar 28, 2003
    Posts: 11422
        
      85

    Hi Richard,
    You have described reading in the schema (header) information. Have you also tried retrieving the data from the data file? You will have to loop through the file until you reach it's end, reading in record after record. You have already read in the numbers of fields, and their sizes, so as long as you have stored them someplace, you can use that information to work out how much data to read from disk for each field.
    Regards, Andrew
    Richard Jackson
    Ranch Hand

    Joined: Jun 25, 2003
    Posts: 128
    Hi,Andrew
    As your said,should I specify the location of file pointer in the database file(.db file)?
    You will have to loop through the file until you reach it's end, reading in record after record.

    At the same time,I don't know how to loop through the file until you reach it's end
    Could you please explain the ways?
    Wait for your suggestion...
    Regards,
    Richard
    Andrew Monkhouse
    author and jackaroo
    Marshal Commander

    Joined: Mar 28, 2003
    Posts: 11422
        
      85

    Hi Richard,
    Since you are using read only access to the database, I assume you are planning on storing all the records in memory (some sort of Cache as Vlad, Philippe and Jim are doing). Then you would overwrite the entire database at some time in the future, saving all the modified records to it.
    (The alternative would be to use a RandomAccessFile which will allow you to read or write individual records as required.)
    So basically your concept is going to be:

    Regards, Andrew
    [ August 27, 2003: Message edited by: Andrew Monkhouse ]
    Richard Jackson
    Ranch Hand

    Joined: Jun 25, 2003
    Posts: 128
    Hi,Andrew
    When I read records from the database file,I can use many different methods.
    Could you please analyze when reading and writing records with Serilized object(e.g record object)?
    Is it suitable?
    Regards,Richard
    Richard Jackson
    Ranch Hand

    Joined: Jun 25, 2003
    Posts: 128
    Hi,Andrew
    Actually I am going to store data into the data file instead of memory.
    Perhaps I can't say that clearly in my post.
    In your previous states:
    You will have to loop through the file until you reach it's end, reading in record after record.

    The problem is how I can read(or write) to the end of the file.
    I wonder if I should use a file pointer and invoke some methods(e.g seek()).
    Regards,Richard
    Richard Jackson
    Ranch Hand

    Joined: Jun 25, 2003
    Posts: 128
    At the top of this page,there is a part of instructions about Data File Format.
    After the section,there are still parts about Database Schema:
    Is the room smoking or non-smoking smoking
    - 1 Flag indicating if smoking is permitted.
    Valid values are "Y" indicating a smoking room, and "N" indicating a non- smoking room

    1)How should I do to define the smoking field ?
    Because 'Y' and 'N' are both char type instead of boolean type.
    The single night to which this record relates, format is yyyy/mm/dd.

    2)In the date field,I want to declare it as Date object,but how can be showed as yyyy/mm/dd format?
    Regards,
    Richard
    Bharat Ruparel
    Ranch Hand

    Joined: Jul 30, 2003
    Posts: 493
    Hello Richard,
    I will see if I can help you. Andrew is stretched too thin helping just about everyone of us...
    I will be referring to your last two posts:
    you wrote:

    The problem is how I can read(or write) to the end of the file.
    I wonder if I should use a file pointer and invoke some methods(e.g seek()).

    I am assuming that you have developed or will develop utility methods to read or write Fixed Length Strings. I called them readFixedString and writeFixedStrings. I further put them in a Helper class that I call DataHelper - just a way to be better organized.
    Once you do that you should proceed to develop the read and write methods of Data Class.
    Read Method - Andrew has already given you the pseudo-code. Since it is against the JavaRanch policy to post solutions, I will see if I can give you enough direction without violating it.
    1. Yes, you need to be positioned to the beginning of the record which you are trying to read. Many ways of doing it. But first and foremost, you need to define where the header information ends and where the actual data records begin. I called this position "recOffset" short for record offset.
    The way I computed it is by reading to the end of the Header information and getting the filePointer position by using the filePointer position of the RandomAccessFile. To be more efficient and make sure that I don't read the Header information over and over again. I defined a class which I call DataSchema where I read this information only once. If you know anything about patterns, I used the Singleton pattern. If you don't, don't worry about it and just try to define the DataSchema class so that you can al-least read the header information correctly. As Max says on this forum over and over again, make sure that you do what Sun is asking you to do, performance is not a requirement.
    Anyway, once you are positioned correctly, record this position in the field such as the one I called "recOffset" above. Now you can just seek to that position. Don't forget that every record starts off with a deleted flag which indicates whether this record has been deleted or not. You need to read that correctly. I used readByte method of the RandomAccessFile. Once you can read this flag successfully, you can go on reading the Fixed length Strings using the routine that you will develop to do so. The way you compute how many records are there in the file is to add the total number of bytes for each record, and add to it the number of bytes required for the deleted flag. Divide this number into the difference of file length and recOffset that you computed above. Now you know how many records are there in the file.

    Is the room smoking or non-smoking smoking
    - 1 Flag indicating if smoking is permitted.
    Valid values are "Y" indicating a smoking room, and "N" indicating a non- smoking room
    In the date field,I want to declare it as Date object,but how can be showed as yyyy/mm/dd format?

    Remember two things:
    1. You are reading fixedlength Strings regardless of whether they are date fields on anything else. You can always convert the String representation of a Date field to the desired Date format.
    2. I am quoting from my assignment: "All text values, and all fields (which are text only), contain only 8 bit characters, null terminated if less than the maximum length. The character encoding is 8 bit ASCII. This means that Java's default Unicode encoding which takes 2 bytes per character is not to be used here and instead 8 bit US-ASCII character encoding scheme should be used.
    Once you have successfully written the readRecord routine without any consideration to locking etc, I will help you with writing part. Better still, I bet you will be in a position to help yourself.
    A bit of friendly advice (not a critique!): I know how you feel because not too long ago when I started on my assignment, I felt the same way. If you do search on my name "Bharat", you will see that I was asking similar questions. Phil helped me out a great deal initially, then Andrew/Max took over. Vlad also helps, and there are a number of others who will pitch in. I realized quickly that the ball really was in my court, i.e., I needed to come up to speed on how to manipulate Random Access Files whether to use File channels etc. Having gone through Max's book, I thought that I knew enough, but I didn't really. I printed out the JavaDoc API on RandomAccessFiles and went through it many times. I did a search on Google for RandomAccessFiles and printed out many articles which seemed applicable. Most importantly, I spent a couple of Saturdays in the library trying to soak up this knowledge. Once I got to the point where I had some sense of direction with Phil's help and my readings, I started out writing small test programs and "kick tires" as Max says.
    Everybody is trying to help you out, just get in a position to be helped effectively.
    Cheers and good-luck.
    Bharat


    SCJP,SCJD,SCWCD,SCBCD,SCDJWS,SCEA
    Richard Jackson
    Ranch Hand

    Joined: Jun 25, 2003
    Posts: 128
    Thank you!Bharat.
    And at the same time,thanks for other great persons who was always helping and giving some advice for me.
    I really know the difficulty to submit the SCJD assignment correctly.
    Especially important is,I want to learn the real Java technology.
    I'm so exciting to see Bharat's help like all people(e.g Andrew/Mark/Max/Vlad/Philippe/etc.).My belief is that I would like to face many of the difficulties and new challenges.
    If a chance,I'd like to help that people who need help.
    Regards,Richard
    Richard Jackson
    Ranch Hand

    Joined: Jun 25, 2003
    Posts: 128
    Hi,Bharat
    Thanks for your response.
    1. You are reading fixedlength Strings regardless of whether they are date fields on anything else. You can always convert the String representation of a Date field to the desired Date format.

    1)Does it mean that I must create a containing readFixedLength() method?
    2)Also, when I face the choice of "smoking" ,I may assume the following code presentation:

    Can I write the part of code like this?
    3)In your post,there is
    This means that Java's default Unicode encoding which takes 2 bytes per character is not to be used here and instead 8 bit US-ASCII character encoding scheme should be used.

    You mean that the 8 bit US-ASCII char encoding should be used.
    Could you please tell me the procedure to implement that?
    Regards,
    Richard
    Bharat Ruparel
    Ranch Hand

    Joined: Jul 30, 2003
    Posts: 493
    Hello Richard,
    You wrote:

    1)Does it mean that I must create a containing readFixedLength() method?
    2)Also, when I face the choice of "smoking" ,I may assume the following code presentation:

    What I meant to say which you may have misunderstood is that a Helper or Utility method such as readFixedLengthString will be quite useful here for you to read Data records off the RandomAccessFile (RAF). Mind you, you can call this method anything that you want, you can even choose not to develop such method, but if a piece of code is going to be called repeatedly then it is a good idea to put it within a method. That is all.
    I have the RAF and the size of the field to be read as parameters in my methods. I advised you to do a search on my name and look-up the posts(threads) that I posted on this matter, have you done that yet? You will see that I have quite a bit of code posted that I was developing with Phil's help. I strongly recommend that you go through these threads, you will get a lot of useful information.
    Regarding item number 2 in your quote above: I am afraid that you are making too much out of it. All you need to do is read this data from the data file and present it to the user. The smoking flag does get read as "Y" or "N". At this point, I found it enough to present it as such to the user along with the other information such as name, location, .. etc. At best, I can see that you probably would want to expand it to "Yes" or "No" when you display this data in JTable. But there are many more things that we can be concerned with at this point. Is this a "MUST" item in your assignment? My advice is that leave it as it is for the time being and concentrate on getting your RAF IO routines right. You can work on the presentation issues later.

    You mean that the 8 bit US-ASCII char encoding should be used.
    Could you please tell me the procedure to implement that?

    Again, well developed in my earlier posts. Please go through them and refer to them in your questions. Better still, do a search on "ASCII" and you will see just not my posts but other people's posts as well.
    When you start referrring to these posts that I encourage you to refer to, I will be happy to answer any/every questions that you have provided that I can answer them effectively.
    Regards.
    Bharat
    Richard Jackson
    Ranch Hand

    Joined: Jun 25, 2003
    Posts: 128
    Hi,Bharat!
    Thanks for your sincere advice.
    I will concentrate to study and research more about the data access instead of some unimportant things.
    Regards,
    Richard
     
    I agree. Here's the link: http://aspose.com/file-tools
     
    subject: NX:[URLyBird]How to use the specified Data File Format?