URLyBird search for the database records issue - advice needed

[lucian mihai]

Hi everybody! This is my first post here.

I have the URLyBird 1.2.x assignment and I got stuck into a problem.

The assignment specifies that : "It must allow the user to search the data for all records, or for records where the name and/or location fields exactly match values specified by the user."

In my implementation I am using a Room type object to move around ( gui,controller,business layer) the room data to be read/written
from/to the database. In this room object, when loading a value corresponding to a certain field, a validation is performed.
If the data is not valid (other format than specified in the fields description) an exception it thrown describing the situation.

Using this validation, when trying to display all the records(the 2 search fields are blank), I'm skipping the records
where fields have invalid format, or the ones that are empty (excepting the owner one that may be blank as the room might be not booked).

When I implemented the validation, my thought was that some data(starting from a record offset and of record bytes length)
does not represent a record unless it contains at least valid values(by the detailed description of fields from the assignment).
Would it make a solid argument for my choice of using validation when searching?

I am afraid of the automatic failure, by possibly breaking the assignment specification above. Should I?

Would be a good approach to display also the ones with empty fields but not to allow booking?
What approach did you use for these situations?

I would appreciate some advices

Thank you!

[Roel De Nijs]

Hi Lucian,

Welcome to the JavaRanch!

I think the database file is really easy: when the "deleted" flag is true, the record is deleted. Otherwise it is a valid one. When a search is done for all records, all valid (not deleted) ones should be returned.

How can a record in your database file have fields with invalid data? You shouldn't write invalid data to your database file in the 1st place. This should be blocked by your business service layer or your Data class (depending if it is room specific or not).

Kind regards,
Roel

[lucian mihai]

Hi Roel!

Thank you! Nice to be talking to you!

I'm taking into account the deleted/valid record flag.
Writing to database invalid field values is blocked by the business layer.

However I was thinking that fields with invalid data may appear as using other client applications with the database or just
manipulating the database file with other applications(although it is only specified that "data must continue to be manipulated for reports using another custom-written application" - this should not signify modifying the data)

Should I still keep the data validation when reading records from the Database? I ask as it may break that must requirement.
Also records with empty fields should be displayed (if matching search criteria)?
Should be booking allowed on them?

Thanks a lot!

[Roel De Nijs]

Also records with empty fields should be displayed (if matching search criteria)?
Should be booking allowed on them?

Why not? A match is a match. I don't see why it would be different because some other field of the record is empty.

[lucian mihai]

You are right.
Although I think that at least the Date field should be validated to make sure it contains a valid value.

the assignment says:

"It must allow the user to search the data for all records, or for records where the name and/or location fields exactly match values specified by the user."

"It must allow the user to book a selected record, updating the database file accordingly."

Let's say I choose to follow the first spec displaying also the records with empty/invalid Date field.
To conform also to the second spec, as it says, if I select a record with empty/invalid Date field I should be able to book it.
But obviously I can't as among the available for booking criteria I am evaluating it's date to see if it's expired and to
check the 48 hours rule.

How should I handle this situation?
To leave for the reading data, validation only for the Date field?

Thank you!

[lucian mihai]

To comply to the above 2 mentioned assignment must requirements I'm thinking about the solution:

- if searching for all the records (the 2 search fields not filled), all the (valid flagged)records in the database will be
displayed into the table

- for record rows in the table, where the Data field has a invalid value, selection of that row won't be allowed. So it
complies to the second requirement. Only the row records with value Data fields will be allowed to be selected, allowing booking.

What do you think?

Thanks a lot!

[Roel De Nijs]

1 big important advice: don't overcomplicate your solution!

You should not allow invalid dates to be saved into your database file in the first place. So that's something the business layer must prevent. And your application should only be able of searching for rooms and booking a room (updating 1 field), so dates are untouched.

I simply didn't implement this 48-hours business rule (and I justified this decision in my choices.txt). I even don't have Date objects in my application, it is just a String.

Kind regards,
Roel

[lucian mihai]

Thanks a lot for the advice Roel!

This is my main thing in mind also, not to get too complicated. But as I read the specs again I'm starting interpreting
them as in a law-court, by the letter, and start to see them as vulnerable places a more severe assessor might speculate.

[Roel De Nijs]

I will play devil's advocate then: what you gonna do if the other program which is using the database file completely corrupts it and writes data which is not according to your database schema ?

You can assume that any other program which accesses and changes your database file won't write invalid data to the file.

[lucian mihai]

I'm going to start from this assumption, it's quite normal, as you advised there is no need for complications.
Although there could be things that could be corrupting though the file:
- sudden shut-down of the machine the DB Server runs on, some I/O operations on the file at OS level that could corrupt the
it. I mean files do get corrupted from time to time, not much in our reach.

As for the use case exposed above it would be handled by my implementation this way:
- if the cookie number is not the expected one, than server application does not start showing a message
- if the cookie number is the same one. The schema is read (less the fields names that are hardcoded). If enough bytes to read as "schema"(and interpret them as fields number, lengths) it will then function based on them and handle the rest of the presumably "data" section based on them(if not enough then abort the operation showing a message). Eventually when reaching the offset of the last record but not enough bytes to read (length of a record) it will stop and ignore that particular incomplete chunk of "record" data.

[Roel De Nijs]

Eventually when reaching the offset of the last record but not enough bytes to read (length of a record) it will stop and ignore that particular incomplete chunk of "record" data.

That's a corrupted data file, which I don't process. An error message will be shown. Trying to handle this will require some extra code, and is it really needed to work on a corrupted file? Maybe other data got corrupted too (but is still according to the database schema) and turns out to be complete nonsense.

Kind regards,
Roel

[lucian mihai]

Besides the magic number, the 2 byte validity flag at precise locations(if 0x8000 or 0 if it got passed the supposed schema section ) checks and reaching EOF while reading a supposed record I can't know that there is something wrong with the data.
Unless I interpret each field and see the values are invalid or don't ignore validity flags that are different than 0x800 and 0 as I am doing now .If a validity flag is not 0x8000 -deleted- I am not writing on that storage space and if it's not 0 I am not reading from it either, so a possible corrupt data could get away, this could provide a clue that at least that area of the record is corrupted. Did you check this kind of situation?
How did you check if a data file is corrupted? Let's say if format magic number matches, enough bytes to be read as schema.
Did you check for validity flag values other than 0x8000 and 0?

[Roel De Nijs]

Hi Lucian,

I read the database file (dynamically) as follows (I didn't use constants to jump directly to the data section for example):

check magic cookie

read database schema

calculate how many records the database file contains

for every record read the "deleted" flag and all fields (I only check the flag for "deleted" status)

Possible errors (which will end up in an error message and exiting the application):

wrong magic cookie

any IOException

UnsupportedEncodingException (when conversion from byte[] to String fails

Kind regards,
Roel

[Robert Benson]

Hi,
I'm doing the urlybird and have a query about the interface methods:

I have an initial version working. On start-up it displays all records on the gui. I use the interface read method to access the database. However, this method only returns a String array.

This is where I am confused: Because of the way the method is defined, I keep the data as an array until I get back up to the controller. Then I convert it to a room object.

Q1/ Should I try to get the data into a room object sooner ,that is when I read it, or is it ok to pass the data back up as an array?
Q2/ Should I (am I allowed to) extend the interface to add a new read method that returns a room object?

Regards, Robert.

[Roel De Nijs]

Q1: completely your decision
Q2: perfectly legal to extend the interface

Another alternative could be to put a business service between gui and database, which takes care of all business logic (searching for rooms, updating a room) and converts between String[] and room object. I followed this scenario (and it is completely up to you to decide if this business service resides at the client or at the server, making your client a thick/fat or thin one)

Kind regards,
Roel

[Seetharaman Iyer]

Hi Roel,

Roel De Nijs wrote:
I read the database file (dynamically) as follows (I didn't use constants to jump directly to the data section for example):

check magic cookie

read database schema

calculate how many records the database file contains

for every record read the "deleted" flag and all fields (I only check the flag for "deleted" status)

..............

You are saying dynamically reading the database file. I'm doing something like this

My Database format is
Meta-data
4 bytes - magic cookie
4 bytes - offset
2 bytes - number of fields

Schema
2 bytes - field name length
n bytes - field name
2 bytes - field length

data section
2 bytes flag + total fields length

Here, I'm using 2 fixed values while reading. One is Magic cookie value that is 514 and Number of fields is 6.
So I'll check whether magic cookie value matches with 514, if not show message and stop the application.
If magic cookie value matches, then I would be reading the schema section only 6 times (because total number of fields is 6) and don't care about field names (as I would expect that current development version that's this application I'm writing will use only 6 fields in each record)
So Is it safe to assume that, if magic cookie matches, then this database file must have only 6 fields in each record (2 bytes flag is taken care) or based on number of fields, I've to act upon?
I have chosen first option that's assuming 6 fields in each record (and ok with dynamic behaviour on field name length and field length). I don't know how many question will pop up on my head before I sumbit my assignment...
Please assist me

Thanks,
Seetha...

[Roel De Nijs]

I also check the magic cookie against a hard-coded value. It's the only hard-coded value in my Data class. What's the benefit of hard coding the number of fields in your assignment? And not just getting the number of fields from the database file and using this value instead.

[Seetharaman Iyer]

Thanks for quick reply Roel.

Roel De Nijs wrote: What's the benefit of hard coding the number of fields in your assignment?

There are three main reason I felt why to hard code the number of fields per record.
1. number of columns in JTable are configured at client side i.e., fixed number of columns for JTable and column names are pre-populated.

2. When communicating with Server from client using some Object (mine is socket programming-serialized object approach), the type of fields will be used in the transaction (to be sent and expected) are pre-defined.
Example :
class Record {
String name;
String location;
String specialties;
String size;
String rate;
String owner;
(getter and setter of above 6 variables)
}

Think about the scenario where if I receive the database file with same cookie name but 8 fields/record.
How could I know and include those two extra fields dynamically and making getter and setter for those 2?
I don't wish to use something like this
class Record {
String[] fields;
(getter and setter of above variable)
}


3. What's the use of getting more than fixed number of fields, in this case 6, say for example program's receiving 10 as number of fields and my program will run 10 times to get the field information (that's not the problem), but what am I going to do with those extra fields in each record when the client is running?

[Roel De Nijs]

1/ That's not a valid argument to hard-code the number of fields (just my opinion). It has nothing to do with each other. If your database has 8 fields per record and in JTable only 5 must be shown, should be possible (and of course there will be some hard-coding, but the number of fields per record in database file is not necessary).

2/ You are using a socket-based solution. That's very brave and I guess you're 1 in 1000 choosing this approach I have 2 converter methods (object -> String[] and String[] -> object) and when a field (or 2) is added in the database file I have to change these converter methods (again a bit of hard-coding).

When my database file gets 2 extra fields per record, I would think the magic cookie changes too (or at least notify the developer, so he can make appropriate changes). If they don't: when you update/create a record in my application I validate the size of data-array (still 6, because no changes to program) with the number of fields (changed to 8, because dynamically read). So 6 vs 8 is not a match, and an exception will be raised and no changes could be made --> the database file won't get corrupted.
If you hard-code number of fields and you also get 2 extra fields, you'll just keep writing every record to the database file and after 1 write it will be corrupted (because you write 6 fields and the record structure is expecting 8) (unless you throw an exception when the number of fields you read from database file doesn't match with your hard-coded value).

Adding an extra column to the database file requires some small changes I have to make:

magic cookie value (if it has changed)

the 2 converter methods

(if this column has to be viewed at the client I also have to make appropriate changes at client, but that's not necessary because it could be a column used in another program and should just be ignored in this program)

You'll have to do another update: the number of fields (which is a small one too, but 1 you could easily forget).

In the end you read your database schema dynamically, except for the number of fields per record. That's just a bit weird

[Seetharaman Iyer]

I think most important thing I've missed out in my last post. That's I do check, No of fields per record should match with hard-coded number of fields value. So If either case failed, it will throw error message and application will exit.

If you hard-code number of fields and you also get 2 extra fields, you'll just keep writing every record to the database file and after 1 write it will be corrupted (because you write 6 fields and the record structure is expecting 8) (unless you throw an exception when the number of fields you read from database file doesn't match with your hard-coded value).

So we can 100% sure that, the database will never get corrupted. (you already answered in "unless...." phrase )

If they don't: when you update/create a record in my application I validate the size of data-array (still 6, because no changes to program) with the number of fields (changed to 8, because dynamically read). So 6 vs 8 is not a match, and an exception will be raised and no changes could be made

According to this, you are also using hard-coded expected value, but using it in different way.

Mine is check-on-load mechanism. Yours is lazy-check mechanism.

Both of us have our own advantages and disadvantages.
Advantage :
Since mine is of check-on-load mechanism, I won't consume memory and CPU unnecessarily (user won't get unexpected/unwanted behavior, when he/she's trying to update or something), application will exit then and there. User will point to the correct db file and continue.

Yours is, Yours class can read any kind of files with same magic cookie value. (good one ) More reusability.

Disadvantage :
Since, of check-on-load mechanism, I won't be able to read all kinds of database file.

Yours is, your logic/process will simply consume CPU by reading records and keeping it with it with no meaning/reason (say for example, you can't use it for update record - user has to shutdown and point to some other db file which matches same cookie/schema) Because of your lazy-check mechanism, if some how you have managed to show your expected 6 fields (by ignoring extra fields), but what's the point of using the loaded db records, when the user will not be able to update or create the record. He/She has to shutdown and point to some other db file which meets your need.

Can I comment something on your logic?
I think, by adding following suggestion to your existing Data loading logic, yours would be better than everything.
That's let your Data class read any kind of files, but once loaded (I don't go with mixing your logic with this suggestion, because it'll consume so much memory and CPU cycles (think if the total records are relatively very huge in the database file) before checking the suitability of the loaded records), check it out whether is this the one you are really looking for by checking received number of fields per record value.

I may be wrong. Please correct.


(As I planned, for this assignment, I will go with my check-on-load option, and will document advantage and disadvantage of this option)

[Roel De Nijs]

Seetharaman Iyer wrote:That's I do check, No of fields per record should match with hard-coded number of fields value.

So you actually do read the number of fields so why not simply use this value (and make the assumption that if the number of fields changes, the magic cookie value has to change too. (or the development team of this application has to be notified of this change).
As a final remark: I don't think it's necessary to handle these situations, because it's very unlikely that the database schema changes and the development team doesn't know anything about it. So that's why I just check the magic cookie: a file with the expected magic cookie value (the hard-coded check) is considered a valid database file according to the given instructions (with a database schema like described in the instructions).

Seetharaman Iyer wrote:According to this, you are also using hard-coded expected value, but using it in different way.

Mine is check-on-load mechanism. Yours is lazy-check mechanism.

Maybe it was not clear from my previous explanation (I did not explain very well, my mistake), but the main reason of this check, is not protecting my database file from getting corrupted (because the assumption I made with regard to the magic cookie value, see above). There are 2 other (main) reasons:
1/ keep my code simple and readible, without clutter
2/ help a developer which uses my API (Data class) for adding new (or extending existing) functionalities

When a developer passes a String[] with 5 elements (and my database file has 6 fields) the developer will get a clear message informing him about his mistake. This a developer mistake, not a user one. So it results in a runtime exception which is not caught in my application. And code with such a mistake should never make it to production environment, because it should be discovered with automated testing and/or during testing in test environment (and/or user acceptance environment). And also I don't need to clutter my logic with checks to see if the array is null or the number of elements in the array is exactly the number of fields. This simplifies your code a lot (and let other developers, when you have left the company for example, work easily with your API)

[Seetharaman Iyer]

So Is it safe to assume that, if magic cookie matches, then this database file must have only 6 fields in each record (2 bytes flag is taken care) or based on number of fields, I've to act upon?

At last, I got an assurance (but little after nice discussion )