Hardcoded fields or generated by file header?

[chris bajada]

Hi,

I have a bit of a dilemma regarding the structure of how I am going to contain the data in the B&S assignment.

Is it best to generate the database fields from the header information? Or is it better if I choose to make a hardcoded class with the field information? (name, location, specialities etc)

I am concerned about choosing the former method, as I am thinking that since the "IT Director does not anticipate much reuse of the first Java technology system", it is better to place the record structure hardcoded and not complicate matters unnecessarily.

But on the other hand the exam creators provided the format of the database file for a reason- and I think to make us parse and generating the record structure from the header information.

Any suggestions from your side?

[Joe Zhou]

I had the same question. My understanding is:
It would be nice to automatically generate the field names from the header of the data file. But is is not required. The marker may not like the fields being hard coded. But the name and location are required in the search criteria and seems not a good idea to generate them from the file header. So, it doen't make a lot of sense to generate the field names.

Any comments will be appreciated.

[Edwin Dalorzo]

In my database implementation the fields are not hardcoded. The data file can grow in fields nothing would happen to the application. New fields could be added, or field size could change, and as far as the database implementation is concerned fields could even be removed without affecting the class.

My server implementation, on the other hand, is business object. It understands the concept of datatypes, and some other business rules stated in the specification, and do searches based on specific fields. Therefore, my business objects (just like you would propably do with any database) depend on the existance of the fields defined in the instructions.html file.

New fields could be added to the data file without any problem, not affecting the database operation, and when it comes to adding new fields my business objects would not even care about them. It would simply ignore them. Conversely, and yet logically, removing fields of the structure of the data file will ultimately lead to changes in the business objects that depend on those fields.

But the database object itself is capable of reading any data file confirming to the specified header format, irrespective of the fields defined in the file.

I think this approach is more flexible than hardcoding the fields in the database class, because new fields, for other purposes could be added to the file structure and new business objects, inherting from the old ones, could be created depending on these new fields without having to modify any class at all, keeping backward compatibility and allowing scallability.

But, of course, this is just a way to think, and since I have not submitted yet, I cannot say my approach will be well received by the Sun evaluator.

[chris bajada]

What I decided to do is to make the Record class dynamic, and build a wrapper class which has getter methods for the fields which i know about.

In that way if, say a new column is added to the db schema, i would have to just create a new getter method in the wrapper class.

As you (Joe) said, it is not practical for the client to generate the schema dynamically as the GUI has to cater for certain records by pre setting them. For instance if I think that a certain field information has to be put in a combo box for searching, I cannot do this without introducing a certain element of hard coding.

Thus I am going to adopt a middle approach, which would require minimal updates to the coding (especially on the server side), in case a schema change occurs.

I found other threads regarding this topic: search for "b&s schema".

[Bhavik Patel]

I tried the dynamic schema but it didn't make to much sense for me.
Because I have to generate the primary key (to throw Duplicate Exception ) and to deal with such type of things you have to know the schema.In real world , Do you think whoever calls to retrieve any specific record doesn't know what fields they are accessing ,,what type of fields..I guess you know all that before writing a query and accessing the data.

let's say you want to retrieve the user details for the logged in user then .."select * from user where userId=primary key " something like that and in all similar cases you know what is the primary key and what are the constraints on the tables and also the fields of table before writing a query . so I don't think its a good idea to implement dynamic schema.

It only works for File Format database.In my applciation , they say that they will be moving to web based application and in that scenario they might come up with some relational database.

This is my decision though ...

[Jeremy Botha]

A dynamic schema object is not difficult to do, given that the database file specifies the size of the header section anyway.

My implementation builds up a map of DBColumn objects, which it then makes available to a Schema object (which also encodes items such as number of fields in the DB etc).

As far as I can see, it should provide for painless extension of additional fields. Obviously, if fields are removed, development will need to be done. But I've seen so much evil perpetrated through hard coded variables that on principle, if I can make something dynamic, I do.

Jeremy

[chris bajada]

Jeremy, although you actually implemented in the way you described - it's not as how I meant as 'fully dynamic'. An entirely dynamic application would mean that if a new column is added- an appropriate field would appear automatically in the GUI for the new column to add a new record... update it, and provide search functionality by that field.

My reasoning is that a change of schema is a change of schema, and a change of schema does not come overnight...

Thus if it decided to change the schema, its plausible that some change in the application has to be done.

Thus, my opinion is that in such case the changes in the application should be easy to do. Therefore if any new column is to be added, data should still be read from it and put into the basic record class. In that way you would not have to change any lower level classes (which communicate with the db file). In such case changes would only have to be done to the classes such as those which feed the gui with info, and to the gui interface itself.

My design is pretty much as Jeremy described, but in case of new columns added, they won't have any visibility and functionality in the GUI... unless you do some minor updates to the code.

[Edwin Dalorzo]

Thus if it decided to change the schema, its plausible that some change in the application has to be done.

I completely agree with Chris.

Even if the GUI were dynamic enough to recognize a new field in the database, the Sun metadata do not provide information about the data types. There will be no plausible way to perform validations, formatting or give business value to a new field added in the database.

Now, since the application has to provide some business logic, there is no way to implement it without trusting in the existance of certain fields. For instance, in the URLyBird assignment I need the date to check hotel availability, and name and location fields to do searches. The owener record to make reservations.

If one of this fields ever got removed from the schema, even a dynamically created aplication has a lot of possibilities to break.

I belive the real secret lies in understanding the importance of separting the business logic from the database implementation.

In my case I have implemented the DBAccess interface just to provide the services declared in the interface itself

DBAccess
|
x--->Data

Then I created a wrapper class that uses Map and List to manipulate the database. Ultimately this class forwards invocations to the underlying Data class.

Finally, I implemented a DAOBusiness interface that exposes busines logic to the clients using the data wrapper class and returns value objects for clients.

My database clasess do not care if new fields are added or removed from the database. I could use the same Database classes to interact with any other database that conforms to the URLyBird metadata format.

My business object, on the other hand, would ignore new fields added to the database, and with all probability will fail if any field is removed from the underlying database schema.
[ October 19, 2006: Message edited by: Edwin Dalorzo ]

[Joe Zhou]

I think 'dynamic' applies to the table schema, which means the table/data schema is automatically populated to the JTable. There will be no code change when the table columns are shrinked or expanded. It doesn't mean the whole application is totally free of any change to the data file. It is impossible to be 100% dynamic. For example, if name or location column is removed, there will be no way to do search by name and/or location in the GUI.
I am going to implement a dynamic JTable and hope the Sun evaluator/marker likes it.

[Edwin Dalorzo]

I am going to implement a dynamic JTable.

In my JTable I use special renderers depending on the data types, for instance the rate field is formatter with a NumberFormatter for currencies and the date field is formated according to the spec instructions (yyy/DD/mm) using SimpleDateFormat. Smoking field is displayed rendered with Checkboxes, etc, etc, etc.

So, my approach is not dynamic in this regard.

Your approach sounds ambitious, and will surely be well received by the evaluator.

Now, just for curosity, I was wondering, if you make your JTable dynamic, and since the database metadata cotains no information regarding data types.

How will your dynamic JTable decide how to render current and possible new database fields? How would you go about rendering the date field or new date fields added to the database? o how about currency fields?

Since it is dynamic it must have a mechanism to decide rendering dymanically. However, sice the metadata does not provides data type info, this could complicate your design a bit. And just wanted to point it out and see what are your ideas regarding this issue. What do you think?

[Joe Zhou]

Hi Edwin,

I am wondering you have drilled into the JTable much deeper than me. It seems your JTable runs lots of details such as rendering, types... It sounds like that your JTable was created at the early version of Java. For Java2 version 1.4, JTable can be created above a TableModel which hides all the details about the rendering... which means you don't need types in creating a JTable because you don't need to do rendering.

For the TableModel, all you need is the table schema which is simply an array of String. The cell type is generic, the Object type. In my project, I use the AbstractTableModel which is good enough to the assignment.

To refresh the JTable I only to refresh the TableModel with the table schema. I don't need the types of the columns.

I implemented the 'dynamic'. It seems working.

[Edwin Dalorzo]

Hi again Joe.

Actually I am using JDK 1.5.

But now I understand what you are doing. You are treating the fields as String objects all the time. You never convert what you read from the database to its corresponding field type. That is why you do not have a problem with the rendering.

You treat everything as a String and display it as is in the JTable.

Just out of curiosity. How do you validate in your insert and updates that the dates are valid, or the hotel rates correspond to actual double numbers, or that the size are actually integers?
[ October 20, 2006: Message edited by: Edwin Dalorzo ]

[Joe Zhou]

Hi Edwin,

I think my assignment is different. I don't have any requirement for the date type. String is good enough for my assignment.
Thanks for alll the messages.

[chris bajada]

Just out of curiosity. How do you validate in your insert and updates that the dates are valid, or the hotel rates correspond to actual double numbers, or that the size are actually integers?

Cheers Edwin, that was exactly my concern in all this. Joe if you have the bodgitt and scarper assignment- you are telling me that you are you letting the user enter the word "Pizza" and "$150.00" in the 'rate' field just the same (rate = Hourly Charge of subcontractors)... as they are both strings okay. Are you sure that NO validation what so ever is required?

Honestly I would trade the possibility of having a completely dynamic application with full field validation. I'm sure that the examiner would fail me if I let the user enter text, where a number should do etc...

Well thanks for all the feedback!

[Edwin Dalorzo]

Hi, Chris.

It is not only about field validations. All the fields in a table must convey some business value, and that means that you must interpret the fields in some way. Without interpretation the fields have no business value at all, they would be just meaningless data, and applications should deliver information.

For instance, in the URLyBird assignment you know that the rate field is a currency field that represents the value of a hotel room. If you treated the field as a plain String, then you could not distinguish between positive and negative numbers, and the rate of a hotel room should never be a negative number.

In a case like this you would feel compelled to do something about it, and the only logical thing to do is to validate the data type and its domain or the field would not have any business value to the application. Filed validations are an important aspect of the business domain of any application.

To set another example, what about the dates?

In my URLyBird assignment, the hotel rooms can only be booked within 48 hours of availability, therefore you need to do date comparisons and determine if a date is grater then the current system date. You cannot determine if a String holding a date is greater than the current system date just using Strings. You will need to create a Date object and compare them.

This comparison is part of the business value conveyed by the application to the end user. It is part of the purpose and nature of the application. If this field were removed from the database or if its data type were changed, the client application would inevitably have to break. Becuase the business rules for the application would have changed.

What if you want to sort the columns in the JTable? Sorting dates or numbers as String objects will not produce accurate results.

Now, when it comes to rendering... all the fields must have buisness value, and that means you can see a field in many different ways to convey that value to the user.

In my URLyBird assignment it is required that dates are displayed in yyyy/MM/dd format. Therefore I must use special rendering for dates. The rate field is a currency in USD (US Dollars) therefore I need to render the numbers accordingly. And the smoking field is a flag that accepts Y o N, I can display a checkbox, or Yes/No, or True/False, or 1/0 or simply Y/N.

If tomorrow I add a new field with the picture of the client that booked a room, what would be displayed in the JTable? A bunch of bytes?

How do I know that new fields added to the JTable are to be displayed in this form, they could probably be used by other applications consuming information from the same database.

So, it is the client application the one that gives value to fields. It is Ok that the client application interpret those fields somehow.

So, what I say is that, the database application should be smart enough to work with any data file conforming to the specification so that any client applications could consume its services to deliver business value to different kind of users. But the client application is intended to deliver some business value, and that implies knowning the structure and data types of the tables. And logically any changes in the data types and fields consumed by the application might break its fuctionality. That would happen even in any kind application.

I strongly believe, like you do Chris, that I could not do all this just using String objects without a great amount of effort that ultimately will lead me to do some data type conversions that evidently would break the "dynamic concept" that Joe seems to pursue for his JTable. I daresay this level of dynamism might even be impossible, because it implies no interpretation of the fields, and therefore there is no way to convey business value to the user.

My own instructions.html file details what are the corresponding data types for every field anyway, therefore I do not see that this data type conversion is breaking any requirements stated in the assignment specification.
[ October 22, 2006: Message edited by: Edwin Dalorzo ]

[Joe Zhou]

Hi Chris and Edwin,

This discussion is getting more interesting and useful.
It seems my GUI is very different than yours. I am doing Bodgitt and Scarper. The 'Instruction.html' only ask me to 'allow the user to book a selected record, updating the database file accrodingly'. Note that this requirement is very simple. It only asks the GUI to update a record. The 'create' and 'delete' operation are not required. To book a record, users only need to enter the Owner ID. So, my GUI only updates the Owner ID.

My GUI does not edit the JTable directely, as you all see, validating the JTable is so complex. Instead, I copy the selected record to a record pane where the data is validated. Since the validation does not happen in the JTable, it is a lot easier. When the user clicks the update button the validated record is written to the data file.

I have a few reasons of doing this:

1. The data file is a plain text file. Basically, at the server side, any data should be able to written to the file.
2. Editing the JTable is complex and difficult to maintain, especially for types etc. Editing a record pane is easy because everything is in text field. It is also easy for operation for delete and create.
3. Validation is against a text. All it is about a function called by the listener. It can validate data format, numeric data, string length, etc.

For Edwin, if you want to sort a JTable, you can alway do it without any types. For instance, you can convert a string to a date type in your comparison class so that the order is accroding to the data instead of string.

Hope you two don't think this implementation is wired.

[Edwin Dalorzo]

The 'Instruction.html' only ask me to 'allow the user to book a selected record, updating the database file accrodingly'. Note that this requirement is very simple. It only asks the GUI to update a record. The 'create' and 'delete' operation are not required.

Joe, those requirements are for the GUI. In my instructions.html file, under the Server section also says:

Your data access class must be called "Data.java", must be in a package called "suncertify.db", and must implement the following interface:

And then it shows the DBAccess interface that must be implemented. I am not sure if throwing UnsupportedOperationException will be considered an implementation.

If I were you I would look into this issue a little bit more. I still have not heard of anybody who has not fully implemented the interface.

The data file is a plain text file.

I am afrraid it is not. The file is a bynary file. For instance, the metadata section is in binary format.

For Edwin, if you want to sort a JTable, you can alway do it without any types. For instance, you can convert a string to a date type in your comparison class so that the order is accroding to the data instead of string.

But Joe, doing such conversion, breaks the dynamism of your JTable, because in the moment you do that you are taking decisions over the data types, and in that moment you recognize that the field is no longer just a plain text field. However your JTable could not take this kind of decision for your over other fields added in the future. New fields will be treated as plain Strings irrespective of their business value.

If tomorrow somebody adds a new field to the database, how would you know what conversion to use? How to sort it? Most important, how would you know if the field should be displayed in this JTable? What if the field is a picture, what will you display, a bunch of bytes? What if the field is a boolean value stored as 0/1 and they wanted displayed as a checkbox or true/false? etc.

My assigment specification also says:

Your user interface should be designed with the expectation of future functionality enhancements.

In this application, future enhancements may require the implementation of add, delete and update operation in the GUI also, since the interface currently do not do that. So, the question is, if you were required to do that, would you have to change your whole approach from plain String manipulation to strong data type validations?

I would not dare to say you approach is incorrect, Joe. But this kind of discussions help us all validate that we are going in the right direction. If you think this design of yours satisfies the requirements, then go ahead, buddy and I wish you success!
[ October 23, 2006: Message edited by: Edwin Dalorzo ]

[Joe Zhou]

Hi Edwin,

I do appreciate your frank messages. Please see the questions from you and the answers of mine:

Q:And then it shows the DBAccess interface that must be implemented. I am not sure if throwing UnsupportedOperationException will be considered an implementation.

A: Thank you for reminding me. Sure, I do implement the DBAccess interface.
No, I don't implement UnsupportedOperationException. I want keep my implementation as simple as possible and try to minimize the unrequired functionalities.

Q:But Joe, doing such conversion, breaks the dynamism of your JTable, because in the moment you do that you are taking decisions over the data types, and in that moment you recognize that the field is no longer just a plain text field. However your JTable could not take this kind of decision for your over other fields added in the future. New fields will be treated as plain Strings irrespective of their business value.

A: The data file does not contain any business rules. As you see all data are bytes. There is no type specified. So matching the JTAble with the data file is reasonable and can make it dynamic. You may add business rules to it JTable. But it will then impossible to make it dynamic(at least very hard to do). If you implement the business rules out of the JTable, then all the good things will come out: the table is dynamic, the business rules are easy to maintain, and the data is correctly written to the data file. cool?

Q:If tomorrow somebody adds a new field to the database, how would you know what conversion to use? How to sort it? Most important, how would you know if the field should be displayed in this JTable? What if the field is a picture, what will you display, a bunch of bytes? What if the field is a boolean value stored as 0/1 and they wanted displayed as a checkbox or true/false? etc.

A: When new field comes out, the current business rules need to be updated no matter where they are implemented. The deal is how easy to update. If the JTable uses strings, then it is inmmute, no code chang! All change only go to the validation and the comparision class. Otherwise the whole JTable pluse the validations need to be rewitten.

Q:In this application, future enhancements may require the implementation of add, delete and update operation in the GUI also, since the interface currently do not do that. So, the question is, if you were required to do that, would you have to change your whole approach from plain String manipulation to strong data type validations?

A: I can easily implement the add/delete in my record pane... simply adding 'Add' and 'Delete' buttons and call the corresponding methond in my implementation od DBAccess interface.
I have carefully studied the add, delete, and update operations to a JTable and realized that is is increasing the complexities at both the GUI level and the implementation level. Note that these operations apply to the data file, not the JTable. So why not display the records in the JTable and do the add/delete/update in a separate record pane.

Q:I would not dare to say you approach is incorrect, Joe. But this kind of discussions help us all validate that we are going in the right direction. If you think this design of yours satisfies the requirements, then go ahead, buddy and I wish you success!

A: I appreciate you comments... actually I like them. If you think again, I am sure you would agree with me.