• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Dealing with Deleted Records

 
Ranch Hand
Posts: 590
Eclipse IDE Chrome Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I am going to use a cache for the records in my database file. But I am wondering about how to handle writing back the contents of the cache to the database file.

At a high level I can think of two approaches:

  • 1) Overwrite the entire section of the database file that contains records with the contents from my cache.
  • 2) Go through each record in the cache and individually update the corresponding record in the database file.

  • What kind of approaches have people here taken?


    Option (1) seems like the simplest approach. But that would require me to store all of the records from the database file in my cache, regardless of whether they had the deleted flag set to true or not.

    I am of course assuming that when a record is deleted that its' deleted flag is set to true rather than the record being removed from the database file. I guess if I decided to remove the record from the database file when it has its' deleted flag set to true, then that would remove the need to store those records in my cache.

    So the only time I would ever encounter a deleted flag set to true is the very first time I read a fresh version of the database file. Because I would never store these records in my cache and any subsequent deletes would result in the entire key\value being removed from the Map for my cache, so I would never be writing a record back to disk with the deleted flag set to true.


    Have people passed with this approach of not storing records that are deleted in the database file (and not storing existing records where the deleted flag is set to true if they exist in the fresh database upon the very first read of the database)?

    Cheers, Sean.
     
    Ranch Hand
    Posts: 170
    Ubuntu
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    my cache is already indexed and overwriting/deleting a record is done programatically.
    then i'll just pass over the cache and write each record (or the flag only if it's deleted.
    the same goes for reading to the cache, if the flag is set i just put a null value)

    option 1 isn't that bad, because if your cache would be of size N it means that there
    were a time when N records where in the database.

     
    Sheriff
    Posts: 11604
    178
    Hibernate jQuery Eclipse IDE Spring MySQL Database AngularJS Tomcat Server Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    Option 1 will cause problems if you want to use that method (or its logic) in a new release when automatic updates of the database file (e.g. each 10 minutes) are added. The record numbers will change, which is not a good thing.

    What I did was simply write each record of my record cache back to the file. When a record was deleted, I just wrote an empty record (together with the deleted flag of course), just to keep my algorithm as simple as possible.
     
    Ranch Hand
    Posts: 101
    Netbeans IDE Oracle Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    Why do you think the flag was provided in the first place? Just for you to mark it on and off. When reading, I constructed deleted records as null and placed them in my cache. When writing back to the database, I iterated over cache.keySet(); testing every value of cache.get(key); if I have a null, I simple mark the delete flag in the database and move on, else I write the contents of the value object at that position. Pretty much like what Jonathan Elkharrat did.
     
    Sean Keane
    Ranch Hand
    Posts: 590
    Eclipse IDE Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    Thanks for the replies guys. If I am understanding you all correctly, you all did the following:

    1) When reading the file, if a record has its deleted flag set to true, then you placed null as the value in the cache.
    2) When deleting an existing record in the cache you replace the value with null.

    Then when writing the cache back to the file you had two different approaches to deal with deleted records:

    A) Update the record in the file by setting the deleted flag to true, don't modifying anything else in the record.
    B) Update the record in the file by setting the deleted flag to true, but also set every other field to be empty.

    Roel, if I am understanding you correctly you adopted approach (B). I don't understand the reason behind this approach - what is the benefit? Why did you go with this approach?
     
    Roel De Nijs
    Sheriff
    Posts: 11604
    178
    Hibernate jQuery Eclipse IDE Spring MySQL Database AngularJS Tomcat Server Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Sean Keane wrote:I don't understand the reason behind this approach - what is the benefit? Why did you go with this approach?


    I think I clearly explained in my previous post, so I just repeat myself. Option 1 will cause problems if you want to use that method (or its logic) in a new release when automatic updates of the database file (e.g. each 10 minutes) are added. The record numbers will change, which is not a good thing. And that's why I opted for the 2nd option. Otherwise you need to redevelop and retest your writing logic again when you are adding automatic updates to the application as a possible solution to prevent data losses when you have a server failure.
     
    Sean Keane
    Ranch Hand
    Posts: 590
    Eclipse IDE Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Roel De Nijs wrote:

    Sean Keane wrote:I don't understand the reason behind this approach - what is the benefit? Why did you go with this approach?


    I think I clearly explained in my previous post, so I just repeat myself. Option 1 will cause problems if you want to use that method (or its logic) in a new release when automatic updates of the database file (e.g. each 10 minutes) are added. The record numbers will change, which is not a good thing. And that's why I opted for the 2nd option. Otherwise you need to redevelop and retest your writing logic again when you are adding automatic updates to the application as a possible solution to prevent data losses when you have a server failure.



    I think you may have misunderstood me Roel. I'm not asking why you didn't overwrite the records section in the file with the contents of the cache (where the cache has no references to deleted records) I am asking you about this:

    Roel De Nijs wrote:
    When a record was deleted, I just wrote an empty record (together with the deleted flag of course)



    It sounds like you update the record in the file by setting the deleted flag to true, but also set every other field to be empty. Whereas the other guys update the record in the file by setting the deleted flag to true, but do NOT modify anything else in the record i.e. the other fields of the record all contain their original values.

    Is my understanding of how you update deleted records in the file correct? If so, what is the idea behind setting all the fields (bar the deleted flag) to be empty?
     
    Jonathan Elkharrat
    Ranch Hand
    Posts: 170
    Ubuntu
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    there's no real difference. it may be easier to just set the flag, indeed, but
    this is not a crucial point...
     
    Roel De Nijs
    Sheriff
    Posts: 11604
    178
    Hibernate jQuery Eclipse IDE Spring MySQL Database AngularJS Tomcat Server Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Sean Keane wrote:I think you may have misunderstood me Roel.

    Sorry, my mistake

    Sean Keane wrote:Is my understanding of how you update deleted records in the file correct? If so, what is the idea behind setting all the fields (bar the deleted flag) to be empty?

    Your understanding is spot-on. The reason why I followed that approach is to keep my logic as simple as possible. I don't have to move the file pointer explicitly, it just moves through the file with writing the file. And I always start with an empty record (which is updated with the actual contents of a non-deleted record).
     
    Sean Keane
    Ranch Hand
    Posts: 590
    Eclipse IDE Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Jonathan Elkharrat wrote:there's no real difference. it may be easier to just set the flag, indeed, but
    this is not a crucial point...



    That is what I was thinking - that there was no real difference. So I was just wondering what the reasoning was behind the different approaches?

    I can think of one reason - how you handle deleted records in your cache.

    * If you set it to null in the cache, then you won't have any values for the fields to write back, so the best you can do is write back an "empty" record with the deleted flag set to true.

    * If you simply mark the deleted flag of that record in the cache, then you will have all the fields of the record when writing back to the file.

    Obviously for the first approach you could get around this by storing the deleted records in a separate data structure which can be accessed when writing back to the file.
     
    Sean Keane
    Ranch Hand
    Posts: 590
    Eclipse IDE Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Roel De Nijs wrote:The reason why I followed that approach is to keep my logic as simple as possible. I don't have to move the file pointer explicitly, it just moves through the file with writing the file. And I always start with an empty record (which is updated with the actual contents of a non-deleted record).



    Thanks Roel. I think I understand your approach now - you write back empty values for the fields because you don't have the real values. You don't have the real values because you store null for the record value in your cache once the record is deleted in the application. Am I correct?

    So, back to the general point of dealing with deleted records. Some of the benefits of storing deleted records in the cache are:

    1) You can overwrite the records section in the data file with the entire contents of your cache.

    - You are able to overwrite the entire records section because your cache will at a minimum contain the same number of records as in the file, or more if records were created since populating the cache. Because of this you can be guaranteed that when you write the cache to the file you will never have additional "unwanted" data at the end of the file

    2) Your index for the records stays the same (assuming your initial index values are generated based on the order you read the records from the file). Because you are never removing records from, and you always write the records back in the order you read them, the file the index for each record will always have the same index regardless of how many records you add or remove.
     
    Roel De Nijs
    Sheriff
    Posts: 11604
    178
    Hibernate jQuery Eclipse IDE Spring MySQL Database AngularJS Tomcat Server Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Sean Keane wrote:Am I correct?


    Yes, you are
     
    Sean Keane
    Ranch Hand
    Posts: 590
    Eclipse IDE Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    Thanks Roel. So I think if you are caching the records you have three choices when dealing with deleted records as you write the cache back to the file:

    1) Write back an "empty record" with the deleted flag set to true

    If you stored null in your cache for deleted records, then this would be the most straight forward approach. It would allow you to find the position in the file where the records start and simply overwrite the rest of the file with the entire contents of the cache.

    2) Write back all the values for the deleted record, with of course the deleted flag set to true

    To do this, you would either need to store the deleted record in your cache (with the deleted flag set to true). Or if you stored the deleted record as null in your cache, then you would need to get the values for each field from the record when writing the cache back to the file - you could do this by querying the file again, or by holding deleted records in a separate data structure.

    With this approach you could also simply overwrite all the records in the file with the contents of the cash.

    3) When writing back a deleted record from your cache, simply update the deleted flag for that record in the file - don't overwrite

    To do this you'd have to control the pointer in the file. You wouldn't be able to simply overwrite the entire records section in the file with the contents of the cache.

    You could overwrite each record in the file with the information from the cache one-by-one. Then when you come across a value of null in your cache, instead of overwriting the record in the file you would simply update its deleted flag to be set to true.



    Looking at those options, I think the solution that Roel adopted is the most straight forward. Of course this solution assumes you don't want to save the information in the fields of deleted records - I guess this is something to document in your choices.
     
    Roel De Nijs
    Sheriff
    Posts: 11604
    178
    Hibernate jQuery Eclipse IDE Spring MySQL Database AngularJS Tomcat Server Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    You forgot a 4th possibility, the 1st approach from your original post

    Sean Keane wrote:Of course this solution assumes you don't want to save the information in the fields of deleted records - I guess this is something to document in your choices.


    A deleted record is simply not there anymore, so why would you store the values of a deleted record? The only thing I assumed was that saving information about a deleted record makes no sense at all (and it needs no further explanation in choices.txt, you can't write any information about a non-existing record )
     
    Sean Keane
    Ranch Hand
    Posts: 590
    Eclipse IDE Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    Roel, in your notes about passing the assignment you mentioned that you reused record numbers for deleted records. I must be missing something, because this seems to be different to the approach I understood you took from this thread. I'll explain what I understood you did, and maybe you can spot what I am missing?

    I understood that you had a cache where the key was the record number and the value was the record data e.g. Map<Integer, String[]>. Then when a record was deleted in your application, you simply replaced the value in the cache with null, the key would still be the same.

    Once your application loads the cache, entries in the cache are never deleted, only their values are updated. So how can you re-use record numbers when you never actually delete a record from the cache, and also never actually delete a record from the file (you simply replace it with a blank record where the delete flag is set to true)?
     
    Roel De Nijs
    Sheriff
    Posts: 11604
    178
    Hibernate jQuery Eclipse IDE Spring MySQL Database AngularJS Tomcat Server Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    When a record is created, I loop over the record cache. When I encounter a null-value the new record will get this record number. When no null value is encountered, there are no record numbers to be re-used and the new record will get the cache size as record number. That's how I'm re-using deleted entries.
     
    Sean Keane
    Ranch Hand
    Posts: 590
    Eclipse IDE Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    Thanks Roel. So is the key that you are using in your Map the record number?

    This is what I understood it to be. But it can't be if you are re-using record numbers.

    For example, say you had two records in your cache, where the key is the record number:

    0 -> ["Palace", "Smallville", ...]
    1 -> null


    So in this example, record with record number "1" is deleted, that is why it's value is null. When writing back the cache to the file we know that the record in position one should have it's deleted flag set to true.

    Say you then create a new record. So you first loop over the cache values to see if there is any record numbers you can re-use. You find a null, so you use it's key value (i.e. 1) as the record number for the new record you are creating. This gives you:

    0 -> ["Palace", "Smallville", ...]
    1 -> ["New Hotel", "New Town", ...]


    This however doesn't make sense with how you describe your approach of dealing with deleted records. You write back an empty record when a record was deleted .-But how do you know it was deleted when you overwrite deleted records in the cache when creating new records?
     
    Roel De Nijs
    Sheriff
    Posts: 11604
    178
    Hibernate jQuery Eclipse IDE Spring MySQL Database AngularJS Tomcat Server Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Sean Keane wrote:This however doesn't make sense with how you describe your approach of dealing with deleted records. You write back an empty record when a record was deleted .-But how do you know it was deleted when you overwrite deleted records in the cache when creating new records?


    I have completely no idea why that makes no sense

    When a record is deleted, an empty record is written back to the file. When a deleted record is re-used, the record is not deleted anymore so just the record details are written back. I didn't say that for each deleted record an empty record is written back to the file (what you are assuming for some mistery reason).
     
    Bartender
    Posts: 2292
    3
    Eclipse IDE Spring Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    Howdy, Sean!

    Let us try to clear your ideas. The idea is to keep the records in a memory cache that is represented by a Map. where the record number is the key and a String []/Room object is the value. If the value is null, then the record is deleted. When the application finishes and it is time to write the records back to the file, you iterate through the Map verifying each entry; if the value is not null, then write the valid flag and the values of the record appropriately. If it is null, then you simply write the deleted flag appropriately and you can then jump off to the next record.

    It is simpler than what you are thinking
     
    Sean Keane
    Ranch Hand
    Posts: 590
    Eclipse IDE Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    Thanks guys. Roel, I misunderstood your approach to be that all your deleted records got written back to the file as empty records.

    I think I understand your approach now:

    1) When a record is deleted, it's value in the Map for the cache is set to null.
    2) When a record is created, a new record number is generated, unless there is a record number which maps to null, in which case one of these record numbers are re-used.

    So a deleted record will only get written back from the cache to the file (as an empty record) if it has not been overwritten by a newly created record.



    What I don't understand now is, why you selectively write deleted records back to the file. What is the reason for this?

    I would have thought you would follow one approach consistently, so either:

    A) Write all deleted records back to the file, setting the deleted flag to true (for why else have a deleted flag?)
    B) Never write a deleted record back to the file (because deleted records shouldn't exist in a database)

    Roel, if I am understanding your approach, then it is doing neither of these.

    Read back on some of your previous posts this seems to be the reason for not following approach (B) above:

    ..if you want to use that method (or its logic) in a new release when automatic updates of the database file (e.g. each 10 minutes) are added. The record numbers will change, which is not a good thing



    What I don't understand from this is what are automatic updates and how do they relate to this project?

    We have one server, it reads in a list of records on start-up and writes them back on shut-down. So would it not be the case that record numbers only relevant in the life time of the server being up?

    Maybe you are talking about periodically writing the cache to the file? But even if you do this, you are only writing the contents of your cache to the file, you aren't changing the cache i.e. you could choose to not write deleted records back to the file, but they'd still exist in the cache whilst the server is up.

    I'm trying to figure out what I am not understanding here
     
    Roberto Perillo
    Bartender
    Posts: 2292
    3
    Eclipse IDE Spring Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Sean Keane wrote:What I don't understand now is, why you selectively write deleted records back to the file. What is the reason for this?



    Did you read with attention what Roel and I wrote?
     
    Sean Keane
    Ranch Hand
    Posts: 590
    Eclipse IDE Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Roberto Perillo wrote:

    Sean Keane wrote:What I don't understand now is, why you selectively write deleted records back to the file. What is the reason for this?



    Did you read with attention what Roel and I wrote?



    Why of course. Is there something you think I have missed?

    My previous post was firstly outlining my understanding of Roel's approach to be sure I understood correctly, as my question at the end is based on me understanding this correctly.

    My question is quite simple. Why selectively write back deleted records to the file? It seems the reason has something to do with "automatic updates" - but I don't understand what the reason is. Could someone enlighten me please ?
     
    Roberto Perillo
    Bartender
    Posts: 2292
    3
    Eclipse IDE Spring Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    Hum... well champ, let's then work to clear your doubts!

    The approach is quite simple. First, we have a Map<Long (or Integer, depending on your interface), Room/String[]> (I myself choose to make my Room object as value). Let's say that, at a given moment, the state of the Map is:



    Record #3 was deleted. Now, let's say that the application is finishing and we have to write the records back to the file. Here's what we can do:



    And that's pretty much it. Please let us know if this clear your thoughts. If you still have doubts, you'll continue trying to help you until your doubts are cleared.
     
    Roel De Nijs
    Sheriff
    Posts: 11604
    178
    Hibernate jQuery Eclipse IDE Spring MySQL Database AngularJS Tomcat Server Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Sean Keane wrote:So a deleted record will only get written back from the cache to the file (as an empty record) if it has not been overwritten by a newly created record.

    When a deleted record is overwritten by a newly created record, it's not a deleted record anymore

    Sean Keane wrote:What I don't understand now is, why you selectively write deleted records back to the file. What is the reason for this?

    I don't know why you keep talking about selectively writing back deleted records. That's not true. If it's time to write records back to the file, all deleted records (that is all entries with a null-value in my record cache) are written back to file (= deleted flag is set to true and an empty record is written).

    Sean Keane wrote:Roel, if I am understanding your approach, then it is doing neither of these.

    Like I said just above: all deleted records (= entry with null-value in the record cache) are written back to the database file.

    Sean Keane wrote:What I don't understand from this is what are automatic updates and how do they relate to this project?

    Using a record cache one of the drawbacks is the possible data loss when server crashes. To prevent this you could use some periodically writing of the cache to the file. With the approach I follow (and not opting for approach B) I can implement this request with my current code. In my case the writeRecordsBackToFile-method is part of the public API of the application (because it's defined in the interface), so this method could be called any time and if you then change the record numbers of the records, you'll introduce big problems. That's why I didn't opt for B.

    Hopefully all your doubts are cleared now
     
    Sean Keane
    Ranch Hand
    Posts: 590
    Eclipse IDE Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    Thanks for the replies guys. I understand technically how your code operates. What I am having difficulty understanding is why bother writing deleted records back to the database at all?

    Your main reason for doing this seems to be because you "could" periodically write the cache back to reduce the probability of data loss if the server were to crash.

    If you periodically write the cache back to the file and only ever write back records that aren't deleted, then how can this cause you problems? You haven't lost any data and you haven't modified the record numbers in the cache.

    The only thing I can see here is that if you applied an index to the records in the file after writing back (i.e. generated the record numbers again based on position in the file), then the index would not match what is in the cache. But why should this matter?

    The index only has relevance for the lifetime that the server is up. The fact that the record numbers will be different when the server is restarted shouldn't cause any problems from what I can see. Especially in the context of a thin client.

    When using a thin client, the record number is never exposed outside the server. If the server dies, so does any thread that is referencing these record numbers. So why does it matter that when the server is restarted some records may have a different record number from the last time the server was running? What problem could this cause within the context of this assignment where we have a single server and using a thin client.




    On why do I say you selectively write back deleted records. I think this is just down to how we are looking at what is means for a record to be deleted. Think about this in database terms. Once a record is deleted, it is deleted. You don't reuse a deleted record in the database - that would be an update. Yes the database may use the space in memory behind the scenes - but this doesn't mean that a record that was deleted is now not deleted, and is reused.

    So viewing deleted records from this point of view - records are either always deleted from the file or always exist in the file regardless of whether deleted in the application. If a record is always deleted then nothing should appear in the file when a record is deleted via the application i.e. no empty record with deleted flag set to true. If it is never deleted, then a record should always appear in the file with the deleted flag set to true when the record is deleted via the application.

    Your approach seems like a half-way house between the concept of being deleted and not deleted. For some records you are updating them (same location in the data file, same record number, just data has changed), for other records you are marking them as deleted by setting the deleted flag to true.

    When a user deletes a record, one of two scenarios could happen in terms of your file, and in a non-deterministic fashion. The record could be set to an empty record with the deleted flag set to true OR the record could be updated with the information from a newly created record (that was created after the previous record was deleted).


    Am I making any sense ?
     
    Roberto Perillo
    Bartender
    Posts: 2292
    3
    Eclipse IDE Spring Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Sean Keane wrote:What I am having difficulty understanding is why bother writing deleted records back to the database at all?



    Hum... well champ, you don't really have to write deleted records back to the file... you just have to write the deleted flag for records that were deleted. And no, our solution didn't include a Thread that writes the records back to the file every now and then. They are read when the application starts and written back when the application finishes.

    Since the file is sequential, it doesn't matter the position of the record in the file. In your algorithm, just include this verification: if the value of the entry in the Map is null, then just write the deleted flag there, otherwise, write the valid flag + the record fields.

    If things aren't yet clear in your solution, please let us know!
     
    Roel De Nijs
    Sheriff
    Posts: 11604
    178
    Hibernate jQuery Eclipse IDE Spring MySQL Database AngularJS Tomcat Server Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Sean Keane wrote:If you periodically write the cache back to the file and only ever write back records that aren't deleted, then how can this cause you problems? You haven't lost any data and you haven't modified the record numbers in the cache.


    It will cause problems if you have in mind (like I had a few years ago) that you write records back to file and then build the record cache again. And then it's really important that record numbers do not change This last step is of course not needed, so writing deleted records to the file is not really needed But if you always skip them it will be hard to verify if your deleted records are handled correctly when building the record cache

    Sean Keane wrote:When using a thin client, the record number is never exposed outside the server.


    Now it gets interesting. I have a thin client and the record number is available at the client. I would otherwise not know how a selected record I have booked can be identified (so I update the correct record in the record cache)? The only unique identifier I have is this record number, so it's needed client-side to identify my records (it's not visible to the user).

    Sean Keane wrote:When a user deletes a record, one of two scenarios could happen in terms of your file, and in a non-deterministic fashion. The record could be set to an empty record with the deleted flag set to true OR the record could be updated with the information from a newly created record (that was created after the previous record was deleted).


    That's what you always get if you decide to re-use deleted entries (and record numbers). And just writing an empty record with a deleted flag set to true, is nothing more than a consequence of the logic used in my write-method. Just setting the deleted flag to true will be enough (or you can also ignore deleted records and only write the valid ones)
     
    Sean Keane
    Ranch Hand
    Posts: 590
    Eclipse IDE Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    Backing Up Cache and Reinitialising It

    Roel De Nijs wrote:It will cause problems if you have in mind (like I had a few years ago) that you write records back to file and then build the record cache again. And then it's really important that record numbers do not change This last step is of course not needed, so writing deleted records to the file is not really needed . But if you always skip them it will be hard to verify if your deleted records are handled correctly when building the record cache



    Ok, so I think I am getting to the root of your reasoning now. The only reason you write back empty records with the deleted flag is to allow for the possibility that a live server would write back the cache to the file and then reinitialise the cache again whilst still running (i.e. not restarting). My first thought is, why would you do this? i.e. why would you read the information that you had just written to the file - I see no point to this? Surely you would just write the cache contents to the file and leave the cache as-is?

    Then my second thought...When designing a solution for this assignment I think a lot of people, you guys included, would have advocated a KISS approach. So why deviate from this approach when it comes to writing the cache back to the file.

    You are adding complexity to allow for functionality that does not exist in your application and may never exist in the future. Even if it was required in the future it could easily be refactored.

    Server Crashing

    Roel De Nijs wrote:Now it gets interesting. I have a thin client and the record number is available at the client. I would otherwise not know how a selected record I have booked can be identified (so I update the correct record in the record cache)? The only unique identifier I have is this record number, so it's needed client-side to identify my records (it's not visible to the user).



    Ah, my mistake here I think. I didn't think about this enough and I haven't gotten around to developing my client yet. Of course the record number has to go to the client as it is the only way of uniquely identifying the record that a client has carried out an operation on when communicating back to the server.

    So taking this into consideration I think I can see a reason for not changing the record numbers of records if the server is restarted. But it depends on what you would expect to happen if the server crashes.

    If the server crashes and restarts would you expect a client to restart or refresh its data from the server? Or would you expect your client to be allowed to continue to operate on the data (list of records) it has already received from the server before it crashes.

    If you expect the client to continue to operate on the list of records that it got from the server before the server crashed and was restarted, then you would need to ensure that the record numbers do not change when the server is restarted, because the client is still using the record numbers that were generated from before the server crashed.

    Conclusion...

    So based on this, I think it is a good idea to ensure record numbers do not change when the server is restarted (cache being reinitialised being the same thing). Not for the reason of you may want to add functionality in the future to periodically write the cache back to the file and reinitialise the cache. But for the second reason discussed here, that the server may crash and clients will still be holding on to records that will contain the record numbers.

    I appreciate your thoughts guys - so what do you think of my reasoning? I could have just followed your approach blindly, but prefer to reason these things out. Also it maybe be useful to others to understand the reasoning behind the approach.
     
    Roel De Nijs
    Sheriff
    Posts: 11604
    178
    Hibernate jQuery Eclipse IDE Spring MySQL Database AngularJS Tomcat Server Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Sean Keane wrote:Then my second thought...When designing a solution for this assignment I think a lot of people, you guys included, would have advocated a KISS approach. So why deviate from this approach when it comes to writing the cache back to the file.


    That's indeed the principle I followed. But what makes you think that it's a deviation from this principle? My code would not be any simpler when not writing deleted records to the file.
     
    Roel De Nijs
    Sheriff
    Posts: 11604
    178
    Hibernate jQuery Eclipse IDE Spring MySQL Database AngularJS Tomcat Server Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Sean Keane wrote:I appreciate your thoughts guys - so what do you think of my reasoning? I could have just followed your approach blindly, but prefer to reason these things out. Also it maybe be useful to others to understand the reasoning behind the approach.


    You should never follow someone's approach blindly. You should decide for yourself if something makes sense or not, and completely understanding the approach will help you decide if something makes sense or not.
    But you could also simply have followed your own approach, e.g. not writing deleted records back to database file and document your decision and you'll be fine (as long as your algorithm to build your record cache is able to handle/ignore deleted records).
     
    Sean Keane
    Ranch Hand
    Posts: 590
    Eclipse IDE Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Roel De Nijs wrote:

    Sean Keane wrote:Then my second thought...When designing a solution for this assignment I think a lot of people, you guys included, would have advocated a KISS approach. So why deviate from this approach when it comes to writing the cache back to the file.


    That's indeed the principle I followed. But what makes you think that it's a deviation from this principle? My code would not be any simpler when not writing deleted records to the file.



    It's a deviation from the KISS principle because you added complexity for no current purpose in your application as you implemented it.

    You said your reason for writing back deleted records in the manner you do was in case at a later point someone implemented the functionality to periodically write the cache to the file when the server is running, and as part of this functionality they would reinitialise the cache.

    You didn't implement this functionality to periodically write the cache to the file and reinitialise the cache from the file contents. So you added code to allow for it in the future, when it could just as easily have been refactored in the future if\when needed. So this is where I think you have deviated from the KISS principle. Would you not agree?
     
    Roel De Nijs
    Sheriff
    Posts: 11604
    178
    Hibernate jQuery Eclipse IDE Spring MySQL Database AngularJS Tomcat Server Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Sean Keane wrote:Would you not agree?


    No, I don't agree

    Like I already said: if I would have decided not to write any deleted records back to database file, my code would not get easier (less complex) than my current code (and my code is even easier than just writing a deleted flag and then moving on to the next record).

    I'll agree that this approach is not following the YAGNI principle
     
    Sean Keane
    Ranch Hand
    Posts: 590
    Eclipse IDE Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    Yep, I agree, your actual code could not have been any simpler (apart from maybe not overwriting deleted records in the cache - which would use more memory, but would be easier for a junior programmer to understand). It's just really how you describe your justification for writing back deleted records. Your reasoning was to allow for future functionality.

    Either way. I think the main and most important point is that the main justification for writing back deleted records is not to allow for some future functionality (periodically writing the cache to the file).

    Rather the main justification is that if you restart the server (either out of choice or because it crashes) you want to ensure that the record numbers are the same as those that were used before the server restarted. The reason being that clients could still be alive with data containing the record numbers from before the server restarted. This would be bad as clients would think they are booking room X, when in reality they would end up booking room Y when the request hits the server.

    This would be the main justification as it actually applies to the application you have implemented. Whereas future functionality does not apply as it doesn't exist in your application, and as we all know, no one can predict the future, so it may never exist in the future either .
     
    Roel De Nijs
    Sheriff
    Posts: 11604
    178
    Hibernate jQuery Eclipse IDE Spring MySQL Database AngularJS Tomcat Server Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator

    Sean Keane wrote:Yep, I agree, your actual code could not have been any simpler (apart from maybe not overwriting deleted records in the cache - which would use more memory, but would be easier for a junior programmer to understand).


    That's just an extra 4-5 lines of simple code (no rocket science), so I'm quiet sure that a junior programmer will not have any problems understanding this code

    Sean Keane wrote:It's just really how you describe your justification for writing back deleted records. Your reasoning was to allow for future functionality.


    I used this future functionality as explanation, because I'm not sure if the scenario you described and apploes to the application that was implemented could happen. Maybe when you restart the server, all clients need to restart because they lost their connection with the server.
     
    Sean Keane
    Ranch Hand
    Posts: 590
    Eclipse IDE Chrome Java
    • Mark post as helpful
    • send pies
      Number of slices to send:
      Optional 'thank-you' note:
    • Quote
    • Report post to moderator
    No, certainly not rocket science to add a few extra lines of code. But in general the simpler the concept & code is, the less chance for errors, and the easier to unit test. So when there's no need to add extra code\complexity then I'd usually be on the side of not adding it.

    When I have my application fully implemented I will certainly test what happens to the clients when the server restarts. If the clients are still able to communicate with the server, then I think it is a far more compelling reason to retain record numbers between restarts of the server.

    If when I test, I discover that the clients can't communicate with the server after a restart of the server, then I think I will go with the option of not writing any deleted records back to the file, as there's no justification for it based on the current implementation.
     
    With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
    reply
      Bookmark Topic Watch Topic
    • New Topic