NX: URLYBird / my approach of the reading problem

[Ulrich Heeger]

Hi everyone,
first I wanted to post to this thread:NX: synchronizing read/write. But then I've seen that Xavier's assignment is the contractor, so I create a new thread.
I adapted my design to Max' solution of the DVDStore. I have a static HashMap and each time a client wants to update or delete a record, he has first to get the get the lock of this map. After this the client checks if the HashMap contains the concerning record, if yes, it means another client has allready locked this Record. So the thread waits to be notified. But I think it's easier to understand when I'm posting the code:

 
public long lock(long recNo)throws RecordNotFoundException{
       
 long cookie;
       //here checking if the recNo is a valid Record Number
       //if no, throw a RecordNotFoundException 
       //wrapping recNo in a Long-instance 
       Long lrecNo = new Long(recNo);
       
       synchronized(reservedRecords){
            //controlling if another client allready owns the lock
            //of the concerning record
            while(reservedRecords.containsKey(lrecNo))
                try{
                    reservedRecords.wait();
    }
    catch(InterruptedException ie){
                     //throws a �Sub�-RuntimeException
   throw new FatalSystemException;
    }
}
 
//creates a new unique lockCookie
      //with help of a Date object;
      //because one one thread can own the lock of
//reservedRecords, the value of lockCookie will be unique
Date date = new Date();
      cookie = date.getTime();
      //wrapping the cookie-value in a Long-Instance
Long lcookie = new Long(cookie);
reservedRecords.put(lrecNo,lcookie);
 
//perhaps a short sleep for ensure
//the uniqueness of cookie?
reservedRecords.notifyAll();
}
return cookie;
}

1. I decided not to take in consideration the case of a possible client crash because it�s not part of the requirements(?)(Should I document my decision?)
So my reading-method would be like:

public String[] readRecord(long recNo) throws RecordNotFoundException{
  String[] data;
       //here checking if the recNo is a valid Record Number
       //if no, throw a RecordNotFoundException 
       //wrapping recNo in a Long-instance 
       Long lrecNo = new Long(recNo);
synchronized(reservedRecords){
            //controlling if another client allready owns the lock
            //of the concerning record
            while(reservedRecords.containsKey(lrecNo))
                try{
                    reservedRecords.wait();
    }
    catch(InterruptedException ie){
                     //throws a �Sub�-RuntimeException
   throw new FatalSystemException;
    }
}
     // Reading the concerning Record and storing the 
     // Result in data[];
     reservedRecords.notifyAll();
      }
return data[];
}

2. So, I can avoid dirty reading, n�est-ce pas?
Hi Bharat, hi everyone
I have a question to you,Bharat, because I think we have quite similar approaches of the locking procedure. But perhaps also someone else can help me?
I don�t either use cache. But when you validate your RecordNumber, how do you check if the concerning Record exists? Because I�m thinking to use the position of the Record in the db.db-file as the RecordNumber. But that would force me to read the whole records in the db.db each time I want to check if the RecordNumber is valid.
3. Is this ok or do you have another idea?
I would be very happy if someone can help me,
Thanks & Regards
Ulrich

[Andrew Monkhouse]

Hi Ulrich,
There is no need to distinguish between the Hotel and Contractors assignments like this. It is probable that someone doing the contractors assignment will have exactly the same requirement as you for locking and reading. It is also probable that someone else doing the hotel assignment will have different requirements than you for locking and reading.
You should be aware that you are not guaranteed to get a unique number by using the Date.getTime() routine. From the API documentation for the System, under the currentTimeMillis() method:

Note that while the unit of time of the return value is a millisecond, the granularity of the value depends on the underlying operating system and may be larger. For example, many operating systems measure time in units of tens of milliseconds.

So if the computer is only accurate to tens of milliseconds, then it is possible to two calls to the lock method may get the same long value.
Whether this matters to you or not is up to you.

I decided not to take in consideration the case of a possible client crash because it?s not part of the requirements(?)(Should I document my decision?)

I think you should document all your decisions.
You don't have to write paragraphs on this. In fact the sentence you have above is probably all you need (although I would probably change "not part of requirements" to "outside of scope" - I think it sounds more professional )

So, I can avoid dirty reading, n?est-ce pas?

Oui / Ja / Yes.
But this does mean that your client doing a search may block between the time that another client gains a lock and then finally releases it. This should be a short period of time for the applications we are writing, but it is not guaranteed to be.
My personal opinion is that it is better to synchronize the read and write from the disk so that it is only while a write is actually taking place that a read will be blocked.

I don?t either use cache. But when you validate your RecordNumber, how do you check if the concerning Record exists? Because I?m thinking to use the position of the Record in the db.db-file as the RecordNumber. But that would force me to read the whole records in the db.db each time I want to check if the RecordNumber is valid.

You could consider a mini cache. Only cache the deleted status of the records. Or only cache the numbers of the records that are deleted.
But for the numbers of records we currently have in the database, the read is not going to be a big performance hit.
Regards, Andrew

[Vlad Rabkin]

Hi Ulrich,
First, I join to all statements from Andrew.
Now some questions:
1) Why do you call notifyAll() in lock() method?
notifyAll is used to notify other threads that the lock is realesed, not acquired!
2) Why do you wait in your read method till user lock is released?
Best,
Vlad
[ September 18, 2003: Message edited by: Vlad Rabkin ]

[Ulrich Heeger]

Hi Andrew,
thank you for your statement.

You should be aware that you are not guaranteed to get a unique number by using the Date.getTime() routine. From the API documentation for the System, under the currentTimeMillis() method:

quote:
--------------------------------------------------------------------------------
Note that while the unit of time of the return value is a millisecond, the granularity of the value depends on the underlying operating system and may be larger. For example, many operating systems measure time in units of tens of milliseconds.
--------------------------------------------------------------------------------
So if the computer is only accurate to tens of milliseconds, then it is possible to two calls to the lock method may get the same long value.

Mmh, I think if I add the RecordNumber to the getTime-Value, I shouldn't have troubles, because of the sequence:
lock, deleting/updating, unlock

You don't have to write paragraphs on this. In fact the sentence you have above is probably all you need (although I would probably change "not part of requirements" to "outside of scope" - I think it sounds more professional )

That's what I will do. Thank you
You quote:

My personal opinion is that it is better to synchronize the read and write from the disk so that it is only while a write is actually taking place that a read will be blocked.

I dont understand, does it mean that I should have a special Class ReaderWriter and have a static instance of it within Data-Class?

Regards, Ulrich

[Ulrich Heeger]

Hi Vlad,
you wrote:

Now some questions:
1) Why do you call notifyAll() in lock() method?
notifyAll is used to notify other threads that the lock is realesed, not acquired!
2) Why do you wait in your read method till user lock is released?

I notify the other waiting threads which wants to acquire the lock of reservedRecords-HashMap. Because I have allready locked the Record with a lockCookie kept as value in the HashMap, I don't need anymore the lock of reservedRecords. So within the unlock-method I will remove the Record from reservedRecords, call also notifyAll(). Thus, other waiting threads will be notified, check again if the concerning Record is in the HashMap. Because it isn't in there, the next thread can lock the Record and itself put the Record in reservedRecords.
Greetings,
Ulrich

[Vlad Rabkin]

Hi Ulrich,
What you said will happen automatically as you thread leaves synchronized() block realeasing releasing the all. synchronized block in java IS a mutex (it calles notify himself: You did invoke wait() for that and you shouldn't invoke notifyAll()).
You need notifyAll() only in unlock method and probably in delete() (depending on your implementation).
You still disn't answer the second question...
Best,
Vlad

[Ulrich Heeger]

Hi Vlad,
sorry, I forgot really to answer your second question.

What you said will happen automatically as you thread leaves synchronized() block realeasing releasing the all. synchronized block in java IS a mutex (it calles notify himself: You did invoke wait() for that and you shouldn't invoke notifyAll()).
You need notifyAll() only in unlock method and probably in delete() (depending on your implementation).

Mmh, now, I'm confused, after thinking a while I think you're right. Because other waiting threads don't need to be notified before the concerning Record has been unlocked, so like you suggest, it's enough to have a notifyAll call within the unlock-method.
But it's perhaps better for a potential client crash. Ok, some lines before, I mentioned that I don't want to take in consideration a client crash. So should I for reasons of consistence omit the notifyAll-call within the lock-method?

2) Why do you wait in your read method till user lock is released?

Here I want to check if the concerning Record is locked. If it is locked, I know that another thread tries to modify it, so I will wait until it's unlocked. So I can avoid dirty reading.
But Andrew suggests to synchronize the read and write-methods. Perhaps you can explain how I should implement it:
I asked him:

I dont understand, does it mean that I should have a special Class ReaderWriter and have a static instance of it within Data-Class?

But this way, I don't ensure that no dirty readings will happen, I'm wrong?
Because it can be that a thread reads a Record, while another has allready locked it and is modifying it (ok, not physically in the file, but he's in the way to do it.) OK, I think it would be impossible to avoid each possible dirty reading. Because even if I'm using my approach, it's possible that on the way back to the client, the concerning Record has allready been locked by another client who is preparing to modify it.
What do you think about my thoughts? Are they comprehensible or am I too confusing?
Thanks a lot for your comments, it's really important to get pointed at the own logical faults, sometimes it's really hard to see them on my own
Regards
Ulrich

[Vlad Rabkin]

Hi Ulrich,

But it's perhaps better for a potential client crash. Ok, some lines before, I mentioned that I don't want to take in consideration a client crash. So should I for reasons of consistence omit the notifyAll-call within the lock-method

First, it is not a good way to prevent deadlock. There many mechanism Unrefenced interface, WeakHashMap, time-outs, "agents" and so on.
I dediced not to do it, but document it. There have people whi made some kind of dead.lock prevention and failed, there have been people whi haven't done it at all and got 100% for locking.
Second, Sun can interpret your intentions for notifyAll in lock method not correctly (As I did ), which can lead you to loose many point if not to fail.
So my personal suggestion: don't use notifyAll() in lock.

I dont understand, does it mean that I should have a special Class ReaderWriter and have a static instance of it within Data-Class?

It is pretty simple. That is what I do.
If client #1 locked a record, but not started to write in database, the client #2 must be still able to read a record. It means user lock (lock/unlock) has nothing to do directly with your thread safety.
Independant where a record is locked or not by any client another client can read the record, but only then , when nobody updates(writes in db) a the momemt. So, you have to prevent it. There many possible solutions.
Here are two simple ones:
1)
to synchronize all (find can be exception):
your Data class is singletone:
just synchronize all methods on this:
public synchronized read();
public synchronized update();
...
public synchronized lock();
in this case you don't even need to synhcronize on reservedRecordy. Instread of then callin reservedRecord.wait() in lock() method and reservedRecord.notifyAll() in unlock, you would incoke this.wait() and this.notifyAll() accordinately.
2) You Data in not a singletone:
instead of synchronizing method use synchronized block use it in all read/write/lock/unlock method on one static object, e.g. reservedRecords.
Actually you do it already: you have synchronized block in your read method.
The same you do in all other methods. It will mean that calles from different clients will block whole database for the time of request execution. It is "slow" design, but simple.

Best,
Vlad

[Max Habibi]

If you guys keep this up, I'm going to be out of a job
ps - I'd like to hear some more thoughts on notifyAll(): I think I know where you guys want to end up with this, but I'd like you to get there on your own. So fight it out
M

[Vlad Rabkin]

Hi Max,

If you guys keep this up, I'm going to be out of a job

Why!? Have I said something wrong?
notifyAll(): Wakes up all threads that are waiting on this object's monitor. A thread waits on an object's monitor by calling one of the wait methods.
I don't see any reason of calling notifyAll.
There two opportunities to lock the record, which I mentioned:

    public void lock(int recNo) throws InterruptedException{
        synchronized(reservedRecords){
                    while( reservedRecords.contains(recNo) ){
                        reservedRecords.wait();
                    }
        reservedRecords.add(upc);

        // I know you have it in your book, I don't see any reason for 
        //reservedRecords.notifyAll();
        }
    }
    
    public synchronized void lock(int recNo) throws InterruptedException{

                    while( reservedRecords.contains(recNo) ){
                        this.wait();
                    }
        this.add(upc);
        // I know you have it in your book, I don't see any reason for 
        //reservedRecords.notifyAll();
    }  

Max, unfortunately your book is at home, and I am now in the office.
May be I would have to be sorry for my statements today, but I am pretty sure at the moment about them...

Best,
Vlad

[Max Habibi]

Hi Vlad,
I just meant that you guys are doing such a good job that you don't need me .
Regarding the following

notifyAll(): Wakes up all threads that are waiting on this object's monitor. A thread waits on an object's monitor by calling one of the wait methods.


Interesting. Can you think of anything other time that thread might need to achieve an Object's monitor?
M

[Vlad Rabkin]

Hi Max,
I at home now. So, you book says:

Line 9 notifies waiting threads thats that reserveDVD is free, and the method returns.

Now, you say:

Interesting. Can you think of anything other time that thread might need to achieve an Object's monitor?

I don't understand the question. Do ask if a thread needs something more to get Object's monitor?
That is my understanding:
There are threads which are waiting outside to get in synchronized block and
there are threads, waiting because a record is locked (lock is already in Vector).
If notifyAll() is called all threads (waiting outside) and in wait state will have to compete to get the Object's monitor.
If the one thread has succesfully acquired Monitor and locked a record (put it Vector)
- it makes no sense to awake threads being inside synchronized block, but in a waiting state, since the lock is in Vector and they will be put again in wait state.
- the threads waiting outside synchronized block have not to be awaked to get in, one of the we get in as soon as the thread(, which has locked the record, get out of synchronized block.
So, I don't really see any reason to call notifyAll!?
There only reason would be if we had used for instance WeakHashMap to prevent dead-locks, but Ulrich decided no to provide client-crach mechanism
(neither does your sample in the book).
Honestly, saying I didn't see first in your book that call notifyAll(), so
I didn't even ask my self if notifyAll() is needed there.
Max, I give up, could you please give at least a hint what do you need notifyAll() there?
Tx,
Vlad

[Max Habibi]

hi Vlad,
I wasn't trying to be cryptic: I just didn�t want to ruin your fun. The issue is that there are threads who are trying just about to enter the synchronization block for the very first time, even as other threads have already done do. These (newbie) threads are blocked, because another thread currently has the monitor for reservedRecords. Now, when that thread calls notifyAll, then the threads who are trying to get the lock on reservedRecords for the very first time will get an opportunity to do so.
Make sense?
M

[Bharat Ruparel]

Hello Ulrich,
Sorry, couldn't get back to you sooner. Just as well since very capable people are answering your question. You wrote:

Hi Bharat, hi everyone
I have a question to you,Bharat, because I think we have quite similar approaches of the locking procedure. But perhaps also someone else can help me?
I don�t either use cache. But when you validate your RecordNumber, how do you check if the concerning Record exists? Because I�m thinking to use the position of the Record in the db.db-file as the RecordNumber. But that would force me to read the whole records in the db.db each time I want to check if the RecordNumber is valid.

I validate my record numbers inside any method that receives a record number as a paramter, lock, unlock, islocked for example.
Regarding checking for the RecordNumber, I do access the deleted record flag on the fly in my validateRecord method. Since I am using a random access file and since I can "seek" to a position directly, I don't have to sequentially read through all records. It is easy to compute the offset mathematically given that the records start at a certain position and that they all have fixed lengths.
I hope that answers your question.
I am a bit mystified per to why you decided not to use the "Data.this" as the key to the hashmap? It solves a number of problems for you all in one-fell-swoop! 1. It gives you the unique key that you are looking for (assuming off-course that you are using one instance of the Data class for each client strategy), 2. More importantly, how do you ascertain that a client calling unlock method is indeed the client that locked the record in the first place? Since you are computing your key (cookie, whatever) value inside the method lock, how do you make sure that it is indeed the same value available to you when you attempt to unlock? In my opinion, that value is long gone. OK, so you store it in a HashMap, but how do you re-create it in the unlock method to compare it with the stored value? 3. Finally, coupled with the WeakHashMap use (only the declaration is different than a regular HashMap, otherwise the usage for our purposes is exactly the same as the regular HashMap), you get the deadlock resolution for free!
I understand your reluctance to use something which is unnatural, i.e., record numbers "appear" to be more natural instantly than the unique instance of an Object, but the ability to "invert" a problem and looking at it from the other side is equally important. To me, it is not a big deal, considering that performance is NOT an important criteria.
Most of your other questions have been answered already, so I will leave them alone.
Regards.
Bharat
Andrew,
I do owe you an apology. I think I unknowingly started a trend of asking people to limit their responses/questions to a certain assignment. I think that Ulrich was following my lead. We definitely need to stop doing that.

[Vlad Rabkin]

Hi Max,

I wasn't trying to be cryptic: I just didn’t want to ruin your fun.

I didn't. Your statements make just think that I have some whole in my understanding of threads and I want to us a chance to let you erase them

Now, when that thread calls notifyAll, then the threads who are trying to get the lock on reservedRecords for the very first time will get an opportunity to do so.
Make sense?

Max, it is propably very pretty embarrassing for me, but NOPE: It make no sense to me.
"New comming" threads trying to get the lock on reservedRecords for the very first time will get an opportunity to do so anyway:
If the current thread inside synchronized bloc locked the record (put it in vector) and left synchronized block, one New Comer is allowed to get in without notifying!? Is it not correct!?
Max, sorry for these questions, but I really miss the idea
Tx,
Vlad

[Max Habibi]

Let me ask you a question. Why call notifyAll @ all, ever? Don't all threads automatically get an opportunity to lock the resource when a given thread exits the synch block?
M

[Vlad Rabkin]

Hi Max,
Ok. I will try.

There two different states: Waiting and Seeking Lock.

Thread gets in Waiting state ONLY if it passed first Seeking Lock and than
Object.wait() was called();
Any thread trying to get into synchronized block must stay in Seeking Lock as long as
Object is locked.
If the thread, which is in waiting state, is awaked , it must go again though the Seeking Lock to obtain a lock on the object.

Running->Seeking Lock -> Condition True -> Go -> Realease Lock
Running->Seeking Lock -> Condition False-> Waiting

Waiting-> Seeking Lock -> Go -> Realease lock
If the thread is in Waiting state the only way to awake it - is notify/notifyAll.
(Notify/notifyAll can be inoked only by the thre who owns the lock.)
This is my answer to your question.
BUT, one of the threads, which are in Seeking Lock state will get the lock on the object immedeatelly as soon as lock is available, they don't need to be notified!!!
You said you call notifyAll for "New Comers". "New Comers" are in the state of Seeking Lock and don't need to be notified.
Only threads in "Waiting" state need to be notified, which is performed by Data.unlock().
Max, I am not arguing with you, I am trying to understand the point.

Many Tx,
Vlad
[ September 18, 2003: Message edited by: Vlad Rabkin ]

[Ulrich Heeger]

Hi Bharat,
thanks a lot for your comments.
You wrote:

Regarding checking for the RecordNumber, I do access the deleted record flag on the fly in my validateRecord method. Since I am using a random access file and since I can "seek" to a position directly, I don't have to sequentially read through all records. It is easy to compute the offset mathematically given that the records start at a certain position and that they all have fixed lengths.

Concerning this point I have two questions:
1.Where do you store the position of the records. I was thinking about to use this position as the RecordNumber. What do you think about that?
1.a:Just another question: For the implementation of the createRecord-method, which position in the file will you choose to add the new Record, at the end of the file?
2.In the requirement it's written:

Any methods that throw RecordNotFoundException should do so if a specified record does not exist or is marked as deleted in the database file.

So don't we have to check not only if a Record is marked as deleted but also to check if it exists. So, I would have to check for each RecordNumber if the concerning Record is in the file? If the position number is the RecordNumber it would be possible to verify it very fast.

I am a bit mystified per to why you decided not to use the "Data.this" as the key to the hashmap? It solves a number of problems for you all in one-fell-swoop! 1. It gives you the unique key that you are looking for (assuming off-course that you are using one instance of the Data class for each client strategy),
...
3. Finally, coupled with the WeakHashMap use (only the declaration is different than a regular HashMap, otherwise the usage for our purposes is exactly the same as the regular HashMap), you get the deadlock resolution for free!

Yes, I was thinking also to go this way. Perhaps I'm a little bit afraid to stumble about something what isn't really required for this assignment.
I'm also a little bit afraid because my Data-class won't be the Remote-implementing Class. Like Max for the Danny's DVDStore, I want to create an Adapter-Class, so for example the booking-method would be very similar to yours, which you posted in another thread. So if the client crashes, there will still be a reference to Data.this in the Adapter-instance. But probably now, the instance of the Adapter-Class can also be garbage collected and then I'm fine?
I'm still thinking to go this way. But then I will have the question if I should also use a ReferenceQueue or something else to notify potential endless waiting threads, because no other thread entries in the synchronized block.
In the thread: NX: Thread safe and WeakHashMap vs. Unrefence Interface you posted:

do not know ReferenceQueue and how it works. I have not done it and am not going to do it until (if at all) I take care of the other requirements.

You know, I don't want to complicate my solutions, because when I take in consideration that, perhaps I should also take in consideration another that. But you can really comment my thoughts, perhaps I'm too lazy and wrong
You wrote:

2. More importantly, how do you ascertain that a client calling unlock method is indeed the client that locked the record in the first place? Since you are computing your key (cookie, whatever) value inside the method lock, how do you make sure that it is indeed the same value available to you when you attempt to unlock? In my opinion, that value is long gone. OK, so you store it in a HashMap, but how do you re-create it in the unlock method to compare it with the stored value?

I think we have different signatures of our lock() and unlock()-method like we noticed in NX: Thread safe and WeakHashMap vs. Unrefence Interface
Perhaps I can justify my choice by showing my code of the book-method in the Adapter-Class which is very similar to yours which you posted in another thread:

public void book(int recNo, String[] data)            
    throws IOException, RecordNotFoundException, SecurityException{        
                                 
      long lockCookie;       
                                   
      lockCookie = db.lock(recNo);                                         
      db.update(recNo,guidata,lockCookie);            
      db.unlock(recNo, lockCookie);        
                          
}

You can see, I store the lockCookie-value within a local variable.
I hope it wasn't too much,
thanks & bests
Ulrich

[Ulrich Heeger]

Hi Vlad, hi Max,
I appreciate very much your discussion an I will post my thoughts tomorrow.
Now, I have to get some sleep, I will surely have dreams (or nightmares? concerning locking, notifying etc.
Good night
Ulrich

[Vlad Rabkin]

Hi Ulrich,
You are not alone
Vlad

[Philippe Maquet]

Hi Vlad,
As far as notifyAll() is concerned, I understand things like you do. A thread entering a synchronized block blocks till it acquires the lock. No need for him to be notified in any way, it's automatic : it will acquire that "first level" (monitor) lock after a (or multiple) competition(s) with other incoming threads when the thread which owns the lock comes out of its synchronized block or enters in wait() (which releases the lock too). Now, only threads which are in that wait state need to be notified IMO. When there are, they compete with all "incoming" threads to acquire a lock on the monitor, before attempting to acquire the db lock.
Best,
Phil.

[Vlad Rabkin]

Hi Phil,
Finally, at least one challenger to support me !
But, there must be a reason why Max used notifyAll() in lock method (reserveDVDs()) in his book and you can kill me, but I don't understand what for!?

Best,
Vlad

[Ulrich Heeger]

Good morning,
Thanks Vlad for your words but I couldn't avoid this night my head to create and block threads
To ensure clean read/write, what do you think of my idea I posted before:

does it mean that I should have a special Class ReaderWriter and have a static instance of it within Data-Class?

If I synchronize the read and write-method I should be fine?
Concerning your discussion with Max, I would agree with you. In Max's book: A blocked thread is waiting for an event to occur. This event in that case isn't it the fact that the thread is leaving the synchronized block and so releasing the lock? But I'm sure, Max has surely a trump he hasn't played allready, I'm right?
Regards
Ulrich

[Philippe Maquet]

Hi Vlad,
Vlad:
But, there must be a reason why Max used notifyAll() in lock method (reserveDVDs()) in his book and you can kill me, but I don't understand what for!?
I see now what you are talking about : it's on p. 129 of Max's book, line 9 (notifyAll()) of his reserveDVD() method. Well, that line is even commented : "Line 9 notifies waiting threads that reservedDVDs is free, and the method returns". I do think it just should be mentioned in the errata : that line is useless IMO. Now maybe Max will explain why all of us are wrong... (I hope so !).

    public void reserveDVD(String upc) throws InterruptedException {
        synchronized (reservedDVDs) {
            //if the record is already locked, then wait
            while (reservedDVDs.contains(upc)) {
                reservedDVDs.wait();
            }
            
            reservedDVDs.add(upc);
            reservedDVDs.notifyAll();
        }
    }

Best,
Phil.

[Vlad Rabkin]

Hi,
Phil:

that line is useless IMO. Now maybe Max will explain why all of us are wrong... (I hope so !).

Ulrich:

But I'm sure, Max has surely a trump he hasn't played allready, I'm right?

We will see. There are two possible scenarios:
1) Max will explain us the reason and I will feel sorry for my hint given to Ulrich.
2) Max will give up and I will get drunk, because it will be the first time I would manage to defend my idea under his agression

does it mean that I should have a special Class ReaderWriter and have a static instance of it within Data-Class?
If I synchronize the read and write-method I should be fine?

It depends what you mean. Do you mean ReadWriteLock mechanism (concurrent reads, exclusive writes)? Or do you mean just a mutex (exclusive reads, exclusive writes) ? I guess you ment a mutex.
If so, you don't need a separate static object. you have already one: reservedRecords. You can synchronize all you public methods on this objects.
To my opinion it should be Ok. That is what I am doing.
Best,
Vlad
[ September 19, 2003: Message edited by: Vlad Rabkin ]

[Bharat Ruparel]

Hello Ulrich,
Sorry could not respond quicker. Have been busy both at the work and home fronts. You wrote:

Concerning this point I have two questions:
1.Where do you store the position of the records. I was thinking about to use this position as the RecordNumber. What do you think about that?
1.a:Just another question: For the implementation of the createRecord-method, which position in the file will you choose to add the new Record, at the end of the file?

One thing that I might not have clarified for you is that I have NO CACHING AT ALL! I was considering going the route of caching the deleted record numbers, but it all boils down to the same thing. You better allow for latency. I am not saying that it cannot be done. It seems like most people are doing it already. I just chose not to do any caching at all! I figured since performance is NOT a criteria, I will keep my data access code as simple as possible, and see what happens. It seems like at least on my "old" laptop, a Compaq Evo N600c which is a Pentium III machine, the performance is quite acceptable. One key thing that you have to realize is that you are dealing with a RandomAccessFile (I also chose not to do anything with NIO, another simplifying factor). With those clarifications, let me see if I can answer your questions above:
1.Where do you store the position of the records. I was thinking about to use this position as the RecordNumber. What do you think about that? - We have a data file with a "fixed" header. You need to compute those fixed number of bytes and store it in an instance variable. I called it "recOffset", you can call it whatever that you want. I am again assuming that you have a class similar to my "DataSchema" class that stores all this "header" information - I have made it a singleton since it needs to do all this computation only once and this information doesn't change. The data class holds a reference to the DataSchema singleton instance. You can always get the total length of the file anytime you want. Subtract the recOffset into the length of the file to get the record data bytes. I also keep the total length of the record information in my DataSchema class instance. Add one to the record length to account for the "deleted" byte flag. Now divide it into the record data bytes that you computed. This will give you total number of records current the data file both deleted and active. Now how do you count the records is upto you, but since everything in Java is zero based, I chose to count the records starting with zero to keep everything in sync with the data structures (ArrayList, Array t name a few) that I use. Therefore record zero is really record one and so on. Now, consider the delete method signature for example: it takes recNo as a parameter - i.e., delete(int recNo). You do not have to store the record numbers
1. Validate the record number, i.e., see if the record number is in range and it is not already deleted - throw RecordNotFoundException if either of these two tests fail.
2. Set the deleted flag for this record as "on" by setting the byte to whatever value you have been asked to - my instructions say "1".
You can see nowhere did I have to store the record numbers!
1.a:Just another question: For the implementation of the createRecord-method, which position in the file will you choose to add the new Record, at the end of the file? - I access the first record (OK record number zero to be precise) deleted flag and see if it deleted if not I access the next record's flag and so on. Again, you do not have to actually read all record fields ONLY the deleted flag using "seek". Therefore, it all works out!
Regards.
Bharat

[Ulrich Heeger]

Hi Vlad,
thanks for your comments and for your help
You quote:

It depends what you mean. Do you mean ReadWriteLock mechanism (concurrent reads, exclusive writes)? Or do you mean just a mutex (exclusive reads, exclusive writes) ? I guess you ment a mutex.
If so, you don't need a separate static object. you have already one: reservedRecords. You can synchronize all you public methods on this objects.

I haven't heard something about mutex. Does it mean that the sense of a mutex consists in acquiring its lock to ensure exclusive methods? I mean there wouldn't be necessarily actions on this object, the goal ist just to get its lock?
Thanks a lot in advance,sorry to continue annoying you
Ulrich

[Ulrich Heeger]

Hi Bharat,
perhaps I annoy you when I always say thank you, but I feel really so much thankfulness toward you, Vlad, Max, Andrew, Phil and the others

One thing that I might not have clarified for you is that I have NO CACHING AT ALL!

That's also my way

It seems like at least on my "old" laptop, a Compaq Evo N600c which is a Pentium III machine, the performance is quite acceptable.

We have a data file with a "fixed" header. You need to compute those fixed number of bytes and store it in an instance variable. I called it "recOffset", you can call it whatever that you want. I am again assuming that you have a class similar to my "DataSchema" class that stores all this "header" information - I have made it a singleton since it needs to do all this computation only once and this information doesn't change. The data class holds a reference to the DataSchema singleton instance. You can always get the total length of the file anytime you want. Subtract the recOffset into the length of the file to get the record data bytes. I also keep the total length of the record information in my DataSchema class instance. Add one to the record length to account for the "deleted" byte flag. Now divide it into the record data bytes that you computed. This will give you total number of records current the data file both deleted and active. Now how do you count the records is upto you, but since everything in Java is zero based, I chose to count the records starting with zero to keep everything in sync with the data structures (ArrayList, Array t name a few) that I use. Therefore record zero is really record one and so on. Now, consider the delete method signature for example: it takes recNo as a parameter - i.e., delete(int recNo). You do not have to store the record numbers

can I just resume?
So you store the position where the Records-bytes begin in the variable recOffset, then you store the number of bytes of one Record, 160, in another variable, for example recLen. The size of the ArrayList determine the number of the Records and the RecNo is the position in the ArrayList. I'm right?
Just another question: For accessing the Data-File, within Data.class do you read suncertify.properties to get the position of the db.file?
Thanks & regards
Ulrich

[Bharat Ruparel]

Hello Ulrich,
You don't annoy me at all! I wouldn't worry about it if I were you. I have bugged Andrew, Max, Phil a whole lot more than you have asked me questions. Glad to be of service. Just pass it around and share your knowledge with all of us especially the newcomers.
You wrote:

So you store the position where the Records-bytes begin in the variable recOffset, then you store the number of bytes of one Record, 160, in another variable, for example recLen. The size of the ArrayList determine the number of the Records and the RecNo is the position in the ArrayList. I'm right?

Quite correct. This may be just details but the size of record really is 159 bytes plus one for deleted flag. As long as you keep that in mind, we are OK. One thing that I forgot to mention or reply to is that for the create method you sequentially start accessing the deleted flag for record number 0,1,2,3... and so on. If you find a deleted record, stop, update it (really overwrite it) with your new records data keeping in mind the fixed length strings. If you don't find a deleted record, just append it to the end of file as we normally do.
You wrote:

Just another question: For accessing the Data-File, within Data.class do you read suncertify.properties to get the position of the db.file?

I am trying to. That is the only significant piece that I have left. Unfortunately, my attempts to create a dialog-box derived from the JDialog box have not been successful thus far. See my last post in URLy Bird 1.3.1 Suncertify.properties thread.
Regards.
Bharat

[Philippe Maquet]

Hi Bharat,
Just one question : what are you doing with ArrayList structure(s) indexed by recNo if you have no cache ?
Best,
Phil.

[Ulrich Heeger]

Hi Bharat,
you wrote:

One thing that I forgot to mention or reply to is that for the create method you sequentially start accessing the deleted flag for record number 0,1,2,3... and so on. If you find a deleted record, stop, update it (really overwrite it) with your new records data keeping in mind the fixed length strings.

I am trying to. That is the only significant piece that I have left. Unfortunately, my attempts to create a dialog-box derived from the JDialog box have not been successful thus far. See my last post in URLy Bird 1.3.1 Suncertify.properties thread.

Good luck, I wanted also to join this thread but then I recognized that I would put myself in confusion because I'm still busy in working out my data access layer... I try always to solve all problems all at once, so I have to learn to concentrate myself just on a small scope
Thanks & regards
Ulrich

[Bharat Ruparel]

Hello Phil,
You wrote:

Just one question : what are you doing with ArrayList structure(s) indexed by recNo if you have no cache ?

In my Data class, I make use of ArrayList structure only in the find method where I am not sure how many records will match a given criteria. Once I have populated the ArrayList with desired record set, I create an array of "int" to return to the calling class (DataAdapter in my case) so that it matches the method signature specified by the DBMain interface. I could have just easily used a TreeSet instead.
Regards.
Bharat

[Philippe Maquet]

Hi Bharat,

I could have just easily used a TreeSet instead.

Or even an HashSet. I think that, for your purpose, both are better IMO than an ArrayList indexed by recNo.
Best,
Phil.

[Ulrich Heeger]

Hi Bharat,
you wrote:

You don't annoy me at all! I wouldn't worry about it if I were you

you shouln't have written it, because now I will bombard you with questions
No, I will try to skimp them and to get them as consise as I can
1.Some lines before you wrote:

I am a bit mystified per to why you decided not to use the "Data.this" as the key to the hashmap? It solves a number of problems for you all in one-fell-swoop! 1. It gives you the unique key that you are looking for (assuming off-course that you are using one instance of the Data class for each client strategy),
...
3. Finally, coupled with the WeakHashMap use (only the declaration is different than a regular HashMap, otherwise the usage for our purposes is exactly the same as the regular HashMap), you get the deadlock resolution for free!

After reflecting a while, I think I will also go this way.
But I worried about the situation which I described above:
I'm also a little bit afraid because my Data-class won't be the Remote-implementing Class. Like Max for the Danny's DVDStore, I want to create an Adapter-Class, so for example the booking-method would be very similar to yours, which you posted in another thread. So if the client crashes, there will still be a reference to Data.this in the Adapter-instance. But probably now, the instance of the Adapter-Class can also be garbage collected and then I'm fine?

Because I have seen that you have the same implementation, you think we wouldn't be in trouble?
2. I don't know if you have followed the discussion between Vlad, Max and me above. Because at Topic: NX: URLy Bird 1.3.1 Explicit Fatal Exception Handling I saw that you also include a notifyAll()-call in your lock-method. We are still waiting for Max's reply,

In my Data class, I make use of ArrayList structure only in the find method where I am not sure how many records will match a given criteria. Once I have populated the ArrayList with desired record set, I create an array of "int" to return to the calling class (DataAdapter in my case) so that it matches the method signature specified by the DBMain interface

3. I have a problem understanding the use of your ArrayList in the Data-Class:
Supposed a client is connecting to the server, then there will be a call to get all the Records of the File to be shown at the GUI, I name it getRecords(). I store the Records as String-Arrays, including the RecordNumber, in an ArrayList, check which are valid and give them back as a Collection to the client.
So each time a client makes a request of booking, I got the RecordNumber from him and I can directly access the Record at the concerning position in the File
But if the client wants to find Records with search-criterias, I will read once more the File, store all once more all Records in an ArrayList, create a new local ArrayList and iterates through it to get the RecordNumbers which match with the search-criterias and give the int-Array with the RecordNumbers back to the client.
Is that the way it works?
Thanks & Regards,
Ulrich

[Bharat Ruparel]

Hello Ulrich,
You wrote:

After reflecting a while, I think I will also go this way.
But I worried about the situation which I described above:
I'm also a little bit afraid because my Data-class won't be the Remote-implementing Class. Like Max for the Danny's DVDStore, I want to create an Adapter-Class, so for example the booking-method would be very similar to yours, which you posted in another thread. So if the client crashes, there will still be a reference to Data.this in the Adapter-instance. But probably now, the instance of the Adapter-Class can also be garbage collected and then I'm fine?

Your Data-class doesn't need to be the remote implementing class. I also wrap the Data class inside a DataAdapter class just as Max does. Even DataAdapter doesn't need to be a Remote Object! Rather, the DataAdapter class is used inside a class I called RoomConnector which has two static methods getLocal() and getRemote(). getLocal() directly instantiates an instance of the DataAdapter class but returns it as an interface (I called it DBClient, this interface defines only the business methods book() and find()). The other method getRemote() returns an instance of the ConnectionFactory. The object that implements the ConnectionFactory interface is a Remote object (conventionally it is given a name postfixed with Impl, .e.g., the ConnectionFactory implementing class is called ConnectionFactoryImpl).
Now your client can call a remote method on the ConnectionFactory which is implemented by its implementation class, i.e., ConnectionFactoryImpl. This remote method, I called it create since I like the EJB analogy that Max has been mentioning in is past posts (I have some J2EE experience) will in turn return the remote object which contains an instance of the DataAdapter class. Therefore DataAdapter which wrapps Data, is also blissfully unaware of whether it is being used by a remote client or a local client! It is a bit hard to get your head around this concept until you go through the fundamentals of RMI course. Andrew had posted a link to Sun's RMI tutorial which is excellent. The link is as follows:
http://developer.java.sun.com/developer/onlineTraining/rmi/RMI.html
I strongly recommend that you spend sometime going through it. Once you understand it, things will fall in place for you.
A reference to the Data class is contained in the DataAdapter class which stays around until the client is around and disappears when the client terminates the session normally or abnormally (it crashes!). In both of these situations, the DataAdapter class and hence the Data class are eligible for the garbage collection. Therefore, the WeakHashMap will eventually remove this instance of the Data Class. Now, I haven't simulated this crash-test yet, but am willing to take Max's word for it until I get to the point (quite soon!) where I can test it myself!
You wrote:

I don't know if you have followed the discussion between Vlad, Max and me above. Because at Topic: NX: URLy Bird 1.3.1 Explicit Fatal Exception Handling I saw that you also include a notifyAll()-call in your lock-method. We are still waiting for Max's reply,

I am not doing anything unusual here, just following Max's lead. Did you follow the DVD example lock method in his book? It is the same concept.
You wrote:

I have a problem understanding the use of your ArrayList in the Data-Class:
Supposed a client is connecting to the server, then there will be a call to get all the Records of the File to be shown at the GUI, I name it getRecords(). I store the Records as String-Arrays, including the RecordNumber, in an ArrayList, check which are valid and give them back as a Collection to the client.
So each time a client makes a request of booking, I got the RecordNumber from him and I can directly access the Record at the concerning position in the File
But if the client wants to find Records with search-criterias, I will read once more the File, store all once more all Records in an ArrayList, create a new local ArrayList and iterates through it to get the RecordNumbers which match with the search-criterias and give the int-Array with the RecordNumbers back to the client.
Is that the way it works?

Think about it. The ArrayList is a dynamic data structure which will store the "live" records matching the supplied "criteria". It does not mean that the zeroth postion of the ArrayList stores record number zero and so on. Suppose for example, the record numer zero, one, and two are deleted records in the datafile and also suppose that the third record in the data file is a live record and it also matches the find criteria supplied. You will store its number, i.e., 3 in the zeroth cell for the ArrayList. Now when you convert the ArrayList into an Array of ints (or longs if that is what your method signature says), then the zeroth cell of the record number array will contain the value 3! Another thing to keep in mind is that you want to declare an ArrayList as an instance variable in the DataAdapter class and set the returned array from the Data class to this ArrayList. This is the ArrayList that is given to the GUI TableModel to construct the tabular display of the rooms. Depending on whichever row (proxy for the room to be booked) is selected by the user, you look-up the record number in the ArrayList. Therefore, the ArrayList stored in the DataAdapter class is really a list of pointers! It is really cool! I owe it to Andrew/Max/Phil/Vlad to walk me through this process. Take your time doing it, you will be quite pleased with how elegant this all turns out to be.
Regards.
Bharat

[Philippe Maquet]

Hi Bharat,

You (Ulrich) wrote:
quote:
--------------------------------------------------------------------------------
I don't know if you have followed the discussion between Vlad, Max and me above. Because at Topic: NX: URLy Bird 1.3.1 Explicit Fatal Exception Handling I saw that you also include a notifyAll()-call in your lock-method. We are still waiting for Max's reply,
--------------------------------------------------------------------------------
I am not doing anything unusual here, just following Max's lead. Did you follow the DVD example lock method in his book? It is the same concept.

It's nice to follow Max's lead. But the discussion Ulrich is talking about is in this thread (the one you are reading), is still pending, and is about the DVD example lock method in his book Vlad, Ulrich and I disagree with (or at least don't understand). Let's wait what Max will reply ...
Best,
Phil.

[Ulrich Heeger]

Hi Bharat,
thanks for your statements
1.
You wrote:

I called it DBClient, this interface defines only the business methods book() and find().

Why do you define only two methods in this interface? Shouldn't you have at least getRecords()?
Because I wrote:

Supposed a client is connecting to the server, then there will be a call to get all the Records of the File to be shown at the GUI, I name it getRecords().

Isn't that the way your implementation works?
2.
you wrote:

A reference to the Data class is contained in the DataAdapter class which stays around until the client is around and disappears when the client terminates the session normally or abnormally (it crashes!). In both of these situations, the DataAdapter class and hence the Data class are eligible for the garbage collection. Therefore, the WeakHashMap will eventually remove this instance of the Data Class. Now, I haven't simulated this crash-test yet, but am willing to take Max's word for it until I get to the point (quite soon!) where I can test it myself!

You have conviced me I will also go this way

[B3.[/B]
You wrote:

Think about it. The ArrayList is a dynamic data structure which will store the "live" records matching the supplied "criteria". It does not mean that the zeroth postion of the ArrayList stores record number zero and so on. Suppose for example, the record numer zero, one, and two are deleted records in the datafile and also suppose that the third record in the data file is a live record and it also matches the find criteria supplied. You will store its number, i.e., 3 in the zeroth cell for the ArrayList. Now when you convert the ArrayList into an Array of ints (or longs if that is what your method signature says), then the zeroth cell of the record number array will contain the value 3! Another thing to keep in mind is that you want to declare an ArrayList as an instance variable in the DataAdapter class and set the returned array from the Data class to this ArrayList. This is the ArrayList that is given to the GUI TableModel to construct the tabular display of the rooms. Depending on whichever row (proxy for the room to be booked) is selected by the user, you look-up the record number in the ArrayList. Therefore, the ArrayList stored in the DataAdapter class is really a list of pointers! It is really cool! I owe it to Andrew/Max/Phil/Vlad to walk me through this process. Take your time doing it, you will be quite pleased with how elegant this all turns out to be.

I don't know if it's ok if I show my code, but I will just do it
within the Data.class:

public long[] findByCriteria(String[] criteria){

       long[] retVal;
ArrayList matchingRecords = new ArrayList(); 

        //Here some code for storing the matching
        //Records in matchingRecords

        //create a new long[]-Array with the length of 
        //the matching Records
        retVal = new long[matchingRecords.size()];
  
        //convert the ArrayList into an Array of longs 
         for (int i = 0; i < retVal.lenght; i++){
      retVal[i] = ((Long)matchingRecords.get(i)). 
                                               getLong();
         }
  
         return retVal;

     } 

and within the DataAdapter-Class

public Collection find(String[] criteria) throws RuntimeIOException,
RuntimeInterruptedException{
 
     //instance ArrayList null-referencing
     list = null;  
     long[] findNo;
     String[] data;
 
     findNo = db.findByCriteria(criteria);
     try{
 
for (int i = 0; i<findNo.length; i++){
     data = db.readRecord(findNo[i]);
     list.put(data);
}  
     }
     catch(RecordNotFindException re){
 re.printStackTrace();
     }
     return list;
}

Is that similar to your design?
I hope that's not too much for you. If yes you could perhaps give me the link to the thread where this topic has been discussed?
Thanks a lot & regards
Ulrich
[ September 21, 2003: Message edited by: Ulrich Heeger ]

[Bharat Ruparel]

Hello Ulrich,
You wrote:

Why do you define only two methods in this interface? Shouldn't you have at least getRecords()?
Because I wrote:

Sorry, I have a method similar to yours which I call getRoomsUsingCriteria(String [] criteria).
You wrote:

Isn't that the way your implementation works?

Yes.
Your code looks good to me. Make sure that you compile it and run it and more importantly test it. I am not sure what is your RuntimeInterruptedException in the find method within the DataAdapter class?
Looks good! Run with it.
Regards.
Bharat

[Philippe Maquet]

Hi Ulrich,
yesterday, reading your findByCriteria() method, I thought "tomorrow I'll have to comment it". But as I see now that you stripped the interesting lines off, I cannot do it anymore... Notice that that new version is perfect !
Best,
Phil.

[Ulrich Heeger]

Hi Bharat,
thanks a lot, I will compile & test the code.

I am not sure what is your RuntimeInterruptedException in the find method within the DataAdapter class?

You're right, there won't be a RuntimeInterruptedException.
Thanks & regards
Ulrich