NX:Client crashed cause deadlock in LockManager

[Jim Yingst]

[Tony]: If you use File Channels do you have to lock the database at all ?
If you're doing one of the new-format (NX) assignments you probably have to implement a particular API which includes lock() and unlock() methods, and requires that update() and delete() throw a SecurityException if called on an unlocked record. Thus locking is required, regardless of whether you use FileChannel, RAF, or something else.
[Max]: 3. The real advantage of FileChannels (here)are guaranteed atomic actions, Interruptible IO, and better performance.
Eh, I still think you overstate the "guaranteed atomic" part, IMO. See my comments at the end of this thread. But interruptibility and performance are still excellent reasons to learn NIO.

[Tony Collins]

I agree Lock/unlock are implemented by the API, my point is that locking on a file level by sycronizing on the cache is unnecessary.
As I understand their is no possibility of dirty reads when using file channels either(if you write the whole record in one operation). As the write is atomic, the fact that the write to a position is not atomic is irrelevant. As the position of a record will never change and the write once positioned will be atomic.
So I would say file channels are the way to go and the new project seems set up to use them and reap their benefits. Though I may misunderstand something.
Tony

[Vlad Rabkin]

Hi Max,

Why did you decide to synchronize read and find

1) To avoid dirty-reads:
If somebody writes in database, allowing to read from it will cause the user get wrong data.
Instance: while looping the file to find records,
some records can be deleted in meantime (if we don't synchronize on read and find). So, a user get a set of record IDs, containing already non-existant records.
2) To avoid data corruption:
Instance: The hotel name is "Sheraton". Somebody updates the name of the hotel: at the moment he writes in the file new Name (Holydays Inn). Tree letters have been already over-written:
"Holraton"). Somebody reads the record to book the room and the updates the whole record. Corrupted name will be saved:
Hotel: Sheraton Customer: _
Hotel: Holyday Inn Customer: _
Hotel: Holraton Customer: Max
If I understood you correctly, the second problem is not an issue with FileChannel: because record read automaticly blocks (without synchronizing it) till write finishes. Am I right?
We still have the first problem - dirty reads. Hamdling them can be more annoying as making complex locks mechanism....

Thanx,
Vlad.

[Tony Collins]

I think that displaying a deleted record is not a problem, well it is but it's beyond the scope of the project to solve it.
What if the record was deleted after it has been displayed by the client but before the client tries to book the record. We still get the same effect. Deleted records are displayed. The user tries to book the contracter, but it's not there.

Maybe some sort of the Observer pattern could help but is this overkill ?
Tony

[Vlad Rabkin]

Hi Max,
I have forgotten to say the biggest problem by not synchnonizing while find:
Since I have cashed database, I have to iteratate through the HashMap to find all record. The specification of HashMap's iterator says:

The iterators returned by all of this class's "collection view methods" are fail-fast: if the map is structurally modified at any time after the iterator is created, in any way except through the iterator's own remove or add methods, the iterator will throw a ConcurrentModificationException.
...
Fail-fast iterators throw ConcurrentModificationException on a best-effort basis. Therefore, it would be wrong to write a program that depended on this exception for its correctness: the fail-fast behavior of iterators should be used only to detect bugs

So, if allow writes while searching I have 99% opportunity get this Exception, which I am not supposed to handle, since it is only for bug detection...
Vlad

[Vlad Rabkin]

Hi Tony,

What if the record was deleted after it has been displayed by the client but before the client tries to book the record.

That is not a big deal. My client first lock the record. Then tries to read it. If the record is not found or (which very likely) already booked by someone else, I show the appropriate message and ask him to refresh the data on the screen.
Observer pattern is pretty good idea. I know that one of us has done it, because he wanted to have checked exception to be throught by server in case of IOException. To my opinion it is
1. overhead
2. doesn't really good fit in our application. It would be perfect solution if we wanted sreens of each user to be updated, when the state of database changes: let's say new hotel rooms or flights become available, but it is the case.
Cheers,
Vlad

[Tony Collins]

Why not just use a get(record No) to get each record. Loop around from recordNo 0 to recordNo mapName.size().
I think locking each all records on a find could be over the top. I really am relising how tough server side programming can be.
Tony

[Vlad Rabkin]

Hi Tony,

Why not just use a get(record No) to get each record. Loop around from recordNo 0 to recordNo mapName.size().

first, you don't know the sequence of record IDs:
Instance:
1 is active
2 is deleted
3 is active
record 2 is not in HashMap, because it is deleted.
So you can't just loop continiously from 1 to size
second:
Imagine following: HashMap size was 30.
you started to loop over. In the meanwhile somebody deleted 2 records. By reaching your 29 element you will get "IndexOutOfBoundsException",
which is RuntimeException.

All of this are drawbacks of cashing the Database...
Cheers,
Vlad

[Max Habibi]

Originally posted by Jim Yingst:
[Tony]: If you use File Channels do you have to lock the database at all ?
If you're doing one of the new-format (NX) assignments you probably have to implement a particular API which includes lock() and unlock() methods, and requires that update() and delete() throw a SecurityException if called on an unlocked record. Thus locking is required, regardless of whether you use FileChannel, RAF, or something else.
[Max]: 3. The real advantage of FileChannels (here)are guaranteed atomic actions, Interruptible IO, and better performance.
Eh, I still think you overstate the "guaranteed atomic" part, IMO. See my comments at the end of this thread. But interruptibility and performance are still excellent reasons to learn NIO.

I looked over your comments Jim, and posted an answer. When I say 'atomic', it seems like I failed to be completely clear. In older versions of the jdk, you could write to a file, finish, and still not actually have the data in the file. That is, under the covers, the JVM was still negotiating with the OS to get all the data out. Thus, you could think that you were done without actually being done. FileChannels don't do this: they don't return until they're actually done. Hence, atomic writes.
All best,
M

[Max Habibi]

Originally posted by Vlad Rabkin:
Hi Tony,

first, you don't know the sequence of record IDs:
Instance:
1 is active
2 is deleted
3 is active
record 2 is not in HashMap, because it is deleted.
So you can't just loop continiously from 1 to size

Hi Vlad,
This depends on your implementation choices: There's no reason to remove deleted records: just mark then deleted. That is, have a isDeleted method on your Record object.

second:
Imagine following: HashMap size was 30.
you started to loop over. In the meanwhile somebody deleted 2 records. By reaching your 29 element you will get "IndexOutOfBoundsException",
which is RuntimeException.

Your jdbc layer probably doesn't (often) lock down for every read operation, and is often susceptible to the same sort of challenge. It's common problem in Computer science. Let's dig in a bit.
The way this generally works is this. Your thread, say Thread #1, makes a local copy of the map(for efficiency), and does it's work there(this is Thread Memory, vs. Main Memory). Similarly, all other threads do the same thing: they make a local copy, and work from that copy. With synchronized code, these threads agree to certain binding constraints on when and how they update/read the original data from their own local copies. Unsynchronized threads aren't bound by most of those constraints, so it's very dangerous for them to update the data. However, when reading it, they are doing so from their own local copies (which reside in Thread Memory). That local state, even if inaccurate, is safe.
The point here is that your dirty reads may be inaccurate, but I don't think they'll throw the kind of exceptions you're talking about.

All of this are drawbacks of cashing the Database...
Cheers,
Vlad

This isn't a drawback of caching: it's a drawback of reading . Even without caching, you'd still have to decide if you wanted to do a dirty read or a locked reads: it's just a matter of doing File IO or not doing File IO.
All best,
M
[ July 21, 2003: Message edited by: Max Habibi ]

[Vlad Rabkin]

Hi Max,

The way this generally works is this. Your thread, say Thread #1, makes a local copy of the map(for efficiency), and does it's work there(this is Thread Memory, vs. Main Memory).

I thought about this yesterday. It would help...
The problem is if your database contains many records it will KILL the server:
Imagine your hashMap size currently is about 100 MB (which is Ok: it is about 700,000 records).
100 Clients want to search. Your have to make a local copy (clone your HashMap) for every client thread.
Upssss: U suddenly need 100MBx100, which is fast 10GB!!!
Max,
Could you please confirm my assumption:
FileChannel: write block all other operation on the object, it means that reads will also automatically wait till write finishes.
Is it correct?
Thanx,
Vlad
[ July 21, 2003: Message edited by: Jim Yingst ]

[Vlad Rabkin]

Hi Max,
Sorry, I guess I understood not correctly...

makes a local copy of the map

You ment probably local copy in case it is a local instance, but it still impossible, because our HashMap is a member variable, not a local one.

This depends on your implementation choices: There's no reason to remove deleted records: just mark then deleted. That is, have a isDeleted method on your Record object.

Ok, but the sme applies to adding a record (somebody can in the meantime add a new record to the database.)
Vlad

[Max Habibi]

[Max]: The way this generally works is this. Your thread, say Thread #1, makes a local copy of the map(for efficiency), and does it's work there(this is Thread Memory, vs. Main Memory).
[Vlad]: I thought about this yesterday. It would help...
The problem is if your database contains many records it will KILL the server:
Vlad,
I think you might enjoy reading a bit more about this.
http://java.sun.com/docs/books/jls/second_edition/html/memory.doc.html#28458

Imagine your hashMap size currently is about 100 MB (which is Ok: it is about 700,000 records).
100 Clients want to search. Your have to make a local copy (clone your HashMap) for every client thread.
Upssss: U suddenly need 100MBx100, which is fast 10GB!!!
Max,
Could you please confirm my assumption:
FileChannel: write block all other operation on the object, it means that reads will also automatically wait till write finishes.
Is it correct?
Thanx,
Vlad

Not exactly. Write(or read, or anything else that changes position) blocks all other operations that also change position. But operations that don't change position don't get blocked. It's all in the documentation for the FileChannels.
M
[ July 21, 2003: Message edited by: Max Habibi ]
[ July 21, 2003: Message edited by: Jim Yingst ]

[Jim Yingst]

Vlad: please don't use [ code ] blocks to quote someone else's test. Or if you do, make sure you put in line breaks as necessary. When you put [ code ] around text with no line breaks, it means that the text will appear as a single line, which may be exceedingly long. This in turn forces the whole web page to be extra-wide, which means we have to use our scroll bars a lot to read any posts on the page, which is a pain. I've edited a couple posts of yours above to remove [ code ] tags that were causing such problems.

[Vlad Rabkin]

Hi Jim,
Sorry ...
I will not do it again.
Thank you that you fixed the format.
Vlad

[Jim Yingst]

Sorry ...
No apology needed; I know it's far from obvious when you post. And someone needs to make these mistakes so the rest of us can learn from them.

[Jim Yingst]

Oh, and I just remembered that there's a nasty bug in the system when you post something containing a [ quote ] block followed sometime later by a [ code ] block - the system may delete some of the text between the two. So in many cases it's safest to avoid [ quote ] tags entirely (since we certainly need [ code ] for many things; can't drop that. That's why I often use other techniques to quote posters, e.g.
[Vlad]: xxxx
[Max]: yyyy
Sometimes I use [ quote ] anyway, but then I later remember why it can't be trusted...

[Vlad Rabkin]

Hallo Max,
Thank you for the link on Threads. I thought Threads craete own copies of the object only for local variables. It is very usefull information. The style of writing this article is a bit complicated, so I will have to read it couple more times...

Vlad