GeeCON Prague 2014*
The moose likes Distributed Java and the fly likes Freezes - A real puzzle Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


JavaRanch » Java Forums » Java » Distributed Java
Bookmark "Freezes - A real puzzle" Watch "Freezes - A real puzzle" New topic
Author

Freezes - A real puzzle

Joel Sander
Ranch Hand

Joined: Jan 09, 2009
Posts: 38
I have a real puzzle with "freezing" that I've been struggling with for over a year now. This has been perhaps the best java forum that I've been a part of, so I'm hoping that you can help. First, I'll describe the layout (see attached flowchart), then the problem.


The Layout
JSP pages and Server B both make calls to a mysql database through some intermediaries. All calls to the database are brokered by the Database Server. To get Object O from the database, the Server B would make the following rmi call: databaseServer.getObjectO(); A JSP page makes a similar rmi call, but through an additional interface class, JSP Interface Class, which in turn relays the call to Database Server. For example:

JSP page calls jspInterfaceClass.getObjectO(); causing the instance of JSP Interface Class to return the results of the call databaseServer.getObjectO();

The Problem
Occasionally, an update call from Server B randomly hangs as it waits for the update method to return. It is always the same updateMyObjects(Object[] myObjects) method that hangs. The update method doesn't always hang. It succeeds a few thousand times for every hang.

Debug Info
Here are the possibly suspects that I've considered:

bad/invalid data/calls - rerunning the Server B's process in a way that produces identical results does not result in a hang.

mysql deadlocks - I didn't find any mysql deadlocks despite checking the processlist during a hang.

rmic failures - This doesn't seem likely since I'd naively thing that they should happen for all rmi calls equally.

database server crashes - The first line of the updateMyObjects prints out to a log file. I've verified that this line does not print during a hang.

restarting the database server - I put the call to the update method in a try block and put a long pause, reconnect, and recall the same update method in the catch block. Upon getting a hang, if I restart the Database Server the method runs without trouble.

synchronization - I've tried making every method in the Database Server that communicates with relevant mysql table be synchronized. This didn't help the problem at all. Al methods in the JSP Interface Class are synchronized.

Any ideas?



[Thumbnail for flowchart.jpg]

William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12792
    
    5
What is the CPU usage when the system freezes?

Bill
Joel Sander
Ranch Hand

Joined: Jan 09, 2009
Posts: 38
William Brogden wrote:What is the CPU usage when the system freezes?

Bill


Good question. I'm fairly sure that it is normal (I don't remember checking, but it is the kind of thing I'd be unlikely to miss). I'll make sure to check next time it freezes which may be a few days.

To be clear, the entire system doesn't freeze, just Server B's update process. Call from the JSP Pages continue to go through without trouble.
Ganesh Kandisa
Greenhorn

Joined: Jul 27, 2011
Posts: 6

.


Ganesh Kandisa
OCPJP 6(93%),OCPWCD 5(92%)
Joel Sander
Ranch Hand

Joined: Jan 09, 2009
Posts: 38
I've new information from calling jstack during one of these "freezes". Any help would be greatly appreciated!

Here is the relevant portion of a jstack from the Database Server:



And here is the relevant portion from Server B:




What are my options for handling this? "Hanging" is a real issue - I currently am forced to call the shutdown() method on the Database Server causing Server B's rmi request to fail at which point it restarts the Database Server and resubmits the same BasicStructure without a problem. Rmi timeout and retry, or an Exception being thrown would be acceptable behavior that could easily be handled.
Joel Sander
Ranch Hand

Joined: Jan 09, 2009
Posts: 38
Hung twice more today. Any suggestions at all?
 
GeeCON Prague 2014
 
subject: Freezes - A real puzzle