I have a real puzzle with "freezing" that I've been struggling with for over a year now. This has been perhaps the best java forum that I've been a part of, so I'm hoping that you can help. First, I'll describe the layout (see attached flowchart), then the problem.
The Layout JSP pages and Server B both make calls to a mysql database through some intermediaries. All calls to the database are brokered by the Database Server. To get Object O from the database, the Server B would make the following rmi call: databaseServer.getObjectO(); A JSP page makes a similar rmi call, but through an additional interface class, JSP Interface Class, which in turn relays the call to Database Server. For example:
JSP page calls jspInterfaceClass.getObjectO(); causing the instance of JSP Interface Class to return the results of the call databaseServer.getObjectO();
The Problem Occasionally, an update call from Server B randomly hangs as it waits for the update method to return. It is always the same updateMyObjects(Object myObjects) method that hangs. The update method doesn't always hang. It succeeds a few thousand times for every hang.
Debug Info Here are the possibly suspects that I've considered:
bad/invalid data/calls - rerunning the Server B's process in a way that produces identical results does not result in a hang.
mysql deadlocks - I didn't find any mysql deadlocks despite checking the processlist during a hang.
rmic failures - This doesn't seem likely since I'd naively thing that they should happen for all rmi calls equally.
database server crashes - The first line of the updateMyObjects prints out to a log file. I've verified that this line does not print during a hang.
restarting the database server - I put the call to the update method in a try block and put a long pause, reconnect, and recall the same update method in the catch block. Upon getting a hang, if I restart the Database Server the method runs without trouble.
synchronization - I've tried making every method in the Database Server that communicates with relevant mysql table be synchronized. This didn't help the problem at all. Al methods in the JSP Interface Class are synchronized.
I've new information from calling jstack during one of these "freezes". Any help would be greatly appreciated!
Here is the relevant portion of a jstack from the Database Server:
And here is the relevant portion from Server B:
What are my options for handling this? "Hanging" is a real issue - I currently am forced to call the shutdown() method on the Database Server causing Server B's rmi request to fail at which point it restarts the Database Server and resubmits the same BasicStructure without a problem. Rmi timeout and retry, or an Exception being thrown would be acceptable behavior that could easily be handled.