Win a copy of Think Java: How to Think Like a Computer Scientist this week in the Java in General forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Freezes - A real puzzle

 
Joel Sander
Ranch Hand
Posts: 38
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have a real puzzle with "freezing" that I've been struggling with for over a year now. This has been perhaps the best java forum that I've been a part of, so I'm hoping that you can help. First, I'll describe the layout (see attached flowchart), then the problem.


The Layout
JSP pages and Server B both make calls to a mysql database through some intermediaries. All calls to the database are brokered by the Database Server. To get Object O from the database, the Server B would make the following rmi call: databaseServer.getObjectO(); A JSP page makes a similar rmi call, but through an additional interface class, JSP Interface Class, which in turn relays the call to Database Server. For example:

JSP page calls jspInterfaceClass.getObjectO(); causing the instance of JSP Interface Class to return the results of the call databaseServer.getObjectO();

The Problem
Occasionally, an update call from Server B randomly hangs as it waits for the update method to return. It is always the same updateMyObjects(Object[] myObjects) method that hangs. The update method doesn't always hang. It succeeds a few thousand times for every hang.

Debug Info
Here are the possibly suspects that I've considered:

bad/invalid data/calls - rerunning the Server B's process in a way that produces identical results does not result in a hang.

mysql deadlocks - I didn't find any mysql deadlocks despite checking the processlist during a hang.

rmic failures - This doesn't seem likely since I'd naively thing that they should happen for all rmi calls equally.

database server crashes - The first line of the updateMyObjects prints out to a log file. I've verified that this line does not print during a hang.

restarting the database server - I put the call to the update method in a try block and put a long pause, reconnect, and recall the same update method in the catch block. Upon getting a hang, if I restart the Database Server the method runs without trouble.

synchronization - I've tried making every method in the Database Server that communicates with relevant mysql table be synchronized. This didn't help the problem at all. Al methods in the JSP Interface Class are synchronized.

Any ideas?
flowchart.jpg
[Thumbnail for flowchart.jpg]
Flow chart
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13061
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What is the CPU usage when the system freezes?

Bill
 
Joel Sander
Ranch Hand
Posts: 38
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
William Brogden wrote:What is the CPU usage when the system freezes?

Bill


Good question. I'm fairly sure that it is normal (I don't remember checking, but it is the kind of thing I'd be unlikely to miss). I'll make sure to check next time it freezes which may be a few days.

To be clear, the entire system doesn't freeze, just Server B's update process. Call from the JSP Pages continue to go through without trouble.
 
Ganesh Kandisa
Greenhorn
Posts: 6
Eclipse IDE Java Spring
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
.
 
Joel Sander
Ranch Hand
Posts: 38
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I've new information from calling jstack during one of these "freezes". Any help would be greatly appreciated!

Here is the relevant portion of a jstack from the Database Server:



And here is the relevant portion from Server B:




What are my options for handling this? "Hanging" is a real issue - I currently am forced to call the shutdown() method on the Database Server causing Server B's rmi request to fail at which point it restarts the Database Server and resubmits the same BasicStructure without a problem. Rmi timeout and retry, or an Exception being thrown would be acceptable behavior that could easily be handled.
 
Joel Sander
Ranch Hand
Posts: 38
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hung twice more today. Any suggestions at all?
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic