We have faced an issue with production JBoss server. Server stopped responding for almost 10 minutes and returned back to normal state after it. During this interval clients were not able to connect and no error or warning messages were logged to log files. However we have an known and open bug in the deployed application. One connection is not closed and Resultset and Statement is also open on this connection, could this be cause of this problem ? Same application is deployed on other three JBoss server but none of these server had this issue, though waring "Closing a connection for you" is being logged frequently on all the four servers. Datasource is configured in mssql-ds.xml as non transactional DS. How JBoss Handles open resources or its Database Driver's call to handle the open resources? Here are the Environment and server details.
Environment Details :
Os : Linux 2.6.18-194.17.4.el5 (i386) CPU : 1
JVM Name : Java HotSpot(TM) Server VM
JVM version : 1.5.0_03-b07 (Sun Microsystems Inc.)
Sounds like a huge garbage collection pause. Are you by any chance running the CMS collector? There are ways of determining if GC is the issue, but not after the fact. For suggestions on how to gather GC data and analyze it, see these white papers:
Java Garbage Collection Statistical Analysis 101
Java Garbage Collection Performance Analysis 201
Thanks a lot for you reply. We got right direction to work.
Could you please also provide some help on these queries.
1 I monitored logs there was continuous logging in server.log during the interval for which client was not able to connect, so looks like JBoss was working in this duration, So only client was not able to connect for 10 minutes, In what situation this could happen ?
2 How are the resources left opened like Resultset, Statements and Connection are handled by JBoss if they are configured in *-ds.xml in JBoss.(What if developer forgot to close them). ?
Usually if a request doesn't close a datasource connection (or actually, release it back into the connection pool), the app server will print a warning in the log file and release the connection for you. This is, of course, if you are using a data source (have a *-ds.xml) and the code looks up the data source in JNDI.
One way that you could end up not connecting for many minutes is if all of the threads in the thread pool are active, in which case other request will have to wait until one of the threads finishes an becomes free.
What kind of requests are coming in? HTTP to servlets/JSPs/JSF? EJB requests? MDB requests?
Joined: Mar 27, 2011
Yes exactly same warning we are getting for unclosed connection, But I am more interested to know about open Statements and open Resultsets on a pooled connection (configured in *-ds.xml and being retrieved from JNDI).
I think all the threads being active is most promising cause of server not responding as we have multiple long database conversations for a single request (each request spends more than 500 millis in database conversations). So If there are simultaneous long running requests than thread pool will be fully consumed. Am i right ?
Request coming in are HTTP request to a servlet, a stateless session bean works at middle layer.