File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Threads and Synchronization and the fly likes JVM is hung Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Threads and Synchronization
Bookmark "JVM is hung" Watch "JVM is hung" New topic
Author

JVM is hung

Chandra Bhatt
Ranch Hand

Joined: Feb 28, 2007
Posts: 1707
We migrated to JBoss 7.1 recently from JBoss 5.1. JVM gets hung in every couple days and stops accepting any requests. We need to restart to get it back to service. It happens when the total thread utilization goes beyond 6-7K. We take thread dump and find majority of threads (90% of threads) stack traces look like this. Not sure what's going on.
This issue was not there when we were on JBoss 5.1. Any clue/help is appreciated.


"pool-696505-thread-3" prio=10 tid=0x0000000062b08000 nid=0x6c57 waiting on condition [0x0000000078bfc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007038b3400> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:315)
at com.sun.xml.messaging.saaj.util.ParserPool.get(ParserPool.java:88)
at com.sun.xml.messaging.saaj.soap.EnvelopeFactory.createEnvelope(EnvelopeFactory.java:91)
at com.sun.xml.messaging.saaj.soap.ver1_2.SOAPPart1_2Impl.createEnvelopeFromSource(SOAPPart1_2Impl.java:84)
at com.sun.xml.messaging.saaj.soap.SOAPPartImpl.getEnvelope(SOAPPartImpl.java:143)
at com.abc.myproject.integrator.uib.UnifiedMessagingIntegrator.createXMLStr(UnifiedMessagingIntegrator.java:425)
at com.abc.myproject.integrator.uib.UnifiedMessagingIntegrator.makeRemoteSystemsCall(UnifiedMessagingIntegrator.java:650)
at com.abc.myproject.integrator.RemoteRequestDispatcherConcurrent.call(RemoteRequestDispatcherConcurrent.java:69)
at com.abc.myproject.integrator.RemoteRequestDispatcherConcurrent.call(RemoteRequestDispatcherConcurrent.java:56)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)


"pool-696504-thread-3" prio=10 tid=0x00002aaad5d72000 nid=0x6c52 waiting on condition [0x0000000078b57000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007038b3400> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:315)
at com.sun.xml.messaging.saaj.util.ParserPool.get(ParserPool.java:88)
at com.sun.xml.messaging.saaj.soap.EnvelopeFactory.createEnvelope(EnvelopeFactory.java:91)
at com.sun.xml.messaging.saaj.soap.ver1_2.SOAPPart1_2Impl.createEnvelopeFromSource(SOAPPart1_2Impl.java:84)
at com.sun.xml.messaging.saaj.soap.SOAPPartImpl.getEnvelope(SOAPPartImpl.java:143)
at com.abc.myproject.integrator.uib.UnifiedMessagingIntegrator.createXMLStr(UnifiedMessagingIntegrator.java:425)
at com.abc.myproject.integrator.uib.UnifiedMessagingIntegrator.makeRemoteSystemsCall(UnifiedMessagingIntegrator.java:650)
at com.abc.myproject.integrator.RemoteRequestDispatcherConcurrent.call(RemoteRequestDispatcherConcurrent.java:69)
at com.abc.myproject.integrator.RemoteRequestDispatcherConcurrent.call(RemoteRequestDispatcherConcurrent.java:56)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

cmbhatt
William P O'Sullivan
Ranch Hand

Joined: Mar 28, 2012
Posts: 860

I haven't seen this [yet].

But look at the stack traces, are your web services hanging?

WP
Chandra Bhatt
Ranch Hand

Joined: Feb 28, 2007
Posts: 1707
My suspect is on below piece of code, specially processing SOAP response:

William P O'Sullivan
Ranch Hand

Joined: Mar 28, 2012
Posts: 860




What is at that line exactly?

WP
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3950
    
  17

Yes, the line that is causing the problem appears to be: SOAPEnvelope soapEnvelope = soapPart.getEnvelope();
That line executed an EnvelopeFactory method to generate the SOAPEnvelope. That factory uses the ParserPool, a pool of SAXParser instances. It appears the collection is a BlockingQueue and that the Queue is empty, so the threads have to wait for new parsers to become available.

If you assume that the ParserPool is initialized with some number of SAXParsers, then your problem lies in the fact that when you get a SAXParser from the Pool, you never release it. But further research indicates you never have to return the SAXParser to the Pool, you EnvelopeFactory method does that for you. So the problem must be that early in the process, the first few calls to generate the Envelope must get a SAXParser from the pool but never release it. The only way I can see that happening is if the SOAPPart's content blocks when the the Parser tries to read it to generate the Envelope.

That leads me to believe the problem is:

Is there any chance that could produce an empty Stream?


Steve
Chandra Bhatt
Ranch Hand

Joined: Feb 28, 2007
Posts: 1707
I posted the method definition of createXMLSrt in below thread. Basically we are making HTTP connection using Apache HttpClient to remote server, getting SOAP response and processing that:


remoteCallResponse = createXMLStr(IOUtils.toString(post.getResponseBodyAsStream(), "UTF-8"));

Chandra Bhatt
Ranch Hand

Joined: Feb 28, 2007
Posts: 1707
Steve Luke wrote:
Is there any chance that could produce an empty Stream?


thanks for sharing idea.

If your question is if authRespone can be null or blank? Yes it can be.
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3950
    
  17

And if it is null or blank, what happens to the Stream? If you try to read from it does it hang waiting for input, or does it output end-of-stream?
Chandra Bhatt
Ranch Hand

Joined: Feb 28, 2007
Posts: 1707

Steve Luke wrote:And if it is null or blank, what happens to the Stream? If you try to read from it does it hang waiting for input, or does it output end-of-stream?


I run the same piece of code from main passing null and blank and got following error:




BTW, In logs I see below line in abundance. May be this gives some clue.

19:34:05,056 SEVERE [com.sun.xml.messaging.saaj.soap] (pool-14021-thread-2) SAAJ0601.util.newSAXParser.exception
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: JVM is hung
 
Similar Threads
Application Unresponsive : Thread Dump Analysis
Weblogic thread stuck on native call of PCKS11 library. Thread dump attched..
Program hangs
JBoss deployment issues
Server boot time issue