Win a copy of Re-engineering Legacy Software this week in the Refactoring forum
or Docker in Action in the Cloud/Virtualization forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

JVM is hung

 
Chandra Bhatt
Ranch Hand
Posts: 1710
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
We migrated to JBoss 7.1 recently from JBoss 5.1. JVM gets hung in every couple days and stops accepting any requests. We need to restart to get it back to service. It happens when the total thread utilization goes beyond 6-7K. We take thread dump and find majority of threads (90% of threads) stack traces look like this. Not sure what's going on.
This issue was not there when we were on JBoss 5.1. Any clue/help is appreciated.


"pool-696505-thread-3" prio=10 tid=0x0000000062b08000 nid=0x6c57 waiting on condition [0x0000000078bfc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007038b3400> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:315)
at com.sun.xml.messaging.saaj.util.ParserPool.get(ParserPool.java:88)
at com.sun.xml.messaging.saaj.soap.EnvelopeFactory.createEnvelope(EnvelopeFactory.java:91)
at com.sun.xml.messaging.saaj.soap.ver1_2.SOAPPart1_2Impl.createEnvelopeFromSource(SOAPPart1_2Impl.java:84)
at com.sun.xml.messaging.saaj.soap.SOAPPartImpl.getEnvelope(SOAPPartImpl.java:143)
at com.abc.myproject.integrator.uib.UnifiedMessagingIntegrator.createXMLStr(UnifiedMessagingIntegrator.java:425)
at com.abc.myproject.integrator.uib.UnifiedMessagingIntegrator.makeRemoteSystemsCall(UnifiedMessagingIntegrator.java:650)
at com.abc.myproject.integrator.RemoteRequestDispatcherConcurrent.call(RemoteRequestDispatcherConcurrent.java:69)
at com.abc.myproject.integrator.RemoteRequestDispatcherConcurrent.call(RemoteRequestDispatcherConcurrent.java:56)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)


"pool-696504-thread-3" prio=10 tid=0x00002aaad5d72000 nid=0x6c52 waiting on condition [0x0000000078b57000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007038b3400> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:315)
at com.sun.xml.messaging.saaj.util.ParserPool.get(ParserPool.java:88)
at com.sun.xml.messaging.saaj.soap.EnvelopeFactory.createEnvelope(EnvelopeFactory.java:91)
at com.sun.xml.messaging.saaj.soap.ver1_2.SOAPPart1_2Impl.createEnvelopeFromSource(SOAPPart1_2Impl.java:84)
at com.sun.xml.messaging.saaj.soap.SOAPPartImpl.getEnvelope(SOAPPartImpl.java:143)
at com.abc.myproject.integrator.uib.UnifiedMessagingIntegrator.createXMLStr(UnifiedMessagingIntegrator.java:425)
at com.abc.myproject.integrator.uib.UnifiedMessagingIntegrator.makeRemoteSystemsCall(UnifiedMessagingIntegrator.java:650)
at com.abc.myproject.integrator.RemoteRequestDispatcherConcurrent.call(RemoteRequestDispatcherConcurrent.java:69)
at com.abc.myproject.integrator.RemoteRequestDispatcherConcurrent.call(RemoteRequestDispatcherConcurrent.java:56)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
 
William P O'Sullivan
Ranch Hand
Posts: 859
Chrome IBM DB2 Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I haven't seen this [yet].

But look at the stack traces, are your web services hanging?

WP
 
Chandra Bhatt
Ranch Hand
Posts: 1710
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
My suspect is on below piece of code, specially processing SOAP response:

 
William P O'Sullivan
Ranch Hand
Posts: 859
Chrome IBM DB2 Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator



What is at that line exactly?

WP
 
Steve Luke
Bartender
Posts: 4181
21
IntelliJ IDE Java Python
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Yes, the line that is causing the problem appears to be: SOAPEnvelope soapEnvelope = soapPart.getEnvelope();
That line executed an EnvelopeFactory method to generate the SOAPEnvelope. That factory uses the ParserPool, a pool of SAXParser instances. It appears the collection is a BlockingQueue and that the Queue is empty, so the threads have to wait for new parsers to become available.

If you assume that the ParserPool is initialized with some number of SAXParsers, then your problem lies in the fact that when you get a SAXParser from the Pool, you never release it. But further research indicates you never have to return the SAXParser to the Pool, you EnvelopeFactory method does that for you. So the problem must be that early in the process, the first few calls to generate the Envelope must get a SAXParser from the pool but never release it. The only way I can see that happening is if the SOAPPart's content blocks when the the Parser tries to read it to generate the Envelope.

That leads me to believe the problem is:

Is there any chance that could produce an empty Stream?
 
Chandra Bhatt
Ranch Hand
Posts: 1710
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I posted the method definition of createXMLSrt in below thread. Basically we are making HTTP connection using Apache HttpClient to remote server, getting SOAP response and processing that:


remoteCallResponse = createXMLStr(IOUtils.toString(post.getResponseBodyAsStream(), "UTF-8"));

 
Chandra Bhatt
Ranch Hand
Posts: 1710
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Steve Luke wrote:
Is there any chance that could produce an empty Stream?


thanks for sharing idea.

If your question is if authRespone can be null or blank? Yes it can be.
 
Steve Luke
Bartender
Posts: 4181
21
IntelliJ IDE Java Python
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
And if it is null or blank, what happens to the Stream? If you try to read from it does it hang waiting for input, or does it output end-of-stream?
 
Chandra Bhatt
Ranch Hand
Posts: 1710
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Steve Luke wrote:And if it is null or blank, what happens to the Stream? If you try to read from it does it hang waiting for input, or does it output end-of-stream?


I run the same piece of code from main passing null and blank and got following error:




BTW, In logs I see below line in abundance. May be this gives some clue.

19:34:05,056 SEVERE [com.sun.xml.messaging.saaj.soap] (pool-14021-thread-2) SAAJ0601.util.newSAXParser.exception
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic