• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

JVM is hung

 
Ranch Hand
Posts: 1710
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
We migrated to JBoss 7.1 recently from JBoss 5.1. JVM gets hung in every couple days and stops accepting any requests. We need to restart to get it back to service. It happens when the total thread utilization goes beyond 6-7K. We take thread dump and find majority of threads (90% of threads) stack traces look like this. Not sure what's going on.
This issue was not there when we were on JBoss 5.1. Any clue/help is appreciated.


"pool-696505-thread-3" prio=10 tid=0x0000000062b08000 nid=0x6c57 waiting on condition [0x0000000078bfc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007038b3400> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:315)
at com.sun.xml.messaging.saaj.util.ParserPool.get(ParserPool.java:88)
at com.sun.xml.messaging.saaj.soap.EnvelopeFactory.createEnvelope(EnvelopeFactory.java:91)
at com.sun.xml.messaging.saaj.soap.ver1_2.SOAPPart1_2Impl.createEnvelopeFromSource(SOAPPart1_2Impl.java:84)
at com.sun.xml.messaging.saaj.soap.SOAPPartImpl.getEnvelope(SOAPPartImpl.java:143)
at com.abc.myproject.integrator.uib.UnifiedMessagingIntegrator.createXMLStr(UnifiedMessagingIntegrator.java:425)
at com.abc.myproject.integrator.uib.UnifiedMessagingIntegrator.makeRemoteSystemsCall(UnifiedMessagingIntegrator.java:650)
at com.abc.myproject.integrator.RemoteRequestDispatcherConcurrent.call(RemoteRequestDispatcherConcurrent.java:69)
at com.abc.myproject.integrator.RemoteRequestDispatcherConcurrent.call(RemoteRequestDispatcherConcurrent.java:56)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)


"pool-696504-thread-3" prio=10 tid=0x00002aaad5d72000 nid=0x6c52 waiting on condition [0x0000000078b57000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007038b3400> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:315)
at com.sun.xml.messaging.saaj.util.ParserPool.get(ParserPool.java:88)
at com.sun.xml.messaging.saaj.soap.EnvelopeFactory.createEnvelope(EnvelopeFactory.java:91)
at com.sun.xml.messaging.saaj.soap.ver1_2.SOAPPart1_2Impl.createEnvelopeFromSource(SOAPPart1_2Impl.java:84)
at com.sun.xml.messaging.saaj.soap.SOAPPartImpl.getEnvelope(SOAPPartImpl.java:143)
at com.abc.myproject.integrator.uib.UnifiedMessagingIntegrator.createXMLStr(UnifiedMessagingIntegrator.java:425)
at com.abc.myproject.integrator.uib.UnifiedMessagingIntegrator.makeRemoteSystemsCall(UnifiedMessagingIntegrator.java:650)
at com.abc.myproject.integrator.RemoteRequestDispatcherConcurrent.call(RemoteRequestDispatcherConcurrent.java:69)
at com.abc.myproject.integrator.RemoteRequestDispatcherConcurrent.call(RemoteRequestDispatcherConcurrent.java:56)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
 
Ranch Hand
Posts: 859
IBM DB2 Chrome Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I haven't seen this [yet].

But look at the stack traces, are your web services hanging?

WP
 
Chandra Bhatt
Ranch Hand
Posts: 1710
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
My suspect is on below piece of code, specially processing SOAP response:

 
William P O'Sullivan
Ranch Hand
Posts: 859
IBM DB2 Chrome Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator



What is at that line exactly?

WP
 
Bartender
Posts: 4179
22
IntelliJ IDE Python Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Yes, the line that is causing the problem appears to be: SOAPEnvelope soapEnvelope = soapPart.getEnvelope();
That line executed an EnvelopeFactory method to generate the SOAPEnvelope. That factory uses the ParserPool, a pool of SAXParser instances. It appears the collection is a BlockingQueue and that the Queue is empty, so the threads have to wait for new parsers to become available.

If you assume that the ParserPool is initialized with some number of SAXParsers, then your problem lies in the fact that when you get a SAXParser from the Pool, you never release it. But further research indicates you never have to return the SAXParser to the Pool, you EnvelopeFactory method does that for you. So the problem must be that early in the process, the first few calls to generate the Envelope must get a SAXParser from the pool but never release it. The only way I can see that happening is if the SOAPPart's content blocks when the the Parser tries to read it to generate the Envelope.

That leads me to believe the problem is:

Is there any chance that could produce an empty Stream?
 
Chandra Bhatt
Ranch Hand
Posts: 1710
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I posted the method definition of createXMLSrt in below thread. Basically we are making HTTP connection using Apache HttpClient to remote server, getting SOAP response and processing that:


remoteCallResponse = createXMLStr(IOUtils.toString(post.getResponseBodyAsStream(), "UTF-8"));

 
Chandra Bhatt
Ranch Hand
Posts: 1710
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Steve Luke wrote:
Is there any chance that could produce an empty Stream?



thanks for sharing idea.

If your question is if authRespone can be null or blank? Yes it can be.
 
Steve Luke
Bartender
Posts: 4179
22
IntelliJ IDE Python Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
And if it is null or blank, what happens to the Stream? If you try to read from it does it hang waiting for input, or does it output end-of-stream?
 
Chandra Bhatt
Ranch Hand
Posts: 1710
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Steve Luke wrote:And if it is null or blank, what happens to the Stream? If you try to read from it does it hang waiting for input, or does it output end-of-stream?



I run the same piece of code from main passing null and blank and got following error:




BTW, In logs I see below line in abundance. May be this gives some clue.

19:34:05,056 SEVERE [com.sun.xml.messaging.saaj.soap] (pool-14021-thread-2) SAAJ0601.util.newSAXParser.exception
 
Consider Paul's rocket mass heater.
reply
    Bookmark Topic Watch Topic
  • New Topic