• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Application Unresponsive : Thread Dump Analysis

 
Ranch Hand
Posts: 157
1
Android MySQL Database Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I have a java based webservice application written on ApacheCXF. We are running into issues where the application becomes completely unresponsive throwing 503 on every request . At this given point we have seen that Old Gen gets completely filled and all the application does is Full GC all the time from then. At this point of time I was trying to analyze both the thread and the heap dump . I have seen some AJP threads in waiting for the monitor as per the thread stat given below.


"ajp-8009-9--ADDADA121312" daemon prio=10 tid=0x00002aaae7ac4800 nid=0x2b84 in Object.wait() [0x00002aab16e72000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006c81449e8> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1621)
- locked <0x00000006c81449e8> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1646)
at java.lang.Thread.run(Thread.java:662)


I have lot of thread pools in the application so the below thread stat I feel is waiting for some tasks to start executing :-> this I guess is 97% of the total threads and it seem to me as normal.


"MyPool-pool-12-thread-10" prio=10 tid=0x000000005958c800 nid=0xaef waiting on condition [0x000000006f876000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c2c6a338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:315)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

The overall status of these threads is below

Overall Thread Count 2078
Overall Monitor Count 68
Number of threads waiting for a monitor 0
Number of threads locking a monitor 49


Number of threads sleeping on a monitor 2035
Number of deadlocks 0
Number of Monitors without locking threads 0

97% of all threads are sleeping on a monitor.
This might indicate they are waiting for some external resource (e.g. database) which is overloaded or not available or are just waiting to get to do something (idle threads). You should check the sleeping threads with a filter excluding all idle threads.


I am in the process of analyzing the thread dump but is there anything which can be sensed out of the above details.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
reply
    Bookmark Topic Watch Topic
  • New Topic