It's not a secret anymore!*
The moose likes Threads and Synchronization and the fly likes Application Unresponsive : Thread Dump Analysis Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Threads and Synchronization
Bookmark "Application Unresponsive : Thread Dump Analysis" Watch "Application Unresponsive : Thread Dump Analysis" New topic
Author

Application Unresponsive : Thread Dump Analysis

harshvardhan ojha
Ranch Hand

Joined: Jul 26, 2007
Posts: 157
    
    1

I have a java based webservice application written on ApacheCXF. We are running into issues where the application becomes completely unresponsive throwing 503 on every request . At this given point we have seen that Old Gen gets completely filled and all the application does is Full GC all the time from then. At this point of time I was trying to analyze both the thread and the heap dump . I have seen some AJP threads in waiting for the monitor as per the thread stat given below.


"ajp-8009-9--ADDADA121312" daemon prio=10 tid=0x00002aaae7ac4800 nid=0x2b84 in Object.wait() [0x00002aab16e72000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006c81449e8> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:1621)
- locked <0x00000006c81449e8> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1646)
at java.lang.Thread.run(Thread.java:662)


I have lot of thread pools in the application so the below thread stat I feel is waiting for some tasks to start executing :-> this I guess is 97% of the total threads and it seem to me as normal.


"MyPool-pool-12-thread-10" prio=10 tid=0x000000005958c800 nid=0xaef waiting on condition [0x000000006f876000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c2c6a338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:315)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)

The overall status of these threads is below

Overall Thread Count 2078
Overall Monitor Count 68
Number of threads waiting for a monitor 0
Number of threads locking a monitor 49


Number of threads sleeping on a monitor 2035
Number of deadlocks 0
Number of Monitors without locking threads 0

97% of all threads are sleeping on a monitor.
This might indicate they are waiting for some external resource (e.g. database) which is overloaded or not available or are just waiting to get to do something (idle threads). You should check the sleeping threads with a filter excluding all idle threads.


I am in the process of analyzing the thread dump but is there anything which can be sensed out of the above details.
 
 
subject: Application Unresponsive : Thread Dump Analysis
 
Similar Threads
ExecutorService is causing the program to hang for few minutes
Program hangs
Tomcat hanging on shutdown
JBoss deployment issues
Server boot time issue