I have a java based webservice application written on ApacheCXF. We are running into issues where the application becomes completely unresponsive throwing 503 on every request . At this given point we have seen that Old Gen gets completely filled and all the application does is Full GC all the time from then. At this point of time I was trying to analyze both the thread and the heap dump . I have seen some AJP threads in waiting for the monitor as per the thread stat given below.
"ajp-8009-9--ADDADA121312" daemon prio=10 tid=0x00002aaae7ac4800 nid=0x2b84 in Object.wait() [0x00002aab16e72000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000006c81449e8> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
- locked <0x00000006c81449e8> (a org.apache.tomcat.util.net.AprEndpoint$Worker)
I have lot of thread pools in the application so the below thread stat I feel is waiting for some tasks to start executing :-> this I guess is 97% of the total threads and it seem to me as normal.
"MyPool-pool-12-thread-10" prio=10 tid=0x000000005958c800 nid=0xaef waiting on condition [0x000000006f876000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000006c2c6a338> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
The overall status of these threads is below
Overall Thread Count 2078
Overall Monitor Count 68
Number of threads waiting for a monitor 0
Number of threads locking a monitor 49
Number of threads sleeping on a monitor 2035
Number of deadlocks 0
Number of Monitors without locking threads 0
97% of all threads are sleeping on a monitor.
This might indicate they are waiting for some external resource (e.g. database) which is overloaded or not available or are just waiting to get to do something (idle threads). You should check the sleeping threads with a filter excluding all idle threads.
I am in the process of analyzing the thread dump but is there anything which can be sensed out of the above details.