I've got an application that works very relaxed most of the time having short (seconds) peaks of 10-20% CPU but... after working for about 5-6 hours its CPU usage starts to grow without control and if left alone the machine goes crazy and no other process work.
I've added a top -cH -n 1 -b -p PID >> topFile.out and a kill-3 pid to try to search the culprit and i've got this...
PSYoungGen total 26688K, used 21823K [0x00002aaac8980000, 0x00002aaaca530000, 0x00002aaad3420000)
eden space 25024K, 87% used [0x00002aaac8980000,0x00002aaac9ecfd08,0x00002aaaca1f0000)
from space 1664K, 0% used [0x00002aaaca390000,0x00002aaaca390000,0x00002aaaca530000)
to space 1664K, 0% used [0x00002aaaca1f0000,0x00002aaaca1f0000,0x00002aaaca390000)
PSOldGen total 349568K, used 349566K [0x00002aaab3420000, 0x00002aaac8980000, 0x00002aaac8980000)
object space 349568K, 99% used [0x00002aaab3420000,0x00002aaac897f918,0x00002aaac8980000)
PSPermGen total 31616K, used 31318K [0x00002aaaae020000, 0x00002aaaaff00000, 0x00002aaab3420000)
object space 31616K, 99% used [0x00002aaaae020000,0x00002aaaafeb5b30,0x00002aaaaff00000)
I've tried DaCapo benchmarks with all sorts of options with no noticeable results.
Tomorrow I'll try with -Xmx set to 2g. If you see something strange please tell me
You are running out of memory. VM thread idoes Garbage collection among other things. PSOldGen is at 99% which means practically it's full. The GC is spinning continuously and is using up CPU
Most probably you have a memory leak somewhere. Maybe references to some objects are not bening cleared. Get a heapdump when the problem happens again and look at it using a memory analysis tool. You should be able to find the problem pretty easily
Interesting and very relevant thread for the issue I am looking into.
In my case - there are two threads which are hogging CPU - The VM Thread and ConcurrentMarkSweep thread.
I can understand that ConcurrentMarkSweep thread is busy as less space is remaining in tenured gen. But, Jayesh Lalwani has commented that VM thread does GC.
I don't quite agree as in my case the GC thread is clearly labeled with name ConcurrentMarkSweep.
Now the query is - what could this VM thread be doing and how to find it?
Just a post to give interesting point we found out facing the same issue:
If your PermGen is 99% you should maybe increase it to 128Mb for instance, as it takes 64Mb by default.
Because if you're running out of PermGen, it is possible (in a 64 bit Unix like server environment in our case) that the JVM starts using the swap instead of throwing PermGen error.
In this case, when your swap starts being full, the GC thread will use 100% of a CPU by constantly switching to the swap, making all I/O incredibly slow (like up to an hour to write a normal-sized file).