VM thread and high CPU

I've got an application that works very relaxed most of the time having short (seconds) peaks of 10-20% CPU but... after working for about 5-6 hours its CPU usage starts to grow without control and if left alone the machine goes crazy and no other process work.

I've added a top -cH -n 1 -b -p PID >> topFile.out and a kill-3 pid to try to search the culprit and i've got this...

Other threads are at 0% CPU

top - 14:50:02 up 50 days, 15:24, 4 users, load average: 0.79, 0.70, 0.39
Tasks: 40 total, 0 running, 40 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.0%us, 0.3%sy, 0.0%ni, 94.8%id, 0.2%wa, 0.0%hi, 0.6%si, 0.0%st
Mem: 8175332k total, 7552648k used, 622684k free, 468232k buffers
Swap: 4194264k total, 96k used, 4194168k free, 5738040k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7219 root 16 0 918m 518m 9640 S 45.9 6.5 8:56.46 java -Xmx512M -jar
7235 root 15 0 918m 518m 9640 S 2.0 6.5 0:41.07 java -Xmx512M -jar
7209 root 18 0 918m 518m 9640 S 0.0 6.5 0:00.00 java -Xmx512M -jar

As you see no swap memory is used.

And kill -3 shows that the 45,9% CPU usage thread is this one (7219->1c33)

"VM Thread" prio=10 tid=0x0000000050ae0000 nid=0x1c33 runnable

Heap summary at the end of the stack

Heap
PSYoungGen total 26688K, used 21823K [0x00002aaac8980000, 0x00002aaaca530000, 0x00002aaad3420000)
eden space 25024K, 87% used [0x00002aaac8980000,0x00002aaac9ecfd08,0x00002aaaca1f0000)
from space 1664K, 0% used [0x00002aaaca390000,0x00002aaaca390000,0x00002aaaca530000)
to space 1664K, 0% used [0x00002aaaca1f0000,0x00002aaaca1f0000,0x00002aaaca390000)
PSOldGen total 349568K, used 349566K [0x00002aaab3420000, 0x00002aaac8980000, 0x00002aaac8980000)
object space 349568K, 99% used [0x00002aaab3420000,0x00002aaac897f918,0x00002aaac8980000)
PSPermGen total 31616K, used 31318K [0x00002aaaae020000, 0x00002aaaaff00000, 0x00002aaab3420000)
object space 31616K, 99% used [0x00002aaaae020000,0x00002aaaafeb5b30,0x00002aaaaff00000)

I've tried DaCapo benchmarks with all sorts of options with no noticeable results.

Tomorrow I'll try with -Xmx set to 2g. If you see something strange please tell me

You are running out of memory. VM thread idoes Garbage collection among other things. PSOldGen is at 99% which means practically it's full. The GC is spinning continuously and is using up CPU

Most probably you have a memory leak somewhere. Maybe references to some objects are not bening cleared. Get a heapdump when the problem happens again and look at it using a memory analysis tool. You should be able to find the problem pretty easily

Your permgen space is also at 99% capacity ? Are you sure the application does not spew out OutofMemoryErrors ? Perm gen is used to load class definitions among other permanent data.

What does the GC stat for the VM look like ? When the CPU is at 100%, which thread takes up more CPU time ?

That VM thread is the culprit always.

I'm searchig for Hibernate Memory Leak Issues to see if I got some clues.

But I got no info from my colleage today so i seems making the heap 4 times bigger worked fine

But I got no info from my colleage today so i seems making the heap 4 times bigger worked fine

For now

If it is a leak, you will eventually run out of memory. Most app servers allow you to monitor such parameters. Does your app server have such provisions ?