We are facing an issue where the Resident memory of the Java process grows gradually. We have Xmx defined at 4096 MB and XX:MaxPermSize=1536m. The number of active threads ~1500 with an Xss of 256K defined.
When the application server(JBoss 6.1) starts the resident memory used is ~5.6GB(have been using top command to monitor it); it gradually grows(around 0.3 to 0.5 Gb per day) till it grows to ~7.4 Gb, when the kernel's OOM killer kills the process due to shortage of RAM space(The server has 9GB of RAM).
We have been regularly monitoring the thread dump- no suspect of a thread leak. We are still unable to figure out from where this extra memory is coming from.
Pmap output shows a number of Anon blocks(apart from the regular blocks for stack and heap), mostly in arenas of 64 Mb, which are un-accounted for in terms of memory usage by heap, perm gen & stacks.
In the heap dump we have also tried looking for DirectByteBuffers and sun.misc.Unsafe objects, which are generally used for non-heap memory allocation, but the number of objects as well as the memory capacity seem nominal. Is it possible that there can still be un- freed native memory even after these objects are GCed? Any other classes that may result in using up non-heap memory?
Our application does have native calls on its own, but it's possible that some third party libs have them.
Any ideas on what could be causing this? Anything other detail/tool that could further help debugging such an increase? Any known issue that we should look out for? Platform : Jboss 6.1 running on Centos 5.6.
Ashay Vedasen wrote:Any ideas on what could be causing this? Anything other detail/tool that could further help debugging such an increase? Any known issue that we should look out for? Platform : Jboss 6.1 running on Centos 5.6.
I'm not sure, but my understanding was that memory used by the JVM is never normally released back for general system use. And even if it was, it's likely to be in big (and very possibly unpredictable) "chunks" - But I have to admit I'm no expert in this; just an ex-jobbing sysadmin.
What you can do, however, is put a limit on the amount of memory that a JVM will use, so you always have room to run other non-Java processes.
Alternatively, if you're running one of the newer Linuxes (not sure about Centos), you could have the JVM run in its own VM, which you could then "throttle" with standard system commands.
Just some thoughts; not sure if it answers your question.
Winston Gutkowski wrote:
I'm not sure, but my understanding was that memory used by the JVM is never normally released back for general system use.
Several years and versions ago I made an experiment to test this. Whether the JVM was willing to return not longer used memory to the operating system seemed to depend on the Java version and not on the operating system (Windows, Linux). Some version did, some did not.