jQuery in Action, 2nd edition*
The moose likes Performance and the fly likes View the objects which are getting stored in permgen and total non heap space Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of JavaScript Promises Essentials this week in the JavaScript forum!
JavaRanch » Java Forums » Java » Performance
Bookmark "View the objects which are getting stored in permgen and total non heap space " Watch "View the objects which are getting stored in permgen and total non heap space " New topic
Author

View the objects which are getting stored in permgen and total non heap space

Raja Nagendra Kumar
Greenhorn

Joined: Jun 14, 2008
Posts: 27
Hi,

We at TejaSoft have a ant xml, with Groovy and Custom Ant tasks being used for a product build. Afer running the script repeatedly, We see out of memory error at permgen.
Trying to use Jprofiler and YourKit to know what are the objects which are being stored in the permgen space but unable to get good clarity on how to see this..

Could any one explain a good place to know such need of ours. Our idea is very simple, if we know the tostring() version of the objects, which exists in permgen space, we know which is the culprit...In our case they are mostly string or ant properties for sure.

We are trying to use locahost with respective agent and connecting to IDE remote debug option.

JConsole clearly shows the issue is with nonheap growth. Does JCconsole it self has a way to view these objects!!


Raja Nagendra Kumar,
C.T.O, www.tejasoft.com
rajesh bala
Ranch Hand

Joined: Jan 14, 2003
Posts: 66
I would probably try to increase the size of the permsize.

If its giving an OOM even after increasing the permgen to more than 128M or 196M, you might have to look at it more closely.

If the above doesn't work, you can try to enable the following 3 options.

http://my.opera.com/karmazilla/blog/2007/03/13/good-riddance-permgen-outofmemoryerror

-XX:+UseConcMarkSweepGCTell GC to sweep PermGen space
-XX:+CMSPermGenSweepingEnabledrequest JVM to Allow classes to be unloaded
-XX:+CMSClassUnloadingEnabledBedank

BTW, which JVM are you using (JRockit or Sun?).

~Rajesh.B
Raja Nagendra Kumar
Greenhorn

Joined: Jun 14, 2008
Posts: 27
Hi Rajesh,

Thank you for the goold link. We shall try with the options mentioned.

We are using Sun VM.

Our ant uses Groovy and Custom Java Ant Tasks. We are suspecting some strings are not getting released in groovy.

Also, if I am able to know the objects which are surviving in permgen space I would have more confidence to deal with it. Are there any tools which can provide such info.

Regards,
Raja Nagendra Kumar,
C.T.O
www.tejasoft.com



rajesh bala
Ranch Hand

Joined: Jan 14, 2003
Posts: 66
I remember "yourkit" profiler giving details about that. I dont think JProfiler can give such info.

1. Use Yourkit profiler (may be trial version).
2. In memory management, they have a bunch of rules (5 or 6 rules) which can be run against the heap. This is more of memory leak detector.
3. One of the rule would be "find duplicate strings". You can give a try by running some of the rules mentioned there.

Its really hard to explain the internal workings of yourkit here. So I just hinted on its functionality and rules.

If you suspect that the constant pool of JVM is getting full and hence the permspace error, you can find out by running that rule.

But before trying that, I would highly suggest to try those JVM parameters in order to reduce the time spent on it.

PS: Reason for asking for JVM version is that, in Sun Permspace is different than heap. In BEA JRockit, permspace is a part of the max heap you set.
Raja Nagendra Kumar
Greenhorn

Joined: Jun 14, 2008
Posts: 27
Hi Rajesh,

Ok, let me have a look at YourKit..

Rajesh, I tried the options -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled

This now is not failing due to outofmemory on perm.. but still fails with java.lang.OutOfMemoryError. From JConsole we see this coming from Cade Cache non heap space is being filled.

As I understand PermGen only stores Class definitions. What does Code Cache contain. Any good documentation of sun's impl.

I shall also try with JRockit compiler.. hope this compiler supports most of the sun java options

Regards,
Nagendra
Raja Nagendra Kumar
Greenhorn

Joined: Jun 14, 2008
Posts: 27
Also Rajesh, is CMSPermGenSweepingEnabled option deprecated.. as I see this message from JDK 1.6


> Please use CMSClassUnloadingEnabled in place of CMSPermGenSweepingEnabled in the build 1.6.0_14-b08 future
rajesh bala
Ranch Hand

Joined: Jan 14, 2003
Posts: 66
Hi Raja,

It seems like in JVM 1.5 it has to be combined with -XX:+CMSClassUnloadingEnabled in order to take effect (with -XX:+CMSPermGenSweepingEnabled)

Can you add both of these and run it?

Reference: http://blogs.sun.com/watt/resource/jvm-options-list.html

~Rajesh.B
rajesh bala
Ranch Hand

Joined: Jan 14, 2003
Posts: 66
If its code cache issue, can you try tweaking

-XX:ReservedCodeCacheSize=128m -XX:InitialCodeCacheSize=48m (this is just the starting value. You can increment or decrement to suit your needs)

So add the above 2.

Note that XX parameters are not compatible across all JVM releases. Sun has the right to revoke the XX parameter anytime they need. Also, XX parameters in sun may not be compatible with JRockit.

So I would suggest to try the above parameters only in Sun JVM. For JRockit JVM, we have to first check if the problem persists. If so, we have to check for the appropriate parameters in JRockit.

~Rajesh.B
rajesh bala
Ranch Hand

Joined: Jan 14, 2003
Posts: 66
Forgot to ask an important question. Are you running the JVM in client mode or in server mode?

Raja Nagendra Kumar
Greenhorn

Joined: Jun 14, 2008
Posts: 27
Rajesh, We are using it in client mode, which I suppose is the default mode when we run the ant script.

rajesh bala wrote:Forgot to ask an important question. Are you running the JVM in client mode or in server mode?

rajesh bala
Ranch Hand

Joined: Jan 14, 2003
Posts: 66
When you say OOM stemming from code cache, do you see code cache just getting full in jconsole?

1. Whenever Sun JVM throws an OOM, it would create a log file in the directory. Ex: hs_<pid>.log where pid is the process id of the JVM

Did you see from that log that the OOM was due to code cache?. If not, we might be looking at a totally different problem altogether this time. If possible, post the contents of the JVM log here.

~Rajesh.B
rajesh bala
Ranch Hand

Joined: Jan 14, 2003
Posts: 66
Raja Nagendra Kumar wrote:Rajesh, We are using it in client mode, which I suppose is the default mode when we run the ant script.

rajesh bala wrote:Forgot to ask an important question. Are you running the JVM in client mode or in server mode?



client mode is meant for short lived applications like applets (note that client mode does not do many optimizations). It is meant for starting the JVM as fast as possible.

Can you run the JVM in "-server" mode and post the results. To be short, Server mode does a lot of internal optimizations.
rajesh bala
Ranch Hand

Joined: Jan 14, 2003
Posts: 66
Raja Nagendra Kumar wrote:
As I understand PermGen only stores Class definitions. What does Code Cache contain. Any good documentation of sun's impl.


Hi Raja,

JVM's JIT optimizes as the program is running. It internally generates lot of code as it optimizes and puts them in cache. This area is called code cache. In some cases where the recursive inlining optimizations are heavy, it might happen that the code getting generated are big and hence you see the code cache getting filled pretty quickly. This is different from the perm space as per my understanding.

~Rajesh.B
Raja Nagendra Kumar
Greenhorn

Joined: Jun 14, 2008
Posts: 27
Hi Rajesh,

Thank you for your quick support.. you seem to be an expert in this area.. your details have been very very helpful to all of us.

I am seeing the graphical representation of the memory in jconsole memory tab, which I see is full not the log file.

Where should be expect the log file. in the current directory for outofmemory I am not seeing any log file.. for pergen outofmemory I saw one yesterday..

Regards,
Nagendra

Could you let me know which

rajesh bala wrote:When you say OOM stemming from code cache, do you see code cache just getting full in jconsole?

1. Whenever Sun JVM throws an OOM, it would create a log file in the directory. Ex: hs_<pid>.log where pid is the process id of the JVM

Did you see from that log that the OOM was due to code cache?. If not, we might be looking at a totally different problem altogether this time. If possible, post the contents of the JVM log here.

~Rajesh.B

rajesh bala
Ranch Hand

Joined: Jan 14, 2003
Posts: 66
Hi Nagendra,

Did you see any JVM crashing due to OOM?. If JVM crashes and the program exits, it should generate the hs_<pid>.log file.

Usually it would be saved in the directory from where you are running. Or else, the program itself would tell you "please refer hs_<pid>.log for more details".

Worst possible case: Do a complete search in your c:\ (hoping you have installed JDK and your program in c:\) for "hs*.log"

Note that this would be created only when the JVM crashes due to OOM.

~Rajesh.B
Raja Nagendra Kumar
Greenhorn

Joined: Jun 14, 2008
Posts: 27
Hi Rajesh,

The last few lines we see on Command prompt are these..




Also, JConsole graphical memory tab shows code cache is full filled bar CommonMacrosProps.xml has groovy integration which is causing the issues all along. Did you see any such issues on ant with groovy integration for a long and repeated tasks specially when subant, ant antcall or java with fork being used. I searched entire hard disk and unable to find hs*.log files of today. yesterdays and day before yesterdays I could find few. but these files were created when permgen Out of memory error is coming.. not todays one after applying your suggested jvm options. I am trying now with -server flag. Regards, Nagendra
Raja Nagendra Kumar
Greenhorn

Joined: Jun 14, 2008
Posts: 27
Hi Rajesh,

The last few lines we see on Command prompt are these..

f:\tejasoft\work\foundation\common\buildutils\common\macros\CommonMacros.xml:17:
The following error occurred while executing this line:
f:\tejasoft\work\foundation\j2se\buildutils\J2SECommon.xml:3: The following error occurred while executing this line:
f:\tejasoft\work\foundation\common\buildutils\common\macros\CommonMacroProps.xml
:22: java.lang.OutOfMemoryError


Also, JConsole graphical memory tab shows code cache is full filled bar CommonMacrosProps.xml has groovy integration which is causing the issues all along. Did you see any such issues on ant with groovy integration for a long and repeated tasks specially when subant, ant antcall or java with fork being used. I searched entire hard disk and unable to find hs*.log files of today. yesterdays and day before yesterdays I could find few. but these files were created when permgen Out of memory error is coming.. not todays one after applying your suggested jvm options. I am trying now with -server flag. Regards, Nagendra
rajesh bala
Ranch Hand

Joined: Jan 14, 2003
Posts: 66
Raja Nagendra Kumar wrote:



groovy integration for a long and repeated tasks specially when subant, ant antcall or java with fork being used. I searched entire hard disk and unable to find hs*.log files of today.


BTW, since you have said java with fork option. Are your jvmargs specifically configured for this forking java process?. I mean, is this forked java process running with all the jvmargs we have discussed?. If so please run with -server option as well.

It would be great if you can post that forking process's ant script alone here (I understand there could be confidential things which you can't post here. You can change the names etc and post the content of that forking script alone here.)

~Rajesh.B
Raja Nagendra Kumar
Greenhorn

Joined: Jun 14, 2008
Posts: 27
rajesh bala wrote:
Can you run the JVM in "-server" mode and post the results. To be short, Server mode does a lot of internal optimizations.


Hi Rajesh,

-server option solves the problem. We are able to run the build for more than one hour.. where as the old one was failing after 30 min.

However, we still on JConsole that Non-Heap Code Cache and PermGen space is going up equally and at much slower rates.

On seeing all the Heap Space Objects

Total Heap Space is also going up with following sub details

- Memory Pool CMS Old Gen - Going Up with wave pattern..(bottom of wave is linearly up..)
- Par Eden - Wave with const bottom and up
- Par Survior are waves with const up value.. its is perfect horizontal

In summary with -server option
- has great impact given at ant launch I see the build is running for more time
- however there is still constant wave like increase in OldGen
- Both Non Heap Code Cache and PermGen is increasing at a slower phase.

Let me try giving the server option for all the forks and see the results.

B.T.W could you pl. let me know what gets stored in OldGen and they don't get to moved to Eden or Survior.


Regards,
Raja Nagendra Kumar
rajesh bala
Ranch Hand

Joined: Jan 14, 2003
Posts: 66
Hi Nagendra,

Old gen would be cleaned up as a part of Full-GC. So until max-heap is touched, you might observe that the heap growing linearly. Minor-GC will be cleaning up only the young generation.

Objects are allocated in young generation and promoted to old-generation at the time of garbage collection. Its vast topic to discuss about young-gen, old-gen here.

Survivor space is used as a bucket before moving to old-gen.

Looking at the current details, I believe most of your problems should be sorted out with -server option.

~Rajesh.B
Raja Nagendra Kumar
Greenhorn

Joined: Jun 14, 2008
Posts: 27
rajesh bala wrote:Hi Nagendra,
Looking at the current details, I believe most of your problems should be sorted out with -server option.
~Rajesh.B


Yes, Thank you very much for your 'The Expert' guidance and support.
We want to do much more testing with lesser heap size and max permsize with all the forks using the same options.

Also, does javac of ant tasks needs the same options such as -server and -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled options.

Regards,
Nagendra
rajesh bala
Ranch Hand

Joined: Jan 14, 2003
Posts: 66
All these flags are specific to JVM and not for javac.

Though JVM arguments can be set for javac using "-Joption", mostly this might not be required. Until you figure out this javac is failing, you should be all set.

~Rajesh.B
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: View the objects which are getting stored in permgen and total non heap space