Meaningless Drivel is fun!*
The moose likes Performance and the fly likes GC taking lots of time Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Android Security Essentials Live Lessons this week in the Android forum!
JavaRanch » Java Forums » Java » Performance
Bookmark "GC taking lots of time" Watch "GC taking lots of time" New topic
Author

GC taking lots of time

Prashant Chandra
Greenhorn

Joined: Jun 28, 2007
Posts: 3
Hi All,

Ours is retail domain Point Of Sale application. With normally not more than 5 concurrent users accessing application at a time through out the year. We have our application designed using IceFaces 1.8.2 (Open Source) and below are the other environment details.

ICEFaces : 1.8.2 (Open Source)
JSF : 1.2_13
container : tomcat 6.0.20
OS : Linux Red Hat 4.1.2-46
java version "1.6.0_17" HotSpot
database : Oracle Database 10g Release 10.2.0.4.0 - 64bit Production
startup params :
/usr/java/default/bin/java -Djava.util.logging.config.file=/opt/xx-xxx-web/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms512m -Xmx4096m -XX:PermSize=512m -XX:MaxPermSize=2048m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8001 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -verbose:gc -Xloggc:/var/log/xx-xxx-web/gc.log -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:HeapDumpPath=/opt/xx-xxx-web/temp -XX:+HeapDumpOnOutOfMemoryError -Djava.library.path=/usr/lib64 -Dorg.apache.catalina.connector.RECYCLE_FACADES=true -Djava.endorsed.dirs=/opt/xx-xxx-web/endorsed -classpath :/opt/xx-xxx-web/bin/bootstrap.jar -Dcatalina.base=/opt/xx-xxx-web -Dcatalina.home=/opt/xx-xxx-web -Djava.io.tmpdir=/opt/xx-xxx-web/temp org.apache.catalina.startup.Bootstrap start

Problem:
Due to Christmas festival the number of requests have increased 10 times. suddenly the server heap memory utilisation has jumped all time high to 100%(then Full GC happens every 2 minutes which takes 7 seconds, all the users faces slow responses during that time), Perm Gen memory utilisation is never above 512 mb. we have increased number of application servers but still even increasing application servers(from 6 to 10 ) has not solved our problem.
We are planning to migrate to IceFaces EE version. but it is going to take time of 2 months. Please suggest me if we can do any quick fix to avoid this problem.

Deepak Bala
Bartender

Joined: Feb 24, 2006
Posts: 6661
    
    5

The real question is - 'What causes the GC to be triggered once every 2 minutes and what causes it to pause for 7 seconds ?'.

I would begin by putting your application through a stress test and using a profiler to look at which objects are causing a bottle neck. 7 seconds is a lot of time. Have you tried tuning you GC ?


SCJP 6 articles - SCJP 5/6 mock exams - More SCJP Mocks
Winston Gutkowski
Bartender

Joined: Mar 17, 2011
Posts: 7549
    
  18

Prashant Chandra wrote:Problem:
Due to Christmas festival the number of requests have increased 10 times. suddenly the server heap memory utilisation has jumped all time high to 100%(then Full GC happens every 2 minutes which takes 7 seconds, all the users faces slow responses during that time), Perm Gen memory utilisation is never above 512 mb. we have increased number of application servers but still even increasing application servers(from 6 to 10 ) has not solved our problem.

This is pure guesswork, but 6 application servers to run a POS application that normally has 5 concurrent users seems like a fabulous amount of power to me, even allowing for a dedicated db server. Are you sure there aren't other things going on that you haven't told us about?

Winston


Isn't it funny how there's always time and money enough to do it WRONG?
Articles by Winston can be found here
Rojan punn
Greenhorn

Joined: Nov 29, 2011
Posts: 17

Disable all the GC verbose and add the below in the start up option

-XX:+UseParNewGC
-XX:+DisableExplicitGC
-XX:NewRatio=2
Randall Twede
Ranch Hand

Joined: Oct 21, 2000
Posts: 4340
    
    2

this doesn't really address the question(sorry), but i tend to agree with Winston. i had a mock e-commerce site once that ran on a Pentium 1 with a 56K dial-up modem and even with more than one user at a time they didn't complain it was too slow.


SCJP
Visit my download page
chander shivdasani
Ranch Hand

Joined: Oct 09, 2007
Posts: 206

Your best bet would be to use a profiler to see if you have any memory leaks. Another thing is to analyze your GC log file to see what your average heap usage is. If, Full GC is happening every 2 mins, it hints that you might have insufficient heap size. Try to increase it by a GB or two.
Can you run the "top" command along side and make sure that swap space is not being used. A swap space is nothing but a portion of Hard Disk which JVM uses as a backup memory when you dont have enough RAM. Accessing pages on hard disk can increase your pause times by several magnitudes. The chances of this happening is very less, but there is no harm in confirming.


Enjoy, Chander
SCJP 5, Oracle Certified PL/SQL Developer
Chris Hurst
Ranch Hand

Joined: Oct 26, 2003
Posts: 407
    
    1

Try turning gc logging on and try gcviewer or the like , should give you a visualisation of what's going on.

Visual VM is your friend as is Eclipse with the Visual MAT plugin run its leak report on a hprof you take with jmap what's it say the suspects are ?

Memory Analyzer

Leaks can cause this but so can memory living too long, i.e. taking too long to free up unused memory as it gets promoted to old and can sit there till a full GC (no CMS) e.g. remove not just cancel timers, look at every cache. Carefully look at any soft / weak reference usage.


"Eagles may soar but weasels don't get sucked into jet engines" SCJP 1.6, SCWCD 1.4, SCJD 1.5,SCBCD 5
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: GC taking lots of time
 
Similar Threads
Problems with Oracle Connection - WebSphere
slow web page response times
Tomcat fails to restart with "Address already in use" exception
JNI Probelm with Oracle iPlanet 7.0.9 Web Server
An unexpected exception has been detected in native code outside the VM.