• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

GC taking lots of time

 
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi All,

Ours is retail domain Point Of Sale application. With normally not more than 5 concurrent users accessing application at a time through out the year. We have our application designed using IceFaces 1.8.2 (Open Source) and below are the other environment details.

ICEFaces : 1.8.2 (Open Source)
JSF : 1.2_13
container : tomcat 6.0.20
OS : Linux Red Hat 4.1.2-46
java version "1.6.0_17" HotSpot
database : Oracle Database 10g Release 10.2.0.4.0 - 64bit Production
startup params :
/usr/java/default/bin/java -Djava.util.logging.config.file=/opt/xx-xxx-web/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms512m -Xmx4096m -XX:PermSize=512m -XX:MaxPermSize=2048m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8001 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -verbose:gc -Xloggc:/var/log/xx-xxx-web/gc.log -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:HeapDumpPath=/opt/xx-xxx-web/temp -XX:+HeapDumpOnOutOfMemoryError -Djava.library.path=/usr/lib64 -Dorg.apache.catalina.connector.RECYCLE_FACADES=true -Djava.endorsed.dirs=/opt/xx-xxx-web/endorsed -classpath :/opt/xx-xxx-web/bin/bootstrap.jar -Dcatalina.base=/opt/xx-xxx-web -Dcatalina.home=/opt/xx-xxx-web -Djava.io.tmpdir=/opt/xx-xxx-web/temp org.apache.catalina.startup.Bootstrap start

Problem:
Due to Christmas festival the number of requests have increased 10 times. suddenly the server heap memory utilisation has jumped all time high to 100%(then Full GC happens every 2 minutes which takes 7 seconds, all the users faces slow responses during that time), Perm Gen memory utilisation is never above 512 mb. we have increased number of application servers but still even increasing application servers(from 6 to 10 ) has not solved our problem.
We are planning to migrate to IceFaces EE version. but it is going to take time of 2 months. Please suggest me if we can do any quick fix to avoid this problem.

 
Bartender
Posts: 6663
5
MyEclipse IDE Firefox Browser Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The real question is - 'What causes the GC to be triggered once every 2 minutes and what causes it to pause for 7 seconds ?'.

I would begin by putting your application through a stress test and using a profiler to look at which objects are causing a bottle neck. 7 seconds is a lot of time. Have you tried tuning you GC ?
 
Bartender
Posts: 10780
71
Hibernate Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Prashant Chandra wrote:Problem:
Due to Christmas festival the number of requests have increased 10 times. suddenly the server heap memory utilisation has jumped all time high to 100%(then Full GC happens every 2 minutes which takes 7 seconds, all the users faces slow responses during that time), Perm Gen memory utilisation is never above 512 mb. we have increased number of application servers but still even increasing application servers(from 6 to 10 ) has not solved our problem.


This is pure guesswork, but 6 application servers to run a POS application that normally has 5 concurrent users seems like a fabulous amount of power to me, even allowing for a dedicated db server. Are you sure there aren't other things going on that you haven't told us about?

Winston
 
Greenhorn
Posts: 17
Eclipse IDE Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Disable all the GC verbose and add the below in the start up option

-XX:+UseParNewGC
-XX:+DisableExplicitGC
-XX:NewRatio=2
 
Ranch Hand
Posts: 4716
9
Scala Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
this doesn't really address the question(sorry), but i tend to agree with Winston. i had a mock e-commerce site once that ran on a Pentium 1 with a 56K dial-up modem and even with more than one user at a time they didn't complain it was too slow.
 
Ranch Hand
Posts: 206
Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Your best bet would be to use a profiler to see if you have any memory leaks. Another thing is to analyze your GC log file to see what your average heap usage is. If, Full GC is happening every 2 mins, it hints that you might have insufficient heap size. Try to increase it by a GB or two.
Can you run the "top" command along side and make sure that swap space is not being used. A swap space is nothing but a portion of Hard Disk which JVM uses as a backup memory when you dont have enough RAM. Accessing pages on hard disk can increase your pause times by several magnitudes. The chances of this happening is very less, but there is no harm in confirming.

 
Ranch Hand
Posts: 443
3
Eclipse IDE C++ Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Try turning gc logging on and try gcviewer or the like , should give you a visualisation of what's going on.

Visual VM is your friend as is Eclipse with the Visual MAT plugin run its leak report on a hprof you take with jmap what's it say the suspects are ?

Memory Analyzer

Leaks can cause this but so can memory living too long, i.e. taking too long to free up unused memory as it gets promoted to old and can sit there till a full GC (no CMS) e.g. remove not just cancel timers, look at every cache. Carefully look at any soft / weak reference usage.
 
reply
    Bookmark Topic Watch Topic
  • New Topic