This week's giveaway is in the EJB and other Java EE Technologies forum.
We're giving away four copies of EJB 3 in Action and have Debu Panda, Reza Rahman, Ryan Cuprak, and Michael Remijan on-line!
See this thread for details.
The moose likes Performance and the fly likes 100% CPU usage on JBoss with 8GB memory Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of EJB 3 in Action this week in the EJB and other Java EE Technologies forum!
JavaRanch » Java Forums » Java » Performance
Bookmark "100% CPU usage on JBoss with 8GB memory" Watch "100% CPU usage on JBoss with 8GB memory" New topic
Author

100% CPU usage on JBoss with 8GB memory

Dean Pullen
Ranch Hand

Joined: May 30, 2003
Posts: 58
Hi all,

We've got an application that stores a large amount of data in-memory which is then used by a lot of users.

Subsequently we're seeing CPU usage go through the use, and every now and then hit 100% (on a quad core machine with 8GB of mem) which I believe is when the JVM tries to garbage collect. The web app can't be used until JBoss is killed and restarted.

We currently have these JVM opts:
JAVA_OPTS="$JAVA_OPTS -Xms5g -Xmx5g -XX:MaxNewSize=3g -XX:PermSize=512m -XX:MaxP
ermSize=1g -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterv
al=3600000 -Djava.awt.headless=true -d64 -verbose:gc"

Which may or may not be causing the problems - my knowledge of GC management is very limited.


We see something similar to this in the logs:

(lots more)


(lots more)

Any help would be much appreciated.

JBoss 4.2.1GA
SuSE Enterprise 10 SP1
Java 1.6.0_05
Michal Glowacki
Ranch Hand

Joined: Mar 14, 2006
Posts: 114
Are you using EJB in your application? When there is lot of ejb transactions it can result in deadlock and in fact heap runs out of memory. In that case gc can't come to the rescue.
Dean Pullen
Ranch Hand

Joined: May 30, 2003
Posts: 58
Yes, the system is predominately based around EJBs.

I'm not sure why a large use of EJBs would result in deadlock?
rajesh bala
Ranch Hand

Joined: Jan 14, 2003
Posts: 66
Did you try taking a thread dump?. 100% CPU utilization problems can be easily solved by taking a threaddump and viewing what is going in the threads.

Use kill -QUIT or kill -3 for linux based and ctrl+pause-break in windows in the jboss console.

You could post the threaddump over here if you prefer.

~Rajesh.B
Dean Pullen
Ranch Hand

Joined: May 30, 2003
Posts: 58
Cheers rajesh - I'll do that next time it does it.
Michal Glowacki
Ranch Hand

Joined: Mar 14, 2006
Posts: 114
Originally posted by Dean Pullen:
Yes, the system is predominately based around EJBs.

I'm not sure why a large use of EJBs would result in deadlock?


It is bad practice, actually even anti-pattern to use EJB as read-only objects - it puts table locks what increase overhead. Where you do not need transaction management you should use data access objects. I can recommend you a tool to generate DAO for your database - it's http://mdaog.sourceforge.net/. It's very ease to replece your entity bean calls with dao and transfer objects using library generated by this tool.
Dean Pullen
Ranch Hand

Joined: May 30, 2003
Posts: 58
I hadn't actually realised this.

They are in fact not read-only. They are generated and then added to the DB.

Would it be best to use standard EJBs to create them, and then retrieve them as DAO's at a latter point for caching?

Also...if we simply detach them from the hibernate session they effectively become POJOs, isn't this sufficient as opposed to creating a new DAO etc?
Michal Glowacki
Ranch Hand

Joined: Mar 14, 2006
Posts: 114
Not sure about hibernate.

I mean not to use ejb as read-only objects in places where you only read from them. Feel free to use them to insert/update tables in db where you need transaction management.

So what I mean you can (and for some/most cases should) have both dao and entity beans for the table. You use dao when you read (except from some transactions, fx in session beans where you need to be sure the data meanwhile hasn't changed it's state) and ejb to insert/update.

Personally, I use BASICALLY entity beans to insert/update data in tables with balances, etc. But when I for example record login attemp, I use dao to insert record into table. For read - I use everywhere dao, except my session bean methods, which for example make calculations and update important table, again, like balances etc.

Hope that helps.
Dean Pullen
Ranch Hand

Joined: May 30, 2003
Posts: 58
I can imagine from what you're saying that you're using EJB2.1 and not EJB3?
Michal Glowacki
Ranch Hand

Joined: Mar 14, 2006
Posts: 114
yes, 2.1 on JBoss 4.0.5GA
Dean Pullen
Ranch Hand

Joined: May 30, 2003
Posts: 58
The DAO mentality is somewhat different with EJB3 - entities are basically Pojo's removing the need for seperate DAOs.

Trying to figure out how best to cache them is my next step.
Roger Chung-Wee
Ranch Hand

Joined: Sep 29, 2002
Posts: 1683
Yes, the system is predominately based around EJBs

Are you using entity beans? JBoss uses dynamic proxies to generate the container classes at run time, making extensive use of reflection which can really affect performance.


SCJP 1.4, SCWCD 1.3, SCBCD 1.3
Dean Pullen
Ranch Hand

Joined: May 30, 2003
Posts: 58
Originally posted by rajesh bala:
Did you try taking a thread dump?. 100% CPU utilization problems can be easily solved by taking a threaddump and viewing what is going in the threads.

Use kill -QUIT or kill -3 for linux based and ctrl+pause-break in windows in the jboss console.

You could post the threaddump over here if you prefer.

~Rajesh.B


I've done a kill -3 and received the following output:
Stack trace

I notice two threads linked to the class CsiAdminAccessChildren.

Is it possible this is in an infinite loop or similar, thus causing the 100% CPU usage?
rajesh bala
Ranch Hand

Joined: Jan 14, 2003
Posts: 66
Most of the threads seems to be fine in the stacktrace. If you have taken this when the CPU was 100% its good.

1. I hope you have access to the sourcecode.
2. If so, please look at CsiAdminAccessLevelCache.java:169. This is being accessed from 3 threads and all the three are in runnable state.
3. Good part is that, all the threads are rooted from CsiAdminAccessLevelCache only. So worst case scenario, you could try turning it off and try out if possible (Try this option if you dont have source code and you can try it in staging box first)
4. Check what is the size of the TreeMap in the cache.

But otherwise, the memory footprint and others threads are very normal. If possible you could post the code snippet here (provided your company allows it. ..).

~Rajesh.B
Dean Pullen
Ranch Hand

Joined: May 30, 2003
Posts: 58
As I thought - and you too - there was a problem with that class.

Infinite loop. Thus the 100% CPU usage!

Deary me! The kill 3 was a master stroke, I'll be using that more often.

Thanks again.
rajesh bala
Ranch Hand

Joined: Jan 14, 2003
Posts: 66
Good to hear that it solved the problem. You are welcome Dean.

~Rajesh.B
Bobby Sharma
Ranch Hand

Joined: Mar 18, 2008
Posts: 550
    
    1

thanks Rajesh, I also learned that


Back to Java , again.
M Fern�ndez
Greenhorn

Joined: Jul 16, 2008
Posts: 1
Hi all,
we're facing a similar problem in our production environment. First we try to replicate the problem in a test environment, a clone of the production environment, with little success. Fortunately we've discovered this forum thread and the kill -3 option.
We toke a threaddump of the production server, while 3 of the 4 cpu's shows a 100% occupation, and found two methods being accessed from several threads and all them are in runnable state. This two methods have loops, I've pasted two code snippets at the bottom of this post.
If you could have a look at the threaddump and confirm us we are in the right way we will appreciate your help a lot.
Thanks in advance


public void service(HttpServletRequest request, HttpServletResponse response){
try {
...
response.setContentType ("application/download");
response.setHeader ("Content-Disposition", "attachment; filename="+nombreFichero );

java.io.File file = new java.io.File(nombreFichero);
response.setContentLength((int)file.length());
java.io.InputStream in = new java.io.BufferedInputStream (new java.io.FileInputStream(file) );
int length;
byte[] buf = new byte[response.getBufferSize()];
java.io.OutputStream op = response.getOutputStream ();
while ((in != null) && ((length = in.read(buf)) != -1)){
op.write(buf,0,length);
}
if (in != null){
in.close();
}
if ( op != null){
op.close();
}

}catch (Exception e){
...
}
}



private ApplicationObjectInterface buscarFamiliaEnJerarquia(String id) throws Exception{
if(nivelesJerarquia != null){
ApplicationObjectInterface obj = null;

for( Iterator iter= this.nivelesJerarquia.values().iterator();iter.hasNext() ; ) {
obj = (ApplicationObjectInterface) iter.next();
if ( obj != null && obj.getObjetosHijos() != null){
HashMap gamasCat = obj.getObjetosHijos();

for (Iterator iter1=gamasCat.values().iterator(); iter1.hasNext() ; ) {
obj = (ApplicationObjectInterface) iter1.next();

if ( obj != null && obj.getObjetosHijos() != null){
HashMap familiasCat = obj.getObjetosHijos();
for (Iterator iter2=familiasCat.values().iterator(); iter2.hasNext() ; ) {
obj = (ApplicationObjectInterface) iter2.next();
if ( obj != null && obj.getId().equals(id)){
return (ApplicationObjectInterface) obj;
}//if
}//for
}//if
}//for
}//for
}//for
}//if
return null;
}

[ July 16, 2008: Message edited by: Marco Fern�ndez ]

[ July 16, 2008: Message edited by: Marco Fern�ndez ]

[ July 18, 2008: Message edited by: M Fern�ndez ]
[ July 18, 2008: Message edited by: M Fern�ndez ]
steve souza
Ranch Hand

Joined: Jun 26, 2002
Posts: 852
Due to formatting problems it's hard to read your code. It looks like you are looping through a lot of data structures to find one object. Maybe you have the wrong type of object and should instead be using one that has better direct access such as a Map.

Also, you should have opened a new posting and not hijacked this old one.


http://www.jamonapi.com/ - a fast, free open source performance tuning api.
JavaRanch Performance FAQ
 
It is sorta covered in the JavaRanch Style Guide.
 
subject: 100% CPU usage on JBoss with 8GB memory
 
Similar Threads
JVM memory releasing after memory consuming task
Any pointers why my app does so much minor GC?
Full GC problem
Forcing Full GC
Garbage Collection Tuning- Help!