• Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

How to debug JBoss crashing

 
David Cheung
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
We're running an Enterprise VOIP application on JBoss 4.2.3 GA and it's crashing under high call traffic. There's no heap dump and we see in the run.log:

Exception in thread "CompilerThread1" java.lang.OutOfMemoryError: requested 6398
48 bytes for Chunk::new. Out of swap space?

or

Exception java.lang.OutOfMemoryError: requested 15066072 bytes for GrET* in C:/B
UILD_AREA/jdk1.5.0_16/hotspot\src\share\vm\utilities\growableArray.cpp. Out of s
wap space?

What are some strategies to investigate these problems?

Thanks
 
Kees Jan Koster
JavaMonitor Support
Rancher
Posts: 251
5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Dear David,

These are very bizarre messages and not very JBossy. Looks like your machine has run out of physical RAM and cannot allocate 14MB in one chunk, and suggested in the last error.

Are you using JNI calls? Is this a stock JDK or a home-built one (perhaps by your vendor?). How much physical RAM is in the machine? What is Java's -Xmx setting?
 
David Cheung
Greenhorn
Posts: 10
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
It's an Intel Xeon machine with 4 GB of RAM, running Windows Server 2003 32-bit. Yes, there are JNI calls which is what we fear might be bringing down the system. We're using the stock JDK.

-Xmx=1048m

Is there any way to get JBoss to create a heap dump on crash?
 
Peter Johnson
author
Bartender
Posts: 5852
7
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You can tell the JVM to produce a heap dump on an out-of-heap-memory exception using the -XX:+HeapDumpOnOutOfMemoryError option. But you are not getting an out-of-heap-memory exception.

As Kees noted, your machine is out of memory - the JVM has asked the OS for some memory and the OS has said "sorry, no more memory is available." The quick fix is to increase the amount of swap space available (i.e., increase the pagefile size, or add another page file). Another fix is to install more RAM.

There are two possible reasons why the JVM would ask for more memory:

1) The JVM needs to allocate another thread stack because it want to create another thread. There are several things you could do to mitigate this:

a) You could restrict the number of threads created. Usually this is an arduous task because the number of places where thread limits are defined.
b) You can decrease the thread stack size. By default is it 1MB on Windows, usually 1/2 that size is sufficient. Use this JVM option: -Xss512M

2) The JVM attempted to allocate more memory for the heap. This can happen is you set -Xms and -Xmx to different values. I always recommend to set these to the same value in production.
 
Kees Jan Koster
JavaMonitor Support
Rancher
Posts: 251
5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Dear Peter,

Have you ever measured the effects of setting -Xmx and -Xms to either the same or different values? I have tested the idea that setting -Xmx and -Xms make a difference, but I found that after the first full GC the JVM just goes back to (re)allocating memory. http://java-monitor.com/forum/showthread.php?t=427

Others have tested this, and found this to be JVM implementation dependent.

Kees Jan
 
Peter Johnson
author
Bartender
Posts: 5852
7
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
In my experience setting -Xms and -Xmx to the same value kept the heap at a constant size. The last time I remember doing this specifically was earlier this year for a series of tests on various versions of the JDK (from 1.3.1 to 7) in Windows (I used a 1G heap). And yes each run had several full GCs. I guess I would be more convinced by your data if you showed the full command line, and used the GC statistics command line options (at least -vebose:gc) to record actual heap sizes. I find that getting the "current heap size" info from JMX is usually useless, especially for tuning or explaining heap behavior.

However, the Sun JDK has platform specific code for Solaris, Windows and Linux, so variations will creep in. Also, the choice of which garbage collector to use could also be a factor. And there are also JVM options that set/reset other options, so where they appear in the command line is of utmost importance.
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic