wood burning stoves*
The moose likes JBoss/WildFly and the fly likes How to debug JBoss crashing Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Products » JBoss/WildFly
Bookmark "How to debug JBoss crashing" Watch "How to debug JBoss crashing" New topic
Author

How to debug JBoss crashing

David Cheung
Greenhorn

Joined: Jul 31, 2009
Posts: 10
We're running an Enterprise VOIP application on JBoss 4.2.3 GA and it's crashing under high call traffic. There's no heap dump and we see in the run.log:

Exception in thread "CompilerThread1" java.lang.OutOfMemoryError: requested 6398
48 bytes for Chunk::new. Out of swap space?

or

Exception java.lang.OutOfMemoryError: requested 15066072 bytes for GrET* in C:/B
UILD_AREA/jdk1.5.0_16/hotspot\src\share\vm\utilities\growableArray.cpp. Out of s
wap space?

What are some strategies to investigate these problems?

Thanks
Kees Jan Koster
JavaMonitor Support
Rancher

Joined: Mar 31, 2009
Posts: 251
    
    5
Dear David,

These are very bizarre messages and not very JBossy. Looks like your machine has run out of physical RAM and cannot allocate 14MB in one chunk, and suggested in the last error.

Are you using JNI calls? Is this a stock JDK or a home-built one (perhaps by your vendor?). How much physical RAM is in the machine? What is Java's -Xmx setting?


Java-monitor, JVM monitoring made easy <- right here on Java Ranch
David Cheung
Greenhorn

Joined: Jul 31, 2009
Posts: 10
It's an Intel Xeon machine with 4 GB of RAM, running Windows Server 2003 32-bit. Yes, there are JNI calls which is what we fear might be bringing down the system. We're using the stock JDK.

-Xmx=1048m

Is there any way to get JBoss to create a heap dump on crash?
Peter Johnson
author
Bartender

Joined: May 14, 2008
Posts: 5776
    
    7

You can tell the JVM to produce a heap dump on an out-of-heap-memory exception using the -XX:+HeapDumpOnOutOfMemoryError option. But you are not getting an out-of-heap-memory exception.

As Kees noted, your machine is out of memory - the JVM has asked the OS for some memory and the OS has said "sorry, no more memory is available." The quick fix is to increase the amount of swap space available (i.e., increase the pagefile size, or add another page file). Another fix is to install more RAM.

There are two possible reasons why the JVM would ask for more memory:

1) The JVM needs to allocate another thread stack because it want to create another thread. There are several things you could do to mitigate this:

a) You could restrict the number of threads created. Usually this is an arduous task because the number of places where thread limits are defined.
b) You can decrease the thread stack size. By default is it 1MB on Windows, usually 1/2 that size is sufficient. Use this JVM option: -Xss512M

2) The JVM attempted to allocate more memory for the heap. This can happen is you set -Xms and -Xmx to different values. I always recommend to set these to the same value in production.

JBoss In Action
Kees Jan Koster
JavaMonitor Support
Rancher

Joined: Mar 31, 2009
Posts: 251
    
    5
Dear Peter,

Have you ever measured the effects of setting -Xmx and -Xms to either the same or different values? I have tested the idea that setting -Xmx and -Xms make a difference, but I found that after the first full GC the JVM just goes back to (re)allocating memory. http://java-monitor.com/forum/showthread.php?t=427

Others have tested this, and found this to be JVM implementation dependent.

Kees Jan
Peter Johnson
author
Bartender

Joined: May 14, 2008
Posts: 5776
    
    7

In my experience setting -Xms and -Xmx to the same value kept the heap at a constant size. The last time I remember doing this specifically was earlier this year for a series of tests on various versions of the JDK (from 1.3.1 to 7) in Windows (I used a 1G heap). And yes each run had several full GCs. I guess I would be more convinced by your data if you showed the full command line, and used the GC statistics command line options (at least -vebose:gc) to record actual heap sizes. I find that getting the "current heap size" info from JMX is usually useless, especially for tuning or explaining heap behavior.

However, the Sun JDK has platform specific code for Solaris, Windows and Linux, so variations will creep in. Also, the choice of which garbage collector to use could also be a factor. And there are also JVM options that set/reset other options, so where they appear in the command line is of utmost importance.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: How to debug JBoss crashing
 
Similar Threads
IPlanet memory management while tuning for performance
jvm memory - bizzare
Using record cache and booking method
'Out of swap space?' error
OutOfMemory Error