You can tell the JVM to produce a heap dump on an out-of-heap-memory exception using the -XX:+HeapDumpOnOutOfMemoryError option. But you are not getting an out-of-heap-memory exception.
As Kees noted, your machine is out of memory - the JVM has asked the OS for some memory and the OS has said "sorry, no more memory is available." The quick fix is to increase the amount of swap space available (i.e., increase the pagefile size, or add another page file). Another fix is to install more RAM.
There are two possible reasons why the JVM would ask for more memory:
1) The JVM needs to allocate another thread stack because it want to create another thread. There are several things you could do to mitigate this:
a) You could restrict the number of threads created. Usually this is an arduous task because the number of places where thread limits are defined.
b) You can decrease the thread stack size. By default is it 1MB on Windows, usually 1/2 that size is sufficient. Use this JVM option: -Xss512M
2) The JVM attempted to allocate more memory for the heap. This can happen is you set -Xms and -Xmx to different values. I always recommend to set these to the same value in production.
Have you ever measured the effects of setting -Xmx and -Xms to either the same or different values? I have tested the idea that setting -Xmx and -Xms make a difference, but I found that after the first full GC the JVM just goes back to (re)allocating memory. http://java-monitor.com/forum/showthread.php?t=427
Others have tested this, and found this to be JVM implementation dependent.
In my experience setting -Xms and -Xmx to the same value kept the heap at a constant size. The last time I remember doing this specifically was earlier this year for a series of tests on various versions of the JDK (from 1.3.1 to 7) in Windows (I used a 1G heap). And yes each run had several full GCs. I guess I would be more convinced by your data if you showed the full command line, and used the GC statistics command line options (at least -vebose:gc) to record actual heap sizes. I find that getting the "current heap size" info from JMX is usually useless, especially for tuning or explaining heap behavior.
However, the Sun JDK has platform specific code for Solaris, Windows and Linux, so variations will creep in. Also, the choice of which garbage collector to use could also be a factor. And there are also JVM options that set/reset other options, so where they appear in the command line is of utmost importance.