I've got an application that was started with mx512m. My understanding is that this limits the JVM heap to 512MB of space. While watching the actual java process (using top) I am seeing the memory usage for the process go up to over 1GB of memory. Once the memory gets this high I start seeing java.lang.OutOfMemory errors.
One of the packages I'm using seems to be using the File.deleteOnExit() method. A bug on Sun's site indicates that this method causes native heap memory leaks. I can understand that that would cause my process memory to go up but would it cause OutOfMemoryErrors?.
I thought that the OutOfMemoryError meant that I didn't have anymore JVM heap space left (limited to 512mb in my case). Is it possible that I'm seeing the OutOfMemoryError due to the size of the native heap?
Have you monitored garbage collection to see how frequent it is being performed? OutOfMemoryErrors can be caused any time garbage collection (assuming it ran) fails to re-allocate enough memory to continue.
I'd run your application through a 3rd party profiling tool and monitor memory allocation and use. The memory leak could be coming from anywhere.