When we start a java program with the -Xmx=1024m option on a machine with 4GB of RAM, is it possible for the java process to cause a memory leak, such that the entire 4GB is used up? Won't it throw an java.lang.OutOfMemory error, rather than try to allocate memory outside the 1024m limit?
The -Xmx value is not the memory limit on the java task, it is only the maximum allowed heap size. The java process also uses memory for:
a) the permanent generation (class-level data)
b) The thread stacks
c) The C/C++ data structures used internally by the JVM
d) The DLLs/SOs loaded by the java process
e) The JIT-compiled code
f) The file handles (and other OS data structures) used by the process (not sure which command you are using to see the memory usages, and I'm not sure which commands include this in their totals)
Also note that while item 'd' is shared by all java processes, commands like top report the memory for every process, thus counting some memory twice. There was a long discussion on this topic several years ago in this forum, let me see if I can find it. Found it:
http://www.coderanch.com/t/111262/Linux-UNIX/read-memory-usage-process-running
Thus you can end up using much more memory than the 1GB you allocated for the heap.