When we start a java program with the -Xmx=1024m option on a machine with 4GB of RAM, is it possible for the java process to cause a memory leak, such that the entire 4GB is used up? Won't it throw an java.lang.OutOfMemory error, rather than try to allocate memory outside the 1024m limit?
The -Xmx value is not the memory limit on the java task, it is only the maximum allowed heap size. The java process also uses memory for:
a) the permanent generation (class-level data)
b) The thread stacks
c) The C/C++ data structures used internally by the JVM
d) The DLLs/SOs loaded by the java process
e) The JIT-compiled code
f) The file handles (and other OS data structures) used by the process (not sure which command you are using to see the memory usages, and I'm not sure which commands include this in their totals)