I was expecting this to remain well within 2.5GB, however when I run top I see RES=4GB. I don't get any OutOfMemory errors and the VM does not show any leaks. The app is simple jsp, No EJBs, No db access.
I assume that you used an operating system utility to determine memory usage for the java process. Well, for a java process the memory usage is a combination of:
the java heap (which you set to 1.5G)
the permgen, which holds class-level info (add another 304MB)
the thread stack - each thread uses up from 512KB to 2MB of space depending on the OS, and an application server runs a large number of threads
the C/C++ data structures used by the JVM
the code for the JVM, including the executable itself and any libraries (.so, .dll) used by it
I probably forgot something. But when you add all that up, it could be 4GB.
So 1536m + 304m + 237 threads at -Xss256k is still less than 2 GB.
That would mean that 2GB of memory is being used by the C/C++ data structures and code for JVM. That is more than even my full heap. I find that very strange.
A similar process that I manage on Solaris 10 with similar settings and more threads does not go beyound 2.3 GB.