Kri,
You are either running out of memory for real, or running out of process address space.
Good luck.
setting your Thread Stack Size to a value lower than 1 mb might help you.. but if you are leaking threads fix that instead and don't sweat it.
-XX:ThreadStackSize=256
Should be adequate... if you are running
java 1.4.2 or higher on HP or Sun