Running on a linux box with 8 processors and 8G of RAM, my app server uses only stateless session bean. Originally, the max number of SLSB was 30 and the heap size was 0.5 G. To handle additional requests, I set the max to 200 and increased the heap size to 1G. The app server was shutdown due to OutOfMemoryError. On two consecutive days, I noticed that the server was down about two hours after the peak time because JVM could not allocate memory to create native thread. After I reset heap size back to 0.5G (kept the max number of SLSB to 200), my app server ran fine. My question is that why my app server was not shutdown when the heap size was small? Was that because the default GC ran better with a small heap size?
I don't think you would get specific answer here but to give my 2 cents, may be due to more memory + the nature of the peak time traffic the heap got more fragmented which made it run out of memory.
Here the issue is, the same peak traffic can't be reproduced. So we can't reproduce the same memory requirements with changing the max heap size next time. Though if we can consistently prove that with the higher heap settings it "always" runs out of memory after peak hours and when its set to low max heap size it works okay THEN we can try investigate it more and see why this behavior.
One time behavior doesn't provide us with good use case to analyse.
You can use JMeter to stress test your application. Also, please check that you are releasing the resources and not creating db and other connections every time unnecessary. If it is a traffic problem use queuing mechanism.