I have a JEE EAR application deployed on 2 clusters (with 2 instances each) in a GlassFish v3.1 application server loadbalanced by an apache server with mod_jk. One of the issues that I am currently facing is that every morning I have almost 100 users accessing the application at 8 am when they start their work and then they access it on a regular basis through the rest of the day. The performance on a normal one person usage scenario is pretty consistent with less than 5 second response time for loading the main page. However, the performance in the mornings when there is a mass hit is pretty bad. Can you advise me on what I can do to improve this?
What you can do to improve this is make sure that you understand what specific performance requirements are required and also make sure that you understand what "performance" means in the context of your application. This should be clearly documented in your technical specifications documents.
Obviously, you have to examine the application to identify exactly (1) the location of the bottleneck and (2) the specific code statements that are causing the bottleneck.
Once you clearly and deeply understand where and what the problem is, you can start to rewrite the relevant portions in a more efficient way. It sounds like your current version is failing to meet customer Quality of Service requirements.