File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Tomcat and the fly likes Need suggestions for improving the performance of the Tomcat server in production environment Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Products » Tomcat
Bookmark "Need suggestions for improving the performance of the Tomcat server in production environment" Watch "Need suggestions for improving the performance of the Tomcat server in production environment" New topic
Author

Need suggestions for improving the performance of the Tomcat server in production environment

Hemanth Kumar Simhadri
Ranch Hand

Joined: Apr 22, 2010
Posts: 32
Hello All,

We are facing severe tomcat server performance problems in production environment. Somany users are using the application in the morning from 8 AM to 11 AM. During this time some of the users are not able to do perform some operations like adding new data in database, modifying the data in the database. But users are able to do some search operations. When users are performing the adding of new data in the database, they are getting the errors like:

'System Warning: #5000001: java.lang.ArrayIndexOutOfBoundsException: -1'

Are there anyways to improve the performance of the tomcat server? The details of the server are as below:

Mode of server: (Tomcat Cluster) Using 4 tomcat servers in parallel managed by the hardware load balancer
OS: Linux
Tomcat version: 6.0.18
Java version: jdk1.6.0_18
JAVA_OPTIONS: -Xss512k -Xms640m -Xmx640m -XX:NewSize=320m -XX:MaxNewSize=320m -XX:MaxPermSize=128m -XX:SurvivorRatio=2 -XX:+DisableExplicitGC
JAVA_OPTS: -Xss512k -Xms640m -Xmx640m -XX:NewSize=320m -XX:MaxNewSize=320m -XX:MaxPermSize=128m -XX:SurvivorRatio=2 -XX:+DisableExplicitGC -Xloggc:/appl/enovianxp/tomcat/plmprd1/server2/logs/gc/GC_tomcat.log -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
Oracle: Oracle 11G

we have a test server also but used by very few users. For testing purpose I would like to performing some bulk operations to put the same kind of load that the production server is facing. But how can we do this. Please suggest if there are anyways to test the load test in tomcat server. I want to make this test server so busy and to do some testing to analyse the performance with the peak load.

Thank you in advance for the help
Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 15964
    
  19

The #1 way of "improving the performance of the Tomcat server" is to improve the performance of the web applications. They're where most of the work gets done.

Tomcat performance issues come into play more in things like running out of request-handler threads, limits of the size/number of deployable apps, and stuff like that.


Customer surveys are for companies who didn't pay proper attention to begin with.
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12761
    
    5
I have used the HttpClient toolkit to create custom load tests.

Seems to me your highest priority should be locating the source of this error:
'System Warning: #5000001: java.lang.ArrayIndexOutOfBoundsException: -1'

because it suggests bad logic somewhere.

Presumably you have also gotten familiar with profiling tools such as JAmon.

Bill
Hemanth Kumar Simhadri
Ranch Hand

Joined: Apr 22, 2010
Posts: 32
Thank you william and Tim for the reply. The same problem still occuring. We are thinking that the reason would be the insufficient resources of database. We have two other servers which are accessing the database other than the tomcat application server. one is The Advance search engine which will index the database to search and get the results faster and the other is the license server which checks the license of every login user. We feel that these two servers are gaining more control over the database and so we are getting some warning messages like Connections insufficient. Creating new database connection, Connections pool size insufficient e.t.c.. But still didnt came to any conclusion about the root cause.
Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 15964
    
  19

That's when you break out the performance measuring tools. Java comes with some decent stuff built in, but if time is more important than money, you're best off buying something.
Hemanth Kumar Simhadri
Ranch Hand

Joined: Apr 22, 2010
Posts: 32
Hello William and Tim,
Its been long time i visited this forum. I would like to share my exeprience and knowledge how I solved this issue.

In our web application, there is a peculiar design where the java code reads some XML files and store the XML data in cache. We are using the tomcat application in cluster mode (4 tomcat servers running parallelly). The XML data is key for the validation when any user tries to do any kind of data creation or modification. We have set several business rules which will be validated each time when a user tries to do any modification of the data or creation of the existing data. As it would be a performance issue to read the XML files, we designed the application in such a way that all XML data (of all XML files) will be read at once and stored in cahce (basically in a Private Static variable).

We used the Apache xPath APIs for the XML parsing. Unfortunately the methods of this API (xalan.jar) are not thread safe as they mentioned in their API documentation.

I designed a sample java program in Eclipse to simulate this java.lang.ArrayIndexOutOfBoundsException. In this sample program I created several threads and tried to access the methods of this API for XML parsing and luckily I was able to replicate the exact errors. With that I concluded that the problem is due to the multiple threads accessing the XML parsing methods.

To solve this I did Synchronization of these methods and ran the program again. There are no errors thrown even with a million threads. The sample program is as below




If we remove the synchronized (lock1) and Synchronization(lock2) in the above code, it will give you the same error

The sample XML data is as below: (cannot give the actual XML data due to violations)

Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 15964
    
  19

So what you really had was a race condition, not a performance issue as such. Race conditions tend to make themselves more apparent as the load increases (as you saw).

Unfortunately, they can be hard to diagnose, as they're often difficult to reproduce for analysis. One of my particular specialities is design and programming for environments like this, but sadly, I've been accused of spending too much time worrying about "things that won't happen" - a/k/a "Just git 'er dun!".

Unfortunately, Murphy's Law is quite clear on that point. The things that "won't happen" - will, and at the worst possible time.
Hemanth Kumar Simhadri
Ranch Hand

Joined: Apr 22, 2010
Posts: 32
Tim Holloway wrote:So what you really had was a race condition, not a performance issue as such. Race conditions tend to make themselves more apparent as the load increases (as you saw).

Unfortunately, they can be hard to diagnose, as they're often difficult to reproduce for analysis. One of my particular specialities is design and programming for environments like this, but sadly, I've been accused of spending too much time worrying about "things that won't happen" - a/k/a "Just git 'er dun!".

Unfortunately, Murphy's Law is quite clear on that point. The things that "won't happen" - will, and at the worst possible time.



You are correct Tim. We dont know when the issue happens and when it doesn't. It is always intermittent. I struggled a lot to identify this. It required lot of investigation and understanding of design. I spent 2 months to conclude the root cause. When we simulated the errors in Eclipse, some times the program was successful (when tested with out synchronized (lock)) and sometimes it thrown errors. The worst part is that, the consequences are unpredictable.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Need suggestions for improving the performance of the Tomcat server in production environment
 
Similar Threads
Tomcat crashes with Out of Memory Error!
JVM Initilisation failure - Could not reserve heap
Weblogic's performance tunning
Memory Args:What does -Xmn stand for ?
jvm memory - bizzare