We are running a Tomcat 5.5 version through proxy_ajp mode via a regular Apache webserver.
Within last 2 weeks we had 2 strange downtimes of our system. Our application is running on Tomcat 5.5.
One morning I woke up and noticed our website is down. I logged into the server and checked for running processes.
The servers load was around 4% (while normally it's around 1%). I checked Tomcat and it was still running, however, the webserver did not respond to any queries. The browser was unable to load our website.
So I tried to restart Tomcat:
1) to stop the service
2) start it.
When I ran a command: service tomcat stop - the Tomcat tried to shut down. However it stopped responding... No status on command line, nothing. It looked like the server halted. So I had to re-login to the server once again and then again to run the same command to stop tomcat.
Tomcat started to respond only the 2nd time and it stopped.
Then I ran a command to start tomcat: service tomcat start
Tomcat did not start for the first time. Well, it started, however, it did not respond.
So I had to stop it once again and then ran a start command once again to get it up and running.
This time Tomcat started successfully and the application started to work.
However, the question is: who caused the problem with Tomcat not responding? It did not crash, however, the web server was no functioning and unable to process requests.
I checked our ping software's logs to see for a downtime and noticed that downtime started to appear something like this:
5 minute intervals
until i finally restarted the server.
Tomcat did not fully crashed, however, it was not serving the requests.
This week I woke up again in the morning and I noticed that Tomcat is serving everything really slow... It was taking 2 minutes or so to load the website, however, this time it was serving (at least!).
I went to restart Tomcat and application this time again and got the same issues as described in CASE 1. Not responding, etc.
I had to stop tomcat, relogin, stop it again, then start it, stop it and finally start it again and it started working.
So my question: what should we do in order to find a reason what's causing these Tomcat's downtimes? How to prevent them?
We tried checking all tomcats logs, etc. but did not find any real problems.
Please, help us to identify the problems so we could fix it... Any suggestions?