• Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Tomcat crashing

 
Randy Monnin
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

I'm running a small website with Tomcat 5.5 serving up jsp and the (relatively few) http requests.

Recently, I've begun seeing problems in which the site will suddenly stop responding. I don't see any error messages written to the catalina.out file or any other file in the tomcat/logs directory. But attempts to access the page (even requests on the same server via localhost) eventually just time out.

I'm at my wits end. Nothing changed in the tomcat configuration files, and nothing else is running on the server. (CentOS 5.2).

I'm not a Tomcat expert by any means, so any help would be appreciated.

Thanks!

-Randy
 
Kees Jan Koster
JavaMonitor Support
Rancher
Posts: 251
5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Dear Randy,

Is the process still alive or not? Does Tomcat still listen on the server port or not? Does Tomcat consume 100% cpu or 0%?

If Tomcat is alive and listening on the port, take a thread dump to see what it is doing. Here is some basic discussion on what to do with thread dumps.

Hope this gives you some corners to peel at. Does this help?
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13058
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You might use the Tomcat Management application to monitor the server. You might see a trend - for example I caught a bug in which the request Threads ended up waiting indefinately for a service that had been removed. Eventually all available request Threads were consumed.

Bill
 
Randy Monnin
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Kees Jan Koster wrote:Dear Randy,

Is the process still alive or not? Does Tomcat still listen on the server port or not? Does Tomcat consume 100% cpu or 0%?

If Tomcat is alive and listening on the port, take a thread dump to see what it is doing. Here is some basic discussion on what to do with thread dumps.

Hope this gives you some corners to peel at. Does this help?


Thanks for your help. The process is still alive and is consuming 0% of the CPU. I'm not sure if it's still listening on the server point at that time or not. I'll follow those instructions the next time it happens and will let you know.
 
Randy Monnin
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
William Brogden wrote:You might use the Tomcat Management application to monitor the server. You might see a trend - for example I caught a bug in which the request Threads ended up waiting indefinately for a service that had been removed. Eventually all available request Threads were consumed.

Bill


Thanks, Bill. Can you point me to any how-to guides for using the Tomcat Management application to monitor the server? I've never done that before. If all request threads were consumed, wouldn't an error be written to the log file? I'm not seeing any errors being written out.

-Randy
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13058
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The manager application is not part of the standard distribution, you have to download and install it separately and configure a user with access.

Here is the howto for Tomcat 5.5 - which is the one I use, probably you can find a similar version for Tomcat 6, I dont remember if Tomcat 4 can use it.

As I recall, when all of the allocated request Threads are busy, Tomcat just does not answer, no error message.

Bill

 
Kees Jan Koster
JavaMonitor Support
Rancher
Posts: 251
5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Dear All,

When Tomcat's HTTP thread pool is exhausted it starts responding with HTTP response code 502 to indicate the server's overloaded status.

As for monitoring threads, you can also use Java-monitor, but I am biased. ;-)
 
Randy Monnin
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for helping!

The problems occurred again last night, right on schedule. I ran a netstat and a jstack each time when the site wasn't responding, and nothing looked out of the ordinary.

Netstat shows one line: "tcp 0 0 :::80 :::* LISTEN 20604/java" indicating that the process is still listening on port 80, and it shows 16 other entries talking to port 80, either in a "FIN_WAIT2", "FIN_WAIT1", "CLOSE_WAIT", or "ESTABLISHED" state.

The jstack dump shows 36 threads, 5 with a state of "IN_WAITING", the rest with a state of "BLOCKED".

The strange thing is that apart from last weekend, when we saw the problem all weekend long, on week days the problem only occurs between 10:00 pm and 1:00 am. Some nights the problems stop as early as 11:20 pm, and some nights as late as 1:10 am. There's nothing scheduled in the crontab to run between those hours.

-Randy
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13058
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What external resource does this servlet access?

Can you get a response when addressing a simple HTML page?

Bill
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic