Possible causes of Intermittent/unexpected Tomcat 5.5.25 shut-downs?

[Doug Fielding]

My Apache-Tomcat 5.5.25 server appears to be shutting itself down automatically several times a week, and sometimes 2-3 times in one day. There are no errors being reported in the /logs, only indicating that the server has shut itself down gracefully. Since I have already secured the shutdown port and command in server.xml, what else might be signaling apache-tomcat to stop without errors?

Running the following:

Solaris 8
apache-tomcat-5.5.25
jdk1.6.0_02

Notes:

* no other tomcat servers running
* shutdown port and command changed to be more secure
* intermittently shutting down without manual execution of ./shutdown.sh

Ports and Logs:

--- (fyi) for security, this is how server.xml shutdown port is currently configured (not really, but like this) ---

<Server port="8038" shutdown="uu87iss_SSHUTss1ool_DOWNN">

--- after unexpected shutdown, here is the catalina.out messages - gracefully closing without additional errors ---

Jan 17, 2008 1:59:55 PM org.apache.coyote.http11.Http11BaseProtocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8088
Jan 17, 2008 1:59:55 PM org.apache.coyote.http11.Http11BaseProtocol pause
INFO: Pausing Coyote HTTP/1.1 on http-8443
Jan 17, 2008 1:59:56 PM org.apache.catalina.core.StandardService stop
INFO: Stopping service Catalina

[ January 17, 2008: Message edited by: Bear Bibeault ]
[ January 17, 2008: Message edited by: Doug Fielding ]
[ January 18, 2008: Message edited by: Doug Fielding ]

[Ben Souther]

Two things to look for:

1.) Users issuing the 'killall java' command from either root or the user under which Tomcat is running (We've seen that a few times here).

2.) Check your source code for any System.exit() calls.
This can be blocked if you run Tomcat under a security manager.

[Doug Fielding]

Thanks Ben for your response. In talks with some of my coworkers, it seems unlikely that a 'users' would be be able to connect to this server (telnet/ssh) with the ability to run a 'killall java'. As you suggested, I did check our source code for System.exit() calls...and there were none to be found. Any more ideas would be appreciated. Anyone...please

Thanks again!
[ January 18, 2008: Message edited by: Doug Fielding ]

[William Brogden]

That is confusing for sure - if this was my problem I would be monitoring Tomcat with the HTML based management application, looking in particular at number of Threads, memory use, session count, request count and error count.

Bill

[Ben Souther]

Can the shutdown port (8005) on that machine open to the outside world?

[David O'Meara]

I would also try:
1) make sure your logging is synchronous (asynch logs can get lost)
2) redirect output streams to a file eg catalina.sh run >logs.txt in case something is getting lost
3) check the rest of the logs. It's worth a look
4) try to reproduce the behaviour outside the production environment

System.out is usually the first place I look, it may still be worth trying a slightly more restrictive security manager.