This week's book giveaway is in the Servlets forum.
We're giving away four copies of Murach's Java Servlets and JSP and have Joel Murach on-line!
See this thread for details.
The moose likes Tomcat and the fly likes tomcat suddenly stopped working for half an your without any error in log Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Murach's Java Servlets and JSP this week in the Servlets forum!
JavaRanch » Java Forums » Products » Tomcat
Bookmark "tomcat suddenly stopped working for half an your without any error in log" Watch "tomcat suddenly stopped working for half an your without any error in log" New topic
Author

tomcat suddenly stopped working for half an your without any error in log

Alexander Rondel
Greenhorn

Joined: Apr 13, 2012
Posts: 3
Hi folks,



I have two tomcats here working in parallel on separate servers. Both have exactly the same configuration and get requests by a load balancer. On each tomcat my struts webapp is deployed. Both webapps are accessing the same database. And both suddenly stopped working without any kind of error message.

What happened yesterday in detail is that suddenly server A stopped working. The logs stopped at 09:55am . No error message, nothing. When I tried to connect to the server my connection timed out. Half an hour later server B also stopped working. Exactly the same way, no connection possible no error in log. After resetting Server A at 10:40am the other server (B) suddenly continued working. There where some minor error in the log caused by lost connections... but thats all. Server B just kept working as if nothing has happened.

My first suggestion was that there was an deadlock situation at the database...but during the 'pause'-time there weren't any waiting connections. Next to this even if the database isn't available the tomcat should do something, shouldn't it? Like accepting the connection and do some logging (I do some logging every time a user connects, independent whether he's accessing data from the db or not)....Additionally the database in the tomcat runs in a timeout after 10s.

I am always thinking about the DB since it's the only thing how both servers are connected. There is no interaction between the two servers except via the database.

Since the restart of tomcat A everything is working perfectly again.



Does anyone of you have any idea what could has happened here?



Thanks in advance for your help!
Alex

Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 15951
    
  19

Welcome to the JavaRanch, Alexander!

There's another possibility. There might be some other, unrelated process running on those servers that's stealing all the CPU, disk, and/or network resources from the Tomcat server and its apps. In a worst-case scenario, your servers might have been "pwned", and are serving (unbeknownst to you) as nodes in a botnet.

If this problem recurs, you should monitor the systems as a whole to see if you can detect unusual loads.


Customer surveys are for companies who didn't pay proper attention to begin with.
Alexander Rondel
Greenhorn

Joined: Apr 13, 2012
Posts: 3
Hi Tim,


thanks for your answer!

I will have a look at this.

Unfortunately bot machines run on VMs and aren't connected to the world wide web. So I think (& hope) we didn't get pwned....
The underlying machine was watched and didn't have any performance problems....

I hope this does not happen again....


Cheers
Alex
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12759
    
    5
If this was my problem I would use the Management app to look at the memory and thread use.

I had this - all connection attempts timeout - thing happen once, the management app revealed that all available threads were working on a request - some for days!

Turned out my code was trying to connect to a service that was not running and I had failed to account for that possibility.

Bill
Alexander Rondel
Greenhorn

Joined: Apr 13, 2012
Posts: 3
Hi Bill,


in opposition to what the server administration said at first there was an enormous peak at the CPU usage at the error time.

There seems to be something in the applications sources going wrong... I'll put massive logging at it to be able to reproduce it the next time it occurs (which hopefully sin't during the next 20 years ;) )


Thanks a lot for your help
Alex
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: tomcat suddenly stopped working for half an your without any error in log
 
Similar Threads
Tomcat hangs when app start-up fails
is it good to carry the jdbc "Connection" over servlet and Actions ?
Changes in tnsnames and restart application server to refresh the new setting??
HELP: Tomcat PersistenceManager + JDBCStore stuck in getConnection()
Tomcat Issue