Win a copy of Mesos in Action this week in the Cloud/Virtualizaton forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Tomcat fails to restart with "Address already in use" exception

 
Alexey Andreev
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi!

Our custom java web application is running on:

Dedicated Linux server, Tomcat 5.5.23.0, JVM 1.5.0_12

Tomcat run parameters:

java -server -Xms1000m -Xmx2000m -Xmn756m -XX:MaxPermSize=128m -Djava.awt.headless=true -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -Xloggc:/usr/local/tomcat/logs/gc.log -XX:+UseParallelGC -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties -Djava.endorsed.dirs=/usr/local/tomcat/common/endorsed -classpath :/usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/commons-logging-api.jar -Dcatalina.base=/usr/local/tomcat -Dcatalina.home=/usr/local/tomcat -Djava.io.tmpdir=/usr/local/tomcat/temp org.apache.catalina.startup.Bootstrap start

Recently we started getting a strange problem while restarting the tomcat.

After running "service tomcat restart" (or stop-start it actually doesn't matter) the server fails to start and the following is written to catalina.out :



No other software is running on port 8080 on the server. And I even check that there is no active connection before restart, and still get the same.
If I do the second restart of tomcat after several minutes it goes fine and quick, but if I try to do it after an hour it almost everytime fails.

Any help is appreciated.

Thank you very much in advance.



 
Damayanti Samal
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
sometimes it found same , just you change the port number to some other port like 7777 and check .
 
William Brogden
Author and all-around good cowpoke
Rancher
Posts: 13062
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Exactly how are you starting and stopping Tomcat.

Exactly how are you checking the 8080 port for an existing user?

Bill
 
Kees Jan Koster
JavaMonitor Support
Rancher
Posts: 251
5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Dear Alexey,

You can check who is listening using the commands in this article: http://java-monitor.com/forum/showthread.php?t=22

Make sure you check on the right interface on a multi-homed machine. Also, there may be background processes automatically starting Tomcat servers. Finally, it is very easy for developers to make a mistake so that Tomcat will not terminate properly. Thus, after stopping Tomcat, check that is dead before starting. So don't do "restart", but "stop, check, start".
 
Alexey Andreev
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks for your replies.

Before restart I am stopping monitoring software which watches for tomcat process (MONIT). So it can not be started automatically.
I stop the tomcat. Check the process with "ps -ef | grep java". Check the 8080 port with "netstat -t -a -n -v | grep :8080"
Everything is clean and I do start. And get the exception in logs.

Must admit that often stop command takes a long time to complete if it can give you a clue.

I am using s standard tomcat service script for starts/stops.

here is an excerpt from it

START:




STOP:

 
Tim Holloway
Saloon Keeper
Pie
Posts: 18169
53
Android Eclipse IDE Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Tomcat shuts down in an orderly way. If one or more of the installed webapps isn't well-behaved, you'll have problems. One of the most common of these is webapps that run a scheduler service such as Quartz and don't shutdown the Quartz scheduler threads (in the servlet destroy() method). That will hang the Tomcat shutdown process forever - or until you forcibly terminate the JVM, since as long as even one thread is alive, the JVM cannot terminate.

Ordinarily the Tomcat shutdown process happens fairly quickly. The "Tomcat restart" process consists of a catalina stop, immediately followed by catalina start. Both of these operations are asynchronous - they don't wait for any sort of event to be posted before returning. So if shutdown takes too long, the catalina start may fail because the catalina stop is still running and hasn't yet released the server network ports.

The cure for that is to either make the webapp shut down faster, or to introduce a delay between stop and start.
 
Sherif Shehab
Ranch Hand
Posts: 485
Android Eclipse IDE Oracle
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Damayanti Samal wrote:sometimes it found same , just you change the port number to some other port like 7777 and check .


Right try to change the port to anything else rather 8080 and it will works fine
 
Alexey Andreev
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Sherif Shehab wrote:
Damayanti Samal wrote:sometimes it found same , just you change the port number to some other port like 7777 and check .


Right try to change the port to anything else rather 8080 and it will works fine


I will be surprised if that will help, but why not to try. I changed the port to 8084. I will report the situation with restarts in the next couple of days.
 
Alexey Andreev
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Nope, all the same on another port (used 8084).

I also checked quartz and it shutdowns properly.

It's all a big mystery for me.

Maybe something connected directly with Linux OS? Sockets left in TIMED_WAIT state or any tricky things?
 
Tim Holloway
Saloon Keeper
Pie
Posts: 18169
53
Android Eclipse IDE Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Alexey Andreev wrote:

Maybe something connected directly with Linux OS? Sockets left in TIMED_WAIT state or any tricky things?


I doubt it. I've worked with Tomcat since about 2.0. When the apps are all shut down, the rest of the server quickly follows. You might try the JMX console and see if there's something still happening. Or put the whole thing under a debugger and see what threads hang around the longest.

An easy way to tell if it's Tomcat or something in an app is to remove all the webapps and see if it still holds the port too long after shutdown.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic