aspose file tools*
The moose likes Tomcat and the fly likes Server startup in 893899 ms - how can this go faster? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Products » Tomcat
Bookmark "Server startup in 893899 ms - how can this go faster?" Watch "Server startup in 893899 ms - how can this go faster?" New topic
Author

Server startup in 893899 ms - how can this go faster?

Erich Specht
Greenhorn

Joined: Mar 05, 2013
Posts: 11
I'm new to Tomcat and I was told by a software vendor that it needs to be installed without symbolic links. I installed it according to the instructions here http://tomcat.apache.org/tomcat-7.0-doc/setup.html and when it was taking over 1 million ms to load the default projects and one additional, I tried installing Tomcat Native Library, but it still takes incredibly long to start up and is real slow for each page load.

I have the following in my $CATALINA_HOME/bin/setenv.sh file:

JAVA_HOME=/pub/cust/jdk1.7.0_17
CATALINA_HOME=/usr/tomcat7
CATALINA_OPTS="-server -Xms3072m -Xmx3072m -XX:PermSize=288m -XX:MaxPermSize=288m -XX:-DisableExplicitGC -XX:CompileThreshold=8000 -Djava.awt.headless=true -Xloggc:$CATALINA_HOME/logs/gc.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:-HeapDumpOnOutOfMemoryError"

the last few items in CATALINA_OPTS regarding garbage collection were from something I was reading to diagnose the problem.

Is there something I can do to speed things up?
Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 16058
    
  21

Welcome to the JavaRanch, Erich!

Tomcat should not take anywhere near that long to do anything.

Normally I would say that you should tune your applications, since it's applications that normally make "Tomcat" run slow.

However, considering the amount of time you are reporting, I think it would be a good idea to make sure that the server that you are communicating with - and its network connections - are not defective.


Customer surveys are for companies who didn't pay proper attention to begin with.
Rakesh Gupta
Greenhorn

Joined: Mar 06, 2013
Posts: 3
Provide web.xml over here.
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12781
    
    5
Perhaps there is a application initialization that is searching/waiting for a system resource and it is not really Tomcat that is slow.

Is there some sort of database initialization involved?

Bill
Erich Specht
Greenhorn

Joined: Mar 05, 2013
Posts: 11
Raks Kr wrote:Provide web.xml over here.

The web.xml and server.xml are unchanged. And this site won't allow me to attach xml files to the post.
Rakesh Gupta
Greenhorn

Joined: Mar 06, 2013
Posts: 3
ok can you copy and paste first 15-20 line of web.xml over here.
Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 16058
    
  21

Or.... You could click on the Attachments tab down below the message editing area and upload the file!

As far as I know, you don't have to have any special privileges to do that.
Erich Specht
Greenhorn

Joined: Mar 05, 2013
Posts: 11
Raks Kr wrote:ok can you copy and paste first 15-20 line of web.xml over here.


Here are the portions that aren't commented out. Thanks for getting back to me.

Erich

<?xml version="1.0" encoding="ISO-8859-1"?>
<web-app version="3.0" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee">
<servlet> <servlet-name>default</servlet-name> <servlet-class>org.apache.catalina.servlets.DefaultServlet</servlet-class> -<init-param> <param-name>debug</param-name> <param-value>0</param-value> </init-param> -<init-param> <param-name>listings</param-name> <param-value>false</param-value> </init-param> <load-on-startup>1</load-on-startup> </servlet>
<servlet> <servlet-name>jsp</servlet-name> <servlet-class>org.apache.jasper.servlet.JspServlet</servlet-class> -<init-param> <param-name>fork</param-name> <param-value>false</param-value> </init-param> -<init-param> <param-name>xpoweredBy</param-name> <param-value>false</param-value> </init-param> <load-on-startup>3</load-on-startup> </servlet>
<servlet-mapping> <servlet-name>default</servlet-name> <url-pattern>/</url-pattern> </servlet-mapping>
<servlet-mapping> <servlet-name>jsp</servlet-name> <url-pattern>*.jsp</url-pattern> <url-pattern>*.jspx</url-pattern> </servlet-mapping>
<session-config> <session-timeout>30</session-timeout> </session-config>
... LOTS OF MIME MAPPINGS ...
<welcome-file-list> <welcome-file>index.html</welcome-file> <welcome-file>index.htm</welcome-file> <welcome-file>index.jsp</welcome-file> </welcome-file-list>
Jaikiran Pai
Marshal

Joined: Jul 20, 2005
Posts: 10101
    
164

Erich Specht wrote:it was taking over 1 million ms to load the default projects and one additional

What is that 1 additional application? Does the startup time improve if that application isn't included in the deployments?


[My Blog] [JavaRanch Journal]
Erich Specht
Greenhorn

Joined: Mar 05, 2013
Posts: 11
Jaikiran Pai wrote:
Erich Specht wrote:it was taking over 1 million ms to load the default projects and one additional

What is that 1 additional application? Does the startup time improve if that application isn't included in the deployments?


That application is a reporting application we created using LogiXML. I removed that application from the webapps folder and it took the startup time down to 137099 ms, which is a whole lot quicker. The other apps it deployed are host-manager, ROOT, manager, and examples.

I also noticed that it takes a couple minutes during startup before anything shows in the logs. Then this message appears:
INFO: An older version 1.1.24 of the APR based Apache Tomcat Native library is installed, while Tomcat recommends a minimum version of 1.1.27

I installed that using RPM, but I'm going to remove that and reinstall native manually if I can.
Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 16058
    
  21

It should not take Tomcat over 2 minutes to start up when no user apps are installed. In fact, if it takes over 20 seconds to fire up, there's cause for concern unless it's a really, really old machine.

It sounds like you don't have access to the host OS or I'd ask you how much physical RAM the server has and what the CPU load is.
Erich Specht
Greenhorn

Joined: Mar 05, 2013
Posts: 11
Tim Holloway wrote:It sounds like you don't have access to the host OS or I'd ask you how much physical RAM the server has and what the CPU load is.


I do. Here's the output of free -m:
total used free shared buffers cached
Mem: 4096 579 3516 0 0 212
-/+ buffers/cache: 367 3728
Swap: 4096 0 4096

I appreciate you looking this over.

==> EDIT

That was before starting tomcat again. After starting it (in 134800 ms), free -m shows:

total used free shared buffers cached
Mem: 4096 636 3459 0 0 212
-/+ buffers/cache: 424 3671
Swap: 4096 0 4096


What I find interesting is that only 636mb is used although I'm specifying the following in the setenv.sh script:

-Xms3072m -Xmx3072m -XX:PermSize=288m -XX:MaxPermSize=288m

Shouldn't that be taking up most of my RAM right away?

Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 16058
    
  21

If I read that properly, you're pressing the limits for recommended Virtual Memory configuration. Try shrinking the Xms value down to about 512M maybe. Also, you might want to kick the Xmx down to 1024M temporarily.

Back in my mainframe days, an 8-to-1 ration of virtual to real memory was the rule of thumb, but mainframes couldn't afford the RAM and could afford the delays, since most processing was batch. More recently, Linux has recommended a swapfile no larger than physical RAM. On webservers and other highly-interactive systems with large transaction loads, ideally, you wouldn't want swapping at all.
Erich Specht
Greenhorn

Joined: Mar 05, 2013
Posts: 11
Tim Holloway wrote:Try shrinking the Xms value down to about 512M maybe. Also, you might want to kick the Xmx down to 1024M temporarily.


I tried your settings and after tomcat started up in 136900 ms, free -m shows:

total used free shared buffers cached
Mem: 4096 643 3452 0 0 212
-/+ buffers/cache: 431 3664
Swap: 4096 0 4096

I'm thinking of allocating additional RAM, however the memory settings in CATALINA_OPTS don't seem to affect boot up time too much.
Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 16058
    
  21

I think something ate all the RAM before Tomcat got there.
Erich Specht
Greenhorn

Joined: Mar 05, 2013
Posts: 11
Tim Holloway wrote:I think something ate all the RAM before Tomcat got there.


I've doubled the amount of ram to 8GB, but it is still taking 123500 ms to load without my main application sitting in the webapps folder. When I do a free -m, I get the following:

total used free shared buffers cached
Mem: 8192 710 7481 0 0 95
-/+ buffers/cache: 614 7577
Swap: 8192 0 8192

So I'm still only using 710MB of RAM (this is with the following suggested settings: -Xms512m -Xmx1024m.

Upping it to -Xms1024m -Xmx1024m, it takes 138500 ms to load and free -m shows:

total used free shared buffers cached
Mem: 8192 713 7478 0 0 95
-/+ buffers/cache: 617 7574
Swap: 8192 0 8192


So it appears that the -Xms and -Xmx flags aren't doing anything.

The way I'm trying to pass that is with this info in my setenv.sh file:

JAVA_HOME=/pub/cust/jdk1.7.0_17
CATALINA_HOME=/usr/tomcat7
CATALINA_OPTS="-server -d64 -Xms1024m -Xmx1024m -XX:PermSize=288m -XX:MaxPermSize=288m -XX:-DisableExplicitGC -XX:CompileThreshold=8000 -Djava.awt.headless=true -Xloggc:$CATALINA_HOME/logs/gc.log -XX:+PrintHeapAtGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:-HeapDumpOnOutOfMemoryError"


Do you know what I am doing wrong?

Also, does anyone know of any diagnostic commands I can run to get a better idea why this is running so slow?
Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 16058
    
  21

The Xms and Xmx values are the starting and maximum RAM allocation values for the JVM that runs Tomcat and its webapps. The actual OS RAM will be slightly larger, since Tomcat also consumes application memory above and beyond that amount for things like PermGen space. However, Xmx is one of the main things that sets the upper limit on how much total OS RAM the JVM can ever consume.

What bothers me is that you're using 8G of swap space. Apparently Tomcat isn't using that memory, so I wonder what is. With that much swap space in use, chances are that the entire OS is going through Virtual Memory thrashing and Tomcat's just one victim among many.

If you can get the swap usage down, that will probably relieve whatever external factors are beating Tomcat up.

You can use the vmstat, iostat and top commands to help with that. A "top" screen display may provide us with some insight. You can tell top to sort the display by memory usage by using the "m" key.
Erich Specht
Greenhorn

Joined: Mar 05, 2013
Posts: 11
Tim Holloway wrote:What bothers me is that you're using 8G of swap space.


free -m is showing that none of the swap is used - 8G is free
Erich Specht
Greenhorn

Joined: Mar 05, 2013
Posts: 11
Tim Holloway wrote:A "top" screen display may provide us with some insight.

Here's the output from top -m:
Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 16058
    
  21

I added Code tags to make it easier to read that.

OK. I misread the swap usage. It's 8G available, but 0 in use, so swap is OK.

However, you have a CPU reading of 100% and whatever it is, it isn't Tomcat/Java, which is a barely-visible 1.6%.

Can you do the vanilla "top", which shows high-CPU users are the head of the listing?
Erich Specht
Greenhorn

Joined: Mar 05, 2013
Posts: 11
Tim Holloway wrote:However, you have a CPU reading of 100% and whatever it is, it isn't Tomcat/Java, which is a barely-visible 1.6%. Can you do the vanilla "top", which shows high-CPU users are the head of the listing?


Thank you so much. Feels like we're getting somewhere. Here's the output of top:
Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 16058
    
  21

This is not good. You only have maybe 2-3% CPU usage accounted for out of 100% user-state CPU consumed.

It's possible that OS I/O services are the culprit, although I can never remember which parts of the OS show up as "user" work, considering all the OS's I've worked with. But for I/O to be pulling that much CPU, I'd be looking for the possibility that some hardware has gone haywire and is firing hot I/O events.

The other possibility is even uglier. If you have been pwned, a malware process could be eating the OS alive and hiding itself from display.
Erich Specht
Greenhorn

Joined: Mar 05, 2013
Posts: 11
Tim Holloway wrote:This is not good. You only have maybe 2-3% CPU usage accounted for out of 100% user-state CPU consumed.


I found the problem - the LogiXML Scheduler service. Killing that java process brought the CPU usage down to 0. Tomcat (without my reporting app back in webapps) started in 662 ms!

Thank you so much for your help. I'll take this up with the software vendor.
 
 
subject: Server startup in 893899 ms - how can this go faster?