• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Tomcat 6.0.35 hangs during startup

 
Ranch Hand
Posts: 294
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi everyone,

I am trying to upgrade from Tomcat 5.5.34 to 6.0.35 in AIX environment. I am a java developer not a Unix Admin, however, I will find out as much info as I can.

Here is what my catalina.out is showing



I have try this twice already but everytime catalina.out stop logging when it get to log4j:WARN I was not the person downloaded tomcat, but I certainly can ask the admin that I work with. I have not made any change to server.xml or context.xml. I have made changes to tomcat-user.xml by just adding a role. I don't believe that cause problems.
Has anyone encounter similar situation before?

Thanks for any input or thoughts in advance.
 
Saloon Keeper
Posts: 27764
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I see 2 things wrong.

One - and very importantly - the servlet.jar has apparently been included in the deployed WAR. Don't do that. The copy of servlet.jar (the servlet-implementation jar) that you need is part of Tomcat itself.

Two. It's annoyed about the log4j configuration. You should have a log4.xml or log4j.properties file in the WAR's WEB-INF/classes directory. You also need to have at least one appender in it that's set up to be able to handle log requests marked as coming from com.sun.faces.config.ConfigureListener. The fact that you don't means that you probably do have a log4 configuration file, but it doesn't properly set up a default appender.

Just for info, if by any chance you have both a log4j.properties file and a log4j.xml, the xml file takes precedence.
 
Davie Lin
Ranch Hand
Posts: 294
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Tim,

Thanks for your insight, very much appreciated. Before I posted this question yesterday afternoon I had some thoughts about Log4j. keep in mind that I was not the original developer for the web app that I deploy to Tomcat 6. This web app does not have Log4j.xml nor Log4j.properties, I think the Log4j message came from commons-logging.jar. I remove commons-logging.jar out of the project's build path and I believe I saw no error. For the time that I inherited this web app I do not recall there was any sort of logging with Log4j. Do you see any problem if I remove commons-loggins.jar?

Thanks for your reply.
 
Davie Lin
Ranch Hand
Posts: 294
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I just try removing commons-logging.jar out of my WEB-INF/lib directory and I get



Looks like commons-logging.jar stays
 
Tim Holloway
Saloon Keeper
Posts: 27764
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
One problem that you seen in complex webapps is that there can easily be 3 or more different logging systems at work in them. The webapp might use log4j, the Hibernate jars might employ apache-commons-logging, another component might be using JULI. So we ended up with choice of additional logging subsystems whose purpose is to co-ordinate all those loggers!

Leave the commons-logging jar in, but add some sort of minimal log4j config file. That should be sufficient. Apparently the original project was simply missing it and no one cared, since it's a non-fatal error that's only really a problem if you wanted to actually log something!
 
Davie Lin
Ranch Hand
Posts: 294
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Well, I don't know if anyone still reads this thread or not but I notice there is only one of my war not working and started to compare with other web app that works and the web app that doesn't deploy has Richfaces3.1.5 and if I took out those jar files and modify the DD, surprise, surprise, my app is started, of course, the pages that use Richfaces doesn't display anyting and get exception in the log. So what does this mean, I need to upgrade Richfaces?
 
Tim Holloway
Saloon Keeper
Posts: 27764
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Davie Lin wrote:Well, I don't know if anyone still reads this thread or not but I notice there is only one of my war not working and started to compare with other web app that works and the web app that doesn't deploy has Richfaces3.1.5 and if I took out those jar files and modify the DD, surprise, surprise, my app is started, of course, the pages that use Richfaces doesn't display anyting and get exception in the log. So what does this mean, I need to upgrade Richfaces?



Probably not. But if you have both RichFaces and Hibernate in the same app, you have 2 conflicting loggers and you need slf4j to integrate them.
 
Davie Lin
Ranch Hand
Posts: 294
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:Probably not. But if you have both RichFaces and Hibernate in the same app, you have 2 conflicting loggers and you need slf4j to integrate them.



WOW Tim, how do you know I have both RichFaces and Hibernate 3. so slf4j huh..., I will have to look into it tomorrow

Thanks again
 
Tim Holloway
Saloon Keeper
Posts: 27764
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Davie Lin wrote:
WOW Tim, how do you know I have both RichFaces and Hibernate 3.


Because I been there and I dun that and I have the scars to prove it!


 
Davie Lin
Ranch Hand
Posts: 294
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:
Because I been there and I dun that and I have the scars to prove it!



Tim, I was just thinking. Based upon what I have Richfaces 3.1.5 and Hibernate 3 and Tomcat 6.0.35 AIX 6.1 Java 6 64 bit and the problems I am experiencing. Let's say if I need to convince a coworker or manager of mine that this hanging problem is indeed due to a Richface and Hibernate logging contention. My question is, what can I do on my side to convince them it is truely like what you mention on the previous post and thus justifiy in implementing slf4j?

Thanks again.
 
Tim Holloway
Saloon Keeper
Posts: 27764
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I actually don't think that "logging conflicts" are the direct cause of the hangup. However, you're not far off from asking "what can I do on my side to convince them it is truely like what you mention on the previous post and thus justify in implementing hibernate.jar?". You NEED slf4j because without it, your app is broken. At a bare minimum you're probably missing a critical error message that would tell you what the real problem is because your logging infrastructure isn't set up properly. If you're using Maven, getting slf4j and the log4j bridge into your project is fairly easy - just add the dependencies and they'll do the rest themselves.

BTW, I do hope you removed the servlet-api.jar from the Lids webapp.

Tomcat isn't prone to hanging up on its own, so the fault is almost guaranteed to be in the webapp. Logging may or may not reveal the root cause. You need to fix it, but you should also consider taking a diagnostic snapshot of the running system just to see what's going on after Tomcat freezes.
 
Davie Lin
Ranch Hand
Posts: 294
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:BTW, I do hope you removed the servlet-api.jar from the Lids webapp..



Done, servlet-api.jar has been remove out of webapp's lib directory

Tim Holloway wrote:
Tomcat isn't prone to hanging up on its own, so the fault is almost guaranteed to be in the webapp. Logging may or may not reveal the root cause. You need to fix it, but you should also consider taking a diagnostic snapshot of the running system just to see what's going on after Tomcat freezes



Does this mean triggering a java dump? I have generated a java dump by doing kill -3 <pid> but with my limited knowledge, I believe I see no lock contention, here is a snippet



I follow the IBM's tutorial a bit and my thoughts were there is no lock contention since there is only one monitor and only the "GC Daemon" is waiting to be notified. There is more stuff after but I don't understand it and the tutorial didn't elaborate

Thanks again for your help
 
Tim Holloway
Saloon Keeper
Posts: 27764
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Actually, it's more common that something goes CPU-crazy or hangs waiting for I/O or network. So looking at the stacktraces for the threads is where I usually start. And yes, the dump is what I meant. Alternatively, if you can jack into the Tomcat JVM with a remote debugger, that's even better.
 
Davie Lin
Ranch Hand
Posts: 294
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:Actually, it's more common that something goes CPU-crazy or hangs waiting for I/O or network. So looking at the stacktraces for the threads is where I usually start. And yes, the dump is what I meant.



Hi Tim,

sorry to bother you again. I have taken 2 dump files at 2 different time but both are with my problem war file and I started learning how to interpret stack trace from Oracle's site. Here is what I found interesting and I would like to know what you think



After comparing the stack trace out of 2 dump files taken different time with same condition. The condition being only have the problem war file in webapps directory.
I believe this is the current RUNNING thread based upon what I learn from Oracle's site because



State is R, which means Running and while I do have another thread that is also have state R



The priority of "Attach API wait loop" is lower than "Signal Dispatcher" thread. Therefore "Signal Dispatcher" thread is the running thread when I took the dump file both times.

Now looking at the call stack of "Signal Dispatcher" thread, I think it is stuck at



My question here is, would you say this is the IBM Java 6 issue?

Thanks again for all your help
 
Tim Holloway
Saloon Keeper
Posts: 27764
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I usually look for the threads who aren't in the process of calling some sort of "wait" method.

And yes, it can't hurt to try running Tomcat under the Sun/Oracle JDK VM instead of IBM's J9. J9 is supposed to work, but we all have our off-days. Just install the Oracle JDK, set JAVA_HOME to point to it and launch Tomcat.
 
Davie Lin
Ranch Hand
Posts: 294
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:And yes, it can't hurt to try running Tomcat under the Sun/Oracle JDK VM instead of IBM's J9. J9 is supposed to work, but we all have our off-days. Just install the Oracle JDK, set JAVA_HOME to point to it and launch Tomcat.



I am all for trying a different JDK, however, we have other Java process that relies on IBM's J9 so our admin hesitate to change to different Java version. My question here is, is there a way for us to not change the default java version while download a new Oracle's JDK/JRE ( I believe Tomcat 6 will ran under a JRE ) and modify catalina.sh to start tomcat under Oracle's JDK/JRE?

Thanks again Tim
 
Tim Holloway
Saloon Keeper
Posts: 27764
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
One of the reasons I try to avoid vendor-specific features unless there's an overriding need: you can end up in a corner. It's one thing to use J9 for its performance and the neat operational capabilities, but when you start becoming addicted to com.ibm java classes, that's another matter entirely.

It is perfectly allowable to install and concurrently use more than one JVM on a machine at a time. That, in fact, is the reason why Tomcat uses the JAVA_HOME variable; so that you can set its JAVA_HOME to use one brand/version of Java while other apps may be using other brands/versions. You do NOT need to modify Tomcat's standard scripts to do this, nor should you. Although you can simply set JAVA_HOME in a script that invokes the catalina.sh script, you can also create a TOMCAT_HOME/setenv.sh file and assign JAVA_HOME there.

None of that helps if the webapp itself cannot run without J9, but at least it helps. It also allows you to use a different version of J9 if the current one is "broken" for your purposes. Once upon a time when IBM was in its prime, they'd even go so far as to provide patches (PTFs) for products so that you could use those as interim fixes.

I got the impression that Tomcat6 was intended to be able to run using only a JRE, but I don't think it happened. I've been using the JDK, at any rate. The only real difference as far as configuration goes is what you point JAVA_HOME at.
 
Davie Lin
Ranch Hand
Posts: 294
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Tim

We brought this issue to IBM and they want us to try



Do I need to modify catalina.sh to add the above? I am not exactly sure where in catalina.sh to add the above.
 
Tim Holloway
Saloon Keeper
Posts: 27764
196
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Davie Lin wrote:Hi Tim

We brought this issue to IBM and they want us to try



Do I need to modify catalina.sh to add the above? I am not exactly sure where in catalina.sh to add the above.



You should be able to create a TOMCAT_HOME/bin/setenv.sh file and put the following statement in it:



 
reply
    Bookmark Topic Watch Topic
  • New Topic