aspose file tools*
The moose likes JBoss/WildFly and the fly likes Application deployed on JBoss 6.1 crashes Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Products » JBoss/WildFly
Bookmark "Application deployed on JBoss 6.1 crashes" Watch "Application deployed on JBoss 6.1 crashes" New topic
Author

Application deployed on JBoss 6.1 crashes

Varun Agarwal
Greenhorn

Joined: Dec 01, 2011
Posts: 11
Hi All,

I have currently hosted my java application on jboss-6.1.0.Final along with jdk1.7.0_01 set up on a linux server with 32GB of RAM.

Almost daily(once or more) the JBoss server crashes with the following error message in the /var/log/messages file.

Earlier this was working fine with jboss-4.2.2.GA and jdk1.5.0_15, before I had upgraded to the above configuration.

Nov 28 13:37:21 <server host name> kernel: java[14296]: segfault at 0000000047f81ff8 rip 00002aaaab20a364 rsp 0000000047f82020 error 6
Nov 30 11:45:25 <server host name> kernel: java[1762]: segfault at 0000000048d8fff0 rip 00002aaaaafd40c4 rsp 0000000048d90000 error 6
Nov 30 12:30:40 <server host name> kernel: java[12904]: segfault at 000000004b8baff8 rip 0000003699a082d9 rsp 000000004b8bb000 error 6
Nov 30 15:39:16 <server host name> kernel: java[15304]: segfault at 0000000045557fe8 rip 00002aaaab20a385 rsp 0000000045557fe0 error 6

I have given around 10GB memory to the JVM in JBoss and it does not use more than 4-5GB during high load. There are no logs in the server.log file when the server crashes. The only solution is to start the JBoss server after which it works fine.

I have investigated the issue and have not been able to resolve it till now so please guide me to the solution for this above problem.

Thank you,

Regards,
Varun
Jaikiran Pai
Marshal

Joined: Jul 20, 2005
Posts: 10289
    
168

That's a JDK crash. You seem to be using JDK 1.7. Try downgrading to JDK 1.6 and see if that works fine. By the way, what kind of application is deployed on that server? Does it include any JNI code?


[My Blog] [JavaRanch Journal]
Varun Agarwal
Greenhorn

Joined: Dec 01, 2011
Posts: 11
I had first tried to deploy the same application using JDK 1.6 and JBoss 5.1 but faced the same problem.

My application is using JSPs, Struts, Spring MVC, EJB 3 and Oracle as the technologies. No JNI code present in the application.

Could the JVM options in run.sh be the cause? The additional JVM options added by me are:-

JAVA_OPTS="-Dfile.path=/home/project -Xms6g -Xmx12g -XX:PermSize=256m -XX:MaxPermSize=1g -server -Duser.timezone=America/New_York -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false -XX:-DisableExplicitGC -XX:+UseStringCache -XX:+AggressiveOpts -XX:+OptimizeStringConcat -XX:+UseCompressedStrings -Djboss.service.binding.set=ports-01 -Dorg.jboss.resolver.warning=true -Djava.util.Arrays.useLegacyMergeSort=true $JAVA
Peter Johnson
author
Bartender

Joined: May 14, 2008
Posts: 5845
    
    7

Try removing the aggresiveOpts, that option is there mainly to run a benchmark and can often cause instabilities in normal programs.


JBoss In Action
Varun Agarwal
Greenhorn

Joined: Dec 01, 2011
Posts: 11
I had actually removed the following options and it has been over 24 hours since the server crashed but due to the weekend I need to wait till at least Tue/Wed to confirm that one of these options was causing the crash.

-server -XX:-DisableExplicitGC -XX:+UseStringCache -XX:+AggressiveOpts -XX:+OptimizeStringConcat -XX:+UseCompressedStrings -Dorg.jboss.resolver.warning=true

Modified -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 to 1800000
Peter Johnson
author
Bartender

Joined: May 14, 2008
Posts: 5845
    
    7

Actually, you should set the gcintervals to really high values, not lower values. Even the original vakues are not high enough. (Ask me how I know...) But if you disable explicitgc, then you don't have to set the gcintervals at all.
Varun Agarwal
Greenhorn

Joined: Dec 01, 2011
Posts: 11
Following are my current JVM OPTIONS
-----------------------------------------

JAVA_OPTS="-Dsems.file.path=/home/project -Xms4g -Xmx12g -XX:PermSize=256m -XX:MaxPermSize=1g -Duser.timezone=America/New_York -Dsun.rmi.dgc.client.gcInterval=1800000 -Dsun.rmi.dgc.server.gcInterval=1800000 -Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false -Djboss.service.binding.set=ports-01 -Djava.util.Arrays.useLegacyMergeSort=true -XX:ErrorFile=./hs_err_pid_.log $JAVA_OPTS"

Even with the above jvm options the JBoss crashed again, below mentioned is the log for it

/var/log/messagesec 5 14:22:29 <server host name> kernel: java[20833]: segfault at 0000000048789ff8 rip 00002aaaab20a364 rsp 000000004878a020 error 6
/var/log/messagesec 5 14:34:05 <server host name> kernel: java[7358]: segfault at 0000000049395ff8 rip 0000003699a082d2 rsp 0000000049396000 error 6
Peter Johnson
author
Bartender

Joined: May 14, 2008
Posts: 5845
    
    7

Hmm, so you have a set of JAVA_OPTS that works,m and a set that doesn't. I think you need to find out which option is causing the crash and then submit a bug report to Oracle to fix the problem. And until they fix it, avoid that option! After all, a segfault in the JVM is beyond the control of JBoss AS.
Varun Agarwal
Greenhorn

Joined: Dec 01, 2011
Posts: 11
No Peter, none of the java options work, even the new java options that were set faced the segfault error.
I think I have no other option but to downgrade my JVM to 5.

I also tried to record various metrics with respect to the memory and thread usage details using MBeans but none of those metrics indicate any problem.
Jaikiran Pai
Marshal

Joined: Jul 20, 2005
Posts: 10289
    
168

Which exact version of Java 1.6 is this? It may not be related to any JAVA_OPTS but might be some bug in the JVM. If there's any upgrades available on the Oracle download sites, try using that.
Varun Agarwal
Greenhorn

Joined: Dec 01, 2011
Posts: 11
It is not JDK 1.6 but is JDK 7u1.

Initially I had tried JBoss 5 with JDK 1.6 update 26 but that too gave me the same problem.

My next step is to try JBoss 6.1 with JDK 1.6 update 29
Jaikiran Pai
Marshal

Joined: Jul 20, 2005
Posts: 10289
    
168

The other thing that you should do is try and isolate this crash to some specific usecase in your application. See if you have some pattern when this crashes. Also check whether the crash generates and dump file in JBOSS_HOME/bin folder.
Varun Agarwal
Greenhorn

Joined: Dec 01, 2011
Posts: 11
I have tried to gather a lot of statistics but am unable to replicate this problem.
Varun Agarwal
Greenhorn

Joined: Dec 01, 2011
Posts: 11
I am using the bouncycastle jars in my application due to which I had to do some workarounds in my JDK configuration, please suggest if this could be a reason for the crash

Workaround done were taken from http://www.randombugs.com/java/javalangsecurityexception-jce-authenticate-provider-bc.html
Varun Agarwal
Greenhorn

Joined: Dec 01, 2011
Posts: 11
I downgraded my JDK to 1.6 update 11 and it is working fine with the application with NO crashes.
Jaikiran Pai
Marshal

Joined: Jul 20, 2005
Posts: 10289
    
168

Varun Agarwal wrote:I downgraded my JDK to 1.6 update 11


You might want to upgrade to the latest update (30 right now) because I remember there's atleast one very serious bug which has been fixed in the recent update (26 from what I remember).
Varun Agarwal
Greenhorn

Joined: Dec 01, 2011
Posts: 11
I had tried with JDK 1.6 update 29 but that too did not work. I have not tried JDK 6 update 30 and cannot do it now as cannot test it on the live server as we are unable to replicate the crash otherwise.
A. Schmidt
Greenhorn

Joined: Jan 26, 2012
Posts: 1
Hi,

we have got the same problem.
Our Server: RHEL Linux 5.7, Java 1.6.0-30, Tomcat 6.0.35.

Java/Tomcat crashes occasionally with segfault error 6.

Any news here?
Varun Agarwal
Greenhorn

Joined: Dec 01, 2011
Posts: 11
you can try downgrading your JDK to JDK 6 update 11 and see if that work similar to what worked for me.
Peter Johnson
author
Bartender

Joined: May 14, 2008
Posts: 5845
    
    7

JDK 1.6.0_11 is a little old, so I would not go with that. We have noticed issues with 1.6.0_29 and 1.6.0_30, so several of our projects revereted to 1.6.0_26 which seems to be working without issue.
 
It is sorta covered in the JavaRanch Style Guide.
 
subject: Application deployed on JBoss 6.1 crashes