wood burning stoves 2.0*
The moose likes Tomcat and the fly likes my tomcat almost kill me Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Android Security Essentials Live Lessons this week in the Android forum!
JavaRanch » Java Forums » Products » Tomcat
Bookmark "my tomcat almost kill me " Watch "my tomcat almost kill me " New topic
Author

my tomcat almost kill me

Sun Nansong
Greenhorn

Joined: Mar 11, 2010
Posts: 4
I have an web server , nginx + tomcat + mysql ,this tomcat troubles me a lot , It always dies , and ,even kill the Linux server ,

the OS is Linux , FC2 , 8G ram ,
I have set the" fs.file-max = 65535"

maybe the problem was caused by the c3p0 ?

I dont understand java codes at all , some other guys wrote the the java codes .

I hope you gus can help ! thanks alot !

there are some kinds erros in the catalina.out


1st error :
------------------------------------------------------------------------------------------------------------------------------
location=/extend/wmpassport/jsp/ssologin.jsp?logintype=1
2010-3-12 2:06:12 org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
严重: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8080] ignored exception: java.net.SocketException: Too many open f
iles
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:407)
at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:70)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------

2nd error
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
2010-3-12 2:06:41 org.apache.catalina.loader.WebappClassLoader openJARs
警告: Failed to open JAR
java.util.zip.ZipException: Too many open files
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:203)
at java.util.jar.JarFile.<init>(JarFile.java:132)
at java.util.jar.JarFile.<init>(JarFile.java:97)
at org.apache.catalina.loader.WebappClassLoader.openJARs(WebappClassLoader.java:1544)
at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:1763)
at org.apache.catalina.loader.WebappClassLoader.findClassInternal(WebappClassLoader.java:1570)
at org.apache.catalina.loader.WebappClassLoader.findClass(WebappClassLoader.java:850)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1299)
at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1181)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
at org.apache.struts2.dispatcher.Dispatcher.cleanup(Dispatcher.java:254)
at org.apache.struts2.dispatcher.FilterDispatcher.destroy(FilterDispatcher.java:233)
at org.apache.catalina.core.ApplicationFilterConfig.release(ApplicationFilterConfig.java:258)
at org.apache.catalina.core.StandardContext.filterStop(StandardContext.java:3575)
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4249)
at org.apache.catalina.core.StandardContext.reload(StandardContext.java:2924)
at org.apache.catalina.loader.WebappLoader.backgroundProcess(WebappLoader.java:403)
at org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1276)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1568)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1577)
at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1557)
at java.lang.Thread.run(Thread.java:595)

-------------------------------------------------------------------------------------------------------------------------------------------------
3rd

------------------------------------------------------------------------------------------------------------------------------------------------
2010-03-12 09:33:10 [Timer-4] WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner --> com.mchange.v2.async.ThreadPoolAsynchronou
sRunner$DeadlockDetector@1796149 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
2010-03-12 09:33:12 [http-8080-Processor64] WARN com.mchange.v2.c3p0.impl.DefaultConnectionTester --> SQL State '08007' of Exceptio
n tested by statusOnException() implies that the database is invalid, and the pool should refill itself with fresh Connections.
com.mysql.jdbc.exceptions.MySQLNonTransientConnectionException: Communications link failure during commit(). Transaction resolution
unknown.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:980)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:926)
at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1569)

----------------------------------------------------------------------------------------------------------------------------------------------------


4th
----------------------------------------------------------------------------------------------------------------------------------------------------
at java.lang.Thread.run(Thread.java:595)
2010-03-12 09:33:12 [http-8080-Processor65] WARN com.mchange.v2.c3p0.impl.DefaultConnectionTester --> SQL State '08007' of Exceptio
n tested by statusOnException() implies that the database is invalid, and the pool should refill itself with fresh Connections.
com.mysql.jdbc.exceptions.MySQLNonTransientConnectionException: Communications link failure during commit(). Transaction resolution
unknown.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:980)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:926)
at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1569)
at com.mchange.v2.c3p0.impl.NewProxyConnection.commit(NewProxyConnection.java:803)
at org.springframework.jdbc.datasource.DataSourceTransactionManager.doCommit(DataSourceTransactionManager.java:262)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManag
er.java:732)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java
:701)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSup
port.java:321)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy5.getAllSearchRolesNumByCondition(Unknown Source)

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Kees Jan Koster
JavaMonitor Support
Rancher

Joined: Mar 31, 2009
Posts: 251
    
    5
Dear Sun,

This is not a Tomcat problem. There may be two problems:

1) Your setting for max files is not picked up by the tomcat process. Linux uses a layered system for setting the max file descriptors. Try printing the output of 'ulimit -f' just before the Java command is executed in the startup script. Check that the output is actually more than the default.

Alternately, you can install Java-monitor (http://java-monitor.com/install.html) and see the file descriptor graph. (http://java-monitor.com/forum/showthread.php?t=130)

2) You application is using lots of file descriptors, most likely not closing database connections or temporary files. You can find such issues in the code using findbugs (http://java-monitor.com/forum/showthread.php?t=200) Code changes are needed to fix these problems.

Most likely the problem is (2) and you can make Tomcat run for longer using (1), but ultimately it is a code bug and a new release is needed in which the leak is plugged.

Kees Jan


Java-monitor, JVM monitoring made easy <- right here on Java Ranch
Sun Nansong
Greenhorn

Joined: Mar 11, 2010
Posts: 4
Kees Jan Koster wrote:Dear Sun,

This is not a Tomcat problem. There may be two problems:

1) Your setting for max files is not picked up by the tomcat process. Linux uses a layered system for setting the max file descriptors. Try printing the output of 'ulimit -f' just before the Java command is executed in the startup script. Check that the output is actually more than the default.

Alternately, you can install Java-monitor (http://java-monitor.com/install.html) and see the file descriptor graph. (http://java-monitor.com/forum/showthread.php?t=130)

2) You application is using lots of file descriptors, most likely not closing database connections or temporary files. You can find such issues in the code using findbugs (http://java-monitor.com/forum/showthread.php?t=200) Code changes are needed to fix these problems.

Most likely the problem is (2) and you can make Tomcat run for longer using (1), but ultimately it is a code bug and a new release is needed in which the leak is plugged.

Kees Jan

Thankyou for helping me !

(1).
the value of 'ulimit -f' is 'unlimited' ,I think my max.file descriptors is fine, I setting this as the SUN told (http://docs.sun.com/source/819-0084-10/pt_tuningos.html#wp63719)
(2).
the java code writer has used the "c3p0" pool connection , and I think the parameters in the "config-datasource-mysql.properties" is the killing mechine .

may I ask you more to have a look at this file ? thank you very much !

this is the details .

#########################################
# MySql JDBC connection configuration #
#########################################
mysql.jdbc.username=dbuser
mysql.jdbc.passeord=password
mysql.jdbc.url=jdbc\:mysql\://localhost\:3306/db?useUnicode\=true&characterEncoding\=utf8
mysql.jdbc.driverClass=com.mysql.jdbc.Driver
###################################
# c3p0 basic pool configuration #
###################################
# Determines how many connections at a time c3p0 will try to acquire when the pool
# is exhausted. Default: 3
c3p0.acquireIncrement=3

# Number of Connections a pool will try to acquire upon startup. Should be between
# minPoolSize and maxPoolSize. Default: 3
c3p0.initialPoolSize=4

# Maximum number of Connections a pool will maintain at any given time. Default: 15
c3p0.maxPoolSize=64

# Minimum number of Connections a pool will maintain at any given time. Default: 3
c3p0.minPoolSize=4

c3p0.testConnectionOnCheckin = true

c3p0.automaticTestTable=C3P0TestTable

c3p0.idleConnectionTestPeriod = 1800

c3p0.maxIdleTime = 2500

c3p0.testConnectionOnCheckout = true

c3p0.description=A pooled c3p0 DataSource
###########################################################################
Sun Nansong
Greenhorn

Joined: Mar 11, 2010
Posts: 4
Kees Jan Koster wrote:Dear Sun,

Alternately, you can install Java-monitor (http://java-monitor.com/install.html) and see the file descriptor graph. (http://java-monitor.com/forum/showthread.php?t=130)

2) You application is using lots of file descriptors, most likely not closing database connections or temporary files. You can find such issues in the code using findbugs (http://java-monitor.com/forum/showthread.php?t=200) Code changes are needed to fix these problems.


Kees Jan


well , as the install.html said :
I unzip the downloaded file "java-monitor-probe.zip" and copy "java-monitor.war" into the /usr/tomcat/webapps/www.myweb.com/monitor/

and then just go back to the forum and wait ? but there is nothing after a couple minutes .

is there anything else I should to ? add somthing in the "server.xml" ? or edit the iptables ? or something ?

thank you very much !
Kees Jan Koster
JavaMonitor Support
Rancher

Joined: Mar 31, 2009
Posts: 251
    
    5
Dear Sun,

You should put the WAR in /usr/tomcat/webapps/ directly. Tomcat will expand the WAR and load the web app.

Kees Jan
Kees Jan Koster
JavaMonitor Support
Rancher

Joined: Mar 31, 2009
Posts: 251
    
    5
Also: mysql.jdbc.passeord... Should that not read mysql.jdbc.password... ?

Sun Nansong
Greenhorn

Joined: Mar 11, 2010
Posts: 4
Kees Jan Koster wrote:Dear Sun,

You should put the WAR in /usr/tomcat/webapps/ directly. Tomcat will expand the WAR and load the web app.

Kees Jan


oh yes !! thanks !!!
I saw the messages in the CATALINA.OUT

-------------------------------------------------------------------------------------
.......
2010-3-16 11:42:09 org.apache.catalina.startup.HostConfig deployWAR
信息: Deploying web application archive java-monitor.war
......
_____________________________________________________________
Kees Jan Koster
JavaMonitor Support
Rancher

Joined: Mar 31, 2009
Posts: 251
    
    5
Dear Sun,

Excellent, good to hear it works for you. Now, what does it tell you about the file descriptors in your system? How many fd's do you have configured as max? What does the graph look like? If you post the graph to Java-monitor's forum I will reply there also.

Kees Jan
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: my tomcat almost kill me
 
Similar Threads
Trouble in setting up JFORUM with Eclipse as a Tomcat Project
No operations allowed after statement closed.
Help with using getResourceAsStream()
Too many connections
Error when I deploy my Strut application on Tomcat