I have
tomcat app-server running (on Linux) from past 3+ months handling moderate load. I recently an seeing HTTP 500/503 errors. The catalina.out has log entry for "java.net.SocketException: Too many open files". The last change I made was increase the number of threads Tomcat could use and increased the JVM memory.
I checked the number of file descriptors open and they seem to have not exceeded the max limit.
$cat /proc/sys/fs/file-nr
1446 38 524288
Any ideas, what could be wrong or how to debug further will be greatly appreciated.
Here is the excerpt from catalina.out:
Apr 5, 2005 6:15:48 PM org.apache.tomcat.util.net.PoolTcpEndpoint acceptSocket
SEVERE: Endpoint ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=8080] ignored exception: java.net.SocketException: Too many open files
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
at java.net.ServerSocket.implAccept(ServerSocket.java:450)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:368)
at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:549)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:683)
at java.lang.Thread.run(Thread.java:595)