I/O exception (java.net.SocketException) caught when processing request

[Chan sekar]

Hi,
My client application is throwing the below exception intermittently, which is causing client application slowness and getting client application freezed sometime.

The exception is
[pool-10-thread-1]: I/O exception (org.apache.commons.httpclient.NoHttpResponseException) caught when processing request: The server 10.8.4.248 failed to respond
2008-06-18 19:04:23,890 INFO [pool-10-thread-1]: Retrying request


INFO [javawsApplicationMain]: I/O exception (java.net.SocketException) caught when processing request: Software caused connection abort: socket write error
2008-06-18 20:29:52,406 INFO [javawsApplicationMain]: Retrying request

Client application is built using Eclipse RCP and Spring. Server side coding is on Spring. Spring remoting is used to communicate between client and server. Tomcat server on Linux is used.

Using JConsole when monitored there were around 150+ threads active and peaked to 230 threads. Is that something I need to increase the threads and timeout parameter in Tomcat server.xml

<Connector port="8080" maxHttpHeaderSize="8192"
maxThreads="150" minSpareThreads="25" maxSpareThreads="75"
enableLookups="false" redirectPort="8443" acceptCount="100"
connectionTimeout="20000" disableUploadTimeout="true" />
<!-- Note : To disable connection timeouts, set connectionTimeout value
to 0 -->


The issue is happening in live environment and is critical. Thanks in advance for your help!.

[Joe Ess]

Welcome to the JavaRanch.
Does the server logs indicate any reason that it would fail to respond to a request?

[Chan sekar]

The server log did not throw any errors or exceptions.

[Joe Ess]

It sure sounds like you are running out of threads. Try increasing maxThreads and see if it makes a difference. If your spikes are temporary, you may also want to increase acceptCount:

The maximum queue length for incoming connection requests when all possible request processing threads are in use. Any requests received when the queue is full will be refused. The default value is 10.

Tomcat Configuration Reference
Have you also been monitoring your server's CPU usage? If it stays at 100% for any amount of time, you may want to look into better hardware.

[Chan sekar]

Thanks for the suggestion. Increased the maxThreads and monitoring the app.

Would like to know how to monitor the Threads. Gone through some article which says there is a way to monitor the threads by turing on Thread dump. In Linux how to do it. By turing this on and keeping the Thread dumping for a day or two will it hit the performance.

Note: CPU usage is 15% only.

[Joe Ess]

Thread dumps are useful for seeing the state of a virtual machine at a particular time (see here). In your case it may be useful if you suspect that you have a bunch of threads waiting on some serialized resource.
As for your CPU load, it would be helpful to know the usage when the connections are failing. Depending on your OS there are tools that will track system activity, like performance monitor in XP.
One thing you should try is replicating the problem in a test environment. Ideally you'd use identical hardware and a tool like The Grinder to simulate a load.