Broken Pipeline in Tomcat

[Shubh Bose]

Hello Guys,

We are running around 50 apps on Apache Tomcat/7.0.57, all are client API accessible via SOAP call. The problem we are facing is that we are getting broken pipeline error (ClientAbortException). Now I assume I cannot avoid this issue completely, the problem I am facing is the server is not recovering from this issue. It starts giving the error in a series. I need to restart the service/server to rectify the issue, that is not a very good idea for live servers.

Tomcat is running with these parameters:

/usr/java/jdk1.7.0_75/bin/java -Djava.util.logging.config.file=/usr/share/apachetomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms3072m -Xmx3072m -XX:NewSize=1024m -XX:MaxNewSize=1024m -XX:PermSize=1024m -XX:MaxPermSize=1024m -XX:+DisableExplicitGC -XX:-UseGCOverheadLimit -Djava.endorsed.dirs=/usr/share/apachetomcat/endorsed -classpath /usr/share/apachetomcat/bin/bootstrap.jar:/usr/share/apachetomcat/bin/tomcat-juli.jar -Dcatalina.base=/usr/share/apachetomcat -Dcatalina.home=/usr/share/apachetomcat -Djava.io.tmpdir=/usr/share/apachetomcat/temp org.apache.catalina.startup.Bootstrap start

And the error log is as follows:

[ERROR] java.net.SocketException: Broken pipe
org.apache.axis2.AxisFault: java.net.SocketException: Broken pipe
        at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
        at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:78)
        at org.apache.axis2.transport.http.CommonsHTTPTransportSender.sendUsingOutputStream(CommonsHTTPTransportSender.java:364)
        at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:241)
        at org.apache.axis2.engine.AxisEngine.sendFault(AxisEngine.java:526)
        at org.apache.axis2.transport.http.AxisServlet.handleFault(AxisServlet.java:433)
        at org.apache.axis2.transport.http.AxisServlet.processAxisFault(AxisServlet.java:398)
        at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:188)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:503)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.ctc.wstx.exc.WstxIOException: java.net.SocketException: Broken pipe
        at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1692)
        at com.ctc.wstx.sw.BaseStreamWriter.close(BaseStreamWriter.java:288)
        at org.apache.axiom.util.stax.wrapper.XMLStreamWriterWrapper.close(XMLStreamWriterWrapper.java:46)
        at org.apache.axiom.om.impl.MTOMXMLStreamWriter.close(MTOMXMLStreamWriter.java:222)
        at org.apache.axiom.om.impl.llom.OMSerializableImpl.serializeAndConsume(OMSerializableImpl.java:192)
        at org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:74)
        ... 28 more
Caused by: org.apache.catalina.connector.ClientAbortException: java.net.SocketException: Broken pipe
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:407)
        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366)
        at org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:432)
        at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:420)
        at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:91)
        at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:96)
        at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:214)
        at com.ctc.wstx.sw.BufferingXmlWriter.close(BufferingXmlWriter.java:194)
        at com.ctc.wstx.sw.BaseStreamWriter.finishDocument(BaseStreamWriter.java:1690)
        ... 33 more
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
        at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
        at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366)
        at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:240)
        at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:117)
        at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192)
        at org.apache.coyote.Response.doWrite(Response.java:499)
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:402)
        ... 42 more
[INFO] org.apache.axis2.AxisFault: java.net.SocketException: Broken pipe

Please let me know if anyone needs any more information to help me rectify this.

[Lucas Smith]

My guess is that the HTTP connections are pooled and reused and the software is not able to detect the broken connections.

According to https://axis.apache.org/axis2/java/core/docs/http-transport.html

CommonsHTTPTransportSender utilizes HttpClient. Here you have some configuration description:

http://hc.apache.org/httpclient-3.x/threading.html

connectionStaleCheckingEnabled
maxConnectionsPerHost
maxTotalConnections

given that you use MultiThreadedHttpConnectionManager. Anyhow, make sure that Stale Connection check is activated:

http://hc.apache.org/httpclient-3.x/preference-api.html -> http.connection.stalecheck

You will probably need to configure HttpClient through Axis interface.

[Shubh Bose]

Actually, This exception cause a request long time to process, which will make other subsequent requests to be in queue, then they will take more time to process and execute, Later at one point, all request takes more time to process, which hang up the service and further on there are only "Broken Pipe" error exception in the logs.