This week's book giveaway is in the Servlets forum.
We're giving away four copies of Murach's Java Servlets and JSP and have Joel Murach on-line!
See this thread for details.
The moose likes BEA/Weblogic and the fly likes Another Stuck Thread Question - SocketNativeIO.readBytesPinned - PreparedStatement Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Murach's Java Servlets and JSP this week in the Servlets forum!
JavaRanch » Java Forums » Products » BEA/Weblogic
Bookmark "Another Stuck Thread Question - SocketNativeIO.readBytesPinned - PreparedStatement" Watch "Another Stuck Thread Question - SocketNativeIO.readBytesPinned - PreparedStatement" New topic
Author

Another Stuck Thread Question - SocketNativeIO.readBytesPinned - PreparedStatement

M Paul Baker
Greenhorn

Joined: Mar 06, 2013
Posts: 3
We're getting several STUCK threads like the one below. These never clear up, the managed servers must be bounced to remove them. The STUCK threads in question are processing a prepared statement and hang indefinately on the socket read (readBytesPinned). We are tracking down the exact query involved. Enabling traces is a "process" in the environment where this happens.

How can I verify that the DB is actively processing this request?
Would the Oracle DB Alert Logs show that there is a/are long running requests?
Getting a DB Alert Log is a challenge, it has to be requested; should I ask for any particular options that would help identify long running connections or connections that have timed out/dropped?

Would a thread remain in the STUCK state even if the connection dropped? I would like to know what the thread is doing at method readBytesPinned. I'm Stuck

Has anyone tracked down these kind of thread issues? If so, will you please share any lessons learned?

This thread stack taken from a thread dump. We have taken several dumps and the thread never appears to have any progress.

Advisories: Stuck Thread DB Execute Socket Read Web Application Request
"[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" id=15 idx=0x68 tid=13619 prio=1 alive, in native, daemon
at jrockit/net/SocketNativeIO.readBytesPinned(Ljava/io/FileDescriptor;[BIII)I(Native Method)
at jrockit/net/SocketNativeIO.socketRead(SocketNativeIO.java:32)[inlined]
at java/net/SocketInputStream.socketRead0(Ljava/io/FileDescriptor;[BIII)I(SocketInputStream.java)[inlined]
at java/net/SocketInputStream.read(SocketInputStream.java:129)[optimized]
at oracle/net/nt/MetricsEnabledInputStream.read(TcpNTAdapter.java:564)[optimized]
at oracle/net/ns/Packet.receive(Packet.java:295)[inlined]
at oracle/net/ns/DataPacket.receive(DataPacket.java:94)[optimized]
at oracle/net/ns/NetInputStream.getNextPacket(NetInputStream.java:176)[optimized]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:121)[inlined]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:96)[inlined]
at oracle/net/ns/NetInputStream.read(NetInputStream.java:81)[optimized]
at oracle/jdbc/driver/T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)[optimized]
at oracle/jdbc/driver/T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)[inlined]
at oracle/jdbc/driver/T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1087)[optimized]
at oracle/jdbc/driver/T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)[inlined]
at oracle/jdbc/driver/T4C8Oall.receive(T4C8Oall.java:625)[optimized]
at oracle/jdbc/driver/T4CPreparedStatement.doOall8(T4CPreparedStatement.java:205)[inlined]
at oracle/jdbc/driver/T4CPreparedStatement.fetch(T4CPreparedStatement.java:1121)[optimized]
at oracle/jdbc/driver/OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:571)[inlined]
at oracle/jdbc/driver/OracleResultSetImpl.next(OracleResultSetImpl.java:445)[optimized]
^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x14e5531a0[recursive]
^-- Holding lock: oracle/jdbc/driver/T4CConnection@0x14e5531a0[thin lock]
at weblogic/jdbc/wrapper/ResultSet_oracle_jdbc_driver_OracleResultSetImpl.next()Z(Unknown Source)[optimized]
at org/hibernate/loader/Loader.doQuery(Loader.java:697)[inlined]
at org/hibernate/loader/Loader.doQueryAndInitializeNonLazyCollections(Loader.java:236)[inlined]
at org/hibernate/loader/Loader.doList(Loader.java:2213)[inlined]
at org/hibernate/loader/Loader.listIgnoreQueryCache(Loader.java:2104)[optimized]
at org/hibernate/loader/Loader.list(Loader.java:2099)
at org/hibernate/loader/hql/QueryLoader.list(QueryLoader.java:378)
at org/hibernate/hql/ast/QueryTranslatorImpl.list(QueryTranslatorImpl.java:338)
at org/hibernate/engine/query/HQLQueryPlan.performList(HQLQueryPlan.java:172)
at org/hibernate/impl/SessionImpl.list(SessionImpl.java:1121)
at org/hibernate/impl/QueryImpl.list(QueryImpl.java:79)
at org/hibernate/ejb/QueryImpl.getResultList(QueryImpl.java:65)
at org/springframework/orm/jpa/JpaTemplate$11.doInJpa(JpaTemplate.java:351)
at org/springframework/orm/jpa/JpaTemplate.execute(JpaTemplate.java:184)
at org/springframework/orm/jpa/JpaTemplate.executeFind(JpaTemplate.java:151)
at org/springframework/orm/jpa/JpaTemplate.findByNamedQuery(JpaTemplate.java:343)
at aiat/csp/esb2/persistence/PersistenceManagerImpl.findByNamedQuery(PersistenceManagerImpl.java:224)
at com/aiat/odlp/commonutilities/referencedata/dao/ReferenceDataManagerDAO.getAirlineList(ReferenceDataManagerDAO.java:725)[optimized]
at com/aiat/odlp/commonutilities/referencedata/impl/ReferenceDataManagerImpl.listAirlineCodes(ReferenceDataManagerImpl.java:177)[optimized]
at sun/reflect/GeneratedMethodAccessor7000.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
at org/springframework/aop/support/AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at org/springframework/aop/framework/ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at org/springframework/transaction/interceptor/TransactionInterceptor.invoke(TransactionInterceptor.java:106)
at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org/springframework/aop/interceptor/ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at org/springframework/aop/framework/JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy247.listAirlineCodes(Lcom/aiat/odlp/commonutilities/referencedata/interfaces/common/ListAirlinesQueryDTO;)Lcom/aiat/odlp/commonutilities/referencedata/interfaces/common/AirlineListDTO;(Unknown Source)
at com/aiat/odlp/commonutilities/referencedata/endpoint/ReferenceDataRepositoryListEndPoint.listAirlines(ReferenceDataRepositoryListEndPoint.java:78)
at sun/reflect/GeneratedMethodAccessor6999.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
at org/springframework/ws/server/endpoint/MethodEndpoint.invoke(MethodEndpoint.java:115)
at org/springframework/ws/server/endpoint/adapter/MarshallingMethodEndpointAdapter.invokeInternal(MarshallingMethodEndpointAdapter.java:135)
at org/springframework/ws/server/endpoint/adapter/AbstractMethodEndpointAdapter.invoke(AbstractMethodEndpointAdapter.java:58)
at org/springframework/ws/server/MessageDispatcher.dispatch(MessageDispatcher.java:236)
at org/springframework/ws/server/MessageDispatcher.receive(MessageDispatcher.java:177)
at com/aiat/odlp/server/endpoint/jaxws/JaxWsRequestHandler.handleRequest(JaxWsRequestHandler.java:96)
at com/aiat/odlp/server/endpoint/jaxws/VoyagerJaxWsEndpoint.invoke(VoyagerJaxWsEndpoint.java:75)
at com/aiat/odlp/server/endpoint/jaxws/VoyagerJaxWsEndpoint.invoke(VoyagerJaxWsEndpoint.java:38)
at sun/reflect/GeneratedMethodAccessor958.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:597)[optimized]
at weblogic/wsee/jaxws/WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:92)
at weblogic/wsee/jaxws/WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:74)
at com/sun/xml/ws/api/server/Invoker.invokeProvider(Invoker.java:108)
at com/sun/xml/ws/server/InvokerTube$2.invokeProvider(InvokerTube.java:161)
at com/sun/xml/ws/server/provider/SyncProviderInvokerTube.processRequest(SyncProviderInvokerTube.java:78)
at com/sun/xml/ws/api/pipe/Fiber.__doRun(Fiber.java:604)[optimized]
at com/sun/xml/ws/api/pipe/Fiber._doRun(Fiber.java:563)
at com/sun/xml/ws/api/pipe/Fiber.doRun(Fiber.java:548)
at com/sun/xml/ws/api/pipe/Fiber.runSync(Fiber.java:445)
^-- Holding lock: com/sun/xml/ws/api/pipe/Fiber@0x14e5349d0[thin lock]
at com/sun/xml/ws/server/WSEndpointImpl$2.process(WSEndpointImpl.java:373)
at com/sun/xml/ws/transport/http/HttpAdapter$HttpToolkit.handle(HttpAdapter.java:524)
at com/sun/xml/ws/transport/http/HttpAdapter.handle(HttpAdapter.java:255)
at com/sun/xml/ws/transport/http/servlet/ServletAdapter.handle(ServletAdapter.java:140)
at weblogic/wsee/jaxws/WLSServletAdapter.handle(WLSServletAdapter.java:208)
at weblogic/wsee/jaxws/HttpServletAdapter$AuthorizedInvoke.run(HttpServletAdapter.java:310)
at weblogic/wsee/jaxws/HttpServletAdapter.post(HttpServletAdapter.java:223)
at weblogic/wsee/jaxws/JAXWSServlet.doPost(JAXWSServlet.java:124)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:727)
at weblogic/wsee/jaxws/JAXWSServlet.service(JAXWSServlet.java:79)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:820)
at weblogic/servlet/internal/StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
at weblogic/servlet/internal/StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
at weblogic/servlet/internal/ServletStubImpl.execute(ServletStubImpl.java:300)
at weblogic/servlet/internal/TailFilter.doFilter(TailFilter.java:26)
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at oracle/dms/wls/DMSServletFilter.doFilter(DMSServletFilter.java:330)[optimized]
at weblogic/servlet/internal/FilterChainImpl.doFilter(FilterChainImpl.java:56)[optimized]
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.doIt(Lweblogic/servlet/internal/ServletStubImpl;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)Ljava/lang/Object;(Unknown Source)
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run()Ljava/lang/Object;(Unknown Source)
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[inlined]
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)[optimized]
at weblogic/servlet/internal/WebAppServletContext.securedExecute(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Z)V(Unknown Source)
at weblogic/servlet/internal/WebAppServletContext.execute(Lweblogic/servlet/internal/ServletRequestImpl;Lweblogic/servlet/internal/ServletResponseImpl;)V(Unknown Source)
at weblogic/servlet/internal/ServletRequestImpl.run()V(Unknown Source)
at weblogic/work/ExecuteThread.execute(ExecuteThread.java:201)[optimized]
at weblogic/work/ExecuteThread.run(ExecuteThread.java:173)
at jrockit/vm/RNI.c2java(JJJJJ)V(Native Method)


The managed server logs are showing "similar" issues but to make things interesting the line numbers shown in the stack traces are different. I am not sure I am comparing apples to apples. Here is an example of a similar thread excpetion from the Weblogic logs.

<Mar 21, 2014 12:59:35 PM GMT+00:00> <Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "605" seconds working on the request "weblogic.servlet.internal.ServletRequestImpl@2b80c9fa[
POST /rd-1.1.2/ReferenceData-1.1?wsdl HTTP/1.1
Content-Type: text/xml; charset=utf-8
SOAPAction: "http://aiat/odlp/services/commonutilities/ReferenceDataRepository/v.0.0.0-28/action/ListAirlines/"
Transfer-Encoding: chunked
User-Agent: Java1.6.0_33
Accept: text/html, image/gif, image/jpeg, */*; q=.2
Connection: Keep-Alive

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
Thread-15 "[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" <alive, in native, suspended, priority=1, DAEMON> {
jrockit.net.SocketNativeIO.readBytesPinned(SocketNativeIO.java:???)
jrockit.net.SocketNativeIO.socketRead(SocketNativeIO.java:24)
java.net.SocketInputStream.socketRead0(SocketInputStream.java:???)
java.net.SocketInputStream.read(SocketInputStream.java:107)
oracle.net.nt.MetricsEnabledInputStream.read(TcpNTAdapter.java:553)
oracle.net.ns.Packet.receive(Packet.java:249)
oracle.net.ns.DataPacket.receive(DataPacket.java:94)
oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:168)
oracle.net.ns.NetInputStream.read(NetInputStream.java:113)
oracle.net.ns.NetInputStream.read(NetInputStream.java:96)
oracle.net.ns.NetInputStream.read(NetInputStream.java:81)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:120)
oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:75)
oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1081)
oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1062)
oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:588)
oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:70)
oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1091)
oracle.jdbc.driver.OracleResultSetImpl.close_or_fetch_from_next(OracleResultSetImpl.java:480)
oracle.jdbc.driver.OracleResultSetImpl.next(OracleResultSetImpl.java:347)
^-- Holding lock: oracle.jdbc.driver.T4CConnection@2d660b1e[thin lock]
^-- Holding lock: oracle.jdbc.driver.T4CConnection@2d660b1e[thin lock]
weblogic.jdbc.wrapper.ResultSet_oracle_jdbc_driver_OracleResultSetImpl.next(Unknown Source)
org.hibernate.loader.Loader.doQuery(Loader.java:665)
org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:232)
org.hibernate.loader.Loader.doList(Loader.java:2207)
org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2104)
org.hibernate.loader.Loader.list(Loader.java:2092)
org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:377)
org.hibernate.hql.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:321)
org.hibernate.engine.query.HQLQueryPlan.performList(HQLQueryPlan.java:149)
org.hibernate.impl.SessionImpl.list(SessionImpl.java:1110)
org.hibernate.impl.QueryImpl.list(QueryImpl.java:75)
org.hibernate.ejb.QueryImpl.getResultList(QueryImpl.java:65)
org.springframework.orm.jpa.JpaTemplate$11.doInJpa(JpaTemplate.java:345)
org.springframework.orm.jpa.JpaTemplate.execute(JpaTemplate.java:169)
org.springframework.orm.jpa.JpaTemplate.executeFind(JpaTemplate.java:151)
org.springframework.orm.jpa.JpaTemplate.findByNamedQuery(JpaTemplate.java:343)
aiat.csp.esb2.persistence.PersistenceManagerImpl.findByNamedQuery(PersistenceManagerImpl.java:224)
com.aiat.odlp.commonutilities.referencedata.dao.ReferenceDataManagerDAO.getAirlineList(ReferenceDataManagerDAO.java:674)
com.aiat.odlp.commonutilities.referencedata.impl.ReferenceDataManagerImpl.listAirlineCodes(ReferenceDataManagerImpl.java:169)
sun.reflect.GeneratedMethodAccessor7000.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:306)
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:92)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:86)
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:151)
$Proxy247.listAirlineCodes(Unknown Source)
com.aiat.odlp.commonutilities.referencedata.endpoint.ReferenceDataRepositoryListEndPoint.listAirlines(ReferenceDataRepositoryListEndPoint.java:75)
sun.reflect.GeneratedMethodAccessor6999.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
org.springframework.ws.server.endpoint.MethodEndpoint.invoke(MethodEndpoint.java:109)
org.springframework.ws.server.endpoint.adapter.MarshallingMethodEndpointAdapter.invokeInternal(MarshallingMethodEndpointAdapter.java:133)
org.springframework.ws.server.endpoint.adapter.AbstractMethodEndpointAdapter.invoke(AbstractMethodEndpointAdapter.java:58)
org.springframework.ws.server.MessageDispatcher.dispatch(MessageDispatcher.java:211)
org.springframework.ws.server.MessageDispatcher.receive(MessageDispatcher.java:167)
com.aiat.odlp.server.endpoint.jaxws.JaxWsRequestHandler.handleRequest(JaxWsRequestHandler.java:90)
com.aiat.odlp.server.endpoint.jaxws.VoyagerJaxWsEndpoint.invoke(VoyagerJaxWsEndpoint.java:66)
com.aiat.odlp.server.endpoint.jaxws.VoyagerJaxWsEndpoint.invoke(VoyagerJaxWsEndpoint.java:38)
sun.reflect.GeneratedMethodAccessor958.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:575)
weblogic.wsee.jaxws.WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:92)
weblogic.wsee.jaxws.WLSInstanceResolver$WLSInvoker.invoke(WLSInstanceResolver.java:73)
com.sun.xml.ws.api.server.Invoker.invokeProvider(Invoker.java:108)
com.sun.xml.ws.server.InvokerTube$2.invokeProvider(InvokerTube.java:159)
com.sun.xml.ws.server.provider.SyncProviderInvokerTube.processRequest(SyncProviderInvokerTube.java:70)
com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:580)
com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:553)
com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:535)
com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:433)
^-- Holding lock: com.sun.xml.ws.api.pipe.Fiber@2d35f1bf[thin lock]
com.sun.xml.ws.server.WSEndpointImpl$2.process(WSEndpointImpl.java:352)
com.sun.xml.ws.transport.http.HttpAdapter$HttpToolkit.handle(HttpAdapter.java:508)
com.sun.xml.ws.transport.http.HttpAdapter.handle(HttpAdapter.java:230)
com.sun.xml.ws.transport.http.servlet.ServletAdapter.handle(ServletAdapter.java:140)
weblogic.wsee.jaxws.WLSServletAdapter.handle(WLSServletAdapter.java:193)
weblogic.wsee.jaxws.HttpServletAdapter$AuthorizedInvoke.run(HttpServletAdapter.java:310)
weblogic.wsee.jaxws.HttpServletAdapter.post(HttpServletAdapter.java:187)
weblogic.wsee.jaxws.JAXWSServlet.doPost(JAXWSServlet.java:124)
javax.servlet.http.HttpServlet.service(HttpServlet.java:700)
weblogic.wsee.jaxws.JAXWSServlet.service(JAXWSServlet.java:76)
javax.servlet.http.HttpServlet.service(HttpServlet.java:815)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:224)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:108)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:206)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
oracle.dms.wls.DMSServletFilter.doFilter(DMSServletFilter.java:114)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:55)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.doIt(Unknown Source)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(Unknown Source)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:308)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:117)
weblogic.servlet.internal.WebAppServletContext.securedExecute(Unknown Source)
weblogic.servlet.internal.WebAppServletContext.execute(Unknown Source)
weblogic.servlet.internal.ServletRequestImpl.run(Unknown Source)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:198)
weblogic.work.ExecuteThread.run(ExecuteThread.java:165)
}

>

Any help/guidance would be really appreciated.


Start of Managed Server log with envt details:
starting weblogic with Java version:
java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b03)
Oracle JRockit(R) (build R28.2.4-14-151097-1.6.0_33-20120618-1634-linux-x86_64, compiled mode)
Starting WLS with line:
/apps/sw/oracle/mw/jrockit-x64/bin/java -jrockit -Xms4096m -Xmx4096m -Dweblogic.Name=VWS-App-4
-Djava.security.policy=/apps/sw/oracle/mw/wls1033/wlserver_10.3/server/lib/weblogic.policy
-Dweblogic.system.BootIdentityFile=/apps/services/mw/VGR-WLS-STG/servers/VWS-App-4/data/nodemanager/boot.properties
-Dweblogic.nodemanager.ServiceEnabled=true
-Dweblogic.security.SSL.ignoreHostnameVerification=true
-Dweblogic.ReverseDNSAllowed=false
-Dweblogic.Stdout=/logs/servers/VWS-App-4/VWS-App-4.out
-Xverify:none -da -Dplatform.home=/apps/sw/oracle/mw/wls1033/wlserver_10.3
-Dwls.home=/apps/sw/oracle/mw/wls1033/wlserver_10.3/server
-Dweblogic.home=/apps/sw/oracle/mw/wls1033/wlserver_10.3/server
-Ddomain.home=/apps/services/mw/VGR-WLS-STG
-Dcommon.components.home=/apps/sw/oracle/mw/wls1033/oracle_common
-Djrf.version=11.1.1 -Djrockit.optfile=/apps/sw/oracle/mw/wls1033/oracle_common/modules/oracle.jrf_11.1.1/jrocket_optfile.txt
-Doracle.domain.config.dir=/apps/services/mw/VGR-WLS-STG/config/fmwconfig
-Doracle.server.config.dir=/apps/services/mw/VGR-WLS-STG/config/fmwconfig/servers/VWS-App-4
-Doracle.security.jps.config=/apps/services/mw/VGR-WLS-STG/config/fmwconfig/jps-config.xml
-Djava.protocol.handler.pkgs=oracle.mds.net.protocol
-Digf.arisidbeans.carmlloc=/apps/services/mw/VGR-WLS-STG/config/fmwconfig/carml
-Digf.arisidstack.home=/apps/services/mw/VGR-WLS-STG/config/fmwconfig/arisidprovider
-Dweblogic.alternateTypesDirectory=/apps/sw/oracle/mw/wls1033/oracle_common/modules/oracle.ossoiap_11.1.1,/apps/sw/oracle/mw/wls1033/oracle_common/modules/oracle.oamprovider_11.1.1
-Dweblogic.jdbc.remoteEnabled=false -Dem.oracle.home=/apps/sw/oracle/mw/wls1033/oracle_common -Djava.awt.headless=true -Dweblogic.management.discover=false
-Dweblogic.management.server=http://LAVDRXS011R:7001 -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole=
-Dweblogic.ext.dirs=/apps/sw/oracle/mw/wls1033/patch_wls1033/profiles/default/sysext_manifest_classpath:/apps/sw/oracle/mw/wls1033/patch_ocp353/profiles/default/sysext_manifest_classpath
-Xverbose:memory -Xverboselog:/logs/servers/VWS-App-4/verbose-gc.log -Djavax.management.builder.initial=weblogic.management.jmx.mbeanserver.WLSMBeanServerBuilder
-Dweblogic.security.SSL.ignoreHostnameVerification=true -Dweblogic.security.SSL.protocolVersion=SSL3 -Dlog4j.debug -Dtangosol.coherence.mode=prod
-Djavax.xml.soap.MessageFactory=weblogic.xml.saaj.MessageFactoryImpl -DnotificationBroker.instanceMode=PASSIVE -Xverify:none -Dwles.scm.port=
-Dwles.arme.port=9101 -Dwles.config.signer=lavdrxs011r.atlis.aiat.com
-Dlog4j.configuration=file:/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/instance/vgr_wls_app4_ssm_instance/config/log4j.properties
-Dlog4j.ignoreTCL -Dwles.ssl.passwordFile=/apps/sw/oracle/mw/wls1033/ales32-shared/keys/password.xml
-Dwles.ssl.passwordKeyFile=/apps/sw/oracle/mw/wls1033/ales32-shared/keys/password.key
-Dwles.ssl.identityKeyStore=/apps/sw/oracle/mw/wls1033/ales32-shared/keys/identity.jceks
-Dwles.ssl.identityKeyAlias=wles-ssm -Dwles.ssl.identityKeyPasswordAlias=wles-ssm
-Dwles.ssl.trustedCAKeyStore=/apps/sw/oracle/mw/wls1033/ales32-shared/keys/trust.jks
-Dwles.ssl.trustedPeerKeyStore=/apps/sw/oracle/mw/wls1033/ales32-shared/keys/peer.jks
-Djava.io.tmpdir=/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/instance/vgr_wls_app4_ssm_instance/work/jar_temp
-Dwles.ssmws.instance.home=/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/instance/vgr_wls_app4_ssm_instance
-Darme.configuration=/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/instance/vgr_wls_app4_ssm_instance/config/WLESarme.properties
-Dales.blm.home=/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/instance/vgr_wls_app4_ssm_instance -Dweblogic.security.enableNegotiate=true
-Dweblogic.alternateTypesDirectory=/apps/sw/oracle/mw/wls1033/ales32-ssm/wls-ssm/lib/providers/wls/v9
-Dwles.scm.useSSL=true
-XX:PermSize=128m
-XX:MaxPermSize=256m weblogic.Server
[WARN ] Use of -Djrockit.optfile is deprecated and discouraged.
Abhinav Sonkar
Greenhorn

Joined: Nov 20, 2013
Posts: 16
The issue is obviously with the database and not WebLogic. The thread managing the oracle connection is waiting to receive data from a query. After 600 seconds it is marked as STUCK. Try checking the AWR report around the hour when threads start getting stuck. That should give a clue as to which query is causing the problem.
M Paul Baker
Greenhorn

Joined: Mar 06, 2013
Posts: 3
Thank you for your suggestion. We have "requested" an AWR report, I'm hopeful that it will provide some clues to what is happening.

We are working on creating several scenarios that would/could duplicate this situation.

What are some conditions that would cause a thread to be marked "STUCK" at SocketNativeIO.readBytesPinned while reading a DB response.
  • DB Bounce? Y/N
  • DB Timeout? Y/N
  • DB Exceptions? Y/N
  • Socket Connection Issues btwn WLS and DB? Y/N
  • A long running query (i.e. no index)? Y/N


  •  
     
    subject: Another Stuck Thread Question - SocketNativeIO.readBytesPinned - PreparedStatement
     
    Similar Threads
    How to read thread dump
    jboss5 - What error did i really experiencing? - org.jboss.logging.Logger.getMessageLogger
    BEA-000337 StuckThreadMaxTime
    Tomcat 6 crash
    Thread count increase in weblogic..