This week's book giveaway is in the Jobs Discussion forum.
We're giving away four copies of Soft Skills and have John Sonmez on-line!
See this thread for details.
The moose likes BEA/Weblogic and the fly likes BEA-000337 StuckThreadMaxTime Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Soft Skills this week in the Jobs Discussion forum!
JavaRanch » Java Forums » Products » BEA/Weblogic
Bookmark "BEA-000337 StuckThreadMaxTime" Watch "BEA-000337 StuckThreadMaxTime" New topic
Author

BEA-000337 StuckThreadMaxTime

Rita Rege
Greenhorn

Joined: Feb 04, 2010
Posts: 1
Our java application runs on weblogic 8.1 SP6 & uses Oracle DB. We also host webservices that are called by other applications. The app. has been very stable and hasn't caused any issues in the last several months. Last Sunday there was some network change (I am not sure on the details, but we were told its a very insignificant change - our database server was recycled to clear the dns cache or something, and we were told it doesnt even require the recycling of our application servers.

However, since then my application fails every now and then on one server after the other. I have had to recycle each of the 8 servers twice this week! Please find below the two type of errors I see in my weblogic logs ever since.

Error # 1 - This is where our webservice is called. (Looking at the threaddump it looks like it fails when trying to get a database connection, although I am not very sure) We use Hibernate/Spring to make a DB connection. We do not use the connectionpool.

+<Error> <WebLogicServer> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '0' for queue: 'weblogic.kernel.Default' has been busy for "885" seconds working on the request "Http Request: /myapp/MyAppAdminService", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>+

Error # 2 - I am not sure if I need to take this error seriously as it does not seem to be there in the threaddump.

+<Error> <WebLogicServer> <weblogic.health.CoreHealthMonitor> <<WLS Kernel>> <> <BEA-000337> <ExecuteThread: '17' for queue: 'weblogic.kernel.Default' has been busy for "921" seconds working on the request "Http Request: /my-portal/view/templates/generic_template.jsp", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>+

We recycled our application servers, weblogic admin server and managed servers on both the clusters today. However that doesnt seem to fix the problem!!

Any help would be appreciated!!

Could this be Database related? (Our DB Server was recycled to clear some DNS cache but I am not sure what else I can do other than recycling my app servers if this is the issue. Please note that the webservice fails sporadically and does not fail all the time) If this is DB related, what do I check with our DBA?

Could this be Weblogic related?

Could this be App Server / Memory / Load Balancer related?

Would redeploying my app help?

Can I do anything else to debug this further?

Any help would be greatly appreciated!! Please find the threaddump below -

===== FULL THREAD DUMP ===============
Wed Feb 3 14:44:41 2010
BEA JRockit(R) R27.2.0-131-78843-1.4.2_13-20070320-1511-linux-ia32


"ExecuteThread: '0' for queue: 'weblogic.kernel.Default'" id=9 idx=0x38 tid=28257 prio=5 alive, in native, waiting, daemon
-- Waiting for notification on: org/apache/commons/pool/impl/GenericObjectPool@0x1c202e20[fat lock]
at jrockit/vm/Threads.waitForSignal(J)Z(Native Method)
at java/lang/Object.wait(J)V(Native Method)[optimized]
at java/lang/Object.wait(Object.java:429)
at org/apache/commons/pool/impl/GenericObjectPool.borrowObject(GenericObjectPool.java:810)
^-- Lock released while waiting: org/apache/commons/pool/impl/GenericObjectPool@0x1c202e20[fat lock]
at org/apache/commons/dbcp/PoolingDataSource.getConnection(PoolingDataSource.java:95)
at org/apache/commons/dbcp/BasicDataSource.getConnection(BasicDataSource.java:540)
at org/springframework/orm/hibernate3/LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:81)
at org/hibernate/jdbc/ConnectionManager.openConnection(ConnectionManager.java:417)
at org/hibernate/jdbc/ConnectionManager.getConnection(ConnectionManager.java:144)
at org/hibernate/jdbc/BorrowedConnectionProxy.invoke(BorrowedConnectionProxy.java:40)
at $Proxy9.setReadOnly(Z)V(Unknown Source)
at org/springframework/jdbc/datasource/DataSourceUtils.prepareConnectionForTransaction(DataSourceUtils.java:156)
at org/springframework/orm/hibernate3/HibernateTransactionManager.doBegin(HibernateTransactionManager.java:444)
at org/springframework/transaction/support/AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:322)
at org/springframework/transaction/interceptor/TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:255)
at org/springframework/transaction/interceptor/TransactionInterceptor.invoke(TransactionInterceptor.java:102)
at org/springframework/aop/framework/ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:176)
at org/springframework/aop/framework/Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:615)
at com/abc/amt/My/adapter/rrgt/generated/ItemDataHome$$EnhancerByCGLIB$$8e2fb3ac.findByPersonId(Ljava/lang/String;)Ljava/util/List;(<generated>)
at com/abc/amt/My/model/rrgt/RequestFactory.getDetails1(RequestFactory.java:339)
at com/abc/amt/My/adapter/rrgt/webservice/internal/generated/My55h4RequestAdminServiceImpl.getDetails1(My55h4RequestAdminServiceImpl.java:570)
at sun/reflect/GeneratedMethodAccessor271.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object;(Unknown Source)
at sun/reflect/DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)[optimized]
at java/lang/reflect/Method.invoke(Method.java:324)[optimized]
at weblogic/webservice/component/javaclass/JavaClassInvocationHandler.invoke(JavaClassInvocationHandler.java:133)
at weblogic/webservice/core/handler/InvokeHandler.handleRequest(InvokeHandler.java:104)
at weblogic/webservice/core/HandlerChainImpl.handleRequest(HandlerChainImpl.java:143)
at weblogic/webservice/core/DefaultOperation.process(DefaultOperation.java:549)
at weblogic/webservice/server/Dispatcher.process(Dispatcher.java:204)
at weblogic/webservice/server/Dispatcher.doDispatch(Dispatcher.java:176)
at weblogic/webservice/server/Dispatcher.dispatch(Dispatcher.java:96)
at weblogic/webservice/server/WebServiceManager.dispatch(WebServiceManager.java:100)
at weblogic/webservice/server/servlet/WebServiceServlet.serverSideInvoke(WebServiceServlet.java:297)
at weblogic/webservice/server/servlet/ServletBase.doPost(ServletBase.java:498)
at weblogic/webservice/server/servlet/WebServiceServlet.doPost(WebServiceServlet.java:267)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:760)
at javax/servlet/http/HttpServlet.service(HttpServlet.java:853)
at weblogic/servlet/internal/ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077)[inlined]
at weblogic/servlet/internal/ServletStubImpl.invokeServlet(ServletStubImpl.java:465)[inlined]
at weblogic/servlet/internal/ServletStubImpl.invokeServlet(ServletStubImpl.java:348)[optimized]
at weblogic/servlet/internal/WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7047)
at weblogic/security/acl/internal/AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)[optimized]
at weblogic/security/service/SecurityManager.runAs(SecurityManager.java:121)[inlined]
at weblogic/servlet/internal/WebAppServletContext.invokeServlet(WebAppServletContext.java:3902)[optimized]
at weblogic/servlet/internal/ServletRequestImpl.execute(ServletRequestImpl.java:2773)[optimized]
at weblogic/kernel/ExecuteThread.execute(ExecuteThread.java:224)[optimized]
at weblogic/kernel/ExecuteThread.run(ExecuteThread.java:183)
at jrockit/vm/RNI.c2java(IIII)V(Native Method)
-- end of trace
Deepak Bala
Bartender

Joined: Feb 24, 2006
Posts: 6662
    
    5

Well there is no error here as such. The message simply means that the pages are taking more than 10 minutes to load, which is the maximum time that a thread can be kept in the running state before it is marked as STUCK by weblogic.

What is your definition of recycled ? Ask the network guys more details about the change.

Restarting things without knowing the root cause is not advisable.

You have a data source do you not ? What does the monitoring tab on that data source say ? What is the wait time before connections are served and what is the max concurrent connections that were served ? What is the max connections that the DB can serve and is it in line with the number of pooled connections on your data source ?


SCJP 6 articles - SCJP 5/6 mock exams - More SCJP Mocks
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: BEA-000337 StuckThreadMaxTime