Thread stuck problems - StuckThreadMaxTime exceeded

[Navjeet Nehra]

Hi,
I am having the problems running continuous 24 hrs test against Weblogic 8.1. After some time the server starts complaining about the threads stuck
<Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '7' for queue: 'weblogic.kernel.Default' has been busy for "630" seconds working on the request "weblogic.ejb20.internal.JMSMessagePoller@d64412", which is
more than the configured time (StuckThreadMaxTime) of "600" seconds.>
This happens for a number of threads and then the server goes into a continuous period of inactivity trying desperately to free memory and then eventually it start giving "java.lang.OutOfMemoryException"
My queries are
1) How to detect on what the thread is stuck on
2) In case of a stuck thread - how to roll back/stop the thread so that the rest of the processing can continue.
Thanks
Navjeet

[Jack Nicholson]

hi,
I am also facing similar problem.
This is problem reported due to JMS Server.
Cause the warning tells about JMS Server port only.
With me I am not facing OutOfMemoryError.
Navjeet, let me know were you able to solve ur problem or not.
Is there any tuning to be done on JMS server ?

[Purvesh Vora]

Hi,
I am also getting the same problem.
How did you resolve this issue?
Thanks,
Purvesh

[sathish kannan]

Hi I am also getting the same problem while doing batch query on DELETE operation in a DAO class.

removeMetaDataDefinition-------------->DELETE FROM TDMS_METADATA_DEFN WHERE RESCLASSID ='05011132220000180' AND METADATADEFNID ='05011132190000179'
<Feb 5, 2005 12:09:20 PM IST> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '11' for queue: 'weblogic.kernel.Default' has been busy for "864" seconds working on the request "Http Request: /dmsweb/folder.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
<Feb 5, 2005 12:09:21 PM IST> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '13' for queue: 'weblogic.kernel.Default' has been busy for "1,539" seconds working on the request "Http Request: /dmsweb/folder.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
<Feb 5, 2005 12:19:21 PM IST> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '10' for queue: 'weblogic.kernel.Default' has been busy for "828" seconds working on the request "Http Request: /dmsweb/folder.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
<Feb 5, 2005 12:19:21 PM IST> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '11' for queue: 'weblogic.kernel.Default' has been busy for "1,464" seconds working on the request "Http Request: /dmsweb/folder.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>


If you have solved the problem please let me know

thanks in advance

[Mark Spritzler]

"sathish"-
Welcome to the JavaRanch! Please adjust your displayed name to meet the

JavaRanch Naming Policy.

You can change it

here.

Thanks! and welcome to the JavaRanch!

Mark

[Carol Enderlin]

I check on stuck threads in the console. There are other ways. See BEA Documentation.

Of course the console looks a little different in each version, I'm looking at 8.1 right now.

Go to the managed server node of interest in the node tree (list on left -- mine shows a world icon for my domain; under that is the "servers" folder; within the servers folder are my three servers).

Fo direct access to threads and what they are doing -- right click on the server node -- select "View Execute Threads". Look to see what the Default threads are "doing" in the "current request" column.

Alternatively you can go the the server's "Monitoring Tab". Select monitor all active Queues. There you can see the number of threads idle for each type of thread. Your user's requests will go to the default queue unless you configure it differently. Click the link for "weblogic.kernel.Default" to see what those threads are doing.

Then what to do? I have a process that runs a few hours and I get a stuck thread warning (1 thread), but no harm to the server. I don't do anything about it.

Check Tuning WebLogic Server Applications, there's a section on "Using Execute Queues to Control Thread Usage". Maybe you just need more threads. Maybe you need some of your application to use a different queue. Maybe your application needs tuning.

Once you know what your threads are stuck on you could run a profiling tool like Optimizeit or Jprobe. You could have tune your db use.

Regarding stopping threads. That's a tricky business and I'm pretty sure WebLogic doesn't support killing threads.
[ February 09, 2005: Message edited by: Carol Enderlin ]

[Marcos Maia]

Originally posted by Navjeet Nehra:

1) How to detect on what the thread is stuck on
2) In case of a stuck thread - how to roll back/stop the thread so that the rest of the processing can continue.

Thanks
Navjeet

1) The best way to discover it is taking a thread dump, to do that you�ll have to start the server in foreground(so you can see the console messages) or redirect the standart output to a file, then all you need to do is hit CTR+BREAK on windows or use the command kill -3 on unix environments, it�ll provide you with a thread dump where you can see exactly where your thread�s are stucked.

2) You can�t roll back a stucked thread, usually as Carol mentioned you�ll have to tune the Default thread queue on weblogic and also tune your application, most of the problems like this i have faced were solved tunning the DB then the application and then, if still required weblogic where you can tune thread count, garbage collection, etc.
You can also consider programming a threshould so the node manager can reboot the managed server in case of too much thread count usage(this will actually kill all running threads of course).

best regards.

[Vetrik Kumaran]

Hi,

Found this thread to be interesting, I am facing similar problem, my application has 200 as thread queue and when this problem occurs , I was not even able to get into the console.

Could any one of you help me in solving this ?

If tuning in the database is required, what kind of tuning is to be made. I would appreciate your early response.

-- Vetrik

[great veera]

Thread dump would help to nail down the issue, also check is there any DB access your app is doing,is it waiting to get connection.,is there any operation going on in DB at that point of time..

[Muthu Ponmozhi]

Hi,
Our app also having same StuckThread problem.My senior said this is because of the weblogic is redirecting the request to a proxy server then to a external system.if the proxy is hanging the weblogic server is getting warning like "ExecuteThread: '17' for queue: 'default' has been busy for "816" seconds working on the request "de.dl.crl.facade.sb.CRLFacadeBeanEJB_r251p2_EOImpl",which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>"

Now we want to track this problem and return back to the user with proper excpetion related information. Is there a way to get this proxy status from weblogic??
if anything is not clear please tell me.
Thanks in advance.

[J Praveen Reddy]

Hi all,
Any solution for this?

[Fadia Ismael]

Originally posted by J Praveen Reddy:
Hi all,
Any solution for this?

Please have a look at

https://support.bea.com/application_content/product_portlets/support_patterns/wls/JDBCCausesServerHangPattern.html

Where it mentions:
This does not interrupt the thread, as this is just a notification for the administrator

Hope this is useful..

[mahesh vavilla]

we still didn't find the solution for this?

I am having the same problem.

Its been revolving from long back.

Please find a solution for this.

[Vikrant Korde]

Check out this link
http://e-docs.bea.com/wls/docs81/perform/WLSTuning.html#1125714

to be precise, do the following
To configure WebLogic Server stuck thread detection behavior:
1. Start the Administration Server if it is not already running.
2. Access the Administration Console for the domain.
3. Expand the Servers node in the left pane to display the servers configured in your domain.
4. Click the name of the server instance that you want to modify for improved stuck thread detection.
Note: You configure stuck thread detection parameters on a per-server basis, rather than on a per-execute queue basis.
5. Select the Configuration —> Tuning tab in the right pane.
6. Modify the following attributes as necessary to tune thread detection behavior for the server:
* Stuck Thread Max Time: Enter the number of seconds, that a thread must be continually working before this server diagnoses the thread as being stuck. By default, WebLogic Server considers a thread to be "stuck" after 600 seconds of continuous use.
* Stuck Thread Timer Interval: Enter the number of seconds, after which WebLogic Server periodically scans threads to see if they have been continually working for the length of time specified by Stuck Thread Max Time. By default, WebLogic Server sets this interval to 600 seconds.
7. Click Apply to save your changes.
8. Reboot the server to use the new settings.

[Hareesh Ram Chanchali]

I strongly agree with Marcos.

We need to take thread dumps at regular intervals and should analyse them.

You take the thread dumps at an interval of 1 or 2 minutes and take the help of samurai tool to analyse the thread dumps. Then you will be able to know what exactly happenning and possible reasons

1) DB locks
2) Native thread problems if any
3) Huge amount of data being pulled up
4) extended time in getting responses from interface systems

with all these try checking the Garbage collection also.