Hi, I am having the problems running continuous 24 hrs test against Weblogic 8.1. After some time the server starts complaining about the threads stuck <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '7' for queue: 'weblogic.kernel.Default' has been busy for "630" seconds working on the request "weblogic.ejb20.internal.JMSMessagePoller@d64412", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.> This happens for a number of threads and then the server goes into a continuous period of inactivity trying desperately to free memory and then eventually it start giving "java.lang.OutOfMemoryException" My queries are 1) How to detect on what the thread is stuck on 2) In case of a stuck thread - how to roll back/stop the thread so that the rest of the processing can continue. Thanks Navjeet
Joined: Sep 07, 2002
hi, I am also facing similar problem. This is problem reported due to JMS Server. Cause the warning tells about JMS Server port only. With me I am not facing OutOfMemoryError. Navjeet, let me know were you able to solve ur problem or not. Is there any tuning to be done on JMS server ?
Joined: Dec 10, 2003
Hi, I am also getting the same problem. How did you resolve this issue? Thanks, Purvesh
Regards,<br />Purvesh<br />SCJP, SCBCD, IBM Exam #486
Joined: Jul 11, 2002
Hi I am also getting the same problem while doing batch query on DELETE operation in a DAO class.
removeMetaDataDefinition-------------->DELETE FROM TDMS_METADATA_DEFN WHERE RESCLASSID ='05011132220000180' AND METADATADEFNID ='05011132190000179' <Feb 5, 2005 12:09:20 PM IST> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '11' for queue: 'weblogic.kernel.Default' has been busy for "864" seconds working on the request "Http Request: /dmsweb/folder.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.> <Feb 5, 2005 12:09:21 PM IST> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '13' for queue: 'weblogic.kernel.Default' has been busy for "1,539" seconds working on the request "Http Request: /dmsweb/folder.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.> <Feb 5, 2005 12:19:21 PM IST> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '10' for queue: 'weblogic.kernel.Default' has been busy for "828" seconds working on the request "Http Request: /dmsweb/folder.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.> <Feb 5, 2005 12:19:21 PM IST> <Warning> <WebLogicServer> <BEA-000337> <ExecuteThread: '11' for queue: 'weblogic.kernel.Default' has been busy for "1,464" seconds working on the request "Http Request: /dmsweb/folder.do", which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>
I check on stuck threads in the console. There are other ways. See BEA Documentation.
Of course the console looks a little different in each version, I'm looking at 8.1 right now.
Go to the managed server node of interest in the node tree (list on left -- mine shows a world icon for my domain; under that is the "servers" folder; within the servers folder are my three servers).
Fo direct access to threads and what they are doing -- right click on the server node -- select "View Execute Threads". Look to see what the Default threads are "doing" in the "current request" column.
Alternatively you can go the the server's "Monitoring Tab". Select monitor all active Queues. There you can see the number of threads idle for each type of thread. Your user's requests will go to the default queue unless you configure it differently. Click the link for "weblogic.kernel.Default" to see what those threads are doing.
Then what to do? I have a process that runs a few hours and I get a stuck thread warning (1 thread), but no harm to the server. I don't do anything about it.
Check Tuning WebLogic Server Applications, there's a section on "Using Execute Queues to Control Thread Usage". Maybe you just need more threads. Maybe you need some of your application to use a different queue. Maybe your application needs tuning.
Once you know what your threads are stuck on you could run a profiling tool like Optimizeit or Jprobe. You could have tune your db use.
Regarding stopping threads. That's a tricky business and I'm pretty sure WebLogic doesn't support killing threads. [ February 09, 2005: Message edited by: Carol Enderlin ]
Joined: Jan 06, 2001
Originally posted by Navjeet Nehra:
1) How to detect on what the thread is stuck on 2) In case of a stuck thread - how to roll back/stop the thread so that the rest of the processing can continue.
1) The best way to discover it is taking a thread dump, to do that you�ll have to start the server in foreground(so you can see the console messages) or redirect the standart output to a file, then all you need to do is hit CTR+BREAK on windows or use the command kill -3 on unix environments, it�ll provide you with a thread dump where you can see exactly where your thread�s are stucked.
2) You can�t roll back a stucked thread, usually as Carol mentioned you�ll have to tune the Default thread queue on weblogic and also tune your application, most of the problems like this i have faced were solved tunning the DB then the application and then, if still required weblogic where you can tune thread count, garbage collection, etc. You can also consider programming a threshould so the node manager can reboot the managed server in case of too much thread count usage(this will actually kill all running threads of course).
Joined: Jul 01, 2003
Found this thread to be interesting, I am facing similar problem, my application has 200 as thread queue and when this problem occurs , I was not even able to get into the console.
Could any one of you help me in solving this ?
If tuning in the database is required, what kind of tuning is to be made. I would appreciate your early response.
Try Try Try...
Joined: May 14, 2001
Thread dump would help to nail down the issue, also check is there any DB access your app is doing,is it waiting to get connection.,is there any operation going on in DB at that point of time..
Joined: Dec 17, 2004
Hi, Our app also having same StuckThread problem.My senior said this is because of the weblogic is redirecting the request to a proxy server then to a external system.if the proxy is hanging the weblogic server is getting warning like "ExecuteThread: '17' for queue: 'default' has been busy for "816" seconds working on the request "de.dl.crl.facade.sb.CRLFacadeBeanEJB_r251p2_EOImpl",which is more than the configured time (StuckThreadMaxTime) of "600" seconds.>"
Now we want to track this problem and return back to the user with proper excpetion related information. Is there a way to get this proxy status from weblogic?? if anything is not clear please tell me. Thanks in advance.
to be precise, do the following
To configure WebLogic Server stuck thread detection behavior:
1. Start the Administration Server if it is not already running.
2. Access the Administration Console for the domain.
3. Expand the Servers node in the left pane to display the servers configured in your domain.
4. Click the name of the server instance that you want to modify for improved stuck thread detection.
Note: You configure stuck thread detection parameters on a per-server basis, rather than on a per-execute queue basis.
5. Select the Configuration —> Tuning tab in the right pane.
6. Modify the following attributes as necessary to tune thread detection behavior for the server:
* Stuck Thread Max Time: Enter the number of seconds, that a thread must be continually working before this server diagnoses the thread as being stuck. By default, WebLogic Server considers a thread to be "stuck" after 600 seconds of continuous use.
* Stuck Thread Timer Interval: Enter the number of seconds, after which WebLogic Server periodically scans threads to see if they have been continually working for the length of time specified by Stuck Thread Max Time. By default, WebLogic Server sets this interval to 600 seconds.
7. Click Apply to save your changes.
8. Reboot the server to use the new settings.
Hareesh Ram Chanchali
Joined: Jan 31, 2005
I strongly agree with Marcos.
We need to take thread dumps at regular intervals and should analyse them.
You take the thread dumps at an interval of 1 or 2 minutes and take the help of samurai tool to analyse the thread dumps. Then you will be able to know what exactly happenning and possible reasons
1) DB locks
2) Native thread problems if any
3) Huge amount of data being pulled up
4) extended time in getting responses from interface systems
with all these try checking the Garbage collection also.
Hareesh Ram Chanchali
SCJP 5.0, IBM Certified Solution Designer