aspose file tools*
The moose likes Websphere and the fly likes Websphere Troubleshooting.. Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Products » Websphere
Bookmark "Websphere Troubleshooting.." Watch "Websphere Troubleshooting.." New topic
Author

Websphere Troubleshooting..

dhanu manjula
Greenhorn

Joined: Aug 31, 2009
Posts: 19
Hello all,

We have a Websphere server running on i5 with more than 25 applications. Also some application use the connection pool to access the database.
last week, the application profile got stuck 3 times and today again! & we had to end the job manually and restarted.
I can’t figure out what has gone wrong.
Anyone please show me the right direction to troubleshoot this issue?

Thanks
Paul Sturrock
Bartender

Joined: Apr 14, 2004
Posts: 10336


application profile got stuck 3 times

Can you explain what you mean by "got stuck"?


JavaRanch FAQ HowToAskQuestionsOnJavaRanch
dhanu manjula
Greenhorn

Joined: Aug 31, 2009
Posts: 19
Paul Sturrock wrote:
Can you explain what you mean by "got stuck"?


Sorry Paul,
I means any of my applications did not open/responce and also I couldn't open the Administration Console
. But WRKACTJOB shows that profile is allocated only 3% of CPU which is normal..
Paul Sturrock
Bartender

Joined: Apr 14, 2004
Posts: 10336

Sounds like WAS might be waiting for a resource. Any blocks on the database? Any synchronized code?
dhanu manjula
Greenhorn

Joined: Aug 31, 2009
Posts: 19
Paul Sturrock wrote:Sounds like WAS might be waiting for a resource. Any blocks on the database? Any synchronized code?


Thanks paul,
I haven’t found any block on database or we don’t synchronize.
And I analyzed the Systemout.log file saw this exception,



What else / where else should I monitor to make this go away?


Thanks
dhanu manjula
Greenhorn

Joined: Aug 31, 2009
Posts: 19
Hi again,

How can i know that which application brings this problem as there are more than 25 applications running in that profile..
Please let me know how we can resolve this issue.

Thanks again.
Paul Sturrock
Bartender

Joined: Apr 14, 2004
Posts: 10336

I would expect end users to be reporting this unresponsive behaviour. You could try asking them.

I'd also get the apps into your test environment asap, deploy them one at a time and run whatever soak tests you have against them.
Kurtcebe Eroglu
Ranch Hand

Joined: Jan 30, 2007
Posts: 30
Also you may try to use a javacore to figure out how your threads got hung (blocking on a resource, doing a lengthy computation or a synchronization issue etc.). Since a javacore shows call stack of threads, you may probably also infer which application used a thread by checking which packages and classes were used during invocation.

I don't have any experience on i5, but you may check following support document to see how you can manually generate a javacores (also called JVM thread dump);
MustGather: Overview of Application Server, Node Agent, and Deployment Manager Hang Problems for IBM i (i5/OS) , Appendix A.

Javacore files are plain text and you can read them with a text editor but that requires a lot of scrolling back and forward. I suggest after collecting dumps, you download IBM Thread and Monitor Dump Analyzer for Java and check your dumps using this tool, which presents the javacore in a much more understandable fashion. Tool page also contains links for guidance on how to use this tool to analyze hung thread problems.

Generate javacores when you notice WSVR0605W messages that are reporting there are hung threads, as in your post. Then check the javacore generated, using the tool above to see what those threads were doing. Threads appear in the tool with the same name as reported in the warning messages in your log (i.e. WebContainer : 1111, WebContainer 1113... in your previous post), so you may locate them easily.

This is not a trivial approach but may provide an answer to your question. Good luck.
Michael Ergotron
Greenhorn

Joined: Nov 19, 2010
Posts: 5
Make sure that you close all connections correctly.
The system maybe waiting for a new connection which is not available because you already used all connections.
Also validate if you committed all transactions.
Then check DB locks with your DBA.
Check if you have any long running methods… maybe with some heavy calculations or huge xml transformation which keep connections open.


Find friends to hang out with
www.hangoutwithus.org
Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18889
    
    8

Kurtcebe Eroglu wrote:I don't have any experience on i5, but you may check following support document to see how you can manually generate a javacores (also called JVM thread dump).


On the System i you can use the DMPJVM command to get a JVM thread dump, but as Kurtcebe says, analyzing a thread dump is not trivial. There may well be better ways to troubleshoot the problem.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
 
subject: Websphere Troubleshooting..