We have a Websphere server running on i5 with more than 25 applications. Also some application use the connection pool to access the database.
last week, the application profile got stuck 3 times and today again! & we had to end the job manually and restarted.
I can’t figure out what has gone wrong.
Anyone please show me the right direction to troubleshoot this issue?
Also you may try to use a javacore to figure out how your threads got hung (blocking on a resource, doing a lengthy computation or a synchronization issue etc.). Since a javacore shows call stack of threads, you may probably also infer which application used a thread by checking which packages and classes were used during invocation.
Javacore files are plain text and you can read them with a text editor but that requires a lot of scrolling back and forward. I suggest after collecting dumps, you download IBM Thread and Monitor Dump Analyzer for Java and check your dumps using this tool, which presents the javacore in a much more understandable fashion. Tool page also contains links for guidance on how to use this tool to analyze hung thread problems.
Generate javacores when you notice WSVR0605W messages that are reporting there are hung threads, as in your post. Then check the javacore generated, using the tool above to see what those threads were doing. Threads appear in the tool with the same name as reported in the warning messages in your log (i.e. WebContainer : 1111, WebContainer 1113... in your previous post), so you may locate them easily.
This is not a trivial approach but may provide an answer to your question. Good luck.
Make sure that you close all connections correctly.
The system maybe waiting for a new connection which is not available because you already used all connections.
Also validate if you committed all transactions.
Then check DB locks with your DBA.
Check if you have any long running methods… maybe with some heavy calculations or huge xml transformation which keep connections open.