• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Liutauras Vilda
  • Junilu Lacar
  • Jeanne Boyarsky
  • Bear Bibeault
Sheriffs:
  • Knute Snortum
  • Tim Cooke
  • Devaka Cooray
Saloon Keepers:
  • Ron McLeod
  • Stephan van Hulst
  • Tim Moores
  • Tim Holloway
  • Carey Brown
Bartenders:
  • Piet Souris
  • Frits Walraven
  • Ganesh Patekar

Websphere Troubleshooting..

 
Greenhorn
Posts: 19
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello all,

We have a Websphere server running on i5 with more than 25 applications. Also some application use the connection pool to access the database.
last week, the application profile got stuck 3 times and today again! & we had to end the job manually and restarted.
I can’t figure out what has gone wrong.
Anyone please show me the right direction to troubleshoot this issue?

Thanks
 
Bartender
Posts: 10336
Hibernate Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator


application profile got stuck 3 times


Can you explain what you mean by "got stuck"?
 
dhanu manjula
Greenhorn
Posts: 19
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Paul Sturrock wrote:
Can you explain what you mean by "got stuck"?



Sorry Paul,
I means any of my applications did not open/responce and also I couldn't open the Administration Console
. But WRKACTJOB shows that profile is allocated only 3% of CPU which is normal..
 
Paul Sturrock
Bartender
Posts: 10336
Hibernate Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Sounds like WAS might be waiting for a resource. Any blocks on the database? Any synchronized code?
 
dhanu manjula
Greenhorn
Posts: 19
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Paul Sturrock wrote:Sounds like WAS might be waiting for a resource. Any blocks on the database? Any synchronized code?



Thanks paul,
I haven’t found any block on database or we don’t synchronize.
And I analyzed the Systemout.log file saw this exception,



What else / where else should I monitor to make this go away?


Thanks
 
dhanu manjula
Greenhorn
Posts: 19
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi again,

How can i know that which application brings this problem as there are more than 25 applications running in that profile..
Please let me know how we can resolve this issue.

Thanks again.
 
Paul Sturrock
Bartender
Posts: 10336
Hibernate Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I would expect end users to be reporting this unresponsive behaviour. You could try asking them.

I'd also get the apps into your test environment asap, deploy them one at a time and run whatever soak tests you have against them.
 
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Also you may try to use a javacore to figure out how your threads got hung (blocking on a resource, doing a lengthy computation or a synchronization issue etc.). Since a javacore shows call stack of threads, you may probably also infer which application used a thread by checking which packages and classes were used during invocation.

I don't have any experience on i5, but you may check following support document to see how you can manually generate a javacores (also called JVM thread dump);
MustGather: Overview of Application Server, Node Agent, and Deployment Manager Hang Problems for IBM i (i5/OS) , Appendix A.

Javacore files are plain text and you can read them with a text editor but that requires a lot of scrolling back and forward. I suggest after collecting dumps, you download IBM Thread and Monitor Dump Analyzer for Java and check your dumps using this tool, which presents the javacore in a much more understandable fashion. Tool page also contains links for guidance on how to use this tool to analyze hung thread problems.

Generate javacores when you notice WSVR0605W messages that are reporting there are hung threads, as in your post. Then check the javacore generated, using the tool above to see what those threads were doing. Threads appear in the tool with the same name as reported in the warning messages in your log (i.e. WebContainer : 1111, WebContainer 1113... in your previous post), so you may locate them easily.

This is not a trivial approach but may provide an answer to your question. Good luck.
 
Greenhorn
Posts: 5
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Make sure that you close all connections correctly.
The system maybe waiting for a new connection which is not available because you already used all connections.
Also validate if you committed all transactions.
Then check DB locks with your DBA.
Check if you have any long running methods… maybe with some heavy calculations or huge xml transformation which keep connections open.

 
Sheriff
Posts: 24654
58
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Kurtcebe Eroglu wrote:I don't have any experience on i5, but you may check following support document to see how you can manually generate a javacores (also called JVM thread dump).



On the System i you can use the DMPJVM command to get a JVM thread dump, but as Kurtcebe says, analyzing a thread dump is not trivial. There may well be better ways to troubleshoot the problem.
 
Don't count your weasels before they've popped. And now for a mulberry bush related tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!