Help - Application crash.

[Joseph Magerramov]

Every once in a while a weird thing happens to our application running on the production server - it crashes. At this time - any access to the application results in custom error page (specified in EAR file) to come up.
Normally, if I look at the task manager (we're running websphere on win2k) i would see too big java processes - one is websphere server and the second is the application itself. Well, after such a crash - the second process seems to be terminated, but it quickly restarts within a minute or two and the server is backup as if nothing happened. Neither stderr nor stdout logs have anything in them, so i'm a little clueless as to what's going on.
Any ideas?
[ June 18, 2003: Message edited by: Joseph Magerramov ]

[Vinod Bijlani]

ur code has got memory leak
the reason u see a new java process started is coz the adminserver restarts it after the crash. according to my understanding, ur appserver runs out of memory causing the jvm to crash. adminserver sees tht the process is down & restarts it.

[Shalini Komarla]

Yes, memory leak is a possibility.
You can enable Garbage Collection an monitor your app that way. It will give you a clue as to whether it is truly a memory leak or not.
Also look at your JVM heap size. If it is too less, you might want to bump it up. This will not stop your server from re-starting, but will delay the re-start until it reaches the max heap size.

[Joseph Magerramov]

Thanks, these are some good ideas.
It happened again just 10 minutes ago. I don't think the app process was restarted or if memory usage was above normal - I was monitoring it through task manager at the time. Of course I opened task manager as soon as the server went down, so I could've missed the actual crash take place.
Btw, looking at the plugin log, it seems like it cannot connect to the app server. Here're a few lines from the log file:
[Thu Jun 26 09:35:47 2003] 000003dc 000005d0 - ERROR: ws_server_group: serverGroupNextRoundRobinServer: Failed to find a server; all could be down
[Thu Jun 26 09:35:47 2003] 000003dc 000005d0 - ERROR: ws_common: websphereFindServer: Failed to find a server
[Thu Jun 26 09:35:47 2003] 000003dc 000005d0 - ERROR: ws_common: websphereHandleRequest: Failed to find a server
[Thu Jun 26 09:35:51 2003] 000003dc 000005ac - ERROR: ws_server_group: serverGroupNextRoundRobinServer: Failed to find a server; all could be down
[Thu Jun 26 09:35:51 2003] 000003dc 000005ac - ERROR: ws_common: websphereFindServer: Failed to find a server
[Thu Jun 26 09:35:51 2003] 000003dc 000005ac - ERROR: ws_common: websphereHandleRequest: Failed to find a server
[Thu Jun 26 09:35:52 2003] 000003dc 000005e0 - ERROR: ws_server_group: serverGroupNextRoundRobinServer: Failed to find a server; all could be down

This seems to prove the server crash theory. As far as memory leak goes, you suggested to enable garbage collector to minitor our app that way. I was under impression garbage collection was enabled by default. Any other things to try to catch the leak?
[ June 26, 2003: Message edited by: Joseph Magerramov ]

[Shalini Komarla]

Sorry I was typing in a hurry there. What I meant was to enable verbose Garbage Collection ...it will help you monitor the Garbage Collection process and will give you a clue as to if it might be a memory leak. To enable verboseGC, in the command line agruments, add -verboseGC. The output will appear in the stderr file.

[Joseph Magerramov]

Thanks - I will do that. I'm monitoring application heap size right now and so far it's been at 45MB for the past couple of hours. So it must be some isolated function that causes the memory leak.

[Joseph Magerramov]

Just witnessed the whole thing unfold again - this time eliminating memory leak possibility. Process size memory size dind't change at all. In fact, this time the java process didn't even restart.
Websphere plugin simply couldn't locate the server, even though the server was running. I'm leaning towards database problems. We're accessing db2 7.2 database through datasources, I wonder if this could have anything to do with it.

[Vinod Bijlani]

whts ur websphere version?
there where some plugin issues before 4.0.5
4.0.5 patch has some cumlative plugin efixes

[Joseph Magerramov]

Websphere version is 4.0.3 with cummulative plugin patch dated October 2002. The reason we dind't go with 4.0.4 is because it had major plugin problems over SSL connection.
I'll give 4.0.5 a try, thanks for the advice.