We have this Tomcat server (5.5.17) running for quite sometime now and today all of a sudden we start see these crashes upon the usage of this particular dll. We have removed this node from the load balancer. We faced the same exact issue almost a year ago and we tried replacing/installing all our software (i.e. Tomcat, FaceIt libraries etc) but still the server exibited the same problem until we had reload a backed up image. We never found the proper resolution nor the exactly root cause (yes we know its the dll library but why the server started crashing out of no where considering that the system didn't change, no updates no new installations )
It seems that this is caused by FaceItLocate.dll library. This is a bio-metric related library that we use for photo quality evaluation. The stack trace of the crash is at the end.
This crashes the server every single time, occurrence is 100%
If there is anything I can check or any pointers to how to investigate this further would be really helpful and appreciated.
Tomcat Version - 5.5.17
Java Version - 1.4.2_06-b03
If I have missed anything please point it out I will post it ASAP.
Thanks for the reply Ulf - the interesting part is that we are unable to reproduce it in our development environment and there are 5 more servers (exact replicas) successfully running this and handling request.
So basically I am looking or any ideas where I can narrow it down to something cause right now what I have done is that i have taken the checksum of all the libraries from the affected server and compared it with the ones on the working servers they seem to be the same so nothing has changed (virus etc.) we have to get the clients network team to get the system scanned for viruses.
We are also in the process of contacting the L1 System guys whose FaceIt library we are using but the client's primary point contact for who handles the support is not available at the moment.
That definitely looks like a problem in the DLL itself.
If the exact same setup works everywhere else, I'd take a good close look at the physical hardware. You might not have enough RAM, in which case an obscure short-on-memory bug might be exposed. Or, you might have flaky RAM. A thorough memory test is definitely recommended.
Some other possibilities would be disk problems, where critical code is getting corrupted when the DLL - or some element that the DLL trusts to be correct - is read into RAM. Or, the OS may be in need of installation (or removal!) of some sort of patch. It's even possible that the DLL is attempting to run like it's in 64-bit mode on a 32-bit OS and thereby wrapping high memory addresses down to 0.
Customer surveys are for companies who didn't pay proper attention to begin with.