Yesterday one of our applications stopped working. It's working now, but I've no idea why it stopped working, and why it's just started working. I'm posting here to see if anyone can shed any light.
The application runs on tomcat 6.0.20 and uses Hibernate 3.2.1, connecting to Oracle 10.1. The application has a very small number of users (~20 or so, only a few concurrent). The application stopped responding yesterday morning, the browser was showing that the proxy was timing out.
We found the reason for this was the application had a row lock on several tables in Oracle and a queue of requests behind it. Any further requests wern't being processed and tomcat was waiting on these.
We turned off the tomcat server and checked the database again. All connections and locks had cleared. We then restarted the application and tested again. The application was still not working, and created more locks in Oracle. We also tried removing the application folder from tomcat and letting it rebuild from the .war file. This didn't work either.
As there had been no code changes to the application for over a year, I assumed it was a data issue. I got the production data copied to a test environment, with a test tomcat server and tried the application there. It showed the same problem. Next, I connected my local tomcat server to this test data so I could step through the code and see where it was stopping. I turned off the test tomcat server while I did this.
The application ran fine from my local tomcat server, so I couldn't see where the problem was. I then turned my local tomcat off and restarted the test tomcat server. The application then started working fine in the test environment.
I then turned off the production tomcat server and connected my local server to the production database. Again, I wanted to step through the code to see where it was erroring, but again my local version was running without error.
We then restarted the production server and it was working...
Has anyone got any idea what might be happening here? Anyone seen anything like this before?