Occasionally, my stand-alone Java application will just go nuts and drag down the entire server. It's just a Java app running as a Windows Service (using jsl.exe), running on Win 2000, using Sun JRE 1.5.0_17.
So I have this application that listens on a couple of ports with a ServerSocket and SSLServerSocket. Those listening threads, in turn, fork a thread for each socket request that comes in. There's also a couple of threads that go to sleep and wake up periodically to do logging or back-end MQ integration.
So anyway, out of 1000+ of these servers, every once in a while, this will happen (2-5 in a month). In our QA Lab, we haven't been able to reproduce this problem, but I noticed that there seem to be 'dangling threads' that never seem to away, even after a few hours. I'm trying to figure out if this is a quirk of the Sun Java JRE or if I'm doing something wrong? There's no sense to these threads. That is, the dangling threads are striped across different pieces of code, so it doesn't seem like an infinite loop in a specific place. Furthermore, memory doesn't seem to appreciate greatly either.
So I'm implementing a Windows Scheduled Task to restart the Java app every 24 hours, as a safeguard. The theory is that eventually, these dangling threads add up, and after several weeks or months, they eventually start to drag down the JVM, just in terms the bookkeeping cost of keeping them around. Not the greatest theory, but the best I've got right now.
Does anyone know of the JRE 1.5 having a known issue close to this, or have any other theories? Can I blame this on the thread-handling of the JVM, or should I keep hunting for a flaw in my code?
BTW, I don't know the thread state at this time, but I'm going the thing where I get the list of threads and print out their CPU time for each thread. I'm noticing that there are threads that are still on the list several hours after the thread should have been expired and garbage-collected, despite a fair amount of server activity.
Example log output: (Threads 2-11 are my server/JVM threads)
Thread ID=1809, CPU Time=0
Thread ID=1787, CPU Time=0
Thread ID=1758, CPU Time=0
Thread ID=1752, CPU Time=0
Thread ID=738, CPU Time=0
Thread ID=734, CPU Time=0
Thread ID=724, CPU Time=0
Thread ID=11, CPU Time=578125000
Thread ID=10, CPU Time=0
Thread ID=9, CPU Time=0
Thread ID=8, CPU Time=2109375000
Thread ID=7, CPU Time=171875000
Thread ID=4, CPU Time=0
Thread ID=3, CPU Time=156250000
Thread ID=2, CPU Time=46875000