When a calls comes to FS server, it send out RFC messages with a header and key-value pairs in body to the
listener listening on the socket port throughout the life-cycle of the call. Upon receiving these messages, a new thread is spawned for every message it receives.
We had gone with these two approaches
(1) When a call enters the server, a call bean is created and added to hash-map. And on respective call activity such call ringing, call put on hold etc.,
the details are updated in the call bean in the hash-map. When the call hangs up this call details is inserted in MySQL database and finally removed from the call map.
The problem with this approach is that solution is not scalable.
(2) So we decided to move in to database for this, i.e logging and updating call details as and when they are received etc.
Here is the main problem, upon successful testing we put this in to production after every run for nearly 5-6 hours an OutOfMemory Exception is thrown and the application crashes.
Using htop the linux utility, we found that when this happens, the memory utilization is around 1-2 gig out of 48 gig on every run.
So the problem is not with the memory or java heap getting exhausted.
After struggling for few days we found that using htop, when OS thread count increase above 32K the application crashes by throwing an OutOfMemory exception.
But by going with approach (1) the thread count never increase beyond 800 and comes down to around 300.
We had checked our code for any unused reference, any exception caught, moved objects to methods definition but still there are blocked threads when going with approach (2).
Using JConsole and VisualVM is not of much help, as the number of threads spawned is high, an average 20-30 rfc messages are sent on socket for every single call activity.
32K threads is a bad bad bad bad bad idea. The amount of thread contention between the threads is going to be horrific. Threads are not magical devices. THey are not going to create resources for you. You are always going to be constrained by the amount of computing resources on the server. Increasing the threads is not going to magically create more resources for you. Creating threads for every request is a bad idea for any application that needs to scale. Your threads should be fixed even though your requests aren't. Did I mention that 32K threads is a bad idea?
WHat you need to do is have a fixed thread pool, and submit jobs to a queue whenever something needs to be executed in the background comes in. The threads in the pool should poll the queue for a job, and pick it up when one comes in. Java provides an excellent class called ExecutorService that does exactly this. Study the API very very carefully.