File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
The moose likes Threads and Synchronization and the fly likes Rapid Increase In Blocked Threads Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Threads and Synchronization
Bookmark "Rapid Increase In Blocked Threads" Watch "Rapid Increase In Blocked Threads" New topic

Rapid Increase In Blocked Threads

Arpit Gadle
Ranch Hand

Joined: Dec 16, 2008
Posts: 69

Dear All,

First I would like to brief you all about what we are doing. We had developed a automatic call distribution
solution using

a) Open Source Telephony Platform FreeSWITCH (FS).

b) FreeSWITCH ESL(Event Socket Library).

c) A socket application developed in java

c) MySQL Database.

It works this way,

When a calls comes to FS server, it send out RFC messages with a header and key-value pairs in body to the
listener listening on the socket port throughout the life-cycle of the call. Upon receiving these messages, a new thread is spawned for every message it receives.

We had gone with these two approaches

(1) When a call enters the server, a call bean is created and added to hash-map. And on respective call activity such call ringing, call put on hold etc.,
the details are updated in the call bean in the hash-map. When the call hangs up this call details is inserted in MySQL database and finally removed from the call map.
The problem with this approach is that solution is not scalable.

(2) So we decided to move in to database for this, i.e logging and updating call details as and when they are received etc.
Here is the main problem, upon successful testing we put this in to production after every run for nearly 5-6 hours an OutOfMemory Exception is thrown and the application crashes.
Using htop the linux utility, we found that when this happens, the memory utilization is around 1-2 gig out of 48 gig on every run.
So the problem is not with the memory or java heap getting exhausted.

After struggling for few days we found that using htop, when OS thread count increase above 32K the application crashes by throwing an OutOfMemory exception.
But by going with approach (1) the thread count never increase beyond 800 and comes down to around 300.

We had checked our code for any unused reference, any exception caught, moved objects to methods definition but still there are blocked threads when going with approach (2).

Using JConsole and VisualVM is not of much help, as the number of threads spawned is high, an average 20-30 rfc messages are sent on socket for every single call activity.

Don't know what to do.

Any help is greatly appreciated

Thanks In Advance

Jayesh A Lalwani
Saloon Keeper

Joined: Jan 17, 2008
Posts: 2746

32K threads is a bad bad bad bad bad idea. The amount of thread contention between the threads is going to be horrific. Threads are not magical devices. THey are not going to create resources for you. You are always going to be constrained by the amount of computing resources on the server. Increasing the threads is not going to magically create more resources for you. Creating threads for every request is a bad idea for any application that needs to scale. Your threads should be fixed even though your requests aren't. Did I mention that 32K threads is a bad idea?

WHat you need to do is have a fixed thread pool, and submit jobs to a queue whenever something needs to be executed in the background comes in. The threads in the pool should poll the queue for a job, and pick it up when one comes in. Java provides an excellent class called ExecutorService that does exactly this. Study the API very very carefully.
Arpit Gadle
Ranch Hand

Joined: Dec 16, 2008
Posts: 69

Dear Jayesh,

Thanks for you reply. There is some misunderstanding, I am not the one who is spawning threads. I can't control these threads.

They are spawned by the event listener whenever the socket receives a message from the FS Server.

I also mentioned in my post that when going with approach (1) everything is working superb but as soon as I go for logging and updating call details

to database it keeps on increasing.

I agree. Here's the link:
subject: Rapid Increase In Blocked Threads
It's not a secret anymore!