• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Rapid Increase In Blocked Threads

 
Ranch Hand
Posts: 69
MySQL Database Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Dear All,


First I would like to brief you all about what we are doing. We had developed a automatic call distribution
solution using

a) Open Source Telephony Platform FreeSWITCH (FS).

b) FreeSWITCH ESL(Event Socket Library).

c) A socket application developed in java

c) MySQL Database.

It works this way,

When a calls comes to FS server, it send out RFC messages with a header and key-value pairs in body to the
listener listening on the socket port throughout the life-cycle of the call. Upon receiving these messages, a new thread is spawned for every message it receives.

We had gone with these two approaches

(1) When a call enters the server, a call bean is created and added to hash-map. And on respective call activity such call ringing, call put on hold etc.,
the details are updated in the call bean in the hash-map. When the call hangs up this call details is inserted in MySQL database and finally removed from the call map.
The problem with this approach is that solution is not scalable.

(2) So we decided to move in to database for this, i.e logging and updating call details as and when they are received etc.
Here is the main problem, upon successful testing we put this in to production after every run for nearly 5-6 hours an OutOfMemory Exception is thrown and the application crashes.
Using htop the linux utility, we found that when this happens, the memory utilization is around 1-2 gig out of 48 gig on every run.
So the problem is not with the memory or java heap getting exhausted.

After struggling for few days we found that using htop, when OS thread count increase above 32K the application crashes by throwing an OutOfMemory exception.
But by going with approach (1) the thread count never increase beyond 800 and comes down to around 300.

We had checked our code for any unused reference, any exception caught, moved objects to methods definition but still there are blocked threads when going with approach (2).

Using JConsole and VisualVM is not of much help, as the number of threads spawned is high, an average 20-30 rfc messages are sent on socket for every single call activity.


Don't know what to do.

Any help is greatly appreciated



Thanks In Advance

Arpit
 
Rancher
Posts: 2759
32
Eclipse IDE Spring Tomcat Server
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
32K threads is a bad bad bad bad bad idea. The amount of thread contention between the threads is going to be horrific. Threads are not magical devices. THey are not going to create resources for you. You are always going to be constrained by the amount of computing resources on the server. Increasing the threads is not going to magically create more resources for you. Creating threads for every request is a bad idea for any application that needs to scale. Your threads should be fixed even though your requests aren't. Did I mention that 32K threads is a bad idea?

WHat you need to do is have a fixed thread pool, and submit jobs to a queue whenever something needs to be executed in the background comes in. The threads in the pool should poll the queue for a job, and pick it up when one comes in. Java provides an excellent class called ExecutorService that does exactly this. Study the API very very carefully.
 
Arpit Gadle
Ranch Hand
Posts: 69
MySQL Database Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator


Dear Jayesh,

Thanks for you reply. There is some misunderstanding, I am not the one who is spawning threads. I can't control these threads.

They are spawned by the event listener whenever the socket receives a message from the FS Server.

I also mentioned in my post that when going with approach (1) everything is working superb but as soon as I go for logging and updating call details

to database it keeps on increasing.


 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
reply
    Bookmark Topic Watch Topic
  • New Topic