File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Threads and Synchronization and the fly likes Monitoring threads in a pool and evicting hung threads Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Threads and Synchronization
Bookmark "Monitoring threads in a pool and evicting hung threads" Watch "Monitoring threads in a pool and evicting hung threads" New topic
Author

Monitoring threads in a pool and evicting hung threads

Nitesh Kant
Bartender

Joined: Feb 25, 2007
Posts: 1638

I know that this has been discussed quite some time in one form or the other but on searching the forums i could not find an appropriate answer.

Consider the following scenario:
I have a framework where i allow people to drop code(implementing an interface) that can be executed at various places in my application flow. The code is given appropriate context to access the environment its running in. In a nutshell, its a container for running POJOs in an execution context.
The framework, spawns different threads for different application flows and executes the above POJOs in those threads. These POJOs can be stacked one after another and they have an option to run in a different thread or in the same thread as the POJO stack. I have a threadpool that does all this for me. Currently, i am using jdk 5 executors for this.
Now, my problem is the following:
  • Since, these POJOs can come from a third-party completely unaware of multi-threading and other container aspects. So, intentionally or un-intentionally they can just keep waiting on an I/O input or external intervention for indefinite periods.
  • If these POJOs are in "wait"/"sleep" i can send an interrupt signal and if the thread does not ignore the interruption(catching interrupted exception and moving ahead) i can get rid of the thread.
  • However, if its waiting for external intervention(eg: response from an external service), it will keep on waiting and unnecessarily block one of my threads.
  • If such a POJO exist in my application flow, it will result in a denial of service after sometime as i can not keep on increasing the number of threads to address the work load.
  • Waiting for a response from an external service is a valid scenario and if the POJO does not handle it properly then it may just block for indefinite periods, so i just cant disallow such a POJO.


  • What is the correct way of handling such a situation from the framework perspective?
    I can think of the following:
  • Monitor tasks that take unusually long time(configurable) to execute and does not respond to interrupt. Evict threads executing such tasks from the pool and let them finish on their own.
  • However, keep an account of such tasks and do not dedicate more than a configured amount of threads to such tasks till sanity prevails.
  • The above is an analogy to server too busy sort of a behavior in HTTP.


  • Am i treading the correct path? How do app servers handle such a situation?

    Any suggestion are more than welcome.
    Thanks,
    Nitesh


    apigee, a better way to API!
    Edward Harned
    Ranch Hand

    Joined: Sep 19, 2005
    Posts: 291

    And you think someone could answer this in twenty five words or less?

    This could take months to research, develop and test. Frameworks take man years to develop.

    In any case you have little or no control over Java threads. In C++ the thread control comes form libraries where programmers can modify and add modules. This is not the case in Java (thankfully.) The best you can do with a Java thread is to program it to recognize a stop request and return in the run method.


    Ed's latest article: A Java Parallel Calamity http://coopsoft.com/ar/Calamity2Article.html
    Nitesh Kant
    Bartender

    Joined: Feb 25, 2007
    Posts: 1638


    This could take months to research, develop and test. Frameworks take man years to develop.

    If that is the case then it should dive directly from the code base to recycle bin. I have never seen a product that take man years to develop a threadpool.

    The best you can do with a Java thread is to program it to recognize a stop request and return in the run method.

    I am totally cognizant of that fact. All i am trying to do is to make the framework adapt to a scenario where a component is taking enormously long time to execute. I just want to avoid a situation where the threadpool dedicates all its threads to a stuck component and results in a DOS.

    [ June 19, 2007: Message edited by: Nitesh Kant ]

    [ June 19, 2007: Message edited by: Nitesh Kant ]
    [ June 19, 2007: Message edited by: Nitesh Kant ]
    Edward Harned
    Ranch Hand

    Joined: Sep 19, 2005
    Posts: 291

    Six programmers working on a task for two months is a man year.

    Traditional thread pools do not give you control over individual threads. You need to design your own where you have complete control (Java wise anyway) over each thread.

    I work on an open source project that does just that. It defines a Queue with a managed pool of threads. Clients add requests to the Queue and the well-managed threads pick up the requests, use reflection to execute a user-defined Class and return any object to the caller. Each stage of the life of a thread is timed. This way a monitor thread can flag threads that use excessive time.

    This is the product page. There is a link to download the product from SourceForge.net
    Tymeac Product

    You can use any of the code or just the idea.
    Nitesh Kant
    Bartender

    Joined: Feb 25, 2007
    Posts: 1638

    Six programmers working on a task for two months is a man year.

    Sure thats true. With my question i was in no way asking how to make such a framework, instead i was just verifying my thinking when it comes to running a third party component in a framework and handling blocking of such components.
    I surely believe that one does not need a man year to develop a thread pool. Anyways, lets not discuss on that.

    I work on an open source project that does just that. It defines a Queue with a managed pool of threads. Clients add requests to the Queue and the well-managed threads pick up the requests, use reflection to execute a user-defined Class and return any object to the caller. Each stage of the life of a thread is timed. This way a monitor thread can flag threads that use excessive time.

    This is the product page. There is a link to download the product from SourceForge.net
    Tymeac Product

    You can use any of the code or just the idea.

    I appreciate you providing the link to the above product. It sure looks interesting.
    However, what i am asking is a little different, probably an extension of what you are doing. I do not have a problem in identifying the time consuming/blocked task. I just want a few thoughts over how to handle a situation when a task is blocking/time consuming.
    Its pretty clear that with the stop() being deprecated, i can not safely stop a thread.
    Is it a good idea to evict such a thread from the pool.(This will make it not contribute to the pool size. Needless to say that the thread does not respond to an interrupt signal)
    Then monitor that thread separately. Till the time do not service request for similar tasks. When the task completes, assuming that the problem was temporary, bring back the thread to the pool and allow any further requests to such tasks.
    I just wanted a few thoughts on the above. But with the kind of response the topic has got, i think this is not appealing to others as it is to me.
    Anyways, thanks for your response edward
    [ June 21, 2007: Message edited by: Nitesh Kant ]
    Edward Harned
    Ranch Hand

    Joined: Sep 19, 2005
    Posts: 291

    Tymeac does what you are asking. It's a length, complex project and requires time to explore. There is extensive documentation.

    It monitors threads in a pool. When a thread misbehaves, it flags that thread as non-participating and notifies an administrator.

    There is only so much a general purpose framework can do. Sometimes it requires manual intervention.

    Every project has a balance: Software control/administrators.
     
     
    subject: Monitoring threads in a pool and evicting hung threads