• Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Monitoring threads in a pool and evicting hung threads

 
Nitesh Kant
Bartender
Posts: 1638
IntelliJ IDE Java MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I know that this has been discussed quite some time in one form or the other but on searching the forums i could not find an appropriate answer.

Consider the following scenario:
I have a framework where i allow people to drop code(implementing an interface) that can be executed at various places in my application flow. The code is given appropriate context to access the environment its running in. In a nutshell, its a container for running POJOs in an execution context.
The framework, spawns different threads for different application flows and executes the above POJOs in those threads. These POJOs can be stacked one after another and they have an option to run in a different thread or in the same thread as the POJO stack. I have a threadpool that does all this for me. Currently, i am using jdk 5 executors for this.
Now, my problem is the following:
  • Since, these POJOs can come from a third-party completely unaware of multi-threading and other container aspects. So, intentionally or un-intentionally they can just keep waiting on an I/O input or external intervention for indefinite periods.
  • If these POJOs are in "wait"/"sleep" i can send an interrupt signal and if the thread does not ignore the interruption(catching interrupted exception and moving ahead) i can get rid of the thread.
  • However, if its waiting for external intervention(eg: response from an external service), it will keep on waiting and unnecessarily block one of my threads.
  • If such a POJO exist in my application flow, it will result in a denial of service after sometime as i can not keep on increasing the number of threads to address the work load.
  • Waiting for a response from an external service is a valid scenario and if the POJO does not handle it properly then it may just block for indefinite periods, so i just cant disallow such a POJO.


  • What is the correct way of handling such a situation from the framework perspective?
    I can think of the following:
  • Monitor tasks that take unusually long time(configurable) to execute and does not respond to interrupt. Evict threads executing such tasks from the pool and let them finish on their own.
  • However, keep an account of such tasks and do not dedicate more than a configured amount of threads to such tasks till sanity prevails.
  • The above is an analogy to server too busy sort of a behavior in HTTP.


  • Am i treading the correct path? How do app servers handle such a situation?

    Any suggestion are more than welcome.
    Thanks,
    Nitesh
     
    Edward Harned
    Ranch Hand
    Posts: 291
    Java
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    And you think someone could answer this in twenty five words or less?

    This could take months to research, develop and test. Frameworks take man years to develop.

    In any case you have little or no control over Java threads. In C++ the thread control comes form libraries where programmers can modify and add modules. This is not the case in Java (thankfully.) The best you can do with a Java thread is to program it to recognize a stop request and return in the run method.
     
    Nitesh Kant
    Bartender
    Posts: 1638
    IntelliJ IDE Java MySQL Database
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator

    This could take months to research, develop and test. Frameworks take man years to develop.

    If that is the case then it should dive directly from the code base to recycle bin. I have never seen a product that take man years to develop a threadpool.

    The best you can do with a Java thread is to program it to recognize a stop request and return in the run method.

    I am totally cognizant of that fact. All i am trying to do is to make the framework adapt to a scenario where a component is taking enormously long time to execute. I just want to avoid a situation where the threadpool dedicates all its threads to a stuck component and results in a DOS.

    [ June 19, 2007: Message edited by: Nitesh Kant ]

    [ June 19, 2007: Message edited by: Nitesh Kant ]
    [ June 19, 2007: Message edited by: Nitesh Kant ]
     
    Edward Harned
    Ranch Hand
    Posts: 291
    Java
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Six programmers working on a task for two months is a man year.

    Traditional thread pools do not give you control over individual threads. You need to design your own where you have complete control (Java wise anyway) over each thread.

    I work on an open source project that does just that. It defines a Queue with a managed pool of threads. Clients add requests to the Queue and the well-managed threads pick up the requests, use reflection to execute a user-defined Class and return any object to the caller. Each stage of the life of a thread is timed. This way a monitor thread can flag threads that use excessive time.

    This is the product page. There is a link to download the product from SourceForge.net
    Tymeac Product

    You can use any of the code or just the idea.
     
    Nitesh Kant
    Bartender
    Posts: 1638
    IntelliJ IDE Java MySQL Database
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Six programmers working on a task for two months is a man year.

    Sure thats true. With my question i was in no way asking how to make such a framework, instead i was just verifying my thinking when it comes to running a third party component in a framework and handling blocking of such components.
    I surely believe that one does not need a man year to develop a thread pool. Anyways, lets not discuss on that.

    I work on an open source project that does just that. It defines a Queue with a managed pool of threads. Clients add requests to the Queue and the well-managed threads pick up the requests, use reflection to execute a user-defined Class and return any object to the caller. Each stage of the life of a thread is timed. This way a monitor thread can flag threads that use excessive time.

    This is the product page. There is a link to download the product from SourceForge.net
    Tymeac Product

    You can use any of the code or just the idea.

    I appreciate you providing the link to the above product. It sure looks interesting.
    However, what i am asking is a little different, probably an extension of what you are doing. I do not have a problem in identifying the time consuming/blocked task. I just want a few thoughts over how to handle a situation when a task is blocking/time consuming.
    Its pretty clear that with the stop() being deprecated, i can not safely stop a thread.
    Is it a good idea to evict such a thread from the pool.(This will make it not contribute to the pool size. Needless to say that the thread does not respond to an interrupt signal)
    Then monitor that thread separately. Till the time do not service request for similar tasks. When the task completes, assuming that the problem was temporary, bring back the thread to the pool and allow any further requests to such tasks.
    I just wanted a few thoughts on the above. But with the kind of response the topic has got, i think this is not appealing to others as it is to me.
    Anyways, thanks for your response edward
    [ June 21, 2007: Message edited by: Nitesh Kant ]
     
    Edward Harned
    Ranch Hand
    Posts: 291
    Java
    • Mark post as helpful
    • send pies
    • Quote
    • Report post to moderator
    Tymeac does what you are asking. It's a length, complex project and requires time to explore. There is extensive documentation.

    It monitors threads in a pool. When a thread misbehaves, it flags that thread as non-participating and notifies an administrator.

    There is only so much a general purpose framework can do. Sometimes it requires manual intervention.

    Every project has a balance: Software control/administrators.
     
    • Post Reply
    • Bookmark Topic Watch Topic
    • New Topic