wood burning stoves 2.0*
The moose likes Threads and Synchronization and the fly likes RejectedExecutionException thrown in bounded threadpool even when the # of tasks < coresize Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


JavaRanch » Java Forums » Java » Threads and Synchronization
Bookmark "RejectedExecutionException thrown in bounded threadpool even when the # of tasks < coresize" Watch "RejectedExecutionException thrown in bounded threadpool even when the # of tasks < coresize" New topic
Author

RejectedExecutionException thrown in bounded threadpool even when the # of tasks < coresize

Sean Chuang
Greenhorn

Joined: Jan 15, 2014
Posts: 3
I created a bounded threadpool with a coresize of 50, maxsize of 55 and queuesize of 100 like below:

BlockingQueue<Runnable> blockingQueue = new LinkedBlockingQueue<Runnable>(100);

ExecutorService executor =
new ThreadPoolExecutor(50, 55, 30, TimeUnit.MILLISECONDS, blockingQueue, new ThreadPoolExecutor.CallerRunsPolicy());
... Callable pTask = new ServerProcessTask();

Future<IDataProcessor> future = executor.submit(pTask);
.... future.get(); //here RejectedExecutionException would be thrown

java.util.concurrent.ExecutionException:
java.util.concurrent.RejectedExecutionException
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:233)
java.util.concurrent.FutureTask.get(FutureTask.java:94)

but whenever I submit 30 tasks concurrently(some of tasks would run like 10 seconds) I'd see RejectedExecutionException, in which case I think the threadpool should be able to create more threads to process 30 concurrent tasks or have the requests queued.

any help appreciated!


Sean
Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18570
    
    8

Hi Sean, welcome to the Ranch!

I'm going to have to speculate, because the answer to your question isn't obvious. The documentation says this:

New tasks submitted in method execute(java.lang.Runnable) will be rejected when the Executor has been shut down, and also when the Executor uses finite bounds for both maximum threads and work queue capacity, and is saturated.


So that suggests that the Executor is saturated. (I assume you have already checked to make sure it hasn't been shut down?) Perhaps 30 threads is more than your hardware can support, according to the Executor? I did say I was speculating. And it also seems to me that your ThreadPoolExecutor.CallerRunsPolicy should prevent the exception from being thrown in any case. Which is also mysterious. Although the documentation mentions "execute(Runnable)" and you are calling "submit(Callable)", so I guess that documentation quote is maybe not relevant.
Chan Ag
Bartender

Joined: Sep 06, 2012
Posts: 1018
    
  15
Like Paul said, I also find this behavior very strange. I think the CallerRunsPolicy should prevent the RejectedExecutionException from being thrown regardless of the number of tasks submitted or the core pool size or the bounded blocking queue size because a new task wouldn't be submitted till the one that is being run by the caller completes. Even if the tasks are being rejected because the pool is shut down, the task should be silently discarded.

So if you were storing the futures in a list in the submit loop, and you issued a shutdown mid way, the submits wouldn't throw any exception and the submit loop would complete for the number of tasks . But after this loop completes and you iterate over the list of futures to get the result, that loop would continue running, but no exceptions would be thrown. If you try to fetch the result of a cancelled task, you should get a different exception.

Just to be sure, I actually tested a similar program with a couple of combinations. I did not get the RejectedExecutionException for any case. I even increased the number of tasks to 3000 and still did not get the RejectedExecutionException or any other exception for that matter. Here is my test program.



Could you paste your your complete relevant code if that should give more clues or something? I'd be watching over this thread to read what others have to say.

Chan.
Chan Ag
Bartender

Joined: Sep 06, 2012
Posts: 1018
    
  15
And welcome to CodeRanch.
Heena Agarwal
Ranch Hand

Joined: Dec 25, 2013
Posts: 261
    
    4
To add to what has been said, could you also paste the exception stack trace along with the relevant code.
I hope you don't have this line anywhere in your run/call method.

Jaikiran Pai
Marshal

Joined: Jul 20, 2005
Posts: 10141
    
165

Heena Agarwal wrote:
I hope you don't have this line anywhere in your run/call method.



Given the stacktrace, I don't think that's the case.

[My Blog] [JavaRanch Journal]
Heena Agarwal
Ranch Hand

Joined: Dec 25, 2013
Posts: 261
    
    4
Jaikiran Pai wrote:
Heena Agarwal wrote:
I hope you don't have this line anywhere in your run/call method.



Given the stacktrace, I don't think that's the case.


This is interesting. Seriously. I must definitely be missing something very significant cause I don't know much.
But how is that short part of trace so telling?

I added the following line in the code posted above,



as follows -



and changed the number of tasks to 30 instead of 300. This is what I got in the stack trace.

queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
counter =0
counter =4
counter =1
counter =2
counter =3
counter =5
counter =6
counter =7
counter =8
counter =10
counter =11
counter =9
counter =13
counter =12
counter =15
counter =14
counter =16
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
counter =17
counter =18
counter =19
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

Jaikiran Pai
Marshal

Joined: Jul 20, 2005
Posts: 10141
    
165

Heena Agarwal wrote:
Jaikiran Pai wrote:
Heena Agarwal wrote:
I hope you don't have this line anywhere in your run/call method.



Given the stacktrace, I don't think that's the case.


This is interesting. Seriously. Or I must be missing something very significant cause I don't know much.
But how is that short part of trace so telling?


The stacktrace posted by Sean has a small but important difference when compared against the one you are seeing in your test code. Notice the "root" cause in that stacktrace and the line which triggers that exception. In Sean's case it is:

java.util.concurrent.RejectedExecutionException
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:233)


So it's line 233 of FutureTask class (which is a part of the JRE) which is the origin of that exception. Whereas in your case it is:

java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)


So line number 26 from a class within your application.

The relatively hard part lies in traversing the stacktrace to get to the root of it.
Heena Agarwal
Ranch Hand

Joined: Dec 25, 2013
Posts: 261
    
    4
I get it now. Thanks so much, Jaikiran, for spending time to explain it so well.

Like you said, the hard part is still remaining though.. :-)

Henry Wong
author
Sheriff

Joined: Sep 28, 2004
Posts: 18874
    
  40

Heena Agarwal wrote:
Like you said, the hard part is still remaining though.. :-)



IMO, I think we will need more details from the OP, preferably a SSCCE, as this isn't an easily reproducible issue.

Henry


Books: Java Threads, 3rd Edition, Jini in a Nutshell, and Java Gems (contributor)
Jayesh A Lalwani
Bartender

Joined: Jan 17, 2008
Posts: 2383
    
  28

Which version of Java are you using? Many times when I get weird problems like these, I just look at the Java source code to get a clue. All the code is publicly available. It would help if you know what version you are using. It would also help to download the Java sources yourself, and debug though the Java code.

Sean Chuang
Greenhorn

Joined: Jan 15, 2014
Posts: 3
I'm using IBM JDK1.6.0.

I forget to let your guys know about the architecture. I'm building a distributed application on tomcat and Jersey restful services, which includes the clientApp and serverApp. the clientApp has a threadpool that would generate some concurrent restful service requests to a url like:http://locahost:8080/service/, and the serverApp provides the restful service, it also has a thread pool, right now the restful service implementation would just forward each request to be executed in the threadpool.

now I'm testing with my clientApp and serverApp co-located in the same pre-production box. and find that if I let clientApp generate 30 concurrent restful requests through its threadpool, the serverApp's threadpool cannot process the 30 concurrent requests, would throw RejectedExecutionException, while the clientApp threadpool woudn't. both threadpool are created in a similar way:

BlockingQueue<Runnable> blockingQueue = new LinkedBlockingQueue<Runnable>(100);

ExecutorService executor = new ThreadPoolExecutor(50, 55, 30, TimeUnit.MILLISECONDS, blockingQueue, new ThreadPoolExecutor.CallerRunsPolicy());

is it because the JVM only allow one threadpool to work properly?


Sean
Maxim Karvonen
Ranch Hand

Joined: Jun 14, 2013
Posts: 103
    
  11
Sean Chuang wrote:is it because the JVM only allow one threadpool to work properly?


No, JVM have no such restriction. And most likely both following statements are true:
1. Rejected execution is thrown inside your ServerProcessTask.call method. Heena have already pointed on this possibility.
2. Stack trace your provided is incomplete for whatever reason (misconfigured logging, default stack trace format used in IBM jvm, etc...).

Trace you provided (

) looks exactly like a message for ExecutionException (which means that task execution was failed) caused by the RejectedExecutionException (which was raised for unknown-yet reason). Probably this message should be in one line, not two. Small search shows, that there are implementations of the Executors consistent with your stack trace. One of the jsr166 implementations throws ExecutionException with the cause at line 233. And this is not a "rejected execution" of executor in your example, this is an exception thrown by the call (or run) method in an actual task (which you did not show us in the same example).

You should either find real cause in the message (See Heena's example, there is a good stack trace with "Caused-by" section) or wrap all "call" method code in your ServerProcessTask to catch and log all exceptions (you may rethrow them after logging).
Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18570
    
    8

I have to say, I'm so impressed with this thread. So many good responses! If there was a way to give cows to a thread I would be doing that...
Chan Ag
Bartender

Joined: Sep 06, 2012
Posts: 1018
    
  15
What Paul said++.

Thanks Maxim.

[ Edit - I'm not stealing this topic. We're back on the topic. So Sean, I believe the above responses might help you to get to the root cause of the RejectedExecutionException you were getting. Please keep us posted of what you find. ]

Chan.
Sean Chuang
Greenhorn

Joined: Jan 15, 2014
Posts: 3
thank you for all the helpful advice.

I eventually found out the issue. It's because in addition to the two threadpools created using ThreadPoolExecutor class I mentioned before, there's another threadpool created using an in-house solution, which for unknown reasons apparently interfere with the previous two threadpools, as once I removed it, the two ThreadPoolExecutor threadpools would be working properly without RejectedExecutionException.
 
GeeCON Prague 2014
 
subject: RejectedExecutionException thrown in bounded threadpool even when the # of tasks < coresize