• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

RejectedExecutionException thrown in bounded threadpool even when the # of tasks < coresize

 
Greenhorn
Posts: 3
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I created a bounded threadpool with a coresize of 50, maxsize of 55 and queuesize of 100 like below:

BlockingQueue<Runnable> blockingQueue = new LinkedBlockingQueue<Runnable>(100);

ExecutorService executor =
new ThreadPoolExecutor(50, 55, 30, TimeUnit.MILLISECONDS, blockingQueue, new ThreadPoolExecutor.CallerRunsPolicy());
... Callable pTask = new ServerProcessTask();

Future<IDataProcessor> future = executor.submit(pTask);
.... future.get(); //here RejectedExecutionException would be thrown

java.util.concurrent.ExecutionException:
java.util.concurrent.RejectedExecutionException
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:233)
java.util.concurrent.FutureTask.get(FutureTask.java:94)

but whenever I submit 30 tasks concurrently(some of tasks would run like 10 seconds) I'd see RejectedExecutionException, in which case I think the threadpool should be able to create more threads to process 30 concurrent tasks or have the requests queued.

any help appreciated!


Sean
 
Marshal
Posts: 28193
95
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Sean, welcome to the Ranch!

I'm going to have to speculate, because the answer to your question isn't obvious. The documentation says this:

New tasks submitted in method execute(java.lang.Runnable) will be rejected when the Executor has been shut down, and also when the Executor uses finite bounds for both maximum threads and work queue capacity, and is saturated.



So that suggests that the Executor is saturated. (I assume you have already checked to make sure it hasn't been shut down?) Perhaps 30 threads is more than your hardware can support, according to the Executor? I did say I was speculating. And it also seems to me that your ThreadPoolExecutor.CallerRunsPolicy should prevent the exception from being thrown in any case. Which is also mysterious. Although the documentation mentions "execute(Runnable)" and you are calling "submit(Callable)", so I guess that documentation quote is maybe not relevant.
 
Rancher
Posts: 1090
14
  • Likes 3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Like Paul said, I also find this behavior very strange. I think the CallerRunsPolicy should prevent the RejectedExecutionException from being thrown regardless of the number of tasks submitted or the core pool size or the bounded blocking queue size because a new task wouldn't be submitted till the one that is being run by the caller completes. Even if the tasks are being rejected because the pool is shut down, the task should be silently discarded.

So if you were storing the futures in a list in the submit loop, and you issued a shutdown mid way, the submits wouldn't throw any exception and the submit loop would complete for the number of tasks . But after this loop completes and you iterate over the list of futures to get the result, that loop would continue running, but no exceptions would be thrown. If you try to fetch the result of a cancelled task, you should get a different exception.

Just to be sure, I actually tested a similar program with a couple of combinations. I did not get the RejectedExecutionException for any case. I even increased the number of tasks to 3000 and still did not get the RejectedExecutionException or any other exception for that matter. Here is my test program.



Could you paste your your complete relevant code if that should give more clues or something? I'd be watching over this thread to read what others have to say.

Chan.
 
Chan Ag
Rancher
Posts: 1090
14
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
And welcome to CodeRanch.
 
Ranch Hand
Posts: 262
4
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
To add to what has been said, could you also paste the exception stack trace along with the relevant code.
I hope you don't have this line anywhere in your run/call method.

 
Sheriff
Posts: 10445
227
IntelliJ IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Heena Agarwal wrote:
I hope you don't have this line anywhere in your run/call method.



Given the stacktrace, I don't think that's the case.
 
Heena Agarwal
Ranch Hand
Posts: 262
4
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Jaikiran Pai wrote:

Heena Agarwal wrote:
I hope you don't have this line anywhere in your run/call method.



Given the stacktrace, I don't think that's the case.



This is interesting. Seriously. I must definitely be missing something very significant cause I don't know much.
But how is that short part of trace so telling?

I added the following line in the code posted above,



as follows -



and changed the number of tasks to 30 instead of 300. This is what I got in the stack trace.

queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
queue size = 0
counter =0
counter =4
counter =1
counter =2
counter =3
counter =5
counter =6
counter =7
counter =8
counter =10
counter =11
counter =9
counter =13
counter =12
counter =15
counter =14
counter =16
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
counter =17
counter =18
counter =19
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
java.util.concurrent.ExecutionException: java.util.concurrent.RejectedExecutionException
at java.util.concurrent.FutureTask.report(Unknown Source)
at java.util.concurrent.FutureTask.get(Unknown Source)
at threadandsynchronization.TestCallable.main(TestCallable.java:51)
Caused by: java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)
at threadandsynchronization.TestCallable.call(TestCallable.java:1)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)

 
Jaikiran Pai
Sheriff
Posts: 10445
227
IntelliJ IDE Ubuntu
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Heena Agarwal wrote:

Jaikiran Pai wrote:

Heena Agarwal wrote:
I hope you don't have this line anywhere in your run/call method.



Given the stacktrace, I don't think that's the case.



This is interesting. Seriously. Or I must be missing something very significant cause I don't know much.
But how is that short part of trace so telling?



The stacktrace posted by Sean has a small but important difference when compared against the one you are seeing in your test code. Notice the "root" cause in that stacktrace and the line which triggers that exception. In Sean's case it is:


java.util.concurrent.RejectedExecutionException
java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:233)



So it's line 233 of FutureTask class (which is a part of the JRE) which is the origin of that exception. Whereas in your case it is:


java.util.concurrent.RejectedExecutionException
at threadandsynchronization.TestCallable.call(TestCallable.java:26)



So line number 26 from a class within your application.

The relatively hard part lies in traversing the stacktrace to get to the root of it.
 
Heena Agarwal
Ranch Hand
Posts: 262
4
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I get it now. Thanks so much, Jaikiran, for spending time to explain it so well.

Like you said, the hard part is still remaining though.. :-)

 
author
Posts: 23951
142
jQuery Eclipse IDE Firefox Browser VI Editor C++ Chrome Java Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Heena Agarwal wrote:
Like you said, the hard part is still remaining though.. :-)




IMO, I think we will need more details from the OP, preferably a SSCCE, as this isn't an easily reproducible issue.

Henry
 
Rancher
Posts: 2759
32
Eclipse IDE Spring Tomcat Server
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Which version of Java are you using? Many times when I get weird problems like these, I just look at the Java source code to get a clue. All the code is publicly available. It would help if you know what version you are using. It would also help to download the Java sources yourself, and debug though the Java code.

 
Sean Chuang
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I'm using IBM JDK1.6.0.

I forget to let your guys know about the architecture. I'm building a distributed application on tomcat and Jersey restful services, which includes the clientApp and serverApp. the clientApp has a threadpool that would generate some concurrent restful service requests to a url like:http://locahost:8080/service/, and the serverApp provides the restful service, it also has a thread pool, right now the restful service implementation would just forward each request to be executed in the threadpool.

now I'm testing with my clientApp and serverApp co-located in the same pre-production box. and find that if I let clientApp generate 30 concurrent restful requests through its threadpool, the serverApp's threadpool cannot process the 30 concurrent requests, would throw RejectedExecutionException, while the clientApp threadpool woudn't. both threadpool are created in a similar way:

BlockingQueue<Runnable> blockingQueue = new LinkedBlockingQueue<Runnable>(100);

ExecutorService executor = new ThreadPoolExecutor(50, 55, 30, TimeUnit.MILLISECONDS, blockingQueue, new ThreadPoolExecutor.CallerRunsPolicy());

is it because the JVM only allow one threadpool to work properly?


Sean
 
Ranch Hand
Posts: 121
12
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Sean Chuang wrote:is it because the JVM only allow one threadpool to work properly?



No, JVM have no such restriction. And most likely both following statements are true:
1. Rejected execution is thrown inside your ServerProcessTask.call method. Heena have already pointed on this possibility.
2. Stack trace your provided is incomplete for whatever reason (misconfigured logging, default stack trace format used in IBM jvm, etc...).

Trace you provided (

Sean Chuang wrote:
java.util.concurrent.ExecutionException:
java.util.concurrent.RejectedExecutionException


) looks exactly like a message for ExecutionException (which means that task execution was failed) caused by the RejectedExecutionException (which was raised for unknown-yet reason). Probably this message should be in one line, not two. Small search shows, that there are implementations of the Executors consistent with your stack trace. One of the jsr166 implementations throws ExecutionException with the cause at line 233. And this is not a "rejected execution" of executor in your example, this is an exception thrown by the call (or run) method in an actual task (which you did not show us in the same example).

You should either find real cause in the message (See Heena's example, there is a good stack trace with "Caused-by" section) or wrap all "call" method code in your ServerProcessTask to catch and log all exceptions (you may rethrow them after logging).
 
Paul Clapham
Marshal
Posts: 28193
95
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I have to say, I'm so impressed with this thread. So many good responses! If there was a way to give cows to a thread I would be doing that...
 
Chan Ag
Rancher
Posts: 1090
14
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
What Paul said++.

Thanks Maxim.

[ Edit - I'm not stealing this topic. We're back on the topic. So Sean, I believe the above responses might help you to get to the root cause of the RejectedExecutionException you were getting. Please keep us posted of what you find. ]

Chan.
 
Sean Chuang
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
thank you for all the helpful advice.

I eventually found out the issue. It's because in addition to the two threadpools created using ThreadPoolExecutor class I mentioned before, there's another threadpool created using an in-house solution, which for unknown reasons apparently interfere with the previous two threadpools, as once I removed it, the two ThreadPoolExecutor threadpools would be working properly without RejectedExecutionException.
 
Consider Paul's rocket mass heater.
reply
    Bookmark Topic Watch Topic
  • New Topic