File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Threads and Synchronization and the fly likes Producer Consumer Controller Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Soft Skills this week in the Jobs Discussion forum!
JavaRanch » Java Forums » Java » Threads and Synchronization
Bookmark "Producer Consumer Controller" Watch "Producer Consumer Controller" New topic
Author

Producer Consumer Controller

Kasi Viswan
Ranch Hand

Joined: Sep 27, 2008
Posts: 42
Hello,

I need some help with this Producer Consumer Controller code.

I need to perform Task A on all the records of an input file. Input file may contain millions of records.
I need to be able to run this process at a steady thread count of say 20 threads.

I wrote three classes - Producer, Consumer and Controller







I do not know how to make the program wait till the producer and consumer completes their task and then exit the VM.
If I set the threadcount less than the number of records in the file, the program exits properly after processing all the records.
If I set threadcount equal to the number of records in file, the program never exits, even though all the records were processed.

Can someone please tell me how to get past this, Thank you.

Regards
Kasi
Mike Peters
Ranch Hand

Joined: Oct 10, 2009
Posts: 67

Maybe removing c.join() will speed up your shutdown process. You have a deadlock there.


Mike Peters
Jim Hoglund
Ranch Hand

Joined: Jan 09, 2008
Posts: 525
Kasi : Where are you with your project? Have you solved your
problem or do you still need some pointers?

Jim ... ...


BEE MBA PMP SCJP-6
Kasi Viswan
Ranch Hand

Joined: Sep 27, 2008
Posts: 42
Thank you Jim, Mike was right on the mark, removing join statement on the consumer made it work.

It would be great, if you can help me in the same app how to vary number of consumer threads at run time.

E.g. Increase/Decrease the consumer count based on TPS.
Lets say, i started this process, with 10 consumers and i find that based on some parameters that i can start few more consumers or reduce the number of consumers, how can i do that, Please advice.

Thanks
Kasi
Sandeep Sanaboyina
Ranch Hand

Joined: Dec 14, 2009
Posts: 72
Try using Executors. Check out this link.

http://cupi2.uniandes.edu.co/site/images/recursos/javadoc/j2se/1.5.0/docs/api/java/util/concurrent/Executors.html#newCachedThreadPool(java.util.concurrent.ThreadFactory)


They say you have to be the first, the best or different. I say, is it too much to ask for all three.
Kasi Viswan
Ranch Hand

Joined: Sep 27, 2008
Posts: 42
Hi Sandeep,

Sorry, I couldn't figure out how to use newCachedThreadPool() for this scenario.

There is a set of producers injecting tasks into a blockingqueue of size 10.
There is a set of consumers removing these tasks from the queue and executing these tasks.

I want to know how to increase the number of consumers while the application is running.
Do you have a sample code or pseudo code that i can refer?

Thanks
Kasi
Sandeep Sanaboyina
Ranch Hand

Joined: Dec 14, 2009
Posts: 72
I'm giving the sample code for ThreadPoolExecutor. Run it once and see how new threads are created.

Also look up javadocs for ThreadPoolExecutor.

Hope it helps.

Kasi Viswan
Ranch Hand

Joined: Sep 27, 2008
Posts: 42
Thank you Sandeep for your detailed explanation. I understood your suggestion to use for loop and keep calling more execute() in the executor to increase number of consumers.

If there is a way to increase the blockingqueue size at runtime, I need not change any of my producer consumer setup.

Thanks again,
Kasi
Jim Hoglund
Ranch Hand

Joined: Jan 09, 2008
Posts: 525
Kasi : Can you read a parameter file periodically, or pass in a queue size on the command
line at start-up? You could then construct a new (larger/smaller) blocking queue, transfer
any pending jobs to it, and shut down the old queue. What do you think?

Jim ... ...
Kasi Viswan
Ranch Hand

Joined: Sep 27, 2008
Posts: 42
Good Idea Jim, I like the periodic parameter file reading and transferring the tasks to new queue.

Thanks
Kasi
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Producer Consumer Controller