Win a copy of Design for the Mind this week in the Design forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Producer Consumer Controller

 
Kasi Viswan
Ranch Hand
Posts: 42
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hello,

I need some help with this Producer Consumer Controller code.

I need to perform Task A on all the records of an input file. Input file may contain millions of records.
I need to be able to run this process at a steady thread count of say 20 threads.

I wrote three classes - Producer, Consumer and Controller







I do not know how to make the program wait till the producer and consumer completes their task and then exit the VM.
If I set the threadcount less than the number of records in the file, the program exits properly after processing all the records.
If I set threadcount equal to the number of records in file, the program never exits, even though all the records were processed.

Can someone please tell me how to get past this, Thank you.

Regards
Kasi
 
Mike Peters
Ranch Hand
Posts: 67
Debian Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Maybe removing c.join() will speed up your shutdown process. You have a deadlock there.
 
Jim Hoglund
Ranch Hand
Posts: 525
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Kasi : Where are you with your project? Have you solved your
problem or do you still need some pointers?

Jim ... ...
 
Kasi Viswan
Ranch Hand
Posts: 42
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thank you Jim, Mike was right on the mark, removing join statement on the consumer made it work.

It would be great, if you can help me in the same app how to vary number of consumer threads at run time.

E.g. Increase/Decrease the consumer count based on TPS.
Lets say, i started this process, with 10 consumers and i find that based on some parameters that i can start few more consumers or reduce the number of consumers, how can i do that, Please advice.

Thanks
Kasi
 
Sandeep Sanaboyina
Ranch Hand
Posts: 72
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Try using Executors. Check out this link.

http://cupi2.uniandes.edu.co/site/images/recursos/javadoc/j2se/1.5.0/docs/api/java/util/concurrent/Executors.html#newCachedThreadPool(java.util.concurrent.ThreadFactory)
 
Kasi Viswan
Ranch Hand
Posts: 42
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Sandeep,

Sorry, I couldn't figure out how to use newCachedThreadPool() for this scenario.

There is a set of producers injecting tasks into a blockingqueue of size 10.
There is a set of consumers removing these tasks from the queue and executing these tasks.

I want to know how to increase the number of consumers while the application is running.
Do you have a sample code or pseudo code that i can refer?

Thanks
Kasi
 
Sandeep Sanaboyina
Ranch Hand
Posts: 72
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I'm giving the sample code for ThreadPoolExecutor. Run it once and see how new threads are created.

Also look up javadocs for ThreadPoolExecutor.

Hope it helps.

 
Kasi Viswan
Ranch Hand
Posts: 42
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thank you Sandeep for your detailed explanation. I understood your suggestion to use for loop and keep calling more execute() in the executor to increase number of consumers.

If there is a way to increase the blockingqueue size at runtime, I need not change any of my producer consumer setup.

Thanks again,
Kasi
 
Jim Hoglund
Ranch Hand
Posts: 525
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Kasi : Can you read a parameter file periodically, or pass in a queue size on the command
line at start-up? You could then construct a new (larger/smaller) blocking queue, transfer
any pending jobs to it, and shut down the old queue. What do you think?

Jim ... ...
 
Kasi Viswan
Ranch Hand
Posts: 42
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Good Idea Jim, I like the periodic parameter file reading and transferring the tasks to new queue.

Thanks
Kasi
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic