File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Threads and Synchronization and the fly likes producer-multiple consumers Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Threads and Synchronization
Bookmark "producer-multiple consumers" Watch "producer-multiple consumers" New topic
Author

producer-multiple consumers

Ugender Rekulampally
Ranch Hand

Joined: Nov 14, 2005
Posts: 130
Hello All,

I have a situation with my synchronization.

My program has a reader and worker threads. reader should be one thread and workers are more than one.

reader thread reads data into 10 arrays a certain number of elements. lets say in this case, reader reades 3 elements from the file and puts in arrays. once its done with reading, it has to go to wait() mode and writerr/workers should start processing the arrays. once workers finish their processing, reader thread should read another round of data and workers start processing once reader goes wait(). this process should continue till reader finishes the input file.

now what my program doing is

lets assusme I have one reader and two worker/writer threads.
here are the execution steps:

1) reader reads
2) reader goes wait()
3) writer1 processes
4) writer1 goes wait()
5) writer2 goes wait()

6) reader reades
7) reader goes wait()
8)writer2 processes
9)writer2 goes wait()
10)writer1 goes wait()




if you see steps 3, 4 and 5 , writer1 worked correctly but writer 2 went to wait() without processing it..and same as steps 8, 9, 10..




here is my code, I appreciate if any one can throw some light.







all I have to do is before I make readable= true all of my worker threads finish executing just above the readable=true line of code.

I am thiking some thing should be done at the bold letters above(//I HAVE TO DO SOMETHING HERE
// for(int i = 0; i < SIZE; i++)
// workers[i].join();[/)


this program works fine for One Reader and One worker situation but if I have multiple workers, I am having this situation.


Thanks,
Ugender
[ September 01, 2007: Message edited by: Ugender Rekulampally ]
Henry Wong
author
Sheriff

Joined: Sep 28, 2004
Posts: 19073
    
  40

if you see steps 3, 4 and 5 , writer1 worked correctly but writer 2 went to wait() without processing it..and same as steps 8, 9, 10..


Just by a quick-eyeball-look at the code, it doesn't look like your worker threads are willing to share the load. They grab the queue synchronization lock, and work on the queue until everything is done. Basically, the second writer has nothing to do by the time it gets to the queue -- whichever writer gets to the queue first does all the work.

Henry


Books: Java Threads, 3rd Edition, Jini in a Nutshell, and Java Gems (contributor)
Ugender Rekulampally
Ranch Hand

Joined: Nov 14, 2005
Posts: 130
Henry,

Thanks for your response.

No...I have different values in two different arrays. lets say account numbers which end with 0 (zero) goes to array first and which is picked up by the first thread. and account numbers which end with 1 (one) goes to array second which is picked up by thread 2.

below workers[0].setQueue(first) is setting up the first array which is generated by reader thread with account numbers ending zero. and same as for workers[1] thread.




I have sample out put too which is written into two different output files..
first out put by first thread
0
0
0
0
0
0
0
0
0
0
34560
0
0
0
0
0
0
0
0
0
34560
0
0
0
0
0
0
0
0
0


and out put by second thread

34561
34571
0
0
0
0
0
0
0
0

if you observe here ...worker1 executed 3 times ( there are 30 values..10 for each time) but worker2 executed only one time (just 10 values)..

and this is happening very inconsistently. some times worker2 executes 3 times worker one executes 1 time. or worker2 3 times and worker1 2 times...

but the problem is


..worker thread 1 finishing the execution fast and making readable = true before even worker thread 2 starts ..so worker thread 2 directly goes to wait loop. and next cycle worker2 processes and changes readble value to true before even worker thread 1 starts so worker1 directly goes to wait() since the readble = true and reader thread starts.


I think this situation is one producer and multiple consumers situation but producer produces different values for different consumers.
but some of the consumers not able to consume their value because other consumer changing the lock value (here it is readble = true )...


Hope I am making it clear.

I couldnt find a producer with multiple /group of consumer example.

please let me know if you want me to explain anything else..this is kind of urgent for me.

Thanks and appreciate your help.

Ugender
Henry Wong
author
Sheriff

Joined: Sep 28, 2004
Posts: 19073
    
  40

..worker thread 1 finishing the execution fast and making readable = true before even worker thread 2 starts ..so worker thread 2 directly goes to wait loop. and next cycle worker2 processes and changes readble value to true before even worker thread 1 starts so worker1 directly goes to wait() since the readble = true and reader thread starts.


Actually no. Worker thread 1 is not finishing too fast. It can take all the time it needs and still be done before worker thread 2 starts. What is happening is that worker thread 1 has the synchronization lock of the queue. Worker thread 2 can't even start without the synchronization lock of the queue.

Henry
[ September 02, 2007: Message edited by: Henry Wong ]
Henry Wong
author
Sheriff

Joined: Sep 28, 2004
Posts: 19073
    
  40

I couldnt find a producer with multiple /group of consumer example.

please let me know if you want me to explain anything else..this is kind of urgent for me.


Well, what you are doing is not really the producer/consumer design pattern. In that pattern, the producer and consumer are independent of each other -- there is no such thing as a mode of when it is in readable mode or not.

A producer just produces data for a queue. It doesn't know about the number of consumers or other producers. A consumer just consumes data from a queue. It doesn't know about the number of producers or other consumers. It does wait/block, but only because the queue is full or empty.

There is no waiting for mode, or working part of a job -- which requires coordination with other threads. It coordinates with the queue -- to put and take jobs -- and nothing else. All the synchronization is the put/take of jobs/data to/from the queue. Everything else runs concurrently.

Henry
Ugender Rekulampally
Ranch Hand

Joined: Nov 14, 2005
Posts: 130
Hello Henry and other Ranchers,
As you suggested, I changed my synchronization now. I am only waiting on my queues.

reader will read data into queues only if they are not full. if q.size() == full, my reader will go to wait().

and same with writers/workers threads. if q.size() == empty, then they will go wait() till reader puts the data in to the queues.

So far, I am thinking my logic makes sense.

I have a input file with 14 elements and every time reader thread has to read 4 elements and puts 2 elements each in 2 queues (queue0, and queue1) and once reader thread goes to wait, worker threads should write these 2 elements each to two different output files...cycle should go on till reader thread finishes the input file.
currently, my program doing that.
my reader is working as I was expecting. even my workers also doing what I was expecting but,

I am thinking there may be still some thing wrong with workers threads. as I was observing, one writer gets activated, processes and goes to wait then the second writer activates , processes and goes to wait() before reader starts processing.

for now, my queue SIZE is 2.

here is my log

reader processing
reader processing
reader processing
reader processing
writer0 active and processing
writer0 wait()
writer1 active and processing
writer1 wait()
reader processing
reader processing
reader processing
reader processing
writer1 active and processing
writer1 wait()
writer0 active and processing
writer0 wait()
reader processing
reader processing
reader processing
reader processing
writer0 active and processing
writer0 wait()
writer1 active and processing
writer1 wait()



if you observe the log, I was not seeing anything like

writer1 active and processing
writer0 active and processing
writer1 wait()
writer0 wait()




every time I see

writer0 active and processing
writer0 wait()
writer1 active and processing
writer1 wait()



I was not sharing any queues in between the two worker threads but I am not sure why these threads acting like mutual exclusive.


please some one have a look at my code throw some suggestions please.


and here is my input file
1 21874
2 89347
3 09033
4 34560
5 34561
6 34562
7 34565
8 34566
9 34568
10 34569
11 34570
12 34571
13 34580
14 34581


I really appreciate your time and help.

Ugender
[ September 03, 2007: Message edited by: Ugender Rekulampally ]
Henry Wong
author
Sheriff

Joined: Sep 28, 2004
Posts: 19073
    
  40

I was not sharing any queues in between the two worker threads but I am not sure why these threads acting like mutual exclusive.


I might be missing something here, but I don't understand this statement. How can you say that you are not sharing the queues? Here is the code that does it.



And here is the code where the workers are blocked on the lock they are sharing... In fact, the reader thread is not allowed to run concurrently either as it is sharing the same lock.



Of course, you created a few vectors which you are not sharing, but that is not the problem.

Henry
Ugender Rekulampally
Ranch Hand

Joined: Nov 14, 2005
Posts: 130
Henry,

Thanks for your response. Yes you are right, I think that is where I am making mistake.

Queue class is the monitor I am using between Reader Thread and workers threads.

But here is my question, how can I share my monitor object between either one reader thread or all writers threads?

either Reader thread should have the lock and all writers should be waiting
OR all writers should have it at the same time (not one writer after other writer...) and Reader thread should be waiting.

hope I put my idea clearly here.

because of that monitor object, writer threads were getting lock one after the other and that is where I was making mistake.

Thanks,
Ugender
Ugender Rekulampally
Ranch Hand

Joined: Nov 14, 2005
Posts: 130
Can some one help me with the siatuation( above message) I have now please?

I have a reader and workers threads. I have one monitor object 'queue' which will be shared betweem reader and worker classes.

scene 1: my reader thread should have the lock and none of the worker threads.

scene 2: workers threads (all at the same time not one after the other) should have it but not reader thread.

in my current solution
scene 1 works fine.
but in scene 2, all of my workers get the lock one after the other not parallelly though but reader wont be having it.

May be, do I need to use ThreadPools to implement such kind of scenario?

Thanks,
Ugender




Thanks,
Ugender
Henry Wong
author
Sheriff

Joined: Sep 28, 2004
Posts: 19073
    
  40

either Reader thread should have the lock and all writers should be waiting
OR all writers should have it at the same time (not one writer after other writer...) and Reader thread should be waiting.


Well, there is no magic here. You need to implement it.

You need to hold the lock just long enough to setup the correct state and get the shared stuff. You also need to setup stuff to allow the worker threads to work indpendently.

After that, the worker thread leaves the synchronized block and works concurrently. When it is done, it may need to get back into a synchronized block to get the state back to reader mode.

Henry
[ September 04, 2007: Message edited by: Henry Wong ]
Ugender Rekulampally
Ranch Hand

Joined: Nov 14, 2005
Posts: 130
Henry,
Thanks for your response. that helped. I removed synchronized from the method signature of public synchronized void writeIntoFile(Vector q, String fname)and implemented synchronized block where ever it needed inside the method.

Thanks for your advice. btw, I just ordered your Java Threads book in amazon.com today.

Thanks again.

Ugender
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: producer-multiple consumers