File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Threads and Synchronization and the fly likes Getting results for processing and also waiting for all threads to complete Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


JavaRanch » Java Forums » Java » Threads and Synchronization
Bookmark "Getting results for processing and also waiting for all threads to complete" Watch "Getting results for processing and also waiting for all threads to complete" New topic
Author

Getting results for processing and also waiting for all threads to complete

John Vorwald
Ranch Hand

Joined: Sep 26, 2010
Posts: 139
What are the options, and advantages, for waiting for all threads to complete. I have something like the following, and am looking for comments / recommendations for the loop starting at line 104.

Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 4181
    
  21

I am not sure the loop is necessary. You have a loop just above it which waits until all the tasks are done, and before running that loop you had called shutdown(). The result should be that by the time you are done take()ing the last completed task, the last executor thread should be complete and shutdown.

The purpose of it is to make sure the resources used by the method are completely cleaned up by the end of the method, and to block until they are. But a (possibly unending) loop of 30 minutes waits seems excessive to me. If it takes more than a few seconds to shutdown the threads after shutdown() is called and all tasks are complete it would indicate a problem to me, and I would look into more aggressive termination. You are incredibly patient to wait 30 minutes, then continue to wait when it doesn't complete in that period.


Steve
John Vorwald
Ranch Hand

Joined: Sep 26, 2010
Posts: 139
Steve,
It takes about 6 hours for the line 37 to execute, where a set of threads is created and executed. The software is designed so that all the threads should complete and the results are reviewed to generate the next set of conditions to evaluate. The next set of conditions, about 90 conditions (threads) are created.
I didn't have the loop at 104, and the results show that the next set of cases is created before the previous set has completed running. That seems very strange to me since I thought the loop at 93 would wait for the results from each of the tasks. But it doesn't, and it's not clear to me where the error is, or how to correct it.

John
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 4181
    
  21

I suspect there is something in the code that you are hiding behind ellipses. The loop at line 93 will not end until all the tasks that were submitted are complete. But it processes each task one at a time, so it takes the first one, process it then waits on the second one. If the processing of the first one causes a thread to run, then that thread could very well be complete before the second task is. But I don't see how the loop at the end of the method helps prevent that... unless the for loop at line 93 is in a different thread.

I think you need to make an SSCCE which shows the problem (over the time scale of seconds, not hours please).
John Vorwald
Ranch Hand

Joined: Sep 26, 2010
Posts: 139
Steve,
Thanks for the thoughts.
The code that I posted is the current code; I stopped the program last night. Last night, the output from some of the perturbations from generation 5 were printing / posting completion times, after generation 7 had started. The previous code had the thread executor shutdown command right before the await termination command, and there was no loop around await termination. I'm not sure how hot code works in eclipse, if the modifications were incorporated into the running program that I stopped last night or not.
I don't think I'm leaving anything relevant out, line 37 starts the member function process to evaluate the next generation. But, for some reason, the execution is returning to line 37 quite a while before all the perturbations for the generations are evaluated, which is the confusing part. Even generation 5 perturbation was being evaluated after generation 7 start, so that would be several hours difference.
I'm open to suggestions on print / timing statements. I'll work on it over the weekend.

John
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 4181
    
  21

I am at the limit of what I can suggest because I don't know what generation 5 or generation 7 or any of that means. What I could use would be a complete and compileable example that shows the problem so I can see it run. That is what I meant by SSCCE (<- follow that link). Without seeing an example I can run, all I can say is that the detail which probably explains this is left outside of the code you posted, because nearly everything you said in your previous post was (geberation 5, generation 7, printing, posting, completion times, returning to line 37 (which either means a loop or the method being called from someplace else ...)

So I think you are on your own until you can come up with a much simplified version you can post which shows what you mean. One thing out-side-the-box to consider is that your application may not be ending before your restart it. If you start the application, change code, then start the application, the previous run is probably still running. Eclipse allows one or more applications or instances of the application to run at the same time. You can usually choose from all running versions using a selector near where the console is (bottom right side usually).
John Vorwald
Ranch Hand

Joined: Sep 26, 2010
Posts: 139
Steve,
I don't know how to make a simple example from this code, because I'm not sure at what point the problem occurs.
Below is a start at a simple example. The goal is to have a tree, and add random letters at each generation of leaves. The letters from the leaf, including parents, are combined and the number of unique vowels and consonants are counted, and a score assigned. The program should rank the terminal leafs based on score, and generate the next generation of leafs for the top two scores.
However, after generating a list of terminal and executable nodes (top ranked), and generating the workers, and getting the scores, I'm not able to update the date in the baseline tree leafs.

This code runs, but doesn't copy the scores into the leafs. This problem needs to be resolved before illustrating the the waiting on execution problem.

Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 4181
    
  21

The reason you are having trouble coming up with a simple example is because you are still thinking about the application space and not the area that is giving you trouble. Why are you doing leaf/tree analysis stuff when you are trying to isolate a problem waiting for executor services to stop? Why not write a simple application which creates 10 simple threads that wait for random amounts of time, and try to wait for them all to finish.

When you can do that, add some of your other layers - when each task is done, do whatever next relevant thing is (start a new task? collate some data?) Think about the problem space, not the application space. It should help you isolate the problem and where the SSCCE and tests should be done.
 
GeeCON Prague 2014
 
subject: Getting results for processing and also waiting for all threads to complete