File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Java in General and the fly likes parallelism in a for loop and using java 5 concurrency Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of EJB 3 in Action this week in the EJB and other Java EE Technologies forum!
JavaRanch » Java Forums » Java » Java in General
Bookmark "parallelism in a for loop and using java 5 concurrency" Watch "parallelism in a for loop and using java 5 concurrency" New topic
Author

parallelism in a for loop and using java 5 concurrency

manish ahuja
Ranch Hand

Joined: Oct 23, 2003
Posts: 312
All,

I have a scenario wherein I invoke a small custom java application in a for loop from another app.
The for loop can contain 100k elements and there is no inter-dependency b/w the elements contained in the list so processing these elements in parallel should not cause any functional issues. Currently the program unit runs synchronously where we start record 2 after record 1 finishes so on and so forth.

How can I spawn multiple threads within this for loop so at a time I can have 10 or more such elements being executed in parallel. I feel this way we can utilize resources optimally and increase the overall performance by reducing the time it takes to process 100k records.

Please suggest the best practice here. Most threading examples I saw had some counter based scenario which did not deem fit to the scenario I am having.

Thanks.
Manish Singh
Ranch Hand

Joined: Jan 26, 2007
Posts: 160
You can have a thread pool of say 10 threads. All those threads can get the job from a shared location such as job queue.
manish ahuja
Ranch Hand

Joined: Oct 23, 2003
Posts: 312
Manish: Can you elaborate a bit about getting the job from a shared location such as job queue. How can I achieve this programmatic-ally.
Henry Wong
author
Sheriff

Joined: Sep 28, 2004
Posts: 18108
    
  39


It's quite simple. In the current scenario, I am assuming that you use the index of the loop to determine what to do. And by no dependency, I am assuming that you don't depend on previous indexes of the loop.

In your new scenario, you have the same loop -- but many threads are running the same loop (in separate threads). And your index will no longer go from the beginning to the end. Instead, your threads must work with each other to take the next index, do the loop, and come back to take the next index.

Since there is no dependency, your threads can have separate copies of the variables. And as for the loop, you can use synchronization, or one of the atomic classes to distribute the indexes to each thread.

Henry


Books: Java Threads, 3rd Edition, Jini in a Nutshell, and Java Gems (contributor)
David Newton
Author
Rancher

Joined: Sep 29, 2008
Posts: 12617

You'd probably want to do some benchmarking to find the optimal thread count for your particular hardware and JVM.

My intuition believes that anything past the number of cores your JVM will utilize at once won't help--but I'm curious as to the reality, since I haven't done much parallel stuff since the advent of multicore procs.
manish ahuja
Ranch Hand

Joined: Oct 23, 2003
Posts: 312
Henry - This is exactly what I am trying to achieve.

-----
In your new scenario, you have the same loop -- but many threads are running the same loop (in separate threads). And your index will no longer go from the beginning to the end. Instead, your threads must work with each other to take the next index, do the loop, and come back to take the next index.
-----

Keeping the threads in harmony in such a way they maintain the order index as described by you is what I m struggling with.


Henry Wong
author
Sheriff

Joined: Sep 28, 2004
Posts: 18108
    
  39

manish ahuja wrote:
Keeping the threads in harmony in such a way they maintain the order index as described by you is what I m struggling with.


Well, one option is to divide the work prior to starting the threads. For example, if your index normally goes from one to a thousand. With ten threads, the first thread does the first hundred. The second does the second hundred. etc. Of course, this works only the iterations are nearly the same duration. Otherwise, you wind up with threads completing at different time intervals, and hence, not achieve full parallelism.

Now... if you want something more complex than that, you will have to tell us more detail than, it's something that you are "struggling with".

Henry
Mike Simmons
Ranch Hand

Joined: Mar 05, 2008
Posts: 2969
    
    9
In terms of Java 5 concurrency, I would look at Executors, ExecutorService, Runnable and Callable. Here's a simple example to play with:
Michael Angstadt
Ranch Hand

Joined: Jun 17, 2009
Posts: 272



SCJP 6 || SCWCD 5
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: parallelism in a for loop and using java 5 concurrency
 
Similar Threads
(B&S):Lock cookie importance
performance while executing loops
hibernate in a batch program suggestion
do-while loop in real time
how to delete huge amount of records