Win a copy of Think Java: How to Think Like a Computer Scientist this week in the Java in General forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

parallelism in a for loop and using java 5 concurrency

 
manish ahuja
Ranch Hand
Posts: 312
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
All,

I have a scenario wherein I invoke a small custom java application in a for loop from another app.
The for loop can contain 100k elements and there is no inter-dependency b/w the elements contained in the list so processing these elements in parallel should not cause any functional issues. Currently the program unit runs synchronously where we start record 2 after record 1 finishes so on and so forth.

How can I spawn multiple threads within this for loop so at a time I can have 10 or more such elements being executed in parallel. I feel this way we can utilize resources optimally and increase the overall performance by reducing the time it takes to process 100k records.

Please suggest the best practice here. Most threading examples I saw had some counter based scenario which did not deem fit to the scenario I am having.

Thanks.
 
Manish Singh
Ranch Hand
Posts: 160
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You can have a thread pool of say 10 threads. All those threads can get the job from a shared location such as job queue.
 
manish ahuja
Ranch Hand
Posts: 312
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Manish: Can you elaborate a bit about getting the job from a shared location such as job queue. How can I achieve this programmatic-ally.
 
Henry Wong
author
Marshal
Pie
Posts: 21112
78
C++ Chrome Eclipse IDE Firefox Browser Java jQuery Linux VI Editor Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

It's quite simple. In the current scenario, I am assuming that you use the index of the loop to determine what to do. And by no dependency, I am assuming that you don't depend on previous indexes of the loop.

In your new scenario, you have the same loop -- but many threads are running the same loop (in separate threads). And your index will no longer go from the beginning to the end. Instead, your threads must work with each other to take the next index, do the loop, and come back to take the next index.

Since there is no dependency, your threads can have separate copies of the variables. And as for the loop, you can use synchronization, or one of the atomic classes to distribute the indexes to each thread.

Henry
 
David Newton
Author
Rancher
Posts: 12617
IntelliJ IDE Ruby
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You'd probably want to do some benchmarking to find the optimal thread count for your particular hardware and JVM.

My intuition believes that anything past the number of cores your JVM will utilize at once won't help--but I'm curious as to the reality, since I haven't done much parallel stuff since the advent of multicore procs.
 
manish ahuja
Ranch Hand
Posts: 312
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Henry - This is exactly what I am trying to achieve.

-----
In your new scenario, you have the same loop -- but many threads are running the same loop (in separate threads). And your index will no longer go from the beginning to the end. Instead, your threads must work with each other to take the next index, do the loop, and come back to take the next index.
-----

Keeping the threads in harmony in such a way they maintain the order index as described by you is what I m struggling with.


 
Henry Wong
author
Marshal
Pie
Posts: 21112
78
C++ Chrome Eclipse IDE Firefox Browser Java jQuery Linux VI Editor Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
manish ahuja wrote:
Keeping the threads in harmony in such a way they maintain the order index as described by you is what I m struggling with.


Well, one option is to divide the work prior to starting the threads. For example, if your index normally goes from one to a thousand. With ten threads, the first thread does the first hundred. The second does the second hundred. etc. Of course, this works only the iterations are nearly the same duration. Otherwise, you wind up with threads completing at different time intervals, and hence, not achieve full parallelism.

Now... if you want something more complex than that, you will have to tell us more detail than, it's something that you are "struggling with".

Henry
 
Mike Simmons
Ranch Hand
Posts: 3076
13
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
In terms of Java 5 concurrency, I would look at Executors, ExecutorService, Runnable and Callable. Here's a simple example to play with:
 
Michael Angstadt
Ranch Hand
Posts: 277
Eclipse IDE Java PHP
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic