• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

parallelism in a for loop and using java 5 concurrency

 
Ranch Hand
Posts: 312
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
All,

I have a scenario wherein I invoke a small custom java application in a for loop from another app.
The for loop can contain 100k elements and there is no inter-dependency b/w the elements contained in the list so processing these elements in parallel should not cause any functional issues. Currently the program unit runs synchronously where we start record 2 after record 1 finishes so on and so forth.

How can I spawn multiple threads within this for loop so at a time I can have 10 or more such elements being executed in parallel. I feel this way we can utilize resources optimally and increase the overall performance by reducing the time it takes to process 100k records.

Please suggest the best practice here. Most threading examples I saw had some counter based scenario which did not deem fit to the scenario I am having.

Thanks.
 
Ranch Hand
Posts: 160
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You can have a thread pool of say 10 threads. All those threads can get the job from a shared location such as job queue.
 
manish ahuja
Ranch Hand
Posts: 312
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Manish: Can you elaborate a bit about getting the job from a shared location such as job queue. How can I achieve this programmatic-ally.
 
author
Posts: 23951
142
jQuery Eclipse IDE Firefox Browser VI Editor C++ Chrome Java Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

It's quite simple. In the current scenario, I am assuming that you use the index of the loop to determine what to do. And by no dependency, I am assuming that you don't depend on previous indexes of the loop.

In your new scenario, you have the same loop -- but many threads are running the same loop (in separate threads). And your index will no longer go from the beginning to the end. Instead, your threads must work with each other to take the next index, do the loop, and come back to take the next index.

Since there is no dependency, your threads can have separate copies of the variables. And as for the loop, you can use synchronization, or one of the atomic classes to distribute the indexes to each thread.

Henry
 
Author
Posts: 12617
IntelliJ IDE Ruby
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You'd probably want to do some benchmarking to find the optimal thread count for your particular hardware and JVM.

My intuition believes that anything past the number of cores your JVM will utilize at once won't help--but I'm curious as to the reality, since I haven't done much parallel stuff since the advent of multicore procs.
 
manish ahuja
Ranch Hand
Posts: 312
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Henry - This is exactly what I am trying to achieve.

-----
In your new scenario, you have the same loop -- but many threads are running the same loop (in separate threads). And your index will no longer go from the beginning to the end. Instead, your threads must work with each other to take the next index, do the loop, and come back to take the next index.
-----

Keeping the threads in harmony in such a way they maintain the order index as described by you is what I m struggling with.


 
Henry Wong
author
Posts: 23951
142
jQuery Eclipse IDE Firefox Browser VI Editor C++ Chrome Java Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

manish ahuja wrote:
Keeping the threads in harmony in such a way they maintain the order index as described by you is what I m struggling with.



Well, one option is to divide the work prior to starting the threads. For example, if your index normally goes from one to a thousand. With ten threads, the first thread does the first hundred. The second does the second hundred. etc. Of course, this works only the iterations are nearly the same duration. Otherwise, you wind up with threads completing at different time intervals, and hence, not achieve full parallelism.

Now... if you want something more complex than that, you will have to tell us more detail than, it's something that you are "struggling with".

Henry
 
Master Rancher
Posts: 4796
72
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
In terms of Java 5 concurrency, I would look at Executors, ExecutorService, Runnable and Callable. Here's a simple example to play with:
 
Ranch Hand
Posts: 282
Eclipse IDE PHP Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
 
reply
    Bookmark Topic Watch Topic
  • New Topic