wood burning stoves 2.0*
The moose likes Threads and Synchronization and the fly likes Problem is Parallel Processing of Jobs in Java Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of EJB 3 in Action this week in the EJB and other Java EE Technologies forum!
JavaRanch » Java Forums » Java » Threads and Synchronization
Bookmark "Problem is Parallel Processing of Jobs in Java" Watch "Problem is Parallel Processing of Jobs in Java" New topic
Author

Problem is Parallel Processing of Jobs in Java

Deepak Lal
Ranch Hand

Joined: Jul 01, 2008
Posts: 507

Ranchers,
we have a application where Jobs are running concurrently and which is updating the database column in a table.
Hence we are noticing that it is not updating the database column with proper value.
How do we handle this scenario of parallel processing/updating of values in database in java ??

Example :
Job 1 triggers and runs a update query ,it successfully updates the value in DB(oracle)
Job 2 is also triggered parallelely and runs a update query,it is executing the update query but the value is not getting reflected in database and hence in the application

we have checked for connections and other relevant details.all has been properly handled/closed connections.
we have a "Auto commit" enabled ,so inspite of that in the second scenario,the update query is getting executed but the proper value is not getting reflected in database and hence in the application

Example of snippet code

There will be "N" number of jobs which will call the below updateDetails() method parallely.so need help to resolve this in Java.



Need help guys



When The Going Gets Tougher,The Tougher gets Going
Marco Ehrentreich
best scout
Bartender

Joined: Mar 07, 2007
Posts: 1278

Hi Deepak,

are you trying to update the same column in the same row or what? In this case the reason would most probably be that one job is simply overwriting changes of another job but I guess it wouldn't make much sense to update the same database value multiple times directly after one another.

Anyway, it's definitely necessary to control correct synchronization of multiple concurrent jobs if they could possibly interfere with each other, i.e. they update some shared data. One way to do this would be to use correct synchronization mechanisms in the Java code. Another solution would of course be to use correct transaction handling for the database queries. As I don't think you're trying to update exactly the same row and column multiple times it's more likely that the correct values get lost in your code because of thread interference.

Without more information I can hardly imagine any other reason why you should lose some updates. If the outcome of concurrent jobs is more or less unpredictable it would be a typical indicator for multithreading problems.

Marco


Deepak Lal
Ranch Hand

Joined: Jul 01, 2008
Posts: 507

Anyway, it's definitely necessary to control correct synchronization of multiple concurrent jobs if they could possibly interfere with each other, i.e. they update some shared data. One way to do this would be to use correct synchronization mechanisms in the Java code.


yes, it is to do with sychronization as you have indicated.
how do i handle it in this scenario ???
Marco Ehrentreich
best scout
Bartender

Joined: Mar 07, 2007
Posts: 1278


Well, by using correct synchronization But seriously, it's difficult to tell if this problem is better fixed on the Java (thread synchronization) side or database side (transactions). I suppose, it would be appropriate to fix your code as I suspect the root of these problems being there.

Unfortunately correct synchronization and multithreading in general for any non-trivial application is probably one of the most difficult things in software development today. So it's really impossible to give you a full solution here without knowing your application very well. As an advice start by looking for data (ususally object or class members) which could pontentially be updated or read by multiple threads at the same time. Then you have to correctly synchronize every methods that could manipulate these data. This may sound very simple but it isn't. It's not sufficient to put the "synchronized" keyword everywhere and suspect the code to work correctly An easy rule to solve problems with concurrency is often to avoid member variables whenever possible. If it's possible to use local variables instead, use it. They are inherently thread-safe! Immutable objects, i.e. objects which can't change their state after creation, are thread-safe by default, too. Besides that it's hardly possible to give general rules which work for all concurrency problems.

Additionally you will have to learn at least the basics of concurrency in Java. You could start with Sun's tutorial for example.

I'm sorry that I really can't give you more helpful advices but feel free to ask any question that may come up

Marco
Deepak Lal
Ranch Hand

Joined: Jul 01, 2008
Posts: 507

Ranchers,
we are still stuck up with this problem.

There are more than 1 job running in parallel.so we want all the jobs to complete and then based on all jobs status(i.e if OK then update the status of some other column in other table(TABLE2) in database.

The scenario is as below



This is happening only when it is a parallel processing of jobs,so the requirement is that we first want all the 4 jobs to complete their running and become to OK status and then we want the UPDATE QUERY FOR TABLE2 to be fired.

Please advice.How we can apply synchronization to the code.??

Need Help Java Experts...


Deepak



Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3932
    
  17

Sun has a pretty good concurrency tutorial, I think there is already a link to it in this thread. You should read it to find some strategies.

Some tools that may make things easier:
- If the number of tasks you have is known before you start the tasks, then you should look at using a java.util.concurrent.CountDownLatch. You generate the count down latch with the total number of parallel processes you plan on running. At the end of each parallel task they countDown() on the latch. Your final update to table 2 task await()s until all of the other tasks reached countDown() before performing the update.

- If the number of tasks you have are variable, and new tasks can be begun while others are finishing, and you want to make sure the update table 2 task is complete when no other task is running, I would consider a java.util.concurrent.locks.ReentrantReadWriteLock, where each of the parallel tasks takes a read lock, and the update to table 2 task takes a write lock. The update task would then wait for any current parallel tasks to complete. But would allow more parallel tasks to be completed after the update table 2 task.


Steve
Deepak Lal
Ranch Hand

Joined: Jul 01, 2008
Posts: 507

Hi Steve,
Thanks for the inputs,Im new to threads.how should i go about implementation for java.util.concurrent.CountDownLatch

i have below sqls being fired




Since im new to Concurrency can you help me how should i refine my code to handle java.util.concurrent.CountDownLatch
I had one more clarification,how do we implement CountDownLatch in jdk1.4 since i read in api it is since 1.5

Please advice and suggest...

Deepak

Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3932
    
  17

I suggest you read, and understand thoroughly Sun's Concurrency Turorial, found here: http://java.sun.com/docs/books/tutorial/essential/concurrency/index.html

It is too big of a subject to do in a forum. Read the tutorial, write some code and when you have problems come back. This just isn't a good medium for teaching something which requires this much detail.
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3932
    
  17

Deepak Lal wrote:how do we implement CountDownLatch in jdk1.4 since i read in api it is since 1.5


Woops, I missed that part.

First: Java 1.4 is past its End Of Life. You should do what you can to NOT use it. Java 5 is near its End Of Life as well. Do yourself a favor and update to Java 6.
Second: If you can't update to a new version of java, there is a back-port of the java.util.concurrent package. You should do a Google search for 'java.util.concurrent back port' and see if you can't find where it is located.
Marco Ehrentreich
best scout
Bartender

Joined: Mar 07, 2007
Posts: 1278

Hi Deepak,

as Steve pointed out, too, there's really no other way than to learn about the basics (or more) of concurrency yourself! If it would be that easy, I'd be glad to help you fix your application. Believe me

If you only take some advices from us here without really understanding it, it's likely that you will mess up your code completely. Furthermore it's usually not sufficient to change 2 or 3 lines of code and it will work as expected. Concurrency should be considered for the whole application design or at least the parts of your application that are confronted with concurrency effects. This is obviously something that can only be done correctly by someone who knows the application and concurrent programming quite well.

Anyway, this is a subject worth the effort for learning as it will become more and more important with multi-core/multi-processor computers in the future ;-)

Marco
Deepak Lal
Ranch Hand

Joined: Jul 01, 2008
Posts: 507

I do agree with you Marco and Steve,but please atleast help me with this for now.


Atleast in the scenario explained.do you think it is feasible to use a synchronized block for updateTable1() and updateTable2() methods as shown in the Posts earlier.
Im a newbie to all these concepts and im stuck with this...

Please help.

Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3932
    
  17

If we take the code you posted to mean:

You want to do:

n times, and each concurrently. Then when they are all complete, you want to do:

Then the answer is: Yes. Like I previously said, CountDownLatch would work real well here if n is a known value. You would create a number of runnable tasks (perhaps using a ThreadPoolExecutorService to execute them). The tasks that do the dao.updateTable1() would call CountDownLatch#countDown() when they finish. The task that does dao.updateTable2() would call CountDownLatch#await() before it begins. That would mean that the second task must wait for all the other tasks to complete before running.

If n is not known before you start executing taks then you need to use some other tool. For example, the ReadWriteLock method I mentioned earlier, or by using the Future objects returned when the dao.updateTable1() tasks are submitted to wait for them to complete.

I can't change the code you posted to make it work. You will have to read up on how this stuff works to figure that out.
Deepak Lal
Ranch Hand

Joined: Jul 01, 2008
Posts: 507

Hi Steve,
Do you have any Sample Code Example for the same so that i can get an idea as to how to proceed.??
Deepak

Marco Ehrentreich
best scout
Bartender

Joined: Mar 07, 2007
Posts: 1278

Hi Deepak,

your problem seems to be really serious...

I can imagine a relatively simple ad-hoc solution which _COULD_ work, too.

But in any case Steve and I should now if you have checked if it's possible to use the backported version of the java.util.concurrent package or upgrade to Java >= 1.5! Since Java 1.5 there are a lot of very useful tools in the JDK which make working with concurrency much easier and safer, in particular because you're new to this subject! Maybe you can check this and let us know!

Marco
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3932
    
  17

Sure, the java.util.concurrent.CountDownLatch has example code.
Deepak Lal
Ranch Hand

Joined: Jul 01, 2008
Posts: 507

Hi Steve,
I hasd been to below site

http://sourceforge.net/project/platformdownload.php?group_id=153802&sel_platform=4411

it says download backport-util-concurrent-*.zip - works with Java 1.4 and newer
but im unable to find out the relevant backport package.

Could you please help me with this..
Could you tell me the steps for using backport-util-concurrent with JDK 1.4
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3932
    
  17

Sorry, I have never used the backport. Reading that site should give you the information you need though. If you can't figure it out you should check out the contact information for the sourceforge project, or see if there is a forum. If all else fails, download whatever the 'latest' link tells you to.
Marco Ehrentreich
best scout
Bartender

Joined: Mar 07, 2007
Posts: 1278

I have never used it, too. From a quick look at the README which is inside the ZIP you can download it seems to be sufficient to include the JAR file in your project which you can also find inside the ZIP file. According to the documentation there are some limitations because it's a backport version but I think for your needs it should work ;-)

Marco
Deepak Lal
Ranch Hand

Joined: Jul 01, 2008
Posts: 507

Marco and steve.
Im unable to find out the zip version,and how do i proceed once i get it ??
Please help me out.

Deepak
Marco Ehrentreich
best scout
Bartender

Joined: Mar 07, 2007
Posts: 1278

I can't test it at the moment because I don't have enough time and no JDK 1.4 :-)

But I would take this one: backport-util-concurrent-Java60-3.1.zip

Unzip it and add the included JAR in it to your project. Then the necessary classes (e.g. CountDownLatch) for Steve's example should hopefully be available to your application!

Marco
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3932
    
  17

Hi Deepak, that link you posted has a big green button at the center/top of the page with a download link. Did you click that link?
Deepak Lal
Ranch Hand

Joined: Jul 01, 2008
Posts: 507



Hi Steve and Marco Ehrentreich,
Yes,i have downloaded the backport-util-concurrent.jar and configured in my build path.Can you please tell me how to proceed on this ?

How do i use the CountdownLatch and which classes i need to import in the below scenario ?? Please help....



I have another clarification ,


1. what is the number of jobs that can be run concurrently and can be handled by the CountdownLatch as you have suggested ?
2. suppose i have 1000 jobs running concurrently ,will it degrade the performance of the running application /server will crash/ or any other possibility you see for such a huge number of jobs running concurrently.
3. Does the Backport with concurency have any limitations in terms of the above scenario ??


Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3932
    
  17

Deepak Lal wrote:
How do i use the CountdownLatch and which classes i need to import in the below scenario ?? Please help....


The JavaDoc is here: http://backport-jsr166.sourceforge.net/doc/api/. That should tell you what classes to import and how to use the class. Otherwise, any examples of using the Java 6 version of CountDownLatch will work as showing how to use it.



1. what is the number of jobs that can be run concurrently and can be handled by the CountdownLatch as you have suggested ?
2. suppose i have 1000 jobs running concurrently ,will it degrade the performance of the running application /server will crash/ or any other possibility you see for such a huge number of jobs running concurrently.
3. Does the Backport with concurency have any limitations in terms of the above scenario ??



Number of concurrent Jobs? Many. Theoretically, Integer.MAX_VALUE. Realistically, you will be limited by your specific application. Can you have 1000 jobs running concurrently, and will that degrade performance? That depends on the application. For example, if you have an application that spends a lot of time waiting on some external datasource (Input from external applications, web sites, databases, ...) then you can use a lot of threads and get some benefit. If your application is heavy on processing, and takes a lot of processor time rather than waiting time, then you will gain little above having one thread per Processor the computer has, having too much more, in this case, may hurt performance. If your Job takes a lot of memory, then having 2 or more Jobs running simultaneously may cause memory concerns, OutOfMemoryErrors, or cause so many Garbage Collection cycles so as to degrade performance. If your Job accesses files on disks then having even 2 threads concurrently reading from the disk may (almost certainly will) hurt performance.

So how do you know, for your application? The first thing to do is to program the application with a single thread in the Thread Pool, and use a Profiler to test speed, CPU consumption, and memory requirements. Then add more Threads to the Thread Pool and see the affect. It quite often is the case that the actual number of Threads used should depend on the system on which the application is run, and should be configured accordingly. You won't know until you test it out in the real world.
Deepak Lal
Ranch Hand

Joined: Jul 01, 2008
Posts: 507

Hi Steve,
I have written the below code in Java 1.4 using the backport version as suggested by you.



Can you please suggest how should i refine the above code so that it works for my requirement as given below

Requirement as stated in OP:




Need help Steve.....

Please could you refine the above code now....Im stuck up with this for a long time....

Deepak
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3932
    
  17

Okay, Here is what you would do:


You want updateTable1() to run in several threads concurrently, which means you need:
1) Make a number of Threads, which will run concurrently (an ExecutorService suits this need nicely).
2) Each thread will run 1 task - usually a Runnable which does the work you want to do concurrently
3) Create a CountDownLatch with the same number of passes as there are tasks to run, and make sure all the tasks have access to it
4) Each task calls updateTable1()
5) When updateTable1() completes the task counts down once on the CountDownLatch

You then want to call updateTable2() after all updateTable1() calls are done
1) Make a single task to run in a single thread (you can add it to the ExecutorService too if you make the ExecutorService big enough)
2) The task will await() on the CountDownLatch as its first step. So it does nothing till all the other tasks are done.
3) It then calls updateTable2().
Deepak Lal
Ranch Hand

Joined: Jul 01, 2008
Posts: 507

Steve could you please refine the code which i have pasted...
I'm unable to proceed...
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3932
    
  17

Deepak Lal wrote:Steve could you please refine the code which i have pasted...
I'm unable to proceed...


Hi Deepak, JavaRanch is Not a code mill, so I won't write the code for you. Try what I pointed out. If you can't do it, go over to the Java Concurrency Tutorial, specifically the section on high level concurrency objects. If you go through that tutorial it should show you how to use the classes you worked so hard to get a backport for.

Then implement what you want in a very basic sketch, like the code you copy and pasted from the API above. Then fill in the real work. If you have more questions other than 'write this for me' please feel free to come back and ask them.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Problem is Parallel Processing of Jobs in Java
 
Similar Threads
EDITABLE JTable and SQL QUERY
Its Very Urgent !!!! Help Me Now !!!!!!
JPA update not available to some successive queries
Update in For loop.
Multiple Columns in a JCombobox