wood burning stoves 2.0*
The moose likes Java in General and the fly likes running java app as a batch job Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Java in General
Bookmark "running java app as a batch job " Watch "running java app as a batch job " New topic
Author

running java app as a batch job

manish ahuja
Ranch Hand

Joined: Oct 23, 2003
Posts: 312
Hi there,

Currently I have a pure java application using core java which interact with backend database using an OR mapping layer(hibernate). This is synchronous invocation.
Now I want to run this application in a batch. i.e. the input for the application will be in a file so instead of 1 record processing invoked by an external client I will have a file which may consist 1000 records. I have to pick 1 record each from the file and invoke the application.
I want to know your ideas on the same and best practices if any related to this.
Like some of the issues to tackle is not allowing more than 1 batch to run at the same time. We may have to change the file based approach mentioned above to say do a database call and fetch records to be feeded into the java app.
Can we use some of the batch utilities in the java space like Springbatch or quartz.

Thanks in Advance
Jeanne Boyarsky
internet detective
Marshal

Joined: May 26, 2003
Posts: 30537
    
150

Manish,
You could write a wrapper than calls your app for each line of the file. This is likely to be significantly slower than doing batching within the program.

quartz is a scheduler. You'd still need something to divy up the work.


[Blog] [JavaRanch FAQ] [How To Ask Questions The Smart Way] [Book Promos]
Blogging on Certs: SCEA Part 1, Part 2 & 3, Core Spring 3, OCAJP, OCPJP beta, TOGAF part 1 and part 2
Rene Larsen
Ranch Hand

Joined: Oct 12, 2001
Posts: 1179

We may have to change the file based approach mentioned above to say do a database call and fetch records to be feeded into the java app.

You could also use a JMS queue or topic, which your application then have a receiver for.
Each time a new message arrive on the queue/topic your application will automatically get the message.

Can we use some of the batch utilities in the java space like Springbatch or quartz.

You could maybe also use a Windows Service (Windows) or Cron Job (UNIX/Linux/Mac/etc.), to run your application.


Regards, Rene Larsen
Dropbox Invite
manish ahuja
Ranch Hand

Joined: Oct 23, 2003
Posts: 312
Hi Jeanne,

-------------
You could write a wrapper than calls your app for each line of the file. This is likely to be significantly slower than doing batching within the program.
--------------
Can you explain what do you mean by "than doing batching within the program".

-Manish
Jeanne Boyarsky
internet detective
Marshal

Joined: May 26, 2003
Posts: 30537
    
150

Originally posted by manish ahuja:
Can you explain what do you mean by "than doing batching within the program".

Sure. If your Java program read in the file of input values, it could read in X rows at a time (let's say 100) and do a batch update to the database. This cuts down on the number of database round trips.
manish ahuja
Ranch Hand

Joined: Oct 23, 2003
Posts: 312
Thanks Jeanne,

My original intent of the question was can I use something like sub-batches within the parent batch job so as to expedite the whole flow run.
Say If get an input of 100 records (file read or database read), I will typically do a for or while loop iterate every input record and run the application for each such input record.
This would really take long as the volume of input increases. Each input record is independent of the other. With this regard I was wondering if there is anything where I could start multiple such jobs in parallel and accelerate the batch run time. I am not sure multi-threading can greatly help in this situation.

Manish
Jeanne Boyarsky
internet detective
Marshal

Joined: May 26, 2003
Posts: 30537
    
150

Manish,
Now I'm confused. Your original post says "Like some of the issues to tackle is not allowing more than 1 batch to run at the same time. " so I thought you didn't want to parallelize.

If the jobs are independent, there's no reason you couldn't kick off multiple ones at the same time.

Is it safe to assume you've tuned the original job? If not, this often yields more than threading because each update requires less resources.
 
Consider Paul's rocket mass heater.
 
subject: running java app as a batch job