This week's giveaway is in the EJB and other Java EE Technologies forum.
We're giving away four copies of EJB 3 in Action and have Debu Panda, Reza Rahman, Ryan Cuprak, and Michael Remijan on-line!
See this thread for details.
The moose likes EJB and other Java EE Technologies and the fly likes How to execute large low priority batch jobs in EJB3 application Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of EJB 3 in Action this week in the EJB and other Java EE Technologies forum!
JavaRanch » Java Forums » Java » EJB and other Java EE Technologies
Bookmark "How to execute large low priority batch jobs in EJB3 application" Watch "How to execute large low priority batch jobs in EJB3 application" New topic
Author

How to execute large low priority batch jobs in EJB3 application

R van Vliet
Ranch Hand

Joined: Nov 10, 2007
Posts: 144
Hi all,

I was wondering if anyone has any insights when it comes to solving the following issue :

We have a large EJB3 application running and we need to do a big batch of rather time consuming operations. These operations do not have a high priority and we do not mind the entire job taking a few hours. Basically the job has to perform some operation on a very large number of records and persist the result of that operation to another table. We want to avoid is the job interrupting or slowing down the rest of the application. What I'm looking for is a robust way to do this. Theoretically there are 3 options :

1) Somehow make the job thread a lower priority
2) Only allow the job to run if the application "notices" it's not busy
3) Execute small parts of the job at a certain interval

I'm quietly assuming option 1 and 2 are not all that feasible in an EJB3 environment which leaves option 3. I'm planning to implement this using a timer bean but that leaves a couple of issues :

- How do i determine how large each part of the job should be so it doesn't interrupt or slow down the rest of the application
- What happens if the second part wants to start when the first is still busy

Does anyone have any ideas how to approach issues like these?

Thanks!
Masoud Kalali
Author
Ranch Hand

Joined: Jul 08, 2004
Posts: 531

Is there any reason which make you perform this batch processing from within the application server? I think the best bet is to pull this requirement out of the application server and let another separate process perform the task. You can change the second process priority in operating system level and ensure that it will not widely affects your application server performance.

If it is a hard requirement to perform the batch process from within the application server, you might be able to use platform MBeans to check the CPU load and start your process chunk if the load is below a certain value like 20%.






Masoud Kalali
Software Engineer - My Weblog - GlassFish Security
Reza Rahman
author
Ranch Hand

Joined: Feb 01, 2005
Posts: 580
Mr. Van Vliet,

Sorry for the late reply - things have been a little hectic on my end lately.

I have done something very similar for a high-transaction system a little while ago. This is an article I based some of the architecture on: http://www.devx.com/Java/Article/20791. The article does a decent job of covering the issues involved. My own architecture varied in that I used MDBs as worker threads and JMS queues/topics as the coordination backbone. Because bandwidth throttles were placed on the MDBs (limited pool size), they did not hog CPU and could be quite long running. The "jobs" themselves were triggered via the Flux (http://www.fluxcorp.com) commercial scheduler (you can probably also use Quartz).

The big down-sides were that we wound up writing a lot of framework code and rollback accross the worker threads was difficult since there was no "umbrella" JTA transaction across MDBs. We also evaluated Spring Batch (http://static.springframework.org/spring-batch/) in the hopes to cut-down on custom developement, but it was really at a very basic stage at the time and really didn't offer as much as our architecture did for the particular application, especially with Flux/JMS. However, the Spring Batch documentation helped us flesh out a few ideas for the custom framework.

Hope this helps and let me know if you have any more questions.

Best regards,
Reza


Independent Consultant — Author, EJB 3 in Action — Expert Group Member, Java EE 6 and EJB 3.1
R van Vliet
Ranch Hand

Joined: Nov 10, 2007
Posts: 144
Hi Reza,

Thank you for your reply. We were actually toying with that exact idea (MDB/JMS based). I'm glad there's some real world experience with that that (i assume?) was implemented succesfully. I'll look into it ;)
Reza Rahman
author
Ranch Hand

Joined: Feb 01, 2005
Posts: 580
Mr. Van Vliet,

Yes, the project was successful in that it was functional and met the requirements, but with the down-sides I mentioned (however, no one complained but me :-)). The only way it could have been better/more elegent is with something like Spring Batch where all the code could have already been written/tested/maintained/supported by someone else. I don't think this is yet the case, but you should probably investigate it on your own.

Best regards,
Reza
R van Vliet
Ranch Hand

Joined: Nov 10, 2007
Posts: 144
What would you say is the added benefit to Flux or related packages compared to simply using timer beans or the like to trigger jobs?
Reza Rahman
author
Ranch Hand

Joined: Feb 01, 2005
Posts: 580
Mr. Van Vliet,

The question there is primarily about "buy"-vs-build.

Flux includes the capabilty of managing schedules (and workflows) via a GUI (or through a configuration file), a robust Java API for managing jobs, job monitoring/run-time management, pre-built triggers (such as FTP/file system monitoring in addition to timed triggers), load-balancing/fail-over (on top of Java EE/JMS/MDB). For our application, all of this was useful. You can do all of this via EJB timers, but the functionality would either be too rudimentary (which may be fine for your requirements) or you will wind up doing a lot of custom coding.

As a side note, EJB 3.1 is adding Flux/Quartz style declarative timers with cron syntax in addition to the current basic interval based EJB timers.

Hope this helps,
Reza
R van Vliet
Ranch Hand

Joined: Nov 10, 2007
Posts: 144
Hm, good to know. And yes, timers need a little love in EJB3 ;)
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: How to execute large low priority batch jobs in EJB3 application
 
Similar Threads
hello
deadlock
Object Reuse J2me
how to distribute jobs in a clustered server inside a EJB container
Associating objects via object references versus identity map lookups