How to execute large low priority batch jobs in EJB3 application

[R van Vliet]

Hi all,

I was wondering if anyone has any insights when it comes to solving the following issue :

We have a large EJB3 application running and we need to do a big batch of rather time consuming operations. These operations do not have a high priority and we do not mind the entire job taking a few hours. Basically the job has to perform some operation on a very large number of records and persist the result of that operation to another table. We want to avoid is the job interrupting or slowing down the rest of the application. What I'm looking for is a robust way to do this. Theoretically there are 3 options :

1) Somehow make the job thread a lower priority
2) Only allow the job to run if the application "notices" it's not busy
3) Execute small parts of the job at a certain interval

I'm quietly assuming option 1 and 2 are not all that feasible in an EJB3 environment which leaves option 3. I'm planning to implement this using a timer bean but that leaves a couple of issues :

- How do i determine how large each part of the job should be so it doesn't interrupt or slow down the rest of the application
- What happens if the second part wants to start when the first is still busy

Does anyone have any ideas how to approach issues like these?

Thanks!

[Masoud Kalali]

Is there any reason which make you perform this batch processing from within the application server? I think the best bet is to pull this requirement out of the application server and let another separate process perform the task. You can change the second process priority in operating system level and ensure that it will not widely affects your application server performance.

If it is a hard requirement to perform the batch process from within the application server, you might be able to use platform MBeans to check the CPU load and start your process chunk if the load is below a certain value like 20%.

[Reza Rahman]

Mr. Van Vliet,

Sorry for the late reply - things have been a little hectic on my end lately.

I have done something very similar for a high-transaction system a little while ago. This is an article I based some of the architecture on: http://www.devx.com/Java/Article/20791. The article does a decent job of covering the issues involved. My own architecture varied in that I used MDBs as worker threads and JMS queues/topics as the coordination backbone. Because bandwidth throttles were placed on the MDBs (limited pool size), they did not hog CPU and could be quite long running. The "jobs" themselves were triggered via the Flux (http://www.fluxcorp.com) commercial scheduler (you can probably also use Quartz).

The big down-sides were that we wound up writing a lot of framework code and rollback accross the worker threads was difficult since there was no "umbrella" JTA transaction across MDBs. We also evaluated Spring Batch (http://static.springframework.org/spring-batch/) in the hopes to cut-down on custom developement, but it was really at a very basic stage at the time and really didn't offer as much as our architecture did for the particular application, especially with Flux/JMS. However, the Spring Batch documentation helped us flesh out a few ideas for the custom framework.

Hope this helps and let me know if you have any more questions.

Best regards,
Reza

[R van Vliet]

Hi Reza,

Thank you for your reply. We were actually toying with that exact idea (MDB/JMS based). I'm glad there's some real world experience with that that (i assume?) was implemented succesfully. I'll look into it ;)

[Reza Rahman]

Mr. Van Vliet,

Yes, the project was successful in that it was functional and met the requirements, but with the down-sides I mentioned (however, no one complained but me :-)). The only way it could have been better/more elegent is with something like Spring Batch where all the code could have already been written/tested/maintained/supported by someone else. I don't think this is yet the case, but you should probably investigate it on your own.

Best regards,
Reza

[R van Vliet]

What would you say is the added benefit to Flux or related packages compared to simply using timer beans or the like to trigger jobs?

[Reza Rahman]

Mr. Van Vliet,

The question there is primarily about "buy"-vs-build.

Flux includes the capabilty of managing schedules (and workflows) via a GUI (or through a configuration file), a robust Java API for managing jobs, job monitoring/run-time management, pre-built triggers (such as FTP/file system monitoring in addition to timed triggers), load-balancing/fail-over (on top of Java EE/JMS/MDB). For our application, all of this was useful. You can do all of this via EJB timers, but the functionality would either be too rudimentary (which may be fine for your requirements) or you will wind up doing a lot of custom coding.

As a side note, EJB 3.1 is adding Flux/Quartz style declarative timers with cron syntax in addition to the current basic interval based EJB timers.

Hope this helps,
Reza

[R van Vliet]

Hm, good to know. And yes, timers need a little love in EJB3 ;)