• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

coordinate flow of jobs approach

 
Ranch Hand
Posts: 125
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi All,

Currently there is a requirement wherein we have to coordinate the flow of 3 distinct batch jobs. Appreciate if you could provide your thoughts on this.

I have an ETL job which populates some database tables, upon its successful completion there will be a oracle stored procedure job which will read these populated tables, process the data and will populate some other tables. Then at last I have a java batch job which will run a routine continuously for every record by reading the data from the tables populated by the oracle stored procedures.
These job activities typically involve dealing with lot of records so each of these jobs may either complete successfully for all records, fail completely or partial completion.
I intend to follow design approach where in the individual status of each task will not hold the subsequent task to accomplish. what eventually will happen is the subsequent task has nothing to do as its preceding task failed rather than a centralized design approach which kind of monitors all these tasks and decides the flow based on individual status of each task.

Irrespective of how the dependency flow b/w these tasks is determined we want to coordinate the sequential flow of these three activities. What is the best way to wrap a coordination layer around these disparate tasks and impose an execution order on the flow where in the tasks get executed one after another irrespective of the status.
Will using a messaging layer or ESB be an appropriate fit for this sort of coordination activity.

Thanks.
 
Consider Paul's rocket mass heater.
reply
    Bookmark Topic Watch Topic
  • New Topic