Win a copy of Microservices Testing (Live Project) this week in the Spring forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Tim Cooke
  • Ron McLeod
  • Jeanne Boyarsky
  • Paul Clapham
Sheriffs:
  • Liutauras Vilda
  • Henry Wong
  • Devaka Cooray
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Tim Holloway
  • Al Hobbs
  • Carey Brown
Bartenders:
  • Piet Souris
  • Mikalai Zaikin
  • Himai Minh

Spring batch: write as many files as the result from the data base

 
Greenhorn
Posts: 29
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi

I am moderately acquainted with Spring Batch framework. This is a new scenario for me - and googling, reading up Spring batch did not help me find a solution yet.

I need to fetch a list of records from the database in header record- detail records format (this i have done). [Also, i have written it into a flat file]

Now I realize i need to group the data read from the db based on one column and create one file for EACH "group" with the file being named using the value of this column. (All files will have same extension, and same format, only the file name would differ.) Each of these file will have the same header-detail format written to it.

This is where i am stuck.

I know of, and have used MultiResourceItemWriter using delegates to write to multiple files / db. But-

1. how do i configure the itemwriter, now that I do not know the number of writers required? It will be as many as the "groups" found in the db.
2. going a step back - I am now worried about my reader too. It returns a single List right now (ignoring the column / group) - I think i would need even the reader to "split" (not sure if this is the correct word to explain this) the "read" objects according to the columns too. so that one reader gives--> multiple readers at runtime, based on the data found.

Also, I can't / don't want to make this separate job because as i mentioned above - the read/ write/ process / file extension/ file format / source table are all same, only the file name must differe and a new file must be generated for each different value of the said column.

I am not sure if i was able to explain the problem clearly.
Thank you for reading, if you do. Any pointers, suggestions about what / how could this be done will be most helpful.

Thanks & Regards
H.J

Edit: I wonder if writing each "group" to a staging table is an option / a good option. However, i still not have a clear idea, how would i create "as many" writers in this case too

 
reply
    Bookmark Topic Watch Topic
  • New Topic