I am trying to generate reports in my project using spring batch. I have more than 8 million records in my database. Earlier I have set commit-interval as 1 but after reading some articles, I have set commit-interval as 10000 and page-size as 10000 but still, it's taking more than 44 hours to generate the report. In every itertaion, It is taking 3 to 4 minutes to get the records, processes those records and writing in my CSV file.
Please help me, friends. maybe I am doing something wrong.
<!-- each thread will run this job, with different stepExecutionContext
<step id="slave" xmlns="http://www.springframework.org/schema/batch">
<chunk reader="pagingItemReader" writer="flatFileItemWriter"
processor="itemProcessor" commit-interval="10000" />
I would suggest you to analyze each phase - fetching the data, process the data, and writing out the CSV file separately. First of all, you need to discover the bottleneck. Moreover, you should provide more details (if you can) of what your code does, otherwise we can only guess in abstract.
To do a great right, do a little wrong - shakepeare. twisted little ad:
Two software engineers solve most of the world's problems in one K&R sized book