wood burning stoves 2.0*
The moose likes Spring and the fly likes Spring Batch: Problem with commit-interval Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Frameworks » Spring
Bookmark "Spring Batch: Problem with commit-interval" Watch "Spring Batch: Problem with commit-interval" New topic
Author

Spring Batch: Problem with commit-interval

gi tos
Greenhorn

Joined: May 15, 2012
Posts: 12
Hello,

I'm using Spring Batch to read from a file and then to write into a data Base. And there is a file structure example (not real):

98165165132161JeanDupont065400015151
01565466516597JackLor 874854984984
cccccc293000000000000000000000000000

the tow first lines are data lines, and the last is for control:
cccccc means control line
2 the number of lines in the file
93 the sum of values after the name (index 24): 06+87


After reading all lines from the file we have to read the control line and do verification. If OK we commit else we stop (BATCH FAIL).
the problem is that if I have a commit-interval 100 and I have 1000 line in the file. The job will read the 100 first line and it will not find the control line. So the job will stop.
If I keep the verification in the end, the job will commit every 100 line, so if the control is not OK we can't roll back.

I use :
ItemReader: org.springframework.batch.item.file.FlatFileItemReader
lineMapper: org.springframework.batch.item.file.mapping.DefaultLineMapper
lineTokenizer: org.springframework.batch.item.file.transform.FixedLengthTokenizer
itemWriter: org.springframework.batch.item.database.JdbcBatchItemWriter


and step config:

<batch:step id="step1">
<batch:tasklet transaction-manager="transactionManager" >
<batch:chunk reader="myItemReader" processor="myProcessor"
writer="myItemWriter" commit-interval="100">
</batch:chunk>
</batch:tasklet>
</batch:step>

Is there any way to disable the commit interval (no good), or to read by chunk but do commit in the end (if OK).

Thank you in advance.
Mark Spritzler
ranger
Sheriff

Joined: Feb 05, 2001
Posts: 17250
    
    6

Well you could use JobListener and StepListener and I think there is a ChunkListener, and make it not commit. Then in the JobListener at the end of the job have it do the commit. You will have to implement the interfaces yourself, but it is possible.

Mark


Perfect World Programming, LLC - Two Laptop Bag - Tube Organizer
How to Ask Questions the Smart Way FAQ
gi tos
Greenhorn

Joined: May 15, 2012
Posts: 12
Mark Spritzler wrote:Well you could use JobListener and StepListener and I think there is a ChunkListener, and make it not commit. Then in the JobListener at the end of the job have it do the commit. You will have to implement the interfaces yourself, but it is possible.

Mark


thanks for the reply
Which interfaces exactly I have to implement ?

And can I make a tasklet without a chunk ?

My real problem is the chunk

I have to read all the file at the same time. I have to do many controls. so if I have only a chunk I can't do that.
Mark Spritzler
ranger
Sheriff

Joined: Feb 05, 2001
Posts: 17250
    
    6

gi tos wrote:
Mark Spritzler wrote:Well you could use JobListener and StepListener and I think there is a ChunkListener, and make it not commit. Then in the JobListener at the end of the job have it do the commit. You will have to implement the interfaces yourself, but it is possible.

Mark


thanks for the reply
Which interfaces exactly I have to implement ?



In my post, I put the actual interface names in it.

I recommend reading the Spring Batch documentation on all the listener types and I think you will find that there is a way there.

Mark
gi tos
Greenhorn

Joined: May 15, 2012
Posts: 12
Mark,

I've found the solution

I didn't implement any interface. I just changed the completionPolicy instead of using commit interval.

My config is like this now:


<bean id="completionPolicy" class="org.springframework.batch.repeat.policy.DefaultResultCompletionPolicy"/>

<batch:step id="step1">
<batch:tasklet transaction-manager="transactionManager" >
<batch:chunk reader="myItemReader" processor="myProcessor"
writer="myItemWriter" chunk-completion-policy="completionPolicy">
</batch:chunk>
</batch:tasklet>
</batch:step>


Like this the job read all the file at the same time, so I can do my controls and commit at the end if OK.


Mark Spritzler
ranger
Sheriff

Joined: Feb 05, 2001
Posts: 17250
    
    6

Cool. And you can write your own CompletionPolicy to and still use chunks too.

Mark
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Spring Batch: Problem with commit-interval