wood burning stoves 2.0*
The moose likes JDBC and the fly likes Issue with Batch Insertion Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Murach's Java Servlets and JSP this week in the Servlets forum!
JavaRanch » Java Forums » Databases » JDBC
Bookmark "Issue with Batch Insertion" Watch "Issue with Batch Insertion" New topic
Author

Issue with Batch Insertion

sammaiah kyatham
Ranch Hand

Joined: Aug 03, 2003
Posts: 104
Hi All,
can some one please through a best approach to go ahead for below problem:

We have a requirement to insert huge number of records in batches.
In case if one record fails, the application should continue with remaining records with out rollback remaining records.

We are using commit only end of all the records and even not able to trace which record get failed in batch approach.

Thnx,
sam
Martijn Verburg
author
Bartender

Joined: Jun 24, 2003
Posts: 3274
    
    5

If you are not requiring rollback, then maybe you can look at _not_ wrapping the batch job in a transaction.

Hope that helps!


Cheers, Martijn - Blog,
Twitter, PCGen, Ikasan, My The Well-Grounded Java Developer book!,
My start-up.
David O'Meara
Rancher

Joined: Mar 06, 2001
Posts: 13459

I would still consider batches within a transaction, but cut into smaller batches so that you can manage each batch as a transaction so that failures can be managed individually



more or less...
[fixed nested block]
[ September 22, 2008: Message edited by: David O'Meara ]
David O'Meara
Rancher

Joined: Mar 06, 2001
Posts: 13459

I'm not sure what X should be, but if you have few bad records then it can be large (a thousand or more), otherwise you'll need to make it smaller or incur the wrath of individual inserts.
sammaiah kyatham
Ranch Hand

Joined: Aug 03, 2003
Posts: 104
Hi,
Thanks for the response.
The data volume is really high, like 6months past history of customers... It could be more than 1 crore records in real environment...

It would be great if any one could through a best approach...

Thnx,
Sam
David O'Meara
Rancher

Joined: Mar 06, 2001
Posts: 13459

better get started then
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Issue with Batch Insertion
 
Similar Threads
Process the multiple records in a file by Producer/consumer concept using Multithreading
hibernate in a batch program suggestion
Efficient way to calculate the number of threads
Spring Batch: Commit-Interval not honored after roll back during write
Handling massive number of records