This week's book giveaway is in the Jobs Discussion forum.
We're giving away four copies of Soft Skills and have John Sonmez on-line!
See this thread for details.
The moose likes JDBC and the fly likes OutOfMemoryError when deviding to Bulks. Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Soft Skills this week in the Jobs Discussion forum!
JavaRanch » Java Forums » Databases » JDBC
Bookmark "OutOfMemoryError when deviding to Bulks." Watch "OutOfMemoryError when deviding to Bulks." New topic
Author

OutOfMemoryError when deviding to Bulks.

Sharon whipple
Ranch Hand

Joined: Jul 31, 2003
Posts: 294
Hi All,
I have list of million of records,
I want to divide the list to smaller groups for later processing, say take 3000 keys each (Bulk) and insert to Bulk table.
Other process process these Bulks later on.

I based on code sample from Sun website (executeBatch tutorial) and made the following method

On 10000 list I get the Bulk dividing done in less then 2sec.
On 500000 List, the Bulk dividing take like 3 mints.
500000 and up I get out-of-memory Exception.

What do I do wrong?
Please advise.

Thanks
Sharon




Jeanne Boyarsky
author & internet detective
Marshal

Joined: May 26, 2003
Posts: 31054
    
232

Sharon,
Is the out of memory coming from the database part or the rest of the code? If you comment out the prepared statement part and don't get the error, it tells you that the problem is in the database code/driver. In that case, you can do commits more often so as not to build up as large a memory usage.


[Blog] [JavaRanch FAQ] [How To Ask Questions The Smart Way] [Book Promos]
Blogging on Certs: SCEA Part 1, Part 2 & 3, Core Spring 3, OCAJP, OCPJP beta, TOGAF part 1 and part 2
Sharon whipple
Ranch Hand

Joined: Jul 31, 2003
Posts: 294
The out of memory coming from the Java side, problem was solved using -Xmx512m.

But the bulk splitting still take some time..

I added:

after each "addBatch" call so parameters will be removed.

And i added :


So the PreparedStatement will flush to the db every 100 records (6000 characters clob each)
But still this process takes large amount of time (11 seconds for just one million records) and memory space
What am I doing wrong?

Please advice.
Sharon
[ January 10, 2008: Message edited by: Sharon whipple ]
Jeanne Boyarsky
author & internet detective
Marshal

Joined: May 26, 2003
Posts: 31054
    
232

11 seconds to insert one million records isn't bad. An import could take almost that long for a million records.

For the amount of memory used, you can check whether you need to hold all the data in memory at once. If you can do it in stages (read in some records, load them, repeat), this would save a lot of memory.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: OutOfMemoryError when deviding to Bulks.