Win a copy of Design for the Mind this week in the Design forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Best way to handle huge write operation

 
Anil Vupputuri
Ranch Hand
Posts: 527
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,
I've around two million records in database and need to put it in a flat file. What is the best way.
I've tried BufferedWriter but ended up getting java.lang.OutOfMemory exception though I use 'out.flush()' for every record.
And also I invoke java with maximum heap size like,
'java -Xms1024m -Xmx1024m'

Any suggestions to handle this kind of massive operation. Thanks.
 
Stephen Huey
Ranch Hand
Posts: 618
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Maybe you should post some code. Is part of your problem that you've got a humongous ResultSet object? If so, how about trying to select only a subset of the records at a time?
 
Anil Vupputuri
Ranch Hand
Posts: 527
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I can't split the result set as per requirement. Here is the code,

 
Joe Ess
Bartender
Pie
Posts: 9264
10
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I don't see anything that would necessarily give you a memory leak. How big are your records?
Try leaving the StringBuffer out. Just write each element individually. You're using a BufferedWriter so you won't be losing anything on the performance side. You'll probably gain, since BufferedWriter's buffer (8k) may be smaller than your record size.
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Try moving

to be inside the while(rs.next()) loop. The problem right now is that you're adding all the results onto one huge StringBuffer. Also your output is probably something like

a
ab
abc
abcd
abcde
abcdef
abcdefg

rather than

a
b
c
d
e
f
g

Removing the StringBuffer entirely as Joe suggests would also work.
 
Joe Ess
Bartender
Pie
Posts: 9264
10
Linux Mac OS X Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Jim Yingst:
The problem right now is that you're adding all the results onto one huge StringBuffer.


I don't think that's the case. This line:

which _is_ inside the loop should delete the contents of the StringBuffer.
It is good practice to not reuse StringBuffer instances, as deleting the contents of a StringBuffer does not resize the internal storage buffer, so a particularly large record can cause a StringBuffer instance to hog a lot of memory for the duration of it's existance.
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Ah yes, I missed the delete(), thanks. I agree with your other points.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic