aspose file tools*
The moose likes I/O and Streams and the fly likes Best way to handle huge write operation Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » I/O and Streams
Bookmark "Best way to handle huge write operation" Watch "Best way to handle huge write operation" New topic
Author

Best way to handle huge write operation

Anil Vupputuri
Ranch Hand

Joined: Oct 31, 2000
Posts: 527
Hi,
I've around two million records in database and need to put it in a flat file. What is the best way.
I've tried BufferedWriter but ended up getting java.lang.OutOfMemory exception though I use 'out.flush()' for every record.
And also I invoke java with maximum heap size like,
'java -Xms1024m -Xmx1024m'

Any suggestions to handle this kind of massive operation. Thanks.


SCJP 1.5, SCEA, ICED (287,484,486)
Stephen Huey
Ranch Hand

Joined: Jul 15, 2003
Posts: 618
Maybe you should post some code. Is part of your problem that you've got a humongous ResultSet object? If so, how about trying to select only a subset of the records at a time?
Anil Vupputuri
Ranch Hand

Joined: Oct 31, 2000
Posts: 527
I can't split the result set as per requirement. Here is the code,

Joe Ess
Bartender

Joined: Oct 29, 2001
Posts: 8971
    
    9

I don't see anything that would necessarily give you a memory leak. How big are your records?
Try leaving the StringBuffer out. Just write each element individually. You're using a BufferedWriter so you won't be losing anything on the performance side. You'll probably gain, since BufferedWriter's buffer (8k) may be smaller than your record size.


[How To Ask Questions On JavaRanch]
Jim Yingst
Wanderer
Sheriff

Joined: Jan 30, 2000
Posts: 18671
Try moving

to be inside the while(rs.next()) loop. The problem right now is that you're adding all the results onto one huge StringBuffer. Also your output is probably something like

a
ab
abc
abcd
abcde
abcdef
abcdefg

rather than

a
b
c
d
e
f
g

Removing the StringBuffer entirely as Joe suggests would also work.


"I'm not back." - Bill Harding, Twister
Joe Ess
Bartender

Joined: Oct 29, 2001
Posts: 8971
    
    9

Originally posted by Jim Yingst:
The problem right now is that you're adding all the results onto one huge StringBuffer.


I don't think that's the case. This line:

which _is_ inside the loop should delete the contents of the StringBuffer.
It is good practice to not reuse StringBuffer instances, as deleting the contents of a StringBuffer does not resize the internal storage buffer, so a particularly large record can cause a StringBuffer instance to hog a lot of memory for the duration of it's existance.
Jim Yingst
Wanderer
Sheriff

Joined: Jan 30, 2000
Posts: 18671
Ah yes, I missed the delete(), thanks. I agree with your other points.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Best way to handle huge write operation