aspose file tools*
The moose likes Java in General and the fly likes Large amounts of data ... again ... Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Java in General
Bookmark "Large amounts of data ... again ..." Watch "Large amounts of data ... again ..." New topic
Author

Large amounts of data ... again ...

Eivind Fonn
Greenhorn

Joined: Mar 21, 2006
Posts: 6
My basic problem can be found in this thread:

http://www.coderanch.com/t/379562/java/java/Large-arrays

I have now followed the general advice of the replies. I am now reading the data from the input file as I need it (not all at once) and writing output data to the output file as I get it ready. But I'm still running into problems.

My strategy for writing output is to serialize.

My code now looks something like this, largely simplified to get down to the basic problem:

ObjectOutputStream out;
try
{
out = new ObjectOutputStream(new FileOutputStream("C:\\grr.anl.temp"));
for (int i = 0; i < 25; i++)
{
System.out.println(i);
out.writeObject(new double[1000000]);
out.flush();
}
out.close();
}
catch (Exception e)
{
System.out.println(e.toString());
}
I'm trying to make 25 1 million length double arrays, serialize them and write them out one at a time. Why do I still get the Heap Space error? Shouldn't this be possible anyway?

Thanks
[ April 12, 2006: Message edited by: Eivind Fonn ]
Scott Dunbar
Ranch Hand

Joined: Sep 23, 2004
Posts: 245
Remember that a double in Java is 64 bits - 8 bytes. You're trying to allocate 200 * 1 million bytes - a bit over 200MB.

By default the Java virtual machine only allocates so much space for itself. The default is 64MB on most platforms. You can pass something like -Xmx512m to tell it to allocated 512 MB.

Edit: um, no you're not - I missed that you weren't hanging onto the buffer. You're still allocating about 8MB but that shouldn't be too bad, even with the default heap size. Let me try this in my environment.
[ April 12, 2006: Message edited by: Scott Dunbar ]

<a href="http://forums.hotjoe.com/forums/list.page" target="_blank" rel="nofollow">Java forums using Java software</a> - Come and help get them started.
Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18570
    
    8

Originally posted by Scott Dunbar:
Edit: um, no you're not - I missed that you weren't hanging onto the buffer.
True, Eivind's code is not hanging onto the buffer. But the ObjectOutputStream is hanging onto the buffer. It does that so that if you try to serialize the same object twice, it will just write out a sort of pointer to the stream the second time. I believe that if you call the ObjectOutputStream's reset() method, that will allow it to release all the references it's holding. Read its documentation in the API for clarification if you need it.
Eivind Fonn
Greenhorn

Joined: Mar 21, 2006
Posts: 6
Wonderful... it now seems to work! Thanks Paul and Scott.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Large amounts of data ... again ...