Win a copy of Mesos in Action this week in the Cloud/Virtualizaton forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Large amounts of data ... again ...

 
Eivind Fonn
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
My basic problem can be found in this thread:

http://www.coderanch.com/t/379562/java/java/Large-arrays

I have now followed the general advice of the replies. I am now reading the data from the input file as I need it (not all at once) and writing output data to the output file as I get it ready. But I'm still running into problems.

My strategy for writing output is to serialize.

My code now looks something like this, largely simplified to get down to the basic problem:

ObjectOutputStream out;
try
{
out = new ObjectOutputStream(new FileOutputStream("C:\\grr.anl.temp"));
for (int i = 0; i < 25; i++)
{
System.out.println(i);
out.writeObject(new double[1000000]);
out.flush();
}
out.close();
}
catch (Exception e)
{
System.out.println(e.toString());
}
I'm trying to make 25 1 million length double arrays, serialize them and write them out one at a time. Why do I still get the Heap Space error? Shouldn't this be possible anyway?

Thanks
[ April 12, 2006: Message edited by: Eivind Fonn ]
 
Scott Dunbar
Ranch Hand
Posts: 245
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Remember that a double in Java is 64 bits - 8 bytes. You're trying to allocate 200 * 1 million bytes - a bit over 200MB.

By default the Java virtual machine only allocates so much space for itself. The default is 64MB on most platforms. You can pass something like -Xmx512m to tell it to allocated 512 MB.

Edit: um, no you're not - I missed that you weren't hanging onto the buffer. You're still allocating about 8MB but that shouldn't be too bad, even with the default heap size. Let me try this in my environment.
[ April 12, 2006: Message edited by: Scott Dunbar ]
 
Paul Clapham
Sheriff
Posts: 21107
32
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by Scott Dunbar:
Edit: um, no you're not - I missed that you weren't hanging onto the buffer.
True, Eivind's code is not hanging onto the buffer. But the ObjectOutputStream is hanging onto the buffer. It does that so that if you try to serialize the same object twice, it will just write out a sort of pointer to the stream the second time. I believe that if you call the ObjectOutputStream's reset() method, that will allow it to release all the references it's holding. Read its documentation in the API for clarification if you need it.
 
Eivind Fonn
Greenhorn
Posts: 6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Wonderful... it now seems to work! Thanks Paul and Scott.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic