This week's book giveaway is in the Cloud/Virtualizaton forum.
We're giving away four copies of Mesos in Action and have Roger Ignazio on-line!
See this thread for details.
Win a copy of Mesos in Action this week in the Cloud/Virtualizaton forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Reading Large Files

 
Richard West
Ranch Hand
Posts: 127
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi everyone,

I am trying to to read and write large files about 600M.
The thing is when i use the normal FileOutputStream methods an exception gets thrown saying



This is how i am trying to read the file



My computer does have sufficient memory

I tried adding this in the while loop but it does not work and i got the same exception thrown by the JVM again



My question is basically how do i increase the memory programatically
or is there away in which i can read the file without having the above exception thrown by the JVM?

Any help i greatly appreciated

Thank You

Yours Sincerely

Richard West
 
Ernest Friedman-Hill
author and iconoclast
Marshal
Pie
Posts: 24211
35
Chrome Eclipse IDE Mac OS X
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
This code will not cause your program to run out of memory -- the problem lies elsewhere. Is this your real code, or a simplification?
 
Richard Anderson
Ranch Hand
Posts: 61
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Are you passing the min and max heap size arguments to the JVM?

You can do that by....

java -Xms256M -Xmx1G MYCLASS

where -Xms256M allocates 256MB for the mininum heap size and -Xmx1G allocates 1G for the max heap size. I believe by default the JVM sets the max heap size at 256M. Can someone verify this?

Sorry....looks like the default is only 64M.
[ August 11, 2005: Message edited by: Richard Anderson ]
 
Rick O'Shay
Ranch Hand
Posts: 531
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Couple points and then a solution. First, there are approximiately zero reasons to run gc() manually. Second, there are few reasons to read 600 megabytes in to memory in one fell swoop. Use a memory mapped file.

http://javaalmanac.com/egs/java.nio/CreateMemMap.html

P.S. Make sure you set your memory switches and usually it's a good idea to clamp both ends so that re-sizing at runtime is not required: -Xms256m -Xmx256m. A quarter gig of RAM should be enough to run the Goggle mainframe at peak hours. Well, maybe exaggerating but it's a heckuvalotta memory.
 
Stuart Gray
Ranch Hand
Posts: 410
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Sorry to hijack, but the impression I get from the posts here is that this code will try to read all 600MB of data into memory at once.

My understanding was that it would read the buffer size (1K) of data at once, write it out, and then read another 1K, and so on. Have I misunderstood something?
 
Richard West
Ranch Hand
Posts: 127
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi everyone,

Ernest to answer your question, its a simplification. Don't mind if i ask you but what makes you say it would not run out of memory?

Anderson this what you mean



Unfortunately i can't use the above code because the thing is the application is in a jar file on windows platform and users that use this application to run it usually double click on the jar file to run it. Is there a way to do what you suggested programatically maybe by the use of properties?

Stuart, you have understood the question correctly

Hoping to hear from you guys

Richard West
 
Norm Radder
Ranch Hand
Posts: 895
6
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
See posts on this topic on: http://forums.devx.com/forumdisplay.php?f=104
 
Ernest Friedman-Hill
author and iconoclast
Marshal
Pie
Posts: 24211
35
Chrome Eclipse IDE Mac OS X
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Originally posted by West Richard:
Ernest to answer your question, its a simplification. Don't mind if i ask you but what makes you say it would not run out of memory?


Because the code you showed us uses a small, fixed amount of memory. There is nothing here with memory usage that will grow over time. This loop could copy the biggest file your OS could support, and it could do it with -Xmx8m -Xms8m on the command line, because it hardly uses any memory at all.

As a result, the only suggestions you're getting are to up your VM image size, and that's not helping you, right? If you showed us more code, maybe we'd spot where the memory was going, and we could help you fix the real problem.
 
Richard West
Ranch Hand
Posts: 127
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi everyone,

I tried searching somewhere else in the program and you were right Ernest the leak was not coming from the loop but it seemed to come from this part



I know for a fact that the document in the JTextPane spans couple of of thousand pages but it seems that the exception is coming from that area.

Could it be that because the stream is writing the document at one go. If it is then is there a better way to do it?

Richard West
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic