File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Beginning Java and the fly likes Reading Large Files Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Beginning Java
Bookmark "Reading Large Files" Watch "Reading Large Files" New topic
Author

Reading Large Files

Richard West
Ranch Hand

Joined: Jan 07, 2005
Posts: 127
Hi everyone,

I am trying to to read and write large files about 600M.
The thing is when i use the normal FileOutputStream methods an exception gets thrown saying



This is how i am trying to read the file



My computer does have sufficient memory

I tried adding this in the while loop but it does not work and i got the same exception thrown by the JVM again



My question is basically how do i increase the memory programatically
or is there away in which i can read the file without having the above exception thrown by the JVM?

Any help i greatly appreciated

Thank You

Yours Sincerely

Richard West
Ernest Friedman-Hill
author and iconoclast
Marshal

Joined: Jul 08, 2003
Posts: 24184
    
  34

This code will not cause your program to run out of memory -- the problem lies elsewhere. Is this your real code, or a simplification?


[Jess in Action][AskingGoodQuestions]
Richard Anderson
Ranch Hand

Joined: May 20, 2005
Posts: 61
Are you passing the min and max heap size arguments to the JVM?

You can do that by....

java -Xms256M -Xmx1G MYCLASS

where -Xms256M allocates 256MB for the mininum heap size and -Xmx1G allocates 1G for the max heap size. I believe by default the JVM sets the max heap size at 256M. Can someone verify this?

Sorry....looks like the default is only 64M.
[ August 11, 2005: Message edited by: Richard Anderson ]

-Rich, SCJP 1.4
Rick O'Shay
Ranch Hand

Joined: Sep 19, 2004
Posts: 531
Couple points and then a solution. First, there are approximiately zero reasons to run gc() manually. Second, there are few reasons to read 600 megabytes in to memory in one fell swoop. Use a memory mapped file.

http://javaalmanac.com/egs/java.nio/CreateMemMap.html

P.S. Make sure you set your memory switches and usually it's a good idea to clamp both ends so that re-sizing at runtime is not required: -Xms256m -Xmx256m. A quarter gig of RAM should be enough to run the Goggle mainframe at peak hours. Well, maybe exaggerating but it's a heckuvalotta memory.
Stuart Gray
Ranch Hand

Joined: Apr 21, 2005
Posts: 410
Sorry to hijack, but the impression I get from the posts here is that this code will try to read all 600MB of data into memory at once.

My understanding was that it would read the buffer size (1K) of data at once, write it out, and then read another 1K, and so on. Have I misunderstood something?
Richard West
Ranch Hand

Joined: Jan 07, 2005
Posts: 127
Hi everyone,

Ernest to answer your question, its a simplification. Don't mind if i ask you but what makes you say it would not run out of memory?

Anderson this what you mean



Unfortunately i can't use the above code because the thing is the application is in a jar file on windows platform and users that use this application to run it usually double click on the jar file to run it. Is there a way to do what you suggested programatically maybe by the use of properties?

Stuart, you have understood the question correctly

Hoping to hear from you guys

Richard West
Norm Radder
Ranch Hand

Joined: Aug 10, 2005
Posts: 686
See posts on this topic on: http://forums.devx.com/forumdisplay.php?f=104
Ernest Friedman-Hill
author and iconoclast
Marshal

Joined: Jul 08, 2003
Posts: 24184
    
  34

Originally posted by West Richard:
Ernest to answer your question, its a simplification. Don't mind if i ask you but what makes you say it would not run out of memory?


Because the code you showed us uses a small, fixed amount of memory. There is nothing here with memory usage that will grow over time. This loop could copy the biggest file your OS could support, and it could do it with -Xmx8m -Xms8m on the command line, because it hardly uses any memory at all.

As a result, the only suggestions you're getting are to up your VM image size, and that's not helping you, right? If you showed us more code, maybe we'd spot where the memory was going, and we could help you fix the real problem.
Richard West
Ranch Hand

Joined: Jan 07, 2005
Posts: 127
Hi everyone,

I tried searching somewhere else in the program and you were right Ernest the leak was not coming from the loop but it seemed to come from this part



I know for a fact that the document in the JTextPane spans couple of of thousand pages but it seems that the exception is coming from that area.

Could it be that because the stream is writing the document at one go. If it is then is there a better way to do it?

Richard West
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Reading Large Files