• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Writing memory efficient file IO code.

 
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The following java app reads an entire file in using a BufferedReader(new FileReader). Why is it that reading a 5 mb file uses about 35800 K of ram? There must be a better way to do FileIO. Now I know I can use RandomAccessFiles to read chunk by chunk, but I'm just curious why java uses so much memory to read in a 5 meg file and is there are more memory efficient way to read the entire file into ram? I can see maybe using 10 megs, but 35?

FYI, i just put the while(true) there to see the memory usage before the app exits.

Thanks so much,
Gordon


 
author and iconoclast
Posts: 24207
46
Mac OS X Eclipse IDE Chrome
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Gordon,

Welcome to JavaRanch!

Lots of things I can tell you.

First, Java uses Unicode -- 16 bit characters. Therefore, the minimum amount of space needed to store the data is 10M, not 5.

Second, Java, being a garbage collected language, often keeps stuff around that it's not really using, and tries to keep some free space available. So whereas a (space efficient, time inefficient) malloc library might only grow its heap as needed, the Java heap's size is always larger than needed to hold the data.

Third, the JVM itself is kinda big; just loading the core classes takes up a non-negligible amount of space.

Lastly, how big the heap grows depends on how much "object churn" there is in the program -- how many objects are created and destroyed. This one is pretty bad, actually: one String for each line of the file, plus many, many array resizings as the StringBuffer grows incrementally. There are much more efficient ways to write this program.

- You could use the StringBuffer constructor that takes a capacity as an argument. This will avoid the need to ever copy and resize the internal char array.

- More importantly, you could just use FileReader.read() to read data into a char[] buffer, then append these characters directly to the StringBuffer, without ever creating Strings. You could use BufferedReader this way too, but if you're reading in big chunks it shouldn't really matter and will use less memory:



Hope this helps.
 
Gordon Glas
Greenhorn
Posts: 4
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks Ernest,
It makes more sense to reuse a single buffer. I was able to get memory usage down to about 28 megs with your code above, like so:



It's really fast too, because there is only a single read operation (because it's allocating the char buffer by the file's length)

Anyway, I will be using RandomAccessFiles for larger files anyway, but I just wanted to have a better understanding.

Thanks again!
Gordon
 
Greenhorn
Posts: 12
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Nice example Gordon,

Only problem is, when the file is big (35Mo), you'll get a memory problem.
The best solution is to read part of the file, and write it, and then read the rest and so on, in regular same size chunks!
 
Don't get me started about those stupid light bulbs.
reply
    Bookmark Topic Watch Topic
  • New Topic