• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Help! OutOfMemoryError!

 
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Probably I am processing large files most of the time, I got another problem on the jvm memory management: I am trying to read a large file (320M) into memory to be processed later, so I wrote the following program:

public class Test2 {

public static void main(String[] args) {
String filename = "c:\\work\\data\\testfile1.xml";
int size = 20000000;
char buf[] = new char[size];
try {
Reader reader = new BufferedReader(new FileReader(filename));
StringBuffer buffer = new StringBuffer();
long total = 0;
int num;

while ((num = reader.read(buf, 0, size)) > -1) {
buffer.append(buf, 0, num);
total += num;
System.out.println("Total chars read:"+total);
}
reader.close();

} catch (Exception e) {
e.printStackTrace();
}
}

}

The result is as following (jdk is 1.4.2):

C:\work\workspace\test1>java -Xmx800M test1.Test2
Total chars read:20000000
Total chars read:40000000
Total chars read:60000000
Total chars read:80000000
Total chars read:100000000
Total chars read:120000000
Total chars read:140000000
Total chars read:160000000
Exception in thread "main" java.lang.OutOfMemoryError

I have > 1G physical memory in the machine, so this should not be a problem at all. Changing maximum heap size seems not helping at all! I also tried to change the buffer size from 20M to 20K, but it doesn't help at all.

Anybody knows why I could not allocate 320M memory when the heap is much bigger than this? Any suggestion would be appreciated!!!

--James
 
author and iconoclast
Posts: 24207
46
Mac OS X Eclipse IDE Chrome
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Well, first off, note that a Java char is two bytes, so 320 MB of ASCII text takes up 640 MB. That 20 million char buffer takes up another 40 MB. Then there's the fact that to grow a StringBuffer, a new, larger array has to be allocated, and the contents of the old one copied -- meaning two copies of most of the data can exist simultaneously. So by now I'm sure you can see why this isn't enough memory.

Allocate a StringBuffer large enough to hold all the data at once -- use the StringBuffer constructor that takes an int as an argument -- and then make the heap maybe 750MB, and you'll probably be OK.
 
Author and all-around good cowpoke
Posts: 13078
6
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Do you REALLY have to have it all in memory at once or could you take one pass to locate where various bits are and read them as needed as a RandomAccessFile.
Not reading chars if you can process data as bytes can be a big speedup as well as a memory saver since every byte read as a char has to be converted with some sort of encoding.

Bill
 
James Zhang
Ranch Hand
Posts: 30
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks so much for your comments, Ernest! I changed the buffer size to 350M and adjusted the heap size, then it works! Your insights are really useful!

--James
[ January 23, 2007: Message edited by: James Zhang ]
 
Ranch Hand
Posts: 1873
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Cool analysis, Ernest! (as always )

Regards,
Maulin
 
Consider Paul's rocket mass heater.
reply
    Bookmark Topic Watch Topic
  • New Topic