Two Laptop Bag*
The moose likes Performance and the fly likes Help! OutOfMemoryError! Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Android Security Essentials Live Lessons this week in the Android forum!
JavaRanch » Java Forums » Java » Performance
Bookmark "Help! OutOfMemoryError!" Watch "Help! OutOfMemoryError!" New topic
Author

Help! OutOfMemoryError!

James Zhang
Ranch Hand

Joined: Sep 06, 2002
Posts: 30
Probably I am processing large files most of the time, I got another problem on the jvm memory management: I am trying to read a large file (320M) into memory to be processed later, so I wrote the following program:

public class Test2 {

public static void main(String[] args) {
String filename = "c:\\work\\data\\testfile1.xml";
int size = 20000000;
char buf[] = new char[size];
try {
Reader reader = new BufferedReader(new FileReader(filename));
StringBuffer buffer = new StringBuffer();
long total = 0;
int num;

while ((num = reader.read(buf, 0, size)) > -1) {
buffer.append(buf, 0, num);
total += num;
System.out.println("Total chars read:"+total);
}
reader.close();

} catch (Exception e) {
e.printStackTrace();
}
}

}

The result is as following (jdk is 1.4.2):

C:\work\workspace\test1>java -Xmx800M test1.Test2
Total chars read:20000000
Total chars read:40000000
Total chars read:60000000
Total chars read:80000000
Total chars read:100000000
Total chars read:120000000
Total chars read:140000000
Total chars read:160000000
Exception in thread "main" java.lang.OutOfMemoryError

I have > 1G physical memory in the machine, so this should not be a problem at all. Changing maximum heap size seems not helping at all! I also tried to change the buffer size from 20M to 20K, but it doesn't help at all.

Anybody knows why I could not allocate 320M memory when the heap is much bigger than this? Any suggestion would be appreciated!!!

--James
Ernest Friedman-Hill
author and iconoclast
Marshal

Joined: Jul 08, 2003
Posts: 24183
    
  34

Well, first off, note that a Java char is two bytes, so 320 MB of ASCII text takes up 640 MB. That 20 million char buffer takes up another 40 MB. Then there's the fact that to grow a StringBuffer, a new, larger array has to be allocated, and the contents of the old one copied -- meaning two copies of most of the data can exist simultaneously. So by now I'm sure you can see why this isn't enough memory.

Allocate a StringBuffer large enough to hold all the data at once -- use the StringBuffer constructor that takes an int as an argument -- and then make the heap maybe 750MB, and you'll probably be OK.


[Jess in Action][AskingGoodQuestions]
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12761
    
    5
Do you REALLY have to have it all in memory at once or could you take one pass to locate where various bits are and read them as needed as a RandomAccessFile.
Not reading chars if you can process data as bytes can be a big speedup as well as a memory saver since every byte read as a char has to be converted with some sort of encoding.

Bill
James Zhang
Ranch Hand

Joined: Sep 06, 2002
Posts: 30
Thanks so much for your comments, Ernest! I changed the buffer size to 350M and adjusted the heap size, then it works! Your insights are really useful!

--James
[ January 23, 2007: Message edited by: James Zhang ]
Maulin Vasavada
Ranch Hand

Joined: Nov 04, 2001
Posts: 1871
Cool analysis, Ernest! (as always )

Regards,
Maulin
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Help! OutOfMemoryError!
 
Similar Threads
java.net.ConnectException: Connection refused: connect
JSP EDITOR QUETION? NEED HELP
Writing memory efficient file IO code.
file2String faster code
plz delete