File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
The moose likes Performance and the fly likes Help! OutOfMemoryError! Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Performance
Bookmark "Help! OutOfMemoryError!" Watch "Help! OutOfMemoryError!" New topic

Help! OutOfMemoryError!

James Zhang
Ranch Hand

Joined: Sep 06, 2002
Posts: 30
Probably I am processing large files most of the time, I got another problem on the jvm memory management: I am trying to read a large file (320M) into memory to be processed later, so I wrote the following program:

public class Test2 {

public static void main(String[] args) {
String filename = "c:\\work\\data\\testfile1.xml";
int size = 20000000;
char buf[] = new char[size];
try {
Reader reader = new BufferedReader(new FileReader(filename));
StringBuffer buffer = new StringBuffer();
long total = 0;
int num;

while ((num =, 0, size)) > -1) {
buffer.append(buf, 0, num);
total += num;
System.out.println("Total chars read:"+total);

} catch (Exception e) {


The result is as following (jdk is 1.4.2):

C:\work\workspace\test1>java -Xmx800M test1.Test2
Total chars read:20000000
Total chars read:40000000
Total chars read:60000000
Total chars read:80000000
Total chars read:100000000
Total chars read:120000000
Total chars read:140000000
Total chars read:160000000
Exception in thread "main" java.lang.OutOfMemoryError

I have > 1G physical memory in the machine, so this should not be a problem at all. Changing maximum heap size seems not helping at all! I also tried to change the buffer size from 20M to 20K, but it doesn't help at all.

Anybody knows why I could not allocate 320M memory when the heap is much bigger than this? Any suggestion would be appreciated!!!

Ernest Friedman-Hill
author and iconoclast

Joined: Jul 08, 2003
Posts: 24199

Well, first off, note that a Java char is two bytes, so 320 MB of ASCII text takes up 640 MB. That 20 million char buffer takes up another 40 MB. Then there's the fact that to grow a StringBuffer, a new, larger array has to be allocated, and the contents of the old one copied -- meaning two copies of most of the data can exist simultaneously. So by now I'm sure you can see why this isn't enough memory.

Allocate a StringBuffer large enough to hold all the data at once -- use the StringBuffer constructor that takes an int as an argument -- and then make the heap maybe 750MB, and you'll probably be OK.

[Jess in Action][AskingGoodQuestions]
William Brogden
Author and all-around good cowpoke

Joined: Mar 22, 2000
Posts: 13037
Do you REALLY have to have it all in memory at once or could you take one pass to locate where various bits are and read them as needed as a RandomAccessFile.
Not reading chars if you can process data as bytes can be a big speedup as well as a memory saver since every byte read as a char has to be converted with some sort of encoding.

James Zhang
Ranch Hand

Joined: Sep 06, 2002
Posts: 30
Thanks so much for your comments, Ernest! I changed the buffer size to 350M and adjusted the heap size, then it works! Your insights are really useful!

[ January 23, 2007: Message edited by: James Zhang ]
Maulin Vasavada
Ranch Hand

Joined: Nov 04, 2001
Posts: 1873
Cool analysis, Ernest! (as always )

I agree. Here's the link:
subject: Help! OutOfMemoryError!
It's not a secret anymore!