Reading and Writing from Large File, OutOfMemory

[Anant Jagania]

Hi All,

I have a directory which contains large files ~1GB to ~5GB. Recursively i pick up the files and from each file I look for a particular string or sequence of char and create files. The string is the starting and ending point of the sub file which i want to create.

I read some amount of bytes at a time and work on those bytes. However, I am getting OutOfMemory exception after some time. I have set Xmx to 2048. I try to read less number of bytes but then program gets very slow. I have added System.gc() and System.runFinalization() in between the object creation.

I am using RandomFileAccess.

Please let me know how I can over come the problem of OutOfMemory and program to run little faster.

Thanks
Anant

[Jesper de Jong]

Can you show us the source code? You probably have a memory leak somehwere, but it's very hard (or actually, impossible) to debug and point out the exact problem without seeing the source code.

You could try using a profiler, for example JVisualVM (which is included with JDK 6) to find out what is eating up all the memory.

[Anant Jagania]

Hi Jasper,

Thanks for the reply.

Here is the code.

  
import java.io.ByteArrayOutputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.util.Calendar;

public class LargeFileTest {

    public static void main(String[] args) throws FileNotFoundException, IOException {
    	System.out.println("Start " + Calendar.getInstance().getTime());

        args = new String[2];
        //Source file to read
        args[0] = new String("source file");
        //Name of the destination file. This will be repated with count
        args[1] = new String("destination file name");
        //folder name where files need to be stored
        String folderName = "Destination Folder";


        //Number of bytes to ready. Right now its 100MB
        final int MAXREAD = 131072 * 100;
        
        long fileLenght = 0;
        //File handler with read mode
        RandomAccessFile raf = new RandomAccessFile(args[0], "r");
        
        fileLenght = raf.length();

	//Count of files... Actually the count of ABCD found in the source file
        int fileCount = 0;
        RandomAccessFile fos = null;                

	//This array is used for reading the four bytes which should be A B C D
        //This array holds A,B,C or D when we find them while reading bytes one by one
        byte btDataBuff[] = new byte[4];

	//byte array to read 100MB bytes from source file
        byte btData[] = new byte[MAXREAD];

        //Total number of bytes read at each read
        int numberOfData = 0;

        int count = 0;        
		
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        boolean isEOF = false ;        
        
        while(!isEOF)
        {            
            //Run GC each time we read 100MB
            System.gc();
            System.runFinalization();

	    //Read 100MB, it is possible that at the end we do not have 100MB or file is less then 100MB
            //Bytes will be stored in btData
	    numberOfData = raf.read(btData);
		
	    
            if(-1 != numberOfData)
            {

		//See if the ABCD is at the end of the bytes array btData.
                if(btData[numberOfData-4] == 'A' && btData[numberOfData-3] == 'B' && btData[numberOfData-2] == 'C' && btData[numberOfData-1] == 'D')
                {
                    /*Move the pointer of the file to 4 bytes backward. RandomFileAccess have a pointer upto
                     * what bytes are read. So the next read will happen from that pointer.
                     *Next read will happen from this point in file.
                     * The reason behind this is not to worry if any of the character ABCD comes at the end of byte array btData
                     */
                    raf.seek(raf.getFilePointer() - 4) ;
                    numberOfData -= 4 ;
                }
                //See if the ABC is at the end of the bytes raed.
                else if(btData[numberOfData-3] == 'A' && btData[numberOfData-2] == 'B' && btData[numberOfData-1] == 'C')
                {
                    //Move the pointer of the file to 3 bytes backward
                    //Next read will happen from this point in file
                    raf.seek(raf.getFilePointer() - 3) ;
                    numberOfData -= 3 ;
                }
                //See if the AB is at the end of the bytes raed.
                else if(btData[numberOfData-2] == 'A' && btData[numberOfData-1] == 'B')
                {
                    //Move the pointer of the file to 2 bytes backward
                    //Next read will happen from this point in file
                    raf.seek(raf.getFilePointer() - 2) ;
                    numberOfData -= 2 ;
                }
                //See if the A is at the end of the bytes raed.
                else if(btData[numberOfData-1] == 'A')
                {
                    //Move the pointer of the file to 1 byte backward
                    //Next read will happen from this point in file
                    raf.seek(raf.getFilePointer() - 1) ;
                    numberOfData -= 1 ;
                }
                //loop through the entire byte array btData.
                for(int i = 0 ; i < numberOfData ; i++)
                {
                    //Check if the current byte is A.
                    //Count should be 0 because A can come after A in the same file
                    //Count variable can only go upto 4 as we are comparing only four char ABCD
                    if ((char) btData[i] == 'A' && count == 0)
                    {
                        btDataBuff[0] = btData[i];
                        count = 1;
                        continue;
                    } //Check if the current byte is B and current - 1 is A. Count should be 1. AB can repeat after AB
                    else if ((char) btData[i] == 'B' && (char) btDataBuff[0] == 'A' && count == 1)
                    {
                        btDataBuff[1] = btData[i];
                        count = 2;
                        continue;
                    } //Check if the current byte is C, current - 1 is B and current - 2 is A. Count should be 2. ABC can repeat after ABC
                    else if ((char) btData[i] == 'C' && (char) btDataBuff[0] == 'A' && (char) btDataBuff[1] == 'B' && count == 2)
                    {
                        btDataBuff[2] = btData[i];
                        count = 3;
                        continue;
                    } //Check if the current byte is D, current - 1 is C, current - 2 is B and current-3 is A.
                    else if ((char) btData[i] == 'D' && (char) btDataBuff[0] == 'A' && (char) btDataBuff[1] == 'B' && (char) btDataBuff[2]
                            == 'C')
                    {
                        System.out.println("Found ABCD " + fileCount + " " + Calendar.getInstance().getTime());
                    	//This means that we have found the ABCD string in the file. Start writing it to a file until we get another ABCD
                    	btDataBuff[3] = btData[i];
                        count = 4;
                        /*Check if file output stream is not null.
                         * If it is not, which means we had got the ABCD earlier and we have found another ABCD in the same
                         * byte array btData. Write the current byte to file output stream. */
                        if (null != fos)
                        {
                            //BinaryArrayOutputStream holds the bytes what we have read after finding the ABCD.
                            fos.write(baos.toByteArray()) ;
                            baos.reset() ;                            
                            fos.close();
                            System.gc();
                            System.runFinalization();
                            
                        }
                        /*Create new file output stream.
                         * This is required when we first find the ABCD in byte array or subsequent finding of ABCD. */
                        fos = new RandomAccessFile(folderName + args[1] + fileCount + ".abc", "rw");
			//Write the first four bytes ABCD
                        fos.write(btDataBuff);
                        //Increase the file count
                        fileCount++;
                        //Set the first four bytes to 0 in the temp byte array
                        btDataBuff[0] = 0;
                        btDataBuff[1] = 0;
                        btDataBuff[2] = 0;
                        btDataBuff[3] = 0;
                        //Set count to 0
                        count = 0;
                        continue;
                    } //This is the case when we dont find annything of A,AB,ABC,ABCD
                    else if (null != fos)
                    {
                        //Count can be greater than 0 becuase A we have found but next byte was not B
                        if (count > 0)
                        {
                            baos.write(btDataBuff, 0, count);
                        }
                        //Set count to 0 as the any of A,Ab,ABC is written to output stream
                        count = 0;
                        //Set the first four bytes to 0 in the temp byte array
                        btDataBuff[0] = 0;
                        btDataBuff[1] = 0;
                        btDataBuff[2] = 0;
                        btDataBuff[3] = 0;
                        //Write the current byte read to binary array output stream
                        baos.write(btData[i]) ;
                    } 
                    else
                    {
                        btDataBuff[0] = 0;
                        btDataBuff[1] = 0;
                        btDataBuff[2] = 0;
                        btDataBuff[3] = 0;
                        count = 0;
                    }
                }
            }//Once all bytes are read numberOfData will be -1
            else
            {
                //Check if file output stream is still there
                if (null != fos)
                {
                
                    //Write the bytes to FOS. This should be the last destination file int he source file
                    fos.write(baos.toByteArray()) ;
                    baos.reset() ;                    
                    fos.close();                    
                }
                isEOF = true ;
            }            
        }
        raf.close();
        System.out.println("End " + Calendar.getInstance().getTime());
    }
}
 

Please let me know if anything required.
[edit[Break // comments and change to /* */ because the lines in code are too long for screen width. CR[/edit]

[David Newton]

I find that code rather difficult to read, and the comments do more harm than good :(

[Anant Jagania]

Hi David,

I have modified the comments.

I apologize for inconvenience.

If the code or comments still do not make sense, please let me know.. i will refactor it again.

Thanks
Anant

[Wouter Oet]

I agree with David. The code is hard to read. I would refactor/rewrite it to use a more object oriented approach and introduce a lot of new methods. Because a single method of 190 lines of code is hard to comprehend. The calls to System.gc() are unnecessary because the garbage collector is guaranteed to run before a OutOfMemoryError is thrown by the JVM.

[Anant Jagania]

Hey Guys,

Sorry for being late. Here is the complete code. It has got more comments as well. I have refactored little. Please let me know if this is still not readable.

    
import java.io.ByteArrayOutputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.RandomAccessFile;
import java.util.Calendar;

public class LargeFileTest {

	// Number of bytes to ready. Right now its 100MB
	final int MAXREAD = 1048576 * 100;

	// This will be used to set the 0 byte
	final byte bZero = 0;

	// Characters to find
	final char A = 'A';
	final char B = 'B';
	final char C = 'C';
	final char D = 'D';

	// Length of file
	long fileLenght = 0;

	// This array holds A,B,C or D when we find them while reading bytes one by one
	byte btDataBuff[] = new byte[4];

	// byte array to read 100MB bytes from source file
	byte btData[] = new byte[MAXREAD];

	// Total number of bytes read at each read
	int numberOfData = 0;

	// This is count for the bytes found as ABCD
	int count = 0;

	// Destination folder
	String folderName = "Destination Folder";

	// Write bytes temporary to this stream
	ByteArrayOutputStream baos = new ByteArrayOutputStream();

	// RandomAccessFile for writing destination file
	RandomAccessFile fos = null;

	// Count of files... Actually the count of ABCD found in the source file
	int fileCount = 0;

	public static void main(String[] args) throws FileNotFoundException,
			IOException {
		System.out.println("Start " + Calendar.getInstance().getTime());

		args = new String[2];
		// Source file to read
		args[0] = new String("source file"); //For testing. Will be passed as argument to program later
		// Name of the destination file. This will be repeated with count
		args[1] = new String("destination file name"); //For testing. Will be passed as argument to program later

		LargeFileTest lFileTest = new LargeFileTest();
		lFileTest.readSourceFile(args[0], args[1]);
		System.out.println("End " + Calendar.getInstance().getTime());
	}

	private void setZeroBytesAndCnt() {
		btDataBuff[0] = bZero;
		btDataBuff[1] = bZero;
		btDataBuff[2] = bZero;
		btDataBuff[3] = bZero;
		count = 0;
	}

	/**
	 * Reads the source files and creates destination files based on
	 * the string ABCD
	 * @param sourceFileName
	 * @param destFileName
	 * @throws IOException
	 */
	public void readSourceFile(String sourceFileName, String destFileName) throws IOException {
		// File handler with read mode
		RandomAccessFile raf = new RandomAccessFile(sourceFileName, "r");

		// Find out the number of bytes in the file
		fileLenght = raf.length();
		
		boolean isEOF = false;

		while (!isEOF) {

			// Read 100MB. Bytes will be stored in btData
			numberOfData = raf.read(btData);
			
			/* We keep reading 100MB from the source file. In last lets say there are only 
			 * 3 bytes are left to read. The btData array will have only three bytes.
			 * Due to this, nested if condition can fail because we are checking minimum four bytes
			 * If number of bytes read is less than three that means we have reached the end of file. 
                         */
			if (numberOfData > 4) {

				findBytesAtEnd(raf);
				// loop through the entire byte array btData.
				for (int i = 0; i < numberOfData; i++) {
					/* Check if the current byte is A.
					 * Count should be 0 because A can come after A in the same file
					 * Count variable can only go upto 4 as we are comparing
					 * only four char ABCD
                                         */
					if ((char) btData[i] == A && count == 0) {
						btDataBuff[0] = btData[i];
						count = 1;
						continue;
					} // Check if the current byte is B and current - 1 is A. Count should be 1. AB can repeat after AB
					else if ((char) btData[i] == B && (char) btDataBuff[0] == A && count == 1) {
						btDataBuff[1] = btData[i];
						count = 2;
						continue;
					} /* Check if the current byte is C, current - 1 is B and
					   * current - 2 is A. Count should be 2. ABC can repeat after ABC */
					else if ((char) btData[i] == C && (char) btDataBuff[0] == A
							&& (char) btDataBuff[1] == B && count == 2) {
						btDataBuff[2] = btData[i];
						count = 3;
						continue;
					} // Check if the current byte is D, current - 1 is C,
					// current - 2 is B and current-3 is A.
					else if ((char) btData[i] == D && (char) btDataBuff[0] == A
							&& (char) btDataBuff[1] == B && (char) btDataBuff[2] == C) {
						System.out.println("Found ABCD " + fileCount + " "
								+ Calendar.getInstance().getTime());
						// This means that we have found the ABCD string in the
						// file. Start writing it to a file until we get another ABCD
						btDataBuff[3] = btData[i];
						count = 4;
						/* Check if file output stream is not null. If it is not, which means we had got the 
						 * ABCD earlier and we have found another ABCD in the same byte array btData. 
						 * Write the current byte to RandomFileAccess. */
						if (null != fos) {
							/* BinaryArrayOutputStream holds the bytes what we
							 * have read after finding the ABCD. */
							writeFileOnDisc();
						}
						// Create new file output stream. This is required when
						// we first find the ABCD in byte array or subsequent finding of ABCD.
						fos = new RandomAccessFile(folderName + destFileName
								+ fileCount + ".abc", "rw");
						// Write the first four bytes ABCD
						fos.write(btDataBuff);
						// Increase the file count
						fileCount++;
						// Set the first four bytes to 0 in the temp byte array
						setZeroBytesAndCnt();
						continue;
					} // This is the case when we dont find anything of A,AB,ABC,ABCD
					else if (null != fos) {
						/* Count can be greater than 0 because A we have found
						 * but next byte was not B */
						if (count > 0) {
							baos.write(btDataBuff, 0, count);
						}
						/* Set count to 0 as the any of A,Ab,ABC is written to output stream
						 * Set the first four bytes to 0 in the temp byte array */
						setZeroBytesAndCnt();
						// Write the current byte read to binary array output stream
						baos.write(btData[i]);
					} else {
						/* Set count to 0 as the any of A,Ab,ABC is written to output stream
						 * Set the first four bytes to 0 in the temp byte array */
						setZeroBytesAndCnt();
					}
				}
			}// Once all bytes are read numberOfData will be -1
			else {
				// Check if file output stream is still there
				if (null != fos) {

					// Write the bytes to RandomAccessFile. This should be the last
					// destination file in the source file
					writeFileOnDisc();
				}
				isEOF = true;
			}
		}
		raf.close();
	}
	
	/**
	 * Find A,B,C,D in any sequential combination and move the file pointer back.
	 * This is just to make sure that A, AB, ABC, ABCD is not at the end of the 
	 * currently read bytes (we read 100MB bytes at a time)
	 * @param raf
	 * @throws IOException
	 */
	private void findBytesAtEnd(RandomAccessFile raf) throws IOException{
		// See if the ABCD is at the end of the bytes array btData.
		if (btData[numberOfData - 4] == A
				&& btData[numberOfData - 3] == B
				&& btData[numberOfData - 2] == C
				&& btData[numberOfData - 1] == D) {
			/* Move the pointer of the file to 4 bytes backward.
			 * RandomFileAccess have a pointer upto what bytes are read.
			 * So the next read will happen from that pointer.
			 * Next read will happen from this point in file. The reason
			 * behind this is not to worry if any of the character ABCD
			 * comes at the end of byte array btData */
			moveFilePointer(raf, 4);
		}
		// See if the ABC is at the end of the bytes read.
		else if (btData[numberOfData - 3] == A
				&& btData[numberOfData - 2] == B
				&& btData[numberOfData - 1] == C) {
			// Move the pointer of the file to 3 bytes backward
			moveFilePointer(raf, 3);
		}
		// See if the AB is at the end of the bytes read.
		else if (btData[numberOfData - 2] == A
				&& btData[numberOfData - 1] == B) {
			// Move the pointer of the file to 2 bytes backward
			moveFilePointer(raf, 2);
		}
		// See if the A is at the end of the bytes read.
		else if (btData[numberOfData - 1] == A) {
			// Move the pointer of the file to 1 byte backward
			moveFilePointer(raf, 1);		
                }		
	}
	
	/**
	 * Move pointer of the file backward to give number of bytes 
	 * @param raf
	 * @param noOfBytes
	 * @throws IOException
	 */
	private void moveFilePointer(RandomAccessFile raf, int noOfBytes) throws IOException{
		raf.seek(raf.getFilePointer() - noOfBytes);
		numberOfData -= noOfBytes;
		
	}
	
	/**
	 * Writes the bytes from BinaryArrayOutputStream to Random Access File
	 * @throws IOException
	 */
	private void writeFileOnDisc() throws IOException{
		fos.write(baos.toByteArray());
		baos.reset();
		fos.close();
	}
}

[Paul Clapham]

I would use the profiler too. But I don't see why you need a 100 MB buffer to work in. In most situations a buffer size of somewhere around 8 KB is quite sufficient.

And I wouldn't do my own buffering like that. I'm not quite sure what rules you're using to filter the data but it seems to me that you're looking for the string "ABCD" as some kind of sentinel. I would rewrite the code to read the data sequentially using an ordinary FileInputStream. Set up a buffer which always contains the last four bytes read. Then you can compare that buffer to "ABCD" and act accordingly with respect to whether to write data.

[David Newton]

<offtopic>
Way, *way* too many comments. If single lines of code require a paragraph of comments, something has gone wrong.
</offtopic>

[Campbell Ritchie]

Also the comments (don't use // on more than one line) and some of the code is in lines too long to read on screen.

[Jesper de Jong]

Some remarks:

Line 57, 59: You never need to call new String("literal"). Just write "literal" - it makes no sense to create a separate new String object.

If I see this right, when the program doesn't find the sequence ABCD anywhere in the file, it will try to append the whole file to your ByteArrayOutputStream. That could easily cause an OutOfMemoryError if the file is very large.

[Ernest Friedman-Hill]

Also, 131072 * 100 is not 100MB, it's about 1.25 MB. Just an example of what was meant by "the comments hurt more than they help".

[Anant Jagania]

This is what I have to achieve.

There are big files and each file is made with joining many other files. The starting of nested file is ABCD. So I need to go through each byte and see if ABCD comes in the sequence. Once i find it, i need to start writing the bytes to files until i find another ABCD.


Paul, I can do the way you have mentioned but that makes my program very slow. And i have files with the size of 5GB as well. Reading 8/4 bytes at time will be a very slow progress. I will try to use profiler and see what they have to say.

David, Campbell,

I have written comments to get better understanding of the algorithm which i have implemented in the program. I can still shorten it. I will do it and post the code again.

Jasper,

Line 57, 59 are just for testing I have kept. Later on these strings will be passed as arguments to the program from the console.
The program doesn't append the whole file to ByteArrayOutputStream when we do not find ABCD. It will only append the current byte which is read and if any bytes A,AB,ABC where found before the current byte (given that current byte is not D).

File will be written/appended only when we find ABCD and RandomFileAccess object is not null. All the bytes written in BinaryArrayOutputStream will be dumped into RandomAccessFile.

Thanks everybody, I really appreciate your valuable time spending on this post.

I will redo the code/comments and post it again.

Regards,
Anant

[Anant Jagania]

Ernest , Also, 131072 * 100 is not 100MB, it's about 1.25 MB. Just an example of what was meant by "the comments hurt more than they help".

My apology.

Thanks for pointing this out.

I have corrected back the comments and number of bytes to read (MAXREAD) in the code.

Please have a look at it.

[Campbell Ritchie]

Please don't post the code again. My point was not that the comments were too long, but that they were
allwrittenasasinglelinewhi . . . wthescrollbarsappear.

See next post for what it ought to have said

[Campbell Ritchie]

I myself wrote: . . .
all written as a single line which nobody could read because it is too wide for the screen even here at work where I have a screen with 1440px from left to right so it is really difficult to read on an ordinary screen or with large writing in fact it is a good idea to keep all line lengths below 80 keystrokes to make it easier to read and I look forward to Maneesh Godbole posting UseRealWords but you will see how the scroll bars appear.

. . . or would have had I UsedRealWords.

[David Newton]

Well, I think the comments are too long, and don't add to our understanding. They might add to *yours*, but then I'd recommend taking them out so they don't confuse *us*, the folks trying to understand your code.

/** 
 * Move pointer of the file backward to give number of bytes  
 * @param raf 
 * @param noOfBytes 
 * @throws IOException 
 */  
private void moveFilePointer(RandomAccessFile raf, int noOfBytes) throws IOException{  
    raf.seek(raf.getFilePointer() - noOfBytes);  
    numberOfData -= noOfBytes;  
}

The method is named wrong. With the proper name, no comment would be necessary. It also has a side effect, which makes the naming somewhat more difficult, but if it was clear that all file pointer manipulation methods also updated the numberOfData (which is also not a great name), it wouldn't be necessary to include it.

Rather than rename the method, why not just pass in the actual pointer manipulation you want?

moveFilePointer(file, -4)

Otherwise it'd have to be called moveFilePointerBack or something, and it just seems redundant.

/* Move the pointer of the file to 4 bytes backward. 
 * RandomFileAccess have a pointer upto what bytes are read. 
 * So the next read will happen from that pointer. 
 * Next read will happen from this point in file. The reason 
 * behind this is not to worry if any of the character ABCD 
 * comes at the end of byte array btData */  
moveFilePointer(raf, 4);

Here you've described much of what's already known about RandomFileAccess, and again, you are forced to make a comment that would be unnecessary if the method was named appropriately.

// Write the current byte read to binary array output stream  
baos.write(btData[i]);

Here the comment describes precisely what the code does, but takes 2.5x as much text as just the code--this code is self-explanatory. The comment adds unnecessary cognitive overhead.

These same meta-comments apply to much of the code: the code speaks louder than the comments. The code is the ultimate arbiter of what's happening. Comments are difficult to keep in sync with code, because they cannot be (trivially) mechanically checked for correctness. Naming, refactoring, and common sense do more to aid readability than a paragraph for each line of code.

(This is obviously just my opinion, and other folks may feel differently.)

[Andrew Monkhouse]

How many times do you create a new RandomAccessFile and assign it to fos? And how many times do you close the old streams?

You would probably benefit greatly by moving most of your instance variables to method variables. It will then become more obvious as to when they are not being used correctly.

You would probably also benefit from renaming methods and variables (normally if I see a variable named fos, I expect it to be a FileOutputStream - not a RandomAccessFile (often named raf)).
Neither fos nor raf are good descriptions of what the file is being used for though.