wood burning stoves 2.0*
The moose likes I/O and Streams and the fly likes exception while Read very large file > 300 MB Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Android Security Essentials Live Lessons this week in the Android forum!
JavaRanch » Java Forums » Java » I/O and Streams
Bookmark "exception while Read very large file > 300 MB " Watch "exception while Read very large file > 300 MB " New topic
Author

exception while Read very large file > 300 MB

Ganesh Sugumaran
Greenhorn

Joined: Jul 08, 2008
Posts: 10
Hi
I want to read a file which has size of more than 300 MB.
While i m executing my code, i m getting
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

Below is my code

Im getting the exception in the line 2.
Even though i increased the Heap space up to 900 Mb, i m getting this error.
(FYI)
I executed in command prompt like below,
java -Xms128m -Xmx1024m readLargeFile

Kindly give a solution for this. is there any other best way to read large text file ??
I m waiting for your reply..
Thanks in advance
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12761
    
    5
Since UNICODE characters take two bytes, your 300mb file will take at least 600mb.



Starting with 128mb makes no sense if you know it wont work and you will need the maximum.

What do you have to do with the chrBuff when you get it? Could the work be done with a byte[] instead of char?

Bill
Ganesh Sugumaran
Greenhorn

Joined: Jul 08, 2008
Posts: 10
Yes. I knew that minimum is not a big consideration.
The reason why im using CharBuffer is , using this im matching a particular string n no of times.
My code is follows

After Execution of above code i m getting a part of CharSequence from the above CharBuffer for another search(again Matching process).
So i thought that, for these process(file read and matching process) if i use CharBuffer then it will be faster as well.
So Guide me How to overcome HeapSpace Memory Problem.
Is there any other way to read and matching process in efficient way ??
i remember you that file size is more.
steve souza
Ranch Hand

Joined: Jun 26, 2002
Posts: 860
can you read a certain amount of the data into a buffer (say 10k) and perform a match against each of these buffers? of course a match may span a buffer so you would have to provide special logic for this case


http://www.jamonapi.com/ - a fast, free open source performance tuning api.
JavaRanch Performance FAQ
Jimmy Clark
Ranch Hand

Joined: Apr 16, 2008
Posts: 2187
Is there any other way to read and matching process in efficient way ??


Write the program in KornShell or Perl.
Ganesh Sugumaran
Greenhorn

Joined: Jul 08, 2008
Posts: 10
Thanks for suggestion.

But I've to do in JAVA only.
Ganesh Sugumaran
Greenhorn

Joined: Jul 08, 2008
Posts: 10
Thanks for your reply.
My task is to open a large file in READ&WRITE mode and i need to search some portion of text in that file by searching starting and end point.
Then i need to write that searched area of text to a new file and delete that portion from the original file.

The above process i will do more times.
So I thought that for these process, it will be easy by loading the file into memory by CharBuffer and can search easily by MATCHER class.

Above is my task, then now suggest me some efficient ways.
Note that my file size will be more and using JAVA only i've to do these things.

Thanks in Advance...
Amit Ghorpade
Bartender

Joined: Jun 06, 2007
Posts: 2716
    
    6

"Ganesh PS " please check your private messages for an important administrative matter. You can check them by clicking the My Private Messages link above.


SCJP, SCWCD.
|Asking Good Questions|
Duc Vo
Ranch Hand

Joined: Nov 20, 2008
Posts: 254
Ganesh Sugumaran wrote:Thanks for your reply.
My task is to open a large file in READ&WRITE mode and i need to search some portion of text in that file by searching starting and end point.
Then i need to write that searched area of text to a new file and delete that portion from the original file.

The above process i will do more times.
So I thought that for these process, it will be easy by loading the file into memory by CharBuffer and can search easily by MATCHER class.

Above is my task, then now suggest me some efficient ways.
Note that my file size will be more and using JAVA only i've to do these things.

Thanks in Advance...


Well, how about writing your own search instead of using regex? How complex are your starting and ending points? Are they just regular string? or html tags? or something else?

The whole process should look like:




“Everything should be as simple as it is, but not simpler.” Albert Einstein
Kees Jan Koster
JavaMonitor Support
Rancher

Joined: Mar 31, 2009
Posts: 251
    
    5
Dear Ganesh,

In my experience Java is a dreadful language for the kind of file processing you are tasked with. If you insist on using Java, don't whine about Java using gobs of memory to do this, because it is the wrong tool for the job.

Instead, learn about UNIX shell scripting (notably awk and sed) and you'll find it can be done orders of magnitude faster that Java can do this.


Java-monitor, JVM monitoring made easy <- right here on Java Ranch
 
It is sorta covered in the JavaRanch Style Guide.
 
subject: exception while Read very large file > 300 MB
 
Similar Threads
JDBC QUERIES
Write and Update CSV files
exception while Read very large file > 300 MB
Download large files in RMI
java.lang.OutOfMemoryException:java heap space problem