aspose file tools*
The moose likes Beginning Java and the fly likes is it possible to process a 500MB zipfile in java? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Beginning Java
Bookmark "is it possible to process a 500MB zipfile in java?" Watch "is it possible to process a 500MB zipfile in java?" New topic
Author

is it possible to process a 500MB zipfile in java?

pavi
Ranch Hand

Joined: Jan 23, 2008
Posts: 31
In my project i need to split a zipfile of any size into 1MB files and then merge it to get the original file.My java code works fine upto 58MB zipfile but for the file size greater than 58MB i get the following exception

"Exception in thread "main" java.lang.OutOfMemoryError: Java heap space"

Is it possible to process a 500MB zipfile in java?
If possible how to process.
please help me to resolve this.

Thanks in advance,
Latha
Rob Spoor
Sheriff

Joined: Oct 27, 2005
Posts: 19723
    
  20

You can use the -Xmx flag; type in "java -X" in a command window for more info.

Moving to the beginner's forum.


SCJP 1.4 - SCJP 6 - SCWCD 5 - OCEEJBD 6
How To Ask Questions How To Answer Questions
Joe Ess
Bartender

Joined: Oct 29, 2001
Posts: 8927
    
    9

Don't try to read the entire file into memory.
Read a small chunk, write it out and repeat.


"blabbing like a narcissistic fool with a superiority complex" ~ N.A.
[How To Ask Questions On JavaRanch]
pavi
Ranch Hand

Joined: Jan 23, 2008
Posts: 31
i read that The -X options are non-standard and subject to change without notice.

below is a code to find the memory size

System.out.println("MAX MEMORY = " + Runtime.getRuntime().maxMemory() / 1048576 + "MB");
System.out.println("TOTAL MEMORY = " + Runtime.getRuntime().totalMemory() / 1048576 + "MB");


when i run this it prints 63MB as MAX MEMORY and 4MB as total.

and after the below command

java -Xms512m -Xmx512m MyApplication

it prints 512MB.

but again if i just give

java Application
i get the memory size as only 63MB

what does it means?so for every time we run the file should we set the size or what? how to set the size permanantly for the application to handle bigfile?please do help me .

Thanks in advance
Latha
Paul Sturrock
Bartender

Joined: Apr 14, 2004
Posts: 10336


i read that The -X options are non-standard and subject to change without notice.

Indeed. If sun release a new JVM and the rename these to "minimumHeapSize" and "maximumHeapSize" you will need to change how you run your process. Is that an issue?


JavaRanch FAQ HowToAskQuestionsOnJavaRanch
Henry Wong
author
Sheriff

Joined: Sep 28, 2004
Posts: 18896
    
  40

what does it means?so for every time we run the file should we set the size or what? how to set the size permanantly for the application to handle bigfile?please do help me .


For most bigger applications, there are generally a ton more switches than settings than the memory size. And generally, they are done with a batch file (shellscript).

Henry


Books: Java Threads, 3rd Edition, Jini in a Nutshell, and Java Gems (contributor)
pavi
Ranch Hand

Joined: Jan 23, 2008
Posts: 31
Thanks Henry

I created a batch file and is working fine.

now i want to know is there is any maximum size that we can specify or the size can reach upto 1GB or more.do the size depends on the ram size of our system?

thanks in advance
Latha.s
Joe Ess
Bartender

Joined: Oct 29, 2001
Posts: 8927
    
    9

It depends on the memory available.
Of course, if you follow my suggestion and don't read the whole file into memory, you will not have to worry about memory.
Vlado Zajac
Ranch Hand

Joined: Aug 03, 2004
Posts: 245
There is definitely a maximum. Memory used by JVM cannot be higher than what the OS allows to allocate.

Heap size should also be somewhat smaller than total RAM since otherwise garbage collection will cause trashing (slowing down entire system very badly).

It is possible to split arbitrary large file without need for large memory.
pavi
Ranch Hand

Joined: Jan 23, 2008
Posts: 31
Don't try to read the entire file into memory.
Read a small chunk, write it out and repeat.


Hi joe

i didnt get your concept.actually i need to get the file length and then split that file into 100kb file and pass it to queue and from queue another program collects that files and merge all that splitted files into one file to get the original file size.here how to do as you have told.
do help me if you have any suggestions.
thanks in advance
Latha
Joe Ess
Bartender

Joined: Oct 29, 2001
Posts: 8927
    
    9

Originally posted by latha selva:
actually i need to get the file length

java.io.File has a method for that.


and then split that file into 100kb file

read 100kb and write it to a file. Do that over and over until the input stream ends.


and pass it to queue and from queue another program collects that files

Presumably you are doing this now with the files you are creating. Nothing would change.


and merge all that splitted files into one file to get the original file size.

I'm unclear as to whether your program is merging the files, or if the other program is. If your program has to do it, it is simply the reverse of splitting the files. Open an output stream and read each of your 100kb inputs in turn.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: is it possible to process a 500MB zipfile in java?