is it possible to process a 500MB zipfile in java?

[pavi]

In my project i need to split a zipfile of any size into 1MB files and then merge it to get the original file.My java code works fine upto 58MB zipfile but for the file size greater than 58MB i get the following exception

"Exception in thread "main" java.lang.OutOfMemoryError: Java heap space"

Is it possible to process a 500MB zipfile in java?
If possible how to process.
please help me to resolve this.

Thanks in advance,
Latha

[Rob Spoor]

You can use the -Xmx flag; type in "java -X" in a command window for more info.

Moving to the beginner's forum.

[Joe Ess]

Don't try to read the entire file into memory.
Read a small chunk, write it out and repeat.

[pavi]

i read that The -X options are non-standard and subject to change without notice.

below is a code to find the memory size

System.out.println("MAX MEMORY = " + Runtime.getRuntime().maxMemory() / 1048576 + "MB");
System.out.println("TOTAL MEMORY = " + Runtime.getRuntime().totalMemory() / 1048576 + "MB");


when i run this it prints 63MB as MAX MEMORY and 4MB as total.

and after the below command

java -Xms512m -Xmx512m MyApplication

it prints 512MB.

but again if i just give

java Application
i get the memory size as only 63MB

what does it means?so for every time we run the file should we set the size or what? how to set the size permanantly for the application to handle bigfile?please do help me .

Thanks in advance
Latha

[Paul Sturrock]

i read that The -X options are non-standard and subject to change without notice.

Indeed. If sun release a new JVM and the rename these to "minimumHeapSize" and "maximumHeapSize" you will need to change how you run your process. Is that an issue?

[Henry Wong]

what does it means?so for every time we run the file should we set the size or what? how to set the size permanantly for the application to handle bigfile?please do help me .

For most bigger applications, there are generally a ton more switches than settings than the memory size. And generally, they are done with a batch file (shellscript).

Henry

[pavi]

Thanks Henry

I created a batch file and is working fine.

now i want to know is there is any maximum size that we can specify or the size can reach upto 1GB or more.do the size depends on the ram size of our system?

thanks in advance
Latha.s

[Joe Ess]

It depends on the memory available.
Of course, if you follow my suggestion and don't read the whole file into memory, you will not have to worry about memory.

[Vlado Zajac]

There is definitely a maximum. Memory used by JVM cannot be higher than what the OS allows to allocate.

Heap size should also be somewhat smaller than total RAM since otherwise garbage collection will cause trashing (slowing down entire system very badly).

It is possible to split arbitrary large file without need for large memory.

[pavi]

Don't try to read the entire file into memory.
Read a small chunk, write it out and repeat.

Hi joe

i didnt get your concept.actually i need to get the file length and then split that file into 100kb file and pass it to queue and from queue another program collects that files and merge all that splitted files into one file to get the original file size.here how to do as you have told.
do help me if you have any suggestions.
thanks in advance
Latha

[Joe Ess]

Originally posted by latha selva:
actually i need to get the file length

java.io.File has a method for that.

and then split that file into 100kb file

read 100kb and write it to a file. Do that over and over until the input stream ends.

and pass it to queue and from queue another program collects that files

Presumably you are doing this now with the files you are creating. Nothing would change.

and merge all that splitted files into one file to get the original file size.

I'm unclear as to whether your program is merging the files, or if the other program is. If your program has to do it, it is simply the reverse of splitting the files. Open an output stream and read each of your 100kb inputs in turn.