Win a copy of Secure Financial Transactions with Ansible, Terraform, and OpenSCAP this week in the Cloud/Virtualization forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Ron McLeod
  • Paul Clapham
  • Jeanne Boyarsky
  • Bear Bibeault
Sheriffs:
  • Rob Spoor
  • Henry Wong
  • Liutauras Vilda
Saloon Keepers:
  • Tim Moores
  • Carey Brown
  • Stephan van Hulst
  • Tim Holloway
  • Piet Souris
Bartenders:
  • Frits Walraven
  • Himai Minh
  • Jj Roberts

is it possible to process a 500MB zipfile in java?

 
Ranch Hand
Posts: 31
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
In my project i need to split a zipfile of any size into 1MB files and then merge it to get the original file.My java code works fine upto 58MB zipfile but for the file size greater than 58MB i get the following exception

"Exception in thread "main" java.lang.OutOfMemoryError: Java heap space"

Is it possible to process a 500MB zipfile in java?
If possible how to process.
please help me to resolve this.

Thanks in advance,
Latha
 
Sheriff
Posts: 22127
114
Eclipse IDE Spring VI Editor Chrome Java Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
You can use the -Xmx flag; type in "java -X" in a command window for more info.

Moving to the beginner's forum.
 
Bartender
Posts: 9625
16
Mac OS X Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Don't try to read the entire file into memory.
Read a small chunk, write it out and repeat.
 
pavi
Ranch Hand
Posts: 31
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
i read that The -X options are non-standard and subject to change without notice.

below is a code to find the memory size

System.out.println("MAX MEMORY = " + Runtime.getRuntime().maxMemory() / 1048576 + "MB");
System.out.println("TOTAL MEMORY = " + Runtime.getRuntime().totalMemory() / 1048576 + "MB");


when i run this it prints 63MB as MAX MEMORY and 4MB as total.

and after the below command

java -Xms512m -Xmx512m MyApplication

it prints 512MB.

but again if i just give

java Application
i get the memory size as only 63MB

what does it means?so for every time we run the file should we set the size or what? how to set the size permanantly for the application to handle bigfile?please do help me .

Thanks in advance
Latha
 
Bartender
Posts: 10336
Hibernate Eclipse IDE Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator


i read that The -X options are non-standard and subject to change without notice.


Indeed. If sun release a new JVM and the rename these to "minimumHeapSize" and "maximumHeapSize" you will need to change how you run your process. Is that an issue?
 
author
Posts: 23899
142
jQuery Eclipse IDE Firefox Browser VI Editor C++ Chrome Java Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

what does it means?so for every time we run the file should we set the size or what? how to set the size permanantly for the application to handle bigfile?please do help me .



For most bigger applications, there are generally a ton more switches than settings than the memory size. And generally, they are done with a batch file (shellscript).

Henry
 
pavi
Ranch Hand
Posts: 31
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks Henry

I created a batch file and is working fine.

now i want to know is there is any maximum size that we can specify or the size can reach upto 1GB or more.do the size depends on the ram size of our system?

thanks in advance
Latha.s
 
Joe Ess
Bartender
Posts: 9625
16
Mac OS X Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
It depends on the memory available.
Of course, if you follow my suggestion and don't read the whole file into memory, you will not have to worry about memory.
 
Ranch Hand
Posts: 245
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
There is definitely a maximum. Memory used by JVM cannot be higher than what the OS allows to allocate.

Heap size should also be somewhat smaller than total RAM since otherwise garbage collection will cause trashing (slowing down entire system very badly).

It is possible to split arbitrary large file without need for large memory.
 
pavi
Ranch Hand
Posts: 31
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Don't try to read the entire file into memory.
Read a small chunk, write it out and repeat.



Hi joe

i didnt get your concept.actually i need to get the file length and then split that file into 100kb file and pass it to queue and from queue another program collects that files and merge all that splitted files into one file to get the original file size.here how to do as you have told.
do help me if you have any suggestions.
thanks in advance
Latha
 
Joe Ess
Bartender
Posts: 9625
16
Mac OS X Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by latha selva:
actually i need to get the file length


java.io.File has a method for that.


and then split that file into 100kb file


read 100kb and write it to a file. Do that over and over until the input stream ends.


and pass it to queue and from queue another program collects that files


Presumably you are doing this now with the files you are creating. Nothing would change.


and merge all that splitted files into one file to get the original file size.


I'm unclear as to whether your program is merging the files, or if the other program is. If your program has to do it, it is simply the reverse of splitting the files. Open an output stream and read each of your 100kb inputs in turn.
 
yeah, but ... what would PIE do? Especially concerning this tiny ad:
SKIP - a book about connecting industrious people with elderly land owners
https://coderanch.com/t/skip-book
reply
    Bookmark Topic Watch Topic
  • New Topic