• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Ron McLeod
  • Rob Spoor
  • Tim Cooke
  • Junilu Lacar
Sheriffs:
  • Henry Wong
  • Liutauras Vilda
  • Jeanne Boyarsky
Saloon Keepers:
  • Jesse Silverman
  • Tim Holloway
  • Stephan van Hulst
  • Tim Moores
  • Carey Brown
Bartenders:
  • Al Hobbs
  • Mikalai Zaikin
  • Piet Souris

Unable to zip multiple large files in java using jdk1.6

 
Ranch Hand
Posts: 165
Tomcat Server Java Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi, I'm trying to zip multiple large files which are already zipped means multiple zip files into one main(single) zip file using java utils, zipoutputstream but it is not zipping large files. My requirement is to have .zip file only. Just for testing purpose, I have zipped it into .gz using Gzoutputstream which worked fine but im not able to zip using zipoutputstream. Interestingly it is not throwing any exceptions also. It generates .zip file in the end but that .zip file is not correct because when i try to unzip it using winzip or 7zip utility on windows it throws error saying 'end of file not found' something. Total size of these multiple zip files are approximately 3GB. Is there anyway to zip such a large files using JDK1.6. Any suggestions.



It is not printing logger SUCCESSFULLY ZIPPED.

 
Rancher
Posts: 43026
76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Well, what is it printing instead?

It also sounds like you might be running into a 32bit/64bit issue, so be sure to run the 64bit JVM.

Also, the code looks rather complicated. For example, what do you need the ByteArrayInputStream and BufferedInputStream for? You already have the data to be written in the buf[] - which you can write directly to the ZipOutputStream.
 
Sheriff
Posts: 26775
82
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
If I recall correctly, the original PKZIP specification could only handle files which were no greater than 2 MB in size. (The size of the file was stored in a fixed-size field which couldn't store numbers larger than that.) By the time 2-megabyte-files started to appear, Phil Katz didn't have control of the spec any more and various actors implemented various workarounds for that issue. But Java's zip-archive classes continued to implement the original spec rather than any of the extensions. Whether that's changed or not, I don't know.
 
Vinod Vijay
Ranch Hand
Posts: 165
Tomcat Server Java Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Ulf Dittmer wrote:Well, what is it printing instead?

It also sounds like you might be running into a 32bit/64bit issue, so be sure to run the 64bit JVM.

Also, the code looks rather complicated. For example, what do you need the ByteArrayInputStream and BufferedInputStream for? You already have the data to be written in the buf[] - which you can write directly to the ZipOutputStream.


Hi Dittmer, that was my second approach to make the things work.


My original code is like this only:-



Now can you please suggest me if it is JVM 32/64 bit issue or something else? It is not printing the logger "@@@@@@@@@@@@@ Zipped Successfully @@@@@@@@@@@@@@@@", going to finally block, closing all the streams. In the end, .zip is there but as I said, it is not complete zip (Max size of final zip is 1GB). I'm not able to unzip it. Looks like after certain writing to zip file, it is skipping, incomplete only. is there any size limit issue of creating a zip file? Thanks
 
Paul Clapham
Sheriff
Posts: 26775
82
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Vinod Vijay wrote:is there any size limit issue of creating a zip file? Thanks



I thought I mentioned that in my post. Perhaps you overlooked it?
 
Vinod Vijay
Ranch Hand
Posts: 165
Tomcat Server Java Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Paul Clapham wrote:

Vinod Vijay wrote:is there any size limit issue of creating a zip file? Thanks



I thought I mentioned that in my post. Perhaps you overlooked it?



Paul, I have checked that. JVM is 64bit only. Restriction is there as this program is running on production.
 
Ulf Dittmer
Rancher
Posts: 43026
76
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Vinod Vijay wrote:Paul, I have checked that. JVM is 64bit only.


What exactly have you checked? Paul didn't say anything about 32bit/64bit JVMs. He was talking about zip file sizes, and potential limits inherent to the ZIP format.

I doubt that there is such a limit, though, at least not one as low as 2MB. But you can -and should- test that. You should also test whether this works with the same number of files, but if they are all small in size.
 
Sheriff
Posts: 10445
227
IntelliJ IDE Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
This:



can potentially cause a OutOfMemoryError (note that it's an java.lang.Error) and won't be caught in any of the catch blocks which catch java.lang.Exception types. Perhaps you are indeed running into an OutOfMemoryError? To test it out, you can change the catch block to:


 
Vinod Vijay
Ranch Hand
Posts: 165
Tomcat Server Java Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Ulf Dittmer wrote:

Vinod Vijay wrote:Paul, I have checked that. JVM is 64bit only.


What exactly have you checked? Paul didn't say anything about 32bit/64bit JVMs. He was talking about zip file sizes, and potential limits inherent to the ZIP format.

I doubt that there is such a limit, though, at least not one as low as 2MB. But you can -and should- test that. You should also test whether this works with the same number of files, but if they are all small in size.



With small number of files, it works fine but not with huge files with large size
 
Consider Paul's rocket mass heater.
reply
    Bookmark Topic Watch Topic
  • New Topic