This week's book giveaway is in the Mac OS forum.
We're giving away four copies of a choice of "Take Control of Upgrading to Yosemite" or "Take Control of Automating Your Mac" and have Joe Kissell on-line!
See this thread for details.
The moose likes I/O and Streams and the fly likes Split and Merge large zip files Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


JavaRanch » Java Forums » Java » I/O and Streams
Bookmark "Split and Merge large zip files" Watch "Split and Merge large zip files" New topic
Author

Split and Merge large zip files

Vinod Borole
Greenhorn

Joined: Feb 25, 2009
Posts: 26
Hello

Can anyone tel me how can i split and then merge large zip files using java?

Thank you!
Greg Charles
Sheriff

Joined: Oct 01, 2001
Posts: 2853
    
  11

By split you mean unzip, and by merge you mean zip everything into one archive? Check into the java.util.zip package.
Vinod Borole
Greenhorn

Joined: Feb 25, 2009
Posts: 26
I have a .NET application that reads a 1GB size zip file and breaks it into 5 parts of each 200MB, one by one each part is sent to my java application in a multipart request. My java applications downloads each part and store in a location. Now as i have all the parts, i need to merge them together into a 1GB zip file as it was original.
I am not finding a way how to merge them, Can you please suggest?

Note: The .NET application was throwing Out of memory exception when it tried sending a 1GB zip directly, the maximum it could send was 200MB.
Greg Charles
Sheriff

Joined: Oct 01, 2001
Posts: 2853
    
  11

Ah, so it's slicing apart a large file into smaller chunks. In that case, it doesn't really matter that it was a zip file to start with. You need a FileInputStream to read the five files in order, and a FileOutputStream to output them to a single reassembled file. For each file, read some bytes into a byte array (4096 is probably a good size to use) then write that byte array to the assembled file. When you run out of bytes in one file, open the next one and keep going.
Rob Spoor
Sheriff

Joined: Oct 27, 2005
Posts: 19697
    
  20

That will work if the file is split in the simple "let's cut here and continue with the next one" way. Other splitting / merging methods may put some metadata in the split files.


SCJP 1.4 - SCJP 6 - SCWCD 5 - OCEEJBD 6
How To Ask Questions How To Answer Questions
Paul Clapham
Bartender

Joined: Oct 14, 2005
Posts: 18570
    
    8

Greg Charles wrote:For each file, read some bytes into a byte array (4096 is probably a good size to use) then write that byte array to the assembled file. When you run out of bytes in one file, open the next one and keep going.
A SequenceInputStream might help with this process.
Vinod Borole
Greenhorn

Joined: Feb 25, 2009
Posts: 26
I have tried a sample project where I have broken down 67MB zip file into 2 parts of one of 50MB and the other of 17MB. It's taking me more than 30mins to merge them together with the below code.
the zip file after merging that i get is of perfect size, but when i extract it its saying corrupted.

Can any tel me what am i doing wrong? Also, if anyone can suggest me fastest way of doing the same work, as its taking long time to merge with this code.



Greg Charles
Sheriff

Joined: Oct 01, 2001
Posts: 2853
    
  11

Don't use readLine(). That's for reading text, and may be changing the line endings. You don't need anything fancy. Even the DataInput/OutputStreams seem unnecessary, and you will do your own buffering so the the Buffered Streams are overkill as well. Create an array of bytes, read bytes from the input into the array, from the return value figure out how many bytes you read, and write them to the output. It shouldn't take anywhere near that long for 67 megs.
 
GeeCON Prague 2014
 
subject: Split and Merge large zip files