"Corrupt GZIP Trailer"?

[Rudy Rusli]

I got "Corrupt GZIP trailer" exception message in my program. I'm not sure what this is. Could somebody please advice?

The code is actually fine when I run it in my local machine (windows).
It retrieves the data just fine.
But when I run it in the server (linux), it gives me this exception.

Thanks.

-----------------------------------------------------------------------------------------------------

GZIPInputStream gzip_in_stream = null;
BufferedOutputStream destination_out_stream = new BufferedOutputStream(
   new FileOutputStream(weatherFile), BUF_SIZE);
   byte[] input_buffer = new byte[BUF_SIZE];
int byteLength = 0;          
 
InputStream tempStream = ftpClient.retrieveFileStream(zippedFile);
               
gzip_in_stream = new GZIPInputStream(
new BufferedInputStream(tempStream));
 
while ( (byteLength = gzip_in_stream.read(input_buffer, 0, BUF_SIZE)) > 0 )
{
destination_out_stream.write(input_buffer, 0, byteLength);     
}
 
destination_out_stream.flush(); // Ensure all the data is written to the output.
destination_out_stream.close();
gzip_in_stream.close();  

[Guy Allard]

Rudy - is the input file large? By that I mean does it exceed the size supported in the Linux OS you are using? Is it > 4GB?

I think there are known problems in that area.

Try catching that specific exception, logging that it occurred, and ignore it. That might work.

Guy

[Jim Yingst]

I'm not familiar with that particular exception, but from the fact it's complaining about a trailer, my first thought is that the file didn't get transferrred completely. Looking at your code though, I don't see any reason why this might be the case. You correctly flush the streams, which would otherwise be the most likely cause of the problem you describe. So, what else is there to try?

I guess you're using org.apache.commons.net.ftp.FTPClient, correct? Have you used the setFileType() and setFileTransferMode() methods? Those could well be important. In particular I note that the default file type is FTP.ASCII_FILE_TYPE - you should almost certainly change this to FTP.BINARY_FILE_TYPE. If that doesn't work, try experimenting with other options for both file type and file transfer mode.

In the while loop, you might replace > 0 with >= 0. Fundamentally this should be unnecessary, as read() should block until at least one byte is available. But it's possible there's a bug in the implementation. Might was well check.

Simlarly, you could call close() on the FileOutputStream. Again, this should be unnecessary, but try it and see if it makes a difference. You must be careful though to close the BufferendOutputStream first, or you'll introduce a new bug.

If none of this works, try narrowing the problem down a bit by separating the FTP file transfer from the decompression. Try taking the bytes you get from FTP (still in gzipped format) and writing them to a local file. Then read the local file and uncompress it. This takes a little longer, but gives you an intermediate result to look at. Can you also look at the gzipped file on the server? Then you could compare the lengths, to see if they're the same. If you can also transfer files from one machine to another using some method other than you Java program, then you can compare the two files even more direcly - e.g. using unix's cmp for a byte-by-byte comparison. In this way you can at least discover whether the problem is with the FTP transfew process, or the GZIP decompression.

Hope that helps...

[Rudy Rusli]

The file is actually not big. After uncompression, the size only around 30 MB. How do I find out the size supported in the Linux OS I'm using?

Setting the file type into BINARY_FILE_TYPE seems to solve the problem.
Thanks.

- Rudy -