• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Corrupt GZIP trailer

 
Ranch Hand
Posts: 204
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello this seems to work fine for smaller files, ie 10 million or so, but when i try to create a 40+ million file i get that Corrup GZIP trailer!

I've tried many combinations of .close, .finish, .flush ect, but for some reason, i quess it's not writing the last buffer of data!

Thanks very much for an insights!



[code]
try
{
textReader = new BufferedReader(new FileReader(xp.dat_file));

GZIPOutputStream gz = new GZIPOutputStream(new FileOutputStream(xp.xml_file));
PrintWriter outf = new PrintWriter(gz);

String line = null; String sToken = null;

String hdr1=("<?xml version=#1.0# encoding=#UTF-8#?>"); hdr1=hdr1.replace('#','"');

outf.write(hdr1+"\n");
outf.write(" <hlp_data>"+"\n");

int pcnt = 0;
while ( (line = textReader.readLine()) != null )
{
pcnt++; debug_cnt++;
if (line.length() < 5) { System.out.println("File has bad data"); break;}

String[] sa = new String[NUM_COLS];
int previ=0; int t=0;int fcnt=0;

for (int s=0;s<line.length();s++)
{
if (line.charAt(s)=='|')
{
sa[t]=line.substring(previ,s);fcnt++;
t++; previ=s+1;
}
}
sa[t]=line.substring(previ,line.length());fcnt++;

if (NUM_COLS!=fcnt) {throw new NullPointerException("ddl cnt != field cnt: "+NUM_COLS+" != fieldcnt:

"+fcnt);}
outf.write(" <"+FILE_NAME+">"+"\n");

if (NUM_COLS==fcnt)
{

for (int i = 0; i < NUM_COLS; i++)
{
if (i>(sa.length-1)) { sToken = new String(); } else { sToken = sa[i]; }

if (sToken.indexOf('<')>-1 || sToken.indexOf('>')>-1 || sToken.indexOf('"')>-1||

sToken.indexOf('&')>-1 )
{
String sToken2 = new String();
sToken2 = normalize(sToken);
sToken = sToken2;
}
outf.write(" <"+_columnNames[i]+">"+sToken+"</"+_columnNames[i]+">"+"\n");
}

}
outf.write(" </"+FILE_NAME+">"+"\n");
}

outf.write("</hlp_data>"+"\n");

textReader.close();
gz.finish();
gz.close();
outf.flush();
outf.close();
}
catch (IOException e) {System.out.println("HlpValPipExtXml1/Error2/reading dat_file" + e);}
}

[\code]
 
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Bob,

I don't know what file system you're using, but it sounds like ntfs.
And you can't create a zip file greater than 4gb in Windows !

I had a similar problem and I had to change to code so that it would switch to another zip file (z00, z01, etc...) when a certain limit was reached.
 
bob connolly
Ranch Hand
Posts: 204
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks very much Serge, i'll be looking into that!

Have a nice week!
 
bob connolly
Ranch Hand
Posts: 204
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi again Ernest!

I think there might be a 'bug' in the GZIPInputStream pgm, because when i use the following override fix program from Sanger labs, my SAX parse validation works fine!

And actually, the GZIPInputStream works fine for files under 300million or so, but over that it runs into the 'Corrupt Trailer' message!

By the way, I got this override pgm from one of the references in the previous post!

//package uk.ac.sanger.artemis.util;

import java.io.*;

/**
* A wrapper class to work around a bug in GZIPInputStream. The read()
* method sometimes throws a IOException complaining about a Corrupt GZIP
* trailer on the jdk 1.1.8-13 on the Alphas. This wrapper catches and
* ignores that exception.
* @author Kim Rutherford <kmr@sanger.ac.uk>
* @version $Id: WorkingGZIPInputStream.java,v 1.1 2004/06/09 09:53:17 tjc Exp $
**/

public class WorkingGZIPInputStream extends java.util.zip.GZIPInputStream {
/**
* Creates a new input stream with the specified buffer size.
* @param in the input stream
* @param size the input buffer size
* @exception IOException if an I/O error has occurred
**/

public WorkingGZIPInputStream (InputStream in, int size)
throws IOException {
super (in, size);
}

/**
* Creates a new input stream with a default buffer size.
* @param in the input stream
* @exception IOException if an I/O error has occurred
**/
public WorkingGZIPInputStream (InputStream in)
throws IOException {
super (in);
}

/**
* Calls super.read() and then catch and ignore any IOExceptions that
* mention "Corrupt GZIP trailer".
**/
public int read (byte buf[], int off, int len)
throws IOException {
try {
return super.read (buf, off, len);
} catch (IOException e) {
if (e.getMessage ().indexOf ("Corrupt GZIP trailer") != -1) {
return -1;
} else {
throw e;
}
}
}
}

Thanks again Ernest!
 
Don't get me started about those stupid light bulbs.
reply
    Bookmark Topic Watch Topic
  • New Topic