This week's book giveaway is in the Agile and Other Processes forum.
We're giving away four copies of DevSecOps Adventures: A Game-Changing Approach with Chocolate, LEGO, and Coaching Games and have Dana Pylayeva on-line!
See this thread for details.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Ron McLeod
  • Paul Clapham
  • Devaka Cooray
  • Tim Cooke
Sheriffs:
  • Rob Spoor
  • Liutauras Vilda
  • paul wheaton
Saloon Keepers:
  • Tim Holloway
  • Tim Moores
  • Mikalai Zaikin
  • Carey Brown
  • Piet Souris
Bartenders:
  • Stephan van Hulst

Corrupt GZIP trailer

 
Ranch Hand
Posts: 204
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello this seems to work fine for smaller files, ie 10 million or so, but when i try to create a 40+ million file i get that Corrup GZIP trailer!

I've tried many combinations of .close, .finish, .flush ect, but for some reason, i quess it's not writing the last buffer of data!

Thanks very much for an insights!



[code]
try
{
textReader = new BufferedReader(new FileReader(xp.dat_file));

GZIPOutputStream gz = new GZIPOutputStream(new FileOutputStream(xp.xml_file));
PrintWriter outf = new PrintWriter(gz);

String line = null; String sToken = null;

String hdr1=("<?xml version=#1.0# encoding=#UTF-8#?>"); hdr1=hdr1.replace('#','"');

outf.write(hdr1+"\n");
outf.write(" <hlp_data>"+"\n");

int pcnt = 0;
while ( (line = textReader.readLine()) != null )
{
pcnt++; debug_cnt++;
if (line.length() < 5) { System.out.println("File has bad data"); break;}

String[] sa = new String[NUM_COLS];
int previ=0; int t=0;int fcnt=0;

for (int s=0;s<line.length();s++)
{
if (line.charAt(s)=='|')
{
sa[t]=line.substring(previ,s);fcnt++;
t++; previ=s+1;
}
}
sa[t]=line.substring(previ,line.length());fcnt++;

if (NUM_COLS!=fcnt) {throw new NullPointerException("ddl cnt != field cnt: "+NUM_COLS+" != fieldcnt:

"+fcnt);}
outf.write(" <"+FILE_NAME+">"+"\n");

if (NUM_COLS==fcnt)
{

for (int i = 0; i < NUM_COLS; i++)
{
if (i>(sa.length-1)) { sToken = new String(); } else { sToken = sa[i]; }

if (sToken.indexOf('<')>-1 || sToken.indexOf('>')>-1 || sToken.indexOf('"')>-1||

sToken.indexOf('&')>-1 )
{
String sToken2 = new String();
sToken2 = normalize(sToken);
sToken = sToken2;
}
outf.write(" <"+_columnNames[i]+">"+sToken+"</"+_columnNames[i]+">"+"\n");
}

}
outf.write(" </"+FILE_NAME+">"+"\n");
}

outf.write("</hlp_data>"+"\n");

textReader.close();
gz.finish();
gz.close();
outf.flush();
outf.close();
}
catch (IOException e) {System.out.println("HlpValPipExtXml1/Error2/reading dat_file" + e);}
}

[\code]
 
Greenhorn
Posts: 8
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Bob,

I don't know what file system you're using, but it sounds like ntfs.
And you can't create a zip file greater than 4gb in Windows !

I had a similar problem and I had to change to code so that it would switch to another zip file (z00, z01, etc...) when a certain limit was reached.
 
bob connolly
Ranch Hand
Posts: 204
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Thanks very much Serge, i'll be looking into that!

Have a nice week!
 
bob connolly
Ranch Hand
Posts: 204
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi again Ernest!

I think there might be a 'bug' in the GZIPInputStream pgm, because when i use the following override fix program from Sanger labs, my SAX parse validation works fine!

And actually, the GZIPInputStream works fine for files under 300million or so, but over that it runs into the 'Corrupt Trailer' message!

By the way, I got this override pgm from one of the references in the previous post!

//package uk.ac.sanger.artemis.util;

import java.io.*;

/**
* A wrapper class to work around a bug in GZIPInputStream. The read()
* method sometimes throws a IOException complaining about a Corrupt GZIP
* trailer on the jdk 1.1.8-13 on the Alphas. This wrapper catches and
* ignores that exception.
* @author Kim Rutherford <kmr@sanger.ac.uk>
* @version $Id: WorkingGZIPInputStream.java,v 1.1 2004/06/09 09:53:17 tjc Exp $
**/

public class WorkingGZIPInputStream extends java.util.zip.GZIPInputStream {
/**
* Creates a new input stream with the specified buffer size.
* @param in the input stream
* @param size the input buffer size
* @exception IOException if an I/O error has occurred
**/

public WorkingGZIPInputStream (InputStream in, int size)
throws IOException {
super (in, size);
}

/**
* Creates a new input stream with a default buffer size.
* @param in the input stream
* @exception IOException if an I/O error has occurred
**/
public WorkingGZIPInputStream (InputStream in)
throws IOException {
super (in);
}

/**
* Calls super.read() and then catch and ignore any IOExceptions that
* mention "Corrupt GZIP trailer".
**/
public int read (byte buf[], int off, int len)
throws IOException {
try {
return super.read (buf, off, len);
} catch (IOException e) {
if (e.getMessage ().indexOf ("Corrupt GZIP trailer") != -1) {
return -1;
} else {
throw e;
}
}
}
}

Thanks again Ernest!
 
I child proofed my house but they still get in. Distract them with this tiny ad:
New web page for Paul's Rocket Mass Heaters movies
https://coderanch.com/t/785239/web-page-Paul-Rocket-Mass
reply
    Bookmark Topic Watch Topic
  • New Topic