• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Convert OutputStream to InputStream!

 
Ranch Hand
Posts: 204
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello!

This post probably looks like it belongs in the XML category group, but it really has more to do with converting an

OutputStream to an InputStream.

I'm trying to intercept an output file "out.xml" and re-route it into another function, instead of writing it out to an

EXTERNAL file!

The reason i'm trying to do this is that the generated XML file is very large, 2 billion recs, and i only need to run the XML

file thru a SAX.parse() - validation routine and create a report showing which records have invalid data, and would like to

avoid creating a huge XML file when it's really not needed!

So basically, i'm trying to convert and OutPutStream back into an InputStream and i've been trying the PipeOutputStream

approach, but am not making very good progress!

Thanks very much for any ideas!
bc

Here is what Oracle is saying to use for LARGE XML SAX processing and my Pipe.... code follows this example:



Here are my attempts to intercept that "out.xml" file and re-route it to the "sp.parse()" function!

 
bob connolly
Ranch Hand
Posts: 204
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I found this in one of the O'Reilly Java in a Nutshell, so i'll be giving this a once over x 5, but if you may happen to have a simpler solution, i'd be glad to hear about it!

Thanks very much!

 
bob connolly
Ranch Hand
Posts: 204
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello!

This is a followup from a previous post on using PipedInputStreams!

I've been trying to use a PipedInputStream approach as a way to validate some VERY LARGE XML datasets and i'm continuing to get
an "OutofMemory error", which is very suprising because the PipedInputStream approach is supposed to Buffer
the data thru memory!
And in these programs i'm also using 2 seperate threads!

Below i have the progams listed, SX4 uses the OraPipe1 to set up the InputStream and
the OraPipe1 calls the OraPipe2 which retrieves the Oracle data and builds the XML

SX4 > OraPipe1 > OraPipe2

--------------------------------------------------------------------------------------------------------------------------
 
bob connolly
Ranch Hand
Posts: 204
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello!

I think i found the problem, the OutofMemory is from the size of the JDBC Resultset, so i'm now using Oracle's SAXSerializer and routing that output to that PipedOutputSteam in the OraPipe2 and it's working fine!

So thanks much for reviewing this problem!

bc
 
I'm thinking about a new battle cry. Maybe "Not in the face! Not in the face!" Any thoughts tiny ad?
a bit of art, as a gift, that will fit in a stocking
https://gardener-gift.com
reply
    Bookmark Topic Watch Topic
  • New Topic