Win a copy of Spring Boot in Practice this week in the Spring forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Tim Cooke
  • Ron McLeod
  • Jeanne Boyarsky
  • Paul Clapham
Sheriffs:
  • Liutauras Vilda
  • Henry Wong
  • Devaka Cooray
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Tim Holloway
  • Al Hobbs
  • Carey Brown
Bartenders:
  • Piet Souris
  • Mikalai Zaikin
  • Himai Minh

Memory concerns in writting / appending a file

 
Ranch Hand
Posts: 43
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello,

We have an application where I need to put huge amount of records in a flat file. The records are text and binary (images) and the size of the file will probably run int0 700 MB to 800 MB.

I would be firing various queries/SPs and plan to append the data to the output file (The output file will be first created when the first query/SP is fired and subsequently the data will be appended to it).

My question is:

Is this a feasible idea? When I say append to an existing file, I am not sure, if that requires the ENTIRE file to be read in memory and then append the part� Is that how io works in java? If that happens than I will again run into memory issues. OR is it that when I say append to an existing file than io will only flush out my new record to an existing flat file without reading the entire file in memory(and make life simple for me ) . Kindly advice on what is the best approach for this .

Thanks in advance,
Milan
 
Wanderer
Posts: 18671
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
[milan]: Is this a feasible idea?

Sure.

When I say append to an existing file, I am not sure, if that requires the ENTIRE file to be read in memory and then append the part� Is that how io works in java?

No, you can append to a file without reading any of the preceding contents. FileOutputStream has a constructor that allows you to specify append mode - see the API for details. There are several other classes you could use here intead, but I think all can append easily enough. It doesn't sound like this will be a problem for you.
 
Ranch Hand
Posts: 961
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
However if you need to edit a particuar record in the file and you do not want to read the whole file to find the record you want to edit, then you might consider java.io.RandomAccessFile.

But if you are just going to append it will be enough by specifying so in the constructor of your OutputStream.

You could increase performance if you use a BufferedOutputStream when writing the file to disk. Try a buffer of 1024 or 2048 bytes and see how it improves performance over an approach based on writing just one byte at a time. Because BufferedOutpuStream just writes the files to disk until the buffer is full.
 
Bartender
Posts: 9626
16
Mac OS X Linux Windows
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Originally posted by Edwin Dalorzo:
Because BufferedOutpuStream just writes the files to disk until the buffer is full.



Just so we are all clear here, BufferedOutputStream writes files to an array in memory until the array is full, then it flushes the contents to disk. This can increase performance because in general, one big write is less expensive than lots of little writes. As with all performance, your results may vary. There is no substitute for benchmarking and experementing.
The default size of the buffer in BufferedOutputStream (JDK 1.5.0_03) is 8k.
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
reply
    Bookmark Topic Watch Topic
  • New Topic