• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Lots of .nfs files being created

 
Greenhorn
Posts: 15
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I wrote a program that needs to parse through close to 4MB of data that is a compilation of smaller blocks of data which in turn are compilations of even smaller messages. I currently read a block from the file and then write that block to a temp file in order to read/parse through the smaller messages. I noticed that each time I create and then delete the temp file a .nfs[some hex numbers] is created and it creates close to 6000 of these files. I was wondering if there was any way to avoid this because it takes 1-2 minutes to parse a 4 MB file.
 
Sheriff
Posts: 3063
12
Mac IntelliJ IDE Python VI Editor Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Probably you are running Unix or some derivative, and you are creating the temporary files on some remote file system. I could go into why .nfs files get created and what causes them to linger, but it's probably simpler for you to just create your temporary files on the local file system or, even better, don't create temporary files at all ... just parse your smaller messages in memory and then move on.
 
Andrew Robinson
Greenhorn
Posts: 15
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
yes they are on a remote file system. Its the easist way I could think to parse the data. Do you think that the creation of all of those files is what's causing it to run so slow and if I parsed it in memory it would be much quicker??
 
Greg Charles
Sheriff
Posts: 3063
12
Mac IntelliJ IDE Python VI Editor Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Yes, doing things in memory is always faster assuming you have the memory to spare. Disk I/O and network I/O are much slower ... imagine a tortoise racing with a comet. There can be legitimate reasons for using temp files, but definitely avoid them if you can.
 
Andrew Robinson
Greenhorn
Posts: 15
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
The data is a bunch of binary data so shoud I just make a massive byte[] and copy to that rather than writing to a temp file???
 
Greg Charles
Sheriff
Posts: 3063
12
Mac IntelliJ IDE Python VI Editor Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Well, 4 Mb isn't really massive, so bringing it all into memory should be fine. However, you were splitting it up into smaller temporary files before. Wouldn't it be possible to read one chunk of the file, do whatever you need to do with it, and then read the next chunk? Do you need the whole file in memory at once?

I once rewrote a process that was reading a large XML file from a database record, transforming it with a DOM parser, then zipping the result, adding a header, and sending it to a remote client. The way it was implemented, it actually kept six copies of the file in memory in various forms, so as the files started getting larger, the thing became a huge memory hog. Through judicious use of IO streams (and a change to a SAX parser), I was able to read the file, transform, and send it with never more than a single buffer-full of it in memory at any given time.
 
reply
    Bookmark Topic Watch Topic
  • New Topic