File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes I/O and Streams and the fly likes Prepending to a file? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » I/O and Streams
Bookmark "Prepending to a file?" Watch "Prepending to a file?" New topic
Author

Prepending to a file?

Shane Burgel
Ranch Hand

Joined: Sep 09, 2003
Posts: 47
I have a case in which I need to prepend a file.

I'm reading a file full of records in reverse, because I am checking for duplicates and only want to keep the final occurrence so I need a way to prepend the output file with the latest record so that the order is preserved.

So far the only method I have found is to write one file with the current record, and the append the main file to the bottom of it, then copy the whole thing back to the main file. I did this using FileChannels and ByteBuffers but it was far too slow to be practical given the large number of records I'm working with. Is there any way to prepend a file using java or groovy that isn't dog slow?

Thanks

Shane
Shane Burgel
Ranch Hand

Joined: Sep 09, 2003
Posts: 47
I also read that most file systems won't allow a prepend, so I may be out of luck. I'm just curious if anyone has thought of a more creative solution.
Jan Cumps
Bartender

Joined: Dec 20, 2006
Posts: 2510
    
  10

The only performance gain I can propose you is to rename the new file to the old file in stead of copying it to the old one.

Sequence would be:
create temp file with different name with the one record.
Append original file to it.
delete original file.
rename temp file to original filename.

If you create the temp file on the same physical device as the original file, then the last rename is a very cheap operation.

Regards, Jan


OCUP UML fundamental and ITIL foundation
youtube channel
Shane Burgel
Ranch Hand

Joined: Sep 09, 2003
Posts: 47

Hmm can't get File.renameTo() to work for some reason. I doubt the performance gained will be enough but I would like to try it out.
Joe Ess
Bartender

Joined: Oct 29, 2001
Posts: 8964
    
    9

Make sure you've closed the streams/readers you are using on the file. You cannot rename an open file.


[How To Ask Questions On JavaRanch]
Jan Cumps
Bartender

Joined: Dec 20, 2006
Posts: 2510
    
  10

Shane Burgel wrote:... I doubt the performance gained will be enough but I would like to try it out.
Copying a file means writing all the bits of that file to disk. Renaming a file means changing one entry in the filesystem's tables.
Shane Burgel
Ranch Hand

Joined: Sep 09, 2003
Posts: 47
I get that, but I'm still reading the file every single time, and it is getting bigger every single time. I found a workaround I think. I did it like this:

I was able to process 90000 records this way in about 15 minutes. There are some drawbacks to doing it this way (like a very cluttered file system if something goes wrong) but it is by far the fastest way I have come up with.

"text" is a constant I set up that contains one record

( I did this in Groovy)




The other possibility is to use something like Smooks, to run through the file once and create a map of the record numbers/unique identifiers (later duplicate record numbers would replace earlier ones), then walk through a second time and check the unique identifiers and only print out the ones that exist in the map.
Shane Burgel
Ranch Hand

Joined: Sep 09, 2003
Posts: 47
I'm guessing that using Smooks will be faster than the above, because it would be two reads and only one write, whereas even my method above would be two reads and two writes.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Prepending to a file?