fred rosenberger wrote:"best" and "optimum" are subjective terms. Best in terms of
a) program complexity/simplicity?
b) memory consumption?
d) fault tolerance?
several of these are conflicting - in other words, you can't have it all. You need to define what is most important, what is least, and HOW important each is.
your second post indicates that memory may be an issue.
Can you read the file a line (or 10 lines, or 100 lines) at a time, write them to the output file, then get the next 'chunk'?
Oops..my bad.. I should have been more specific.
Well in terms of what i exactly need , mentioned below are the same
a) Memory Consumption and Speed are the first priority
b) Recoverability would be the second priority
The rest follow ... program complexity / simplicity is not an issue as long as my first 2 priorities can be met.
I could read the file in chunks , and then re-write the output file.
But this is something that I want to avoid , since I want to just delete the first line from the file . Loading and rewriting a 1GB file just to delete the first line from it , is what I don't want to do , would like to know any other approach to this problem ( ( like the sed / tail command in Linux ).
Titan Spectra wrote:Can anybody direct me to an optimum method to just delete the first line from the pipe delimited file , without loading / rewriting the file.
There isn't any such method, let alone an "optimum" method. Like Bill says, operating systems don't support that sort of thing.
If you omit the requirements at the end of that sentence then the optimum method is to read the file one line at a time and write out a new version, not writing the first line. There is absolutely no need to read the entire file into memory.
Paul Clapham wrote:There isn't any such method, let alone an "optimum" method. Like Bill says, operating systems don't support that sort of thing.
Some operating systems would let you get very close. But I don't know of any in widespread use today that support it.
VMS (the Vax OS of the 70s and 80s) stored a file not as a string of bytes, but as an array of records, with a binary record descriptor at the start of each record. For normal ASCII files, each record was just a line of the file. On a Vax you did not have a \n to delimit the line, rather the binary record descriptor at the begining of each line/record had the number of bytes in the record, which could be padded. With this, you could make the first record disappear by simply changing the descriptor for the first record to show zero interesting bytes.
This did not, of course, actually make the file smaller. To do that, you have to read each and every byte of the file, and write out the ones you like.
Doing this for files of a gigabyte or two will not take all that long, assuming you don't do a lot of buffer reallocation/garbage collection. Naturally you want to only read in a buffer at a time.