wood burning stoves
The moose likes Linux / UNIX and the fly likes Automate Stripping of a record Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Engineering » Linux / UNIX
Bookmark "Automate Stripping of a record" Watch "Automate Stripping of a record" New topic

Automate Stripping of a record

Muhammad Imad Qureshi
Ranch Hand

Joined: Sep 13, 2005
Posts: 238
I am new to Unix scripting and here is what I am looking for. We are sometime required to strip records from our input data file because the job fails due to erroneous record. We strip by record number. So lets take a typical file. A header record, data and then a trailer record (that contains the total count of file and looks like this: TR5009P6160480001001000023911 (this is a real example).

"TR50" is what all trailer records start with "09P6160480" is what we call a reference number for the job. "001001000023911" is the total number of records in the file.

Now Lets say I want to delete record number "4301" then I would first make a backup copy of the file and call it "filename.backup". Then I would open the original file using "vim". Then I do "set number" and then type "4301 G" to go that record. Then I copy the record to a different file to send it to business so they can look at it and then strip the record out and reduce the count of trailer record by one. In case above the new trailer record would be "TR5009P6160480001001000023910". I save the file and restart the failed jobs.

Can someone help me write a script to automate this. How can I automate the process of copying a record and email it to business before stripping it.

Gerardo Tasistro
Ranch Hand

Joined: Feb 08, 2005
Posts: 362
How do you know which row needs to be deleted?
Muhammad Imad Qureshi
Ranch Hand

Joined: Sep 13, 2005
Posts: 238
When the job fails it gives the record number it failed on. So We know what record to strip. We have been successfully doing that for over a year now. I just want to automate this process.
Gerardo Tasistro
Ranch Hand

Joined: Feb 08, 2005
Posts: 362
Note: I'm asuming your file is called exactly like the trailing record. In this example TR5009P6160480001001000023911 means the file, NOT the trailing record.

I believe awk should provide a solution

This will provide the record which faulted

This will provide the file witout the faulty row

Then you'll just need to create a script that creates the new names

As tip...

This will tell you the number of rows the file has

you can run it before and after awk to obtain the two numbers 23911 and 23910

Then use sed to replace 23911 with 23910 using regular expressions. You'll need to be carefull about bufferin 0s when for example you have 10000 rows turning into 9999 rows.
Andrew Monkhouse
author and jackaroo
Marshal Commander

Joined: Mar 28, 2003
Posts: 11777

You also said you wanted to forward the failed record. Probably something like:

You probably want to automate this a little further, which would probably require automatically finding the record that failed.

Assuming you have a report along the lines of:

You could extract the record number with something like:

And then use that variable with the other statements ...


The Sun Certified Java Developer Exam with J2SE 5: paper version from Amazon, PDF from Apress, Online reference: Books 24x7 Personal blog
Stefan Wagner
Ranch Hand

Joined: Jun 02, 2003
Posts: 1923

Is TR5009P6160480001001000023910 the first line of the file, and the total number of records always running from column 15 to 25?

1 001 000 023 910 is a pretty big number for records. Is that the number of records in a single file?

I don't know how sed handles files with more than billions of records/lines. And the arithmetic fails with leading zeros.

First, 0100100002... is interpreted as an octal number, because of the leading 0, but 3910 contains a 9 which is too big for an octal number.

Second, if your number is 10000000000000 and you reduce it by one, you won't get 099..., but 99..

I agree. Here's the link: http://aspose.com/file-tools
subject: Automate Stripping of a record
jQuery in Action, 3rd edition