Krishnareddy Yeruva wrote:Hi Ulf,
As Ulf mentioned:
Do these characters belong in the files? If so, you'll need to handle them properly. If not, the easiest may be to not put them there in the first place.
These characters doesn't belong to these files. But by mistake these chars are getting placed inside these files. So if we have any unix script to remove these special chars, so that we can validate and remove such chars from that file before the job is going to execute.
The only thing that makes these characters 'special' is that you don't want them in your files. This is where your problem starts. You either have to define the set of the characters you want to remove or you need to define the set that you can accept but either way you to know the character encoding of the file so you can know how to convert the bytes of the files into characters before the filtering. Once you know the character encoding it is a fairly straight forwards to write a small program in almost any language you will find on the Linux box to filter the content of the file. But I stress - you need to know the character encoding to make this safe.
As Ulf says, the best approach is not to put the 'specials' there in the first place and in your position this would be my first point of attack. I would go back to the people who provided the files and ask them to provide clean files with a known character encoding.