Batch updation failure because of duplicate records

[Sandeep Swaminathan]

I'm trying to upload the records from a CSV file into the database. On the process I'm trying to validate those records extracted from the CSV file. Because of performance issue, I thought of using batch update which will return me an array of successfully loaded index. But even if there's a duplication of one record, the entire batch is failing and the records are not inserted into the database. Am I not supposed to use batch update for such cases? Then how do I resolve the performance and cost issue as I dont want to hit the database for each and every insertion of record.

NOTE: I'm not supposed to use LOAD DATA INFILE and use only INSERT INTO command.

[Ireneusz Kordal]

Load data into temporary table, create index(es) on this temp table (to make query faster),
then execute query to see if duplicates exist and remove duplicates,
then update the table.