Algorithem to remove duplicates from multiple xls sheets.

[Bala Tilak]

Hi Everybody,

I am working on task to remove the duplicates from xls sheets that contains mobile numbers, (no mobile number should be repeated with in the xls sheet or accross xls).
Finally i have to generate the csv files with specific number of mobile numbers ( like 1000 numbers in each). Previously i wrote a java program to upload the numbers from xls sheets to db (ms access) by keeping the mobile number as primary key. and finally generate the csv files by reading the mobile number data from db.
But as the data increases the performance of the program is too worst...

Actually my job is to get the mobile numbers data from xls sheets and remove the duplicates accross the sheets and generate the csv files with fixed number of mobile numbers.

Could anyone suggest me the how can i simplify the process....

Regards,

[john price]

This might not be efficient, but it should work. The comments are pseudo-code, since I don't know specifics of your code and files.

 
//read a line (While statement)
        if(!arrayListWithoutDuplicates.Contains(lineToBeRead))  
              arrayListWithoutDuplicates.add(lineToBeRead);  
        } 
//end of while statement

//Break "arrayListWithoutDuplicates" into multiple ArrayLists
//clear() the contents of "arrayListWithoutDuplicates"
//Write the contents of each ArrayList to a separate file

OR

 
//read a line (While statement)
        if(!arrayListWithoutDuplicates.Contains(lineToBeRead))  
              arrayListWithoutDuplicates.add(lineToBeRead);  
        } 
//end of while statement

//while (arrayListWithoutDuplicates.size > 0) {
//take first 1000 entries of ArrayList
//write to a file
//delete first 100 entries of ArrayList
//}

Hopefully this gives you some ideas on how to solve your "issues",
John Price