ejaz khan

Dear Friend,

You can use Jprofiler 7.2.2 for dynamic instrumentation of your Java program
You will be abl to do remotely perform profiling very easily
Download a copy http://www.ej-technologies.com/

If multiple business components changing the database than situation is not easy
You can use dirty bit option in the database against each record. Any component which is making any DML should set the value to '1'
And after 2 minutes you can check only those records which are having dirty bit = '1'
You will refresh your cache and set the dirty bit to '0'
Maintaining two copies of Cache is also a good option

Another point, do not let your Java program to do duplicate hunting, instead create the unique/primary key rules on the DBMS and let the DBMS fail your duplicate records
Secondly, if the records do not have dependency, you can also use split file technique of UNIX. In this way, you will have smaller sized multiple files to process.
You can than use multiple threads to read split files in parallel and it will increase your read efficiency

I would suggest before start processing, open the file in EditPlus 3.41, goto Edit -->Delete-->Delete Duplicate Lines
It will help you remove all the duplicate records quickly

Why can't we use the batch size less than 10k?
Is it a 32 bit or 64 bit system because in 32 bit you will not be able to allocate memory to avoid swap space errors
is it on unix or windows?
stack size tuning helps resolving memory issues
can you implement a check on source DB to provide you data in smaller chunks?
is your code scalable so you can run multiple java processes?