Win a copy of Spring Boot in Practice this week in the Spring forum!

ejaz khan

Greenhorn
+ Follow
since Apr 08, 2013
Cows and Likes
Cows
Total received
0
In last 30 days
0
Total given
0
Likes
Total received
0
Received in last 30 days
0
Total given
2
Given in last 30 days
0
Forums and Threads
Scavenger Hunt
expand Ranch Hand Scavenger Hunt
expand Greenhorn Scavenger Hunt

Recent posts by ejaz khan

Dear Friend,

You can use Jprofiler 7.2.2 for dynamic instrumentation of your Java program
You will be abl to do remotely perform profiling very easily
Download a copy http://www.ej-technologies.com/
9 years ago
If multiple business components changing the database than situation is not easy
You can use dirty bit option in the database against each record. Any component which is making any DML should set the value to '1'
And after 2 minutes you can check only those records which are having dirty bit = '1'
You will refresh your cache and set the dirty bit to '0'
Maintaining two copies of Cache is also a good option
9 years ago
Another point, do not let your Java program to do duplicate hunting, instead create the unique/primary key rules on the DBMS and let the DBMS fail your duplicate records
Secondly, if the records do not have dependency, you can also use split file technique of UNIX. In this way, you will have smaller sized multiple files to process.
You can than use multiple threads to read split files in parallel and it will increase your read efficiency
9 years ago
I would suggest before start processing, open the file in EditPlus 3.41, goto Edit -->Delete-->Delete Duplicate Lines
It will help you remove all the duplicate records quickly
9 years ago
Why can't we use the batch size less than 10k?
Is it a 32 bit or 64 bit system because in 32 bit you will not be able to allocate memory to avoid swap space errors
is it on unix or windows?
stack size tuning helps resolving memory issues
can you implement a check on source DB to provide you data in smaller chunks?
is your code scalable so you can run multiple java processes?
9 years ago