I have a utility program which populates data from table1 to table2 after doing some validation on each row. In table1 i have multiple rows for the same account(Lets say 1 account will have 10 rows,like that i will have many accounts).First I will be fetching all the distinct account from the table1 and then I need to write a mutlithreaded program which can run this utility program concurrently for all the accounts. I am confused how exactly how should i write this program?
I am thinking to use ThreadPoolExecutor and will create specific task for my utility program. But i am confused that should i first create a list of all my task having my account and invoke the method invokeAll(taskList). And in invokeAll Should I loop through all my accounts and invoke the run method on the Callable task.
Also, i want to know that how can i find out that actually my program is multithreaded and its behaving in the same manner?
From your description, the requirement looks something like this
1. The multithread utlity (lets call it that way for now), should accept the number of threads which is dynamic (i.e. one for each account)
2. Each task will operate on one account and take the rows from table1 and populated to table2
The best way to see how it works is to experiment with it in general. Just create a small class and see how it works. There is no shortcut there.
Joined: Nov 13, 2012
Yes you understood correctly.Can you give me a suggestion how should i read the accounts from the table? I will be having lacs of account, should i read account one by one, or read 100 accounts and submit all 100 accounts to process it and then again read 100 more.
Also I want to know how should i go about doing connection pooling.what will be the best way. It is gonna be a standalone program. I can't use tomcat or jboss connection pooling capability for this.
Based on the amount of data you are talking, i strongly feel its a classic example of ETL (extrct, transform, and load). Are you sure you want to handle it the java's way ?
Even though you think concurrency can make it faster, it all drills down to database capabilities too. I see that is where you are going to encounter problems going forward.
Just to get a brief idea, you can do following experiment
create two tables and populate say 1 lakh records into one of the table.
have one simple class that takes data from one table and populate into another
no need to use any synchronization
use straignt forward JDBC since one or two connections are needed, no need to use pooling
That should get the idea of what we are going to get eventually.
If you have already done and very sure then
- reading the accounts is a straight forward way. Is your question about how to code it ?
- processing 100 (or some other number) depends on what your clients are looking for. that said a reasonable time can be arrived at after fine tuning your application
- for pooling you can use commons dbcp. That should do the needful.