I have a 500 mb zip file. I am trying to read the zip file .Then insert the data in to the DB.But it is taking long time. I have to insert all record into DB with in 1 hr.
first i parsed the big file into small files .Then create a thread pool.Each thread in the thread pool read the small files and insert value into the DB.But it takes more time.I have search the google but i did not get any proper solution..
Author and all-around good cowpoke
Joined: Mar 22, 2000
How are you handling the DB connections? Are you using a connection pool? Have you done any monitoring to determine which part of the process takes the longest? Is your application taking a large fraction of the CPU time? Does MySQL live on the same machine or elsewhere on the network?
You won't get valuable suggestions until you can tell people where your bottleneck is. Is it file IO? Is it network IO? Is it a java loop that concatenates strings? Is it database insert statements? You have to find out where your program is slow and then tune it.
There are many tools available for doing this. I wrote one called jamon and so can speak to that. JAMon comes with a jdbc driver that times jdbc methods on Connections/Statements/ResultSets. It also can tell you the performance of every query that you execute.