File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Performance and the fly likes File parsing and insert data into data base Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Spring in Action this week in the Spring forum!
JavaRanch » Java Forums » Java » Performance
Bookmark "File parsing and insert data into data base" Watch "File parsing and insert data into data base" New topic
Author

File parsing and insert data into data base

Vaskar basak
Greenhorn

Joined: Oct 05, 2007
Posts: 1
Hi friends,

I have a 500 mb zip file. I am trying to read the zip file .Then insert the data in to the DB.But it is taking long time. I have to insert all record into DB with in 1 hr.

first i parsed the big file into small files .Then create a thread pool.Each thread in the thread pool read the small files and insert value into the DB.But it takes more time.I have search the google but i did not get any proper solution..

I am using J2sdk1.4 and MYSQL.

pls help me....

Thanks!
Vaskar
[ February 22, 2008: Message edited by: Vaskar basak ]
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12803
    
    5
How are you handling the DB connections? Are you using a connection pool?
Have you done any monitoring to determine which part of the process takes the longest?
Is your application taking a large fraction of the CPU time?
Does MySQL live on the same machine or elsewhere on the network?

Bill
steve souza
Ranch Hand

Joined: Jun 26, 2002
Posts: 861
You won't get valuable suggestions until you can tell people where your bottleneck is. Is it file IO? Is it network IO? Is it a java loop that concatenates strings? Is it database insert statements? You have to find out where your program is slow and then tune it.

There are many tools available for doing this. I wrote one called jamon and so can speak to that. JAMon comes with a jdbc driver that times jdbc methods on Connections/Statements/ResultSets. It also can tell you the performance of every query that you execute.

It works with all types of databases including mysql. It requires not code changes. Just change your jdbc url and driver per the instructions below: http://jamonapi.sourceforge.net/#WhatsNew22

It is also easy to monitory any java interface. http://jamonapi.sourceforge.net/#monitoringinterfaces

You can also embed monitoring statements directly in your code. http://jamonapi.sourceforge.net/#SimpleExamples

That should give you a start.

Whether you use jamon or not the point is the same you must first find out where your program is slow.


http://www.jamonapi.com/ - a fast, free open source performance tuning api.
JavaRanch Performance FAQ
 
It is sorta covered in the JavaRanch Style Guide.
 
subject: File parsing and insert data into data base