wood burning stoves 2.0*
The moose likes Beginning Java and the fly likes best performance / practices for read and parse files ? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » Beginning Java
Bookmark "best performance / practices for read and parse files ?" Watch "best performance / practices for read and parse files ?" New topic
Author

best performance / practices for read and parse files ?

Edward Chen
Ranch Hand

Joined: Dec 23, 2003
Posts: 798

Say, if I have a huge records in external text file (1 GB) , which contain customers credit score. The format will be "peter;400" line by line. These scores are ONLY those that has been updated recently by credit companies.

Then I need read, parse, update records in database. How to get the best performance / practices in pure JDBC ?

Here is my thinking
1. create two threads, one is to read, parse, another is to update records in DB. The pipe is ArrayBlockingQueue.

2. use preparedStatement.

3. use some batch update.

Anything I have missed ? Any suggestions ?

Thanks.
Jesper de Jong
Java Cowboy
Saloon Keeper

Joined: Aug 16, 2005
Posts: 14074
    
  16

It's hard to say in advance if there would be any benefit in making this a multi-threaded program. I'd just start with writing a small and simple single-threaded program that does a batch update for every N records (and you'd have to try out which value of N works best).

If the performance is not good enough, try more complicated things like multi-threading. I suspect that you might find that a much more complicated multi-threaded program doesn't perform a lot better than a simple single-threaded program.

Java Beginners FAQ - JavaRanch SCJP FAQ - The Java Tutorial - Java SE 7 API documentation
Scala Notes - My blog about Scala
Rahul P Kumar
Ranch Hand

Joined: Sep 26, 2009
Posts: 188
Edward Chen wrote:The pipe is ArrayBlockingQueue. .
what is pipe?
Besides how no 2, 3 bar you from using no 1 alternative. These alternatives are not mutually exclusive, isn't it? If multi threading used wisely, actually gives better performance. So, while one thread reads file, processes and is waiting for database connection, other barge in opening file to read from next line onwards. Mind that, I am not saying one is reading another is updating, then you have to handover processed records to other thread to let it update, an inter thread communication. That may be messy.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: best performance / practices for read and parse files ?
 
Similar Threads
Parsing a strange looking text file
Optimize Oracle Performance?
Processing input files with Servlets
reading clob
Efficient java code to process large db set and parse them