• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

best performance / practices for read and parse files ?

 
Ranch Hand
Posts: 798
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Say, if I have a huge records in external text file (1 GB) , which contain customers credit score. The format will be "peter;400" line by line. These scores are ONLY those that has been updated recently by credit companies.

Then I need read, parse, update records in database. How to get the best performance / practices in pure JDBC ?

Here is my thinking
1. create two threads, one is to read, parse, another is to update records in DB. The pipe is ArrayBlockingQueue.

2. use preparedStatement.

3. use some batch update.

Anything I have missed ? Any suggestions ?

Thanks.
 
Java Cowboy
Posts: 16084
88
Android Scala IntelliJ IDE Spring Java
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
It's hard to say in advance if there would be any benefit in making this a multi-threaded program. I'd just start with writing a small and simple single-threaded program that does a batch update for every N records (and you'd have to try out which value of N works best).

If the performance is not good enough, try more complicated things like multi-threading. I suspect that you might find that a much more complicated multi-threaded program doesn't perform a lot better than a simple single-threaded program.
 
Ranch Hand
Posts: 188
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Edward Chen wrote:The pipe is ArrayBlockingQueue. .

what is pipe?
Besides how no 2, 3 bar you from using no 1 alternative. These alternatives are not mutually exclusive, isn't it? If multi threading used wisely, actually gives better performance. So, while one thread reads file, processes and is waiting for database connection, other barge in opening file to read from next line onwards. Mind that, I am not saying one is reading another is updating, then you have to handover processed records to other thread to let it update, an inter thread communication. That may be messy.
reply
    Bookmark Topic Watch Topic
  • New Topic