This week's book giveaways are in the Refactoring and Agile forums.
We're giving away four copies each of Re-engineering Legacy Software and Docker in Action and have the authors on-line!
See this thread and this one for details.
Win a copy of Re-engineering Legacy Software this week in the Refactoring forum
or Docker in Action in the Cloud/Virtualization forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

What is the best way to store List<List<String>> in database?

 
Gaurav Kr. Arora
Ranch Hand
Posts: 37
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi,

Requirement: To store List<List<String>> in database. Here String represents the value fetched from excel file and List<String> represents a row containing cells in excel.
List<List<String>> represents all the records from a excel file.
There can be lacs of records in the excel data file.

Now, the data of this file has to be persisted into database without performance overhead. I have stored data of the file in List<List<String>> object. Table structure in which data needs to be stored is same as that of excel data file.
We can do a batch update for insert queries but since data size is more, I don't find it efficient.

If we can do this using some other technique like JPA or Hibernate or directly saving the file in database table with some kind of mapping between file columns and table columns, please let me know.

Thanks in advance.
 
chris webster
Bartender
Posts: 2407
32
Linux Oracle Postgres Database Python Scala
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The "best" way? Depends on your data and your database.

With Oracle, the fastest and most flexible way to do this with significant volumes of data is probably by loading the data directly from a CSV file into the database using an external table.

The general approach would be:

  • set up an Oracle directory definition in the database (one-off task)
  • copy your CSV file to the physical directory corresponding to the Oracle directory definition
  • create your external table
  • copy the data from the external table to a "real" table

  • Once you've set this up, it will be way faster than any row/cell-based process you can come up with in Java or Hibernate (e.g. I loaded 2.5 million rows of 60 columns on a slow machine in about 30 seconds the other day). The external table will end up with the same basic structure as your CSV file (the CSV header row will appear in the first row of the table if you don't exclude it from the loading process), but it's usually better to copy the data into a "real" table because it's more flexible e.g. you can add more data later, and it allows more performance optimisation e.g. use of indexes etc.

    But if you're not using Oracle, or you can't use a CSV file, you'll need some advice from a Java expert instead.

    Good luck!

     
    It is sorta covered in the JavaRanch Style Guide.
    • Post Reply
    • Bookmark Topic Watch Topic
    • New Topic