File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Object Relational Mapping and the fly likes Use of Hibernate to insert million records from Flat File Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Databases » Object Relational Mapping
Bookmark "Use of Hibernate to insert million records from Flat File" Watch "Use of Hibernate to insert million records from Flat File" New topic
Author

Use of Hibernate to insert million records from Flat File

Mike Boota
Ranch Hand

Joined: Jul 18, 2002
Posts: 82
Hi I am new to Hibernate and have secenario where I have to read a flat file parse it and insert into database. Now the flat file is pretty big that is may contain arround million records. It's taking too long to insert all the records if going through the traditional way so wondering if using hibernate how can I make it faster. But I am not that much familiar with hibernate so wondering if soomeone can help me out here I have used hibernate for simple stuff so far. Can someone provide a small sample or tutorial on this on reading a file and inserting the records into database. Any help is appreciated.

Thanks


MB<br />Sun Certified Programmer for Java2 Platform
Mark Spritzler
ranger
Sheriff

Joined: Feb 05, 2001
Posts: 17256
    
    6

What do you mean by traditional way?

What database are you using, if you are using Oracle the SQLLoader is the fastest way to insert data from a flat file.

Mostly, bulk inserts are better off with straight JDBC than Hibernate, but you can do it, I doubt that it will give you faster speed than whatever "traditional" way you are doing.

Mark


Perfect World Programming, LLC - Two Laptop Bag - Tube Organizer
How to Ask Questions the Smart Way FAQ
pascal betz
Ranch Hand

Joined: Jun 19, 2001
Posts: 547
if you want to "load" the database with millions of records then you better do not go trough hibernate.
rather use the tools comming with your DB or (if you need to do it with java) use straight JDBC.
if, for some reason, you need to use hibernate: flush the session from time to time!
Mark Spritzler
ranger
Sheriff

Joined: Feb 05, 2001
Posts: 17256
    
    6

Originally posted by pascal betz:
if you want to "load" the database with millions of records then you better do not go trough hibernate.
rather use the tools comming with your DB or (if you need to do it with java) use straight JDBC.
if, for some reason, you need to use hibernate: flush the session from time to time!


That time to time should match the JDBC Batch size. So if that is 20 then flush every twenty records.

Mark
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Use of Hibernate to insert million records from Flat File