File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes Agile and Other Processes and the fly likes Database seeding and continuous integration Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Soft Skills this week in the Jobs Discussion forum!
JavaRanch » Java Forums » Engineering » Agile and Other Processes
Bookmark "Database seeding and continuous integration" Watch "Database seeding and continuous integration" New topic
Author

Database seeding and continuous integration

Jorge Phillips
Ranch Hand

Joined: Jun 03, 2001
Posts: 43
Welcome Paul and Andy, it is great to have you in this forum.

I would like to hear your views on using continuous integration for ORM systems where as code evolves so does the schema in the underlying database. I understand well the concept of code evolution, integration and testing.

The issues I am interested in are those that arise from the need to both seed base data into the database as well as second-order data (data that requires complex constraint maintenance and interactions with multiple tables to ensure initial business logic correctness) in order to test system behavior to verify an integration is correct.

Do you have any suggestions on tools, methodologies, etc. to use? Or are we basically thrown into using ad-hoc test and associated seeding protocols?
Paul Duvall
author
Greenhorn

Joined: Jul 17, 2007
Posts: 29
The issues I am interested in are those that arise from the need to both seed base data into the database as well as second-order data (data that requires complex constraint maintenance and interactions with multiple tables to ensure initial business logic correctness) in order to test system behavior to verify an integration is correct.

Do you have any suggestions on tools, methodologies, etc. to use? Or are we basically thrown into using ad-hoc test and associated seeding protocols?


In chapter 5, Continuous Database Integration, we cover many techniques associated with integrating database changes into your build process. If I were to assume that you've scripted all of your DDL and DML, then you can run these scripts as part of your build process so that your data is always in a "known state". In your DDL you can establish these constraints and your build process can manage the mappings for your ORM. After your DDL scripts have been run along with the mappings, you can run your DML scripts for the second-order data you describe, based on these constraints. For additional info, see my blog entry at http://www.testearly.com/2006/06/08/database-integration-in-your-build-scripts Pramod's short cut on "Continuous Database Integration" at http://www.informit.com/store/product.aspx?isbn=032150206X&rl=1 or an article on Evolutionary Database Design at http://www.martinfowler.com/articles/evodb.html
[ August 28, 2007: Message edited by: Paul Duvall ]

Co-author of <a href="http://www.amazon.com/gp/product/0321336380/" target="_blank" rel="nofollow">Continuous Integration: Improving Software Quality and Reducing Risk </a> <br />(Addison-Wesley Martin Fowler Signature Series, 2007). Companion website for the book is <a href="http://www.integratebutton.com/" target="_blank" rel="nofollow">IntegrateButton.com</a>
Andrew Glover
author
Greenhorn

Joined: Dec 19, 2002
Posts: 16
As far as tools go, check out DbUnit-- this framework facilitates database seeding (via XML files). With it, you can write tests that assume a known data structure and model. Of course, if your database model changes often, you'll run into issues associated with managing XML files...


Certified City Slicker
Shiang Wang
Ranch Hand

Joined: Jun 20, 2003
Posts: 96
Jorge,

I am actually facing the same problem in several of our projects. What we did is have a master set of reference data that are required for down stream tables to be populated. The person who writes the junit is still responsible for populating whatever data he needs to test certain scenario and cleaning them up at tearDown().

One of the tools you can extract data from the database is Oracle SQL Developer. I normally use it if I need it to be a DML script.

We also have some in-house utilities to transform data between database and csv or xml. The csv and xml format makes your life easier if you need to modify some of the data to test different cases.


SCBCD, SCWCD, SCJP
Jorge Phillips
Ranch Hand

Joined: Jun 03, 2001
Posts: 43
Paul, thanks for your thoughtful comments and pointers. Andy and Shiang, thanks for the comments on tools. I'll check all these out.
Christophe Fondacci
Greenhorn

Joined: Mar 31, 2011
Posts: 1
Hi there,

Just a quick update : you might be interested in neXtep designer - a database development environment based on the concept of version control. You can automatically fetch and version control any database (both structure and data) and generate any SQL upgrade script from the comparison of 2 versions or from the synchronization between a version and an existing database.

It is an integrated development environment based on Eclipse and, as such, offers you lots of tools and editors to gain speed in your database developments. With our soon to come 1.0.6 release, you'll be able to drive builds from the command-line, allowing you to build a database from scratch up to any version, deploy data (reference data or test data is up to you).

The product is licensed under GPL so it is free. We are trying to gather people around the project, we'll be very pleased to have your feedbacks. It is based on Eclipse RCP and runs on Linux, Windows and Mac and currently supports Oracle, MySql, PostgreSql, DB2 and a MSSQL server support is under development.

More information on our website :
http://www.nextep-softwares.com

The wiki introduces the concepts, tutorials and technical information :
http://www.nextep-softwares.com/wiki

Hope this helps,
Christophe
 
 
subject: Database seeding and continuous integration