wood burning stoves 2.0*
The moose likes Other Java Products and Servers and the fly likes Rules based ETL Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Android Security Essentials Live Lessons this week in the Android forum!
JavaRanch » Java Forums » Products » Other Java Products and Servers
Bookmark "Rules based ETL" Watch "Rules based ETL" New topic
Author

Rules based ETL

Jayesh A Lalwani
Bartender

Joined: Jan 17, 2008
Posts: 2278
    
  28

Does anyone know of any rules based ETL framework that can handle transformation of large amounts of data. The idea is that the transformation logic is specified in a DSL, and the ETL reads the rules in DSL and does the transformation. We would prefer the DSL to be maintained by business users, but it will be ok if the developers do it too. The idea is that we want the rules to be seperated from the Java code, so we can update the rules without doing a complete release.
Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 15964
    
  19

I know something more or less like that.

The Pentaho Business Intelligence suite contains an ETL tool named "Kettle" (also known as Pentaho DI). The ETL rules are storable in an XML file and can be edited by non-programmers via a GUI editor app named "spoon".

Spoon is basically a drag/drop/drool UI where you select sources, destinations, and processing operations into a work area, configure them, and wire them together to make the transformation ruleset.

It is very performant. I have used it to populate databases with hundreds of millions of records at a shot, and that was just basic operation without exploiting its abilities to work with parallelized databases.

The one thing I don't like about it is that some of the processes are fairly non-intuitive. One of them, in fact, used to be Excel input, but I got so fed up with that one that I made modifications to the source code which have since become a permanent part of the Kettle system.

Customer surveys are for companies who didn't pay proper attention to begin with.
Jayesh A Lalwani
Bartender

Joined: Jan 17, 2008
Posts: 2278
    
  28

Awesome! I will look at it. Thanks Tim
 
Consider Paul's rocket mass heater.
 
subject: Rules based ETL
 
Similar Threads
Batch fetching using JPA
Software Factories / MPS
pentaho data integration
Digester in XML
Creating Domain-Specific and General Programming Languages