Win a copy of Rust Web Development this week in the Other Languages forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Tim Cooke
  • Campbell Ritchie
  • Ron McLeod
  • Liutauras Vilda
  • Jeanne Boyarsky
Sheriffs:
  • Junilu Lacar
  • Rob Spoor
  • Paul Clapham
Saloon Keepers:
  • Tim Holloway
  • Tim Moores
  • Jesse Silverman
  • Stephan van Hulst
  • Carey Brown
Bartenders:
  • Al Hobbs
  • Piet Souris
  • Frits Walraven

Kafka vs Traditional ETL/Batch

 
Ranch Hand
Posts: 81
Spring Java Ubuntu
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hi Bill,

Excited to see your upcoming book on latest sensation in enterprise technology area.

I attended an overview session on Kafka recently and I have couple of questions on which I want to request your opinion -

How do you differentiate the traditional Enterprise service buses with ETL combined with Kafka. As an introduction, Kafka is doing real time fast processing of streams but we have other systems like Tibco FTL and IBM MQ which have achieved lightening speed of millions transactions per second. How does kafka provides upper-hand with existing products ?

Another question is do you think that kafka will be able to replace existing batch systems. With scenarios with high availability , kafka may process transactional data quickly but overnight batch may still become bottle neck. What are the few practices you want to share with audiences here about integrating kafka with exiting systems.

Thank You again for bringing the detailed book.
 
Saloon Keeper
Posts: 24826
172
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
I just installed the latest version of Hitachi Pentaho.

You may be interested to know that the Pentaho DI engine (Kettle) now runs under Kafka.
 
Ranch Hand
Posts: 66
1
IntelliJ IDE Java Linux
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator

Tim Holloway wrote:I just installed the latest version of Hitachi Pentaho.

You may be interested to know that the Pentaho DI engine (Kettle) now runs under Kafka.



Pentaho DI does run Kafka out of the box, but it's far from perfect (It really only works for generic AVRO messages + it doesn't have Kerberos support). I had to write my own custom step to successfully solve the requirements we had in our project.
 
You showed up just in time for the waffles! And this tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop
https://coderanch.com/wiki/718759/books/Building-World-Backyard-Paul-Wheaton
reply
    Bookmark Topic Watch Topic
  • New Topic