• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Devaka Cooray
  • Knute Snortum
  • Paul Clapham
  • Tim Cooke
Sheriffs:
  • Liutauras Vilda
  • Jeanne Boyarsky
  • Bear Bibeault
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Ron McLeod
  • Piet Souris
  • Frits Walraven
Bartenders:
  • Ganesh Patekar
  • Tim Holloway
  • salvin francis

Kafka vs Traditional ETL/Batch  RSS feed

 
Ranch Hand
Posts: 81
Java Spring Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Bill,

Excited to see your upcoming book on latest sensation in enterprise technology area.

I attended an overview session on Kafka recently and I have couple of questions on which I want to request your opinion -

How do you differentiate the traditional Enterprise service buses with ETL combined with Kafka. As an introduction, Kafka is doing real time fast processing of streams but we have other systems like Tibco FTL and IBM MQ which have achieved lightening speed of millions transactions per second. How does kafka provides upper-hand with existing products ?

Another question is do you think that kafka will be able to replace existing batch systems. With scenarios with high availability , kafka may process transactional data quickly but overnight batch may still become bottle neck. What are the few practices you want to share with audiences here about integrating kafka with exiting systems.

Thank You again for bringing the detailed book.
 
Bartender
Posts: 20745
124
Android Eclipse IDE Java Linux Redhat Tomcat Server
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I just installed the latest version of Hitachi Pentaho.

You may be interested to know that the Pentaho DI engine (Kettle) now runs under Kafka.
 
Ranch Hand
Posts: 56
1
IntelliJ IDE Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Tim Holloway wrote:I just installed the latest version of Hitachi Pentaho.

You may be interested to know that the Pentaho DI engine (Kettle) now runs under Kafka.



Pentaho DI does run Kafka out of the box, but it's far from perfect (It really only works for generic AVRO messages + it doesn't have Kerberos support). I had to write my own custom step to successfully solve the requirements we had in our project.
 
Get off me! Here, read this tiny ad:
how do I do my own kindle-like thing - without amazon
https://coderanch.com/t/711421/engineering/kindle-amazon
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!