• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Liutauras Vilda
  • Tim Cooke
  • Jeanne Boyarsky
  • Bear Bibeault
Sheriffs:
  • Knute Snortum
  • paul wheaton
  • Devaka Cooray
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Ron McLeod
  • Piet Souris
  • Ganesh Patekar
Bartenders:
  • Tim Holloway
  • Carey Brown
  • salvin francis

Kafka vs Traditional ETL/Batch

 
Ranch Hand
Posts: 81
Spring Java Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi Bill,

Excited to see your upcoming book on latest sensation in enterprise technology area.

I attended an overview session on Kafka recently and I have couple of questions on which I want to request your opinion -

How do you differentiate the traditional Enterprise service buses with ETL combined with Kafka. As an introduction, Kafka is doing real time fast processing of streams but we have other systems like Tibco FTL and IBM MQ which have achieved lightening speed of millions transactions per second. How does kafka provides upper-hand with existing products ?

Another question is do you think that kafka will be able to replace existing batch systems. With scenarios with high availability , kafka may process transactional data quickly but overnight batch may still become bottle neck. What are the few practices you want to share with audiences here about integrating kafka with exiting systems.

Thank You again for bringing the detailed book.
 
Bartender
Posts: 21000
128
Android Eclipse IDE Tomcat Server Redhat Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I just installed the latest version of Hitachi Pentaho.

You may be interested to know that the Pentaho DI engine (Kettle) now runs under Kafka.
 
Ranch Hand
Posts: 59
1
IntelliJ IDE Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator

Tim Holloway wrote:I just installed the latest version of Hitachi Pentaho.

You may be interested to know that the Pentaho DI engine (Kettle) now runs under Kafka.



Pentaho DI does run Kafka out of the box, but it's far from perfect (It really only works for generic AVRO messages + it doesn't have Kerberos support). I had to write my own custom step to successfully solve the requirements we had in our project.
 
What's that smell? I think this tiny ad may have stepped in something.
professionally read, modify and write PDF files from Java
https://products.aspose.com/pdf/java
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!