When looking at Event Streaming, most are talking about Kafka. Is it possible to implement simple event streaming using traditional pub-sub capable messaging system like ActiveMQ, etc.? What are the tradeoffs?
In the book, event streaming is introduced with a very simple example that leverages NSQ, a popular distribted pub/sub messaging platform, so yes it is definitely possible.
However, an interesting aspect of event streaming is the ability to process AND re-process it (in different ways by different components at different times), which pub/sub messaging platforms have a hard time achieving since when a message has been consumed by all subscribers/consumers, it's gone forever, unless you store it somewhere and can reinject it later.
So Kafka and Kinesis are popular because the production of messages and their consumption are completely decoupled and messages are kept on the log for a configurable time, which makes it possible for consumers to process and reprocess messages/events as many times as they'd like.
Thank you for both of your response. It is very helpful for me when I want to explore more on the concept of event streaming and able to use tools that I'm already familiar with like AMQ or RabbitMQ before jumping into Kafka. Just a quick browse on the resource, there are so much of Kafka to figure out.
You showed up just in time for the waffles! And this tiny ad:
Building a Better World in your Backyard by Paul Wheaton and Shawn Klassen-Koop