Win a copy of Five Lines of Code this week in the OO, Patterns, UML and Refactoring forum!

Gavin Roy

Author
+ Follow
since May 19, 2015
Cows and Likes
Cows
Total received
5
In last 30 days
0
Total given
0
Likes
Total received
1
Received in last 30 days
0
Total given
0
Given in last 30 days
0
Forums and Threads
Scavenger Hunt
expand Ranch Hand Scavenger Hunt
expand Greenhorn Scavenger Hunt

Recent posts by Gavin Roy

Pradeep bhatt wrote:RabbitMQ has routing feature. When compared to ESB it lacks message transformation feature. Are there any other differences ?



RabbitMQ has a lot of features; clustering, federation, highly-available queues, multiple patterns for routing as you mentioned, priority queues, message expirations, and too many more to enumerate here. Check out RabbitMQ.com for more info

Pradeep bhatt wrote:Hi author,

In the book which language is used for illustrate code samples. Is it Java or Erlang ?



Primarily Python, though there's a chapter on building plugins with Erlang. I believe the Python code to be straight forward enough that with the Tutorials on RabbitMQ.com, it'd be easy to draw the parallels between the example code and Java code doing the same thing.

Arjun Shastry wrote:hi Gavin,
1) Single point of failure is one problem with Messaging middle-wares. How easy is to set up RabbitMQ for replication,load balancing ?
2) i worked on Tibco EMS in the past. For message persistence we used Oracle and during peak hours, sometimes due to high load, EMS used to stop working causing message clogging. What kind of message persistence(like KahaDB for ActiveMQ) RabbitMQ provides for fast persistence?



1) RabbitMQ has built in clustering and highly-available queues. Additionally it has built-in support via a bundled plugin for cross-datacenter messaging, including federated queues, which should do the trick.
2) The backing store for RabbitMQ is pluggable, so if the MNesia based tables are not fast enough, I know of a few backing stores to choose from including LevelDB and TokyoCabinet. I've not played around with benchmarking them to see what kind of performance gain one would receive in using them, but I imagine it's not trivial.

Paras Jain wrote:Hi Gavin,

Have you used RabbitMQ along with Mule ESB? Do you know anyone using RabbitMQ connector for Mule?

Regards,
Paras



I've not, but they do have a connector here: https://github.com/mulesoft/mule-transport-amqp
Erlang is a language and a VM, and a platform specifically called Erlang/OTP (Open Telecom Platform). The Erlang VM is analogous to the Java VM on the surface and there are other languages that compile down to run on the VM (Elixir, Lisp Flavored Erlang). There's even Ling, a way to run your erlang apps natively on Xen hypervisors (http://erlangonxen.org) without an underlying OS.

Erlang is a soft-realtime system, provides the ability to do hot-code-swapping, native cross-node communication or clustering, and has a focus on high-availability. Here's the marketing blurb from the Erlang website:

"Erlang is a programming language used to build massively scalable soft real-time systems with requirements on high availability. Some of its uses are in telecoms, banking, e-commerce, computer telephony and instant messaging. Erlang's runtime system has built-in support for concurrency, distribution and fault tolerance."

It is platform independent like Java, you run your compiled erlang apps on the VM and the VM is available for Windows, *NIX, OS X, etc.

One of the reasons I'm fond of RabbitMQ is the extensibility of the system. It has a very flexible plugin system and a great ecosystem of plugins for extending functionality.

Re Erlang, I'm a bit of an Erlang fan so I'm bound to come off as biased. I think that it's a great foundation for a messaging broker due to the nature of how it works, the functional aspect of the language and the built-in distributed communication constructs.

Memory is handled like most other compiled languages, though it is functional so a bit different than a procedural or OO language. The VM has the same concerns as the Java VM with gc and the like. State for an application is passed through from function to function when using the OTP constructs in the erlang standard library. With regard to memory limits, you can tune the erlang VM just as you can the java VM and while I haven't specifically looked, I'd be surprised if there was not an option to limit the memory utilization.

From a comparison perspective, I'm not terribly deep on QPid's implementation or benefits outside of being an AMQP broker. While I know RabbitMQ internals and the ecosystem, I can not say the same for QPid and would likely make unfair comparisons


Have you looked at the Headers exchange? I think it'd do the type of thing you're looking for if I'm understanding you correctly. When you're discussing a header that is populated, are you specifically talking about the header table in the AMQP message properties, or is it something in your message body?

I do not know of a filtering exchange, but it'd be fairly trivial to write a plugin that does such a thing.
If I had to pick an overarching theme to the book, it's using RabbitMQ at the center of a event based/messaging oriented architecture. One chapter focuses on the different messaging patterns that are available in RabbitMQ and it includes examples on how and why you would use each. But I'd say that if your primary goal is to learn why you'd use messaging and what problems it solves, it might be the wrong book. That being said, there are specific examples of how to use messaging to solve problems, but that's not the primary focus of the book.

If you do end up checking it out, I'd love to hear your thoughts on if it helped you in that area or not.
While I think that AMQP 1.0 is an interesting protocol and deserves attention, it is a departure from 0-9-1. I wish they would have released a 0-9-2 that addressed some of the shortcomings of 0-9-1. Also, if you think of 1.0 as a different protocol all together (which it generally is), there's not too much fragmentation in the 0-9-1 world. The RabbitMQ folks keep a nice errata page for problems with AMQP 0-9-1 and broker semantics: http://www.rabbitmq.com/amqp-0-9-1-errata.html

That being said, an example of fragmentation is the RabbitMQ team extending AMQP 0-9-1 to add features and address shortcomings in the protocol which other 0-8 or 0-9-1 brokers do not implement. That being said, while the client libraries I maintain support these extensions, you should be able to use them on AMQP 0-9-1 brokers that do not.
No it is not. The RabbitMQ folks distribute a Java driver library and as far as I know (I'm not Java programmer per se) there are no beans involved

https://www.rabbitmq.com/java-client.html

If there are, I'll have to go back and kick myself since I wrote a Java based Apache Flume source and sink plugin to work with RabbitMQ!
Hi there!

I agree with you re similarities in message brokers and MoM in general.

1) I would like to think so. The book starts with a low level discussion around AMQP and the messaging protocol so that the reader can understand the "why" when it comes to how RabbitMQ does things.

2) While my publisher kept me to task on keeping the book approachable, I tried to write the book for people like myself who want to know more of the why than the how. Sure there's a lot of instructional content, but I tried to focus on grokking RabbitMQ holistically and a deeper level than intro content. If you end up reading a copy, I'd love to know what you think.

3) I have worked with a fair number of the and I do agree. I've used multiple brokers "at scale" with large message velocities and complicated topologies. I landed on RabbitMQ because another broker I was using in production at the time was having a lot of issues around memory leaks and gradual throttling of throughput due to said leaks.

Where I think RabbitMQ is different than other systems I've worked with is its focus on extensibility and its roots in AMQP 0-9-1.

Thank you for the questions!
Thank you!

Re question #1 - The book is up to date with RabbitMQ v3.5.2 and I have been working through the MEAP to keep it up to date. One of the more fun aspects of writing a book on RabbitMQ is the team has watched what I've had to say and they have directly addressed problems I've discussed in the book requiring me to go back and deal with it. For example, when I first started RabbitMQ used TCP back pressure to throttle publishers when needed. I had written a whole section on how to detect and deal with this issue. Within about 6 months of the MEAP release of the chapter covering the topic, they added a mechanism to let the client know when they are applying the back pressure and when it could publish again.

Re question #2 - I am aware of Kafka though I have not used it in production. From what I know of Kafka, RabbitMQ is much more flexible. I'm a big fan of using the right tool for the job and I'm sure I'll come across one where I'll be able to put Kafka through its paces some day. I've worked in multiple big-data environments and have had RabbitMQ at the heart of some very large event-based architectures. In my day job, for example, I am feeding Avro datum through RabbitMQ for a large event-based system. The datum are then collected into Avro containers and shipped to S3 for warehousing and EMR processing. I'm currently experimenting with connecting directly up to Apache Spark for the next step in our big-data processing.

I'd say the biggest bottleneck can be when users decide they want to persist messages to disk as part of the publishing contract with RabbitMQ but do not consider that the behavior is exactly the same as a write-heavy OLTP database server. If you want to have persisted messages, you need to provision your servers to support the IOP throughput rates that you would need to perform the same database write throughput velocity that you would expect RabbitMQ to perform.

In both the chapter on publishers and consumers I directly address throughput and performance, providing a map of feature use to performance gain or loss. The chapters have been useful for me and my team in deciding which features we need and want to use for our projects.

RabbitMQ is open-source and designed to be platform neutral. It's highly extensible so you can add plugins for various protocols such as STOMP or MQTT and there are even protocol level plugins like XMPP or SMTP in the ecosystem.

It supports many of the features you're referring to with clustering. I'd recommend you check out http://www.rabbitmq.com for more marketing information than I could provide here.

There are no performance advantages in using RabbitMQ with the Erlang client other than in using Erlang as a language and its VM. With RabbitMQ's goal of being platform neutral, performance is generally directly related to the performance of the language itself. For example, I would expect to see C or Java based clients perform better than Ruby ones.
Hi there,

I think that RabbitMQ in Depth will explain more about why it does what it does than in Action does, but I guess ultimately the reader is the one who is best to make such statements.

The goal with in-Depth was to go deeper into subjects, such as how the AMQP protocol works at the low level at the outset of the book and work your way up to RabbitMQ's features and such instead of the inverse.

While the book uses Python as the example language, the goal is to make the examples and concepts very clear and easy to follow regardless of your language of choice.

I do not compare RabbitMQ to any other brokers in the book.