The publish/subscribe pattern

The publish/subscribe pattern is a communication pattern alternative to the well-known request/reply pattern. Instead of a client (issuing a request) and a server (replying with a response to that request), a publish/subscribe architecture consists of publishers and subscribers.

Each publisher can emit messages. It is of no concern to the publisher who actually gets these messages. This is the concern of the subscribers; each subscriber can subscribe to a certain type of message and be notified whenever a publisher publishes a given type of message. In reverse, each subscriber does not concern itself with where a message actually came from.

The request/reply and the publish/subscribe communication patterns

In practice, many publish/subscribe architectures require a central infrastructure component—the message broker. Publishers publish messages at the message broker, and subscribers subscribe to messages at the message broker. One of the broker's main tasks then is to route published messages to the subscribers that have expressed interest in them.

Typically, messages will be routed topic-based. This means that each publisher specified a topic for a published message (a topic usually just being a string identifier, for example, user.created). Each subscriber will also subscribe to a certain topic. Often, a broker will also allow a subscriber to subscribe to an entire set of topic using wildcard expressions such as user.*.

In contrast to request/reply, the publish/subscribe pattern brings some clear advantages:

• Publishers and subscribers are very loosely coupled. This goes to the extent that they do not even know about one another.
• A pub/sub architecture is very flexible. It is possible to add new subscribers (and, therefore, extend existing processes) without having to modify the publisher. The inverse also applies; you can add new publishers without having to modify the subscribers.
• In case the messages are routed by a message broker, you also gain resiliency. Usually, the message broker stores all messages in a queue, in which they are kept until they have been processed by a subscriber. If a subscriber becomes unavailable (for example, due to a failure or an intentional shutdown), the messages that should have been routed to that subscriber will become queued until the subscriber is available again.
• Often, you will also get some kind of reliability guaranteed by the message broker on a protocol level. For example, RabbitMQ guarantees reliable delivery by requiring each subscriber to acknowledge a received message. Only when the message has been acknowledged, the broker will remove the message from the queue. If the subscriber should fail (for example, by disconnection) when a message had already been delivered, but not yet acknowledged, the message will be put back into the message queue. If another subscriber listens on the same message queue, the message might be routed to that subscriber; otherwise, it will remain in the queue until the subscriber is available again.
• You can easily scale out. In case that too many messages are published for a single subscriber to efficiently handle them, you can add more subscribers and have the message broker load-balance the messages sent to these subscribers.

Of course, introducing a central infrastructure component such as a message broker brings its own risk. When not done right, your message broker might become a single point of failure, taking your entire application down with it in case it fails. When introducing a message broker in a production environment, you should take appropriate measures to ensure high-availability (usually by clustering and automatic failover).

In case your application is run in a cloud environment, you may also take advantage of one of the managed message queuing and delivery services that are offered by the cloud providers, for example, AWS Simple Queue Service (SQS) or the Azure Service Bus.

In this chapter, you will learn how to use two of the most popular open source message brokers—RabbitMQ and Apache Kafka. In Chapter 8, AWS Part II - S3, SQS, API Gateway, and DynamoDB, you will learn about AWS SQS.