The Optimal RabbitMQ Guide (Lovisa Johansson)(Z-Library)
code
No Description
203
Views
1
Downloads
0.00
Total Donations
Registered users can read the full content for free
Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.
Page
1
138
Page
2
THE OPTIMAL RABBITMQ GUIDE From Beginner to Advanced I want to say a big thank you to everyone who has helped me, from the earliest draft of this book, up to this edition. A special thanks go out to my lovely colleagues at 84codes and my tech friends. Finally: A big thank you, to all CloudAMQP users for your feedback and continued support. It has been a great motivator to see our customers’ projects succeed! - Lovisa Johansson, CloudAMQP
Page
3
The Optimal RabbitMQ Guide Book version: 2.1 Author: Lovisa Johansson Email: lovisa@cloudamqp.com Co-author: Elin Vinka Email: elin@cloudamqp.com Graphics: Daniel Marklund, Elin Vinka Published: 2020-03-10 ISBN: 978-91-519-3115-9 Copyright @2020 by 84codes AB
Page
4
I’d love to hear from you Please e-mail me any comments that you might have about the book and let me know what you think should or shouldn’t be included in the next edition. If you have an application using RabbitMQ or a user story that you would like to share, please send me an e-mail!
Page
5
(This page has no text content)
Page
6
This book is dedicated to all the frequent and future users of RabbitMQ, and to the Swedish mentality that makes us queue lovers by heart.
Page
7
(This page has no text content)
Page
8
P A R T O N E Introduction to RabbitMQ ....................................................10 Microservices and RabbitMQ ..................................... 12 What is RabbitMQ? ....................................................... 16 Exchanges, Routing Keys and Bindings ................... 28 RabbitMQ and client libraries .................................... 38 Rabbitmq with Ruby and bunny ............................... 40 RabbitMQ and Node.js with Amqplib .......................46 RabbitMQ and Python with Pika ............................... 56 The Management Interface ....................................... 62 P A R T T W O P A R T T W O Advanced Message Queuing .............................................80 RabbitMQ ..................................................................... 82 Best Practice ................................................................. 82 Best Practices for High Performance ....................... 96 Best Practices for High Availability ..........................100 Queue Federation ....................................................... 104 RabbitMQ Protocols ....................................................110 P A R T T H R E E RabbitMQ User Stories ......................................................... 116 A Monolithic System into Microservices .................118 How RabbitMQ Transformed Farmbot ................... 122 Microservice Architecture Built Upon RabbitMQ . 128 Event-based Communication .................................. 132
Page
9
(This page has no text content)
Page
10
INTRODUCTION Times are changing; reliability and scalability are more important than ever. Because of that, companies are rethinking their architecture. Monoliths are evolving into microservices and servers are moving into the cloud. Message Queues and RabbitMQ in particular, as one of the most widely deployed open-source message brokers, have come to play a significant role in the growing world of microservices. This book is divided into three parts: The first part is an introduction to microservices and RabbitMQ, including a section on how to use RabbitMQ on the CloudAMQP platform. This section includes the most important RabbitMQ concepts and how it allows users to create products that meet current and future industry demands. The second part of the book is for more advanced users, who will learn how to take full advantage of RabbitMQ. This section explores Best Practice recommendations for High Performance and High Availability and common RabbitMQ errors and mistakes. It is also a deep dive into some RabbitMQ concepts and features. The third part gives some real-world user stories from our own experiences with RabbitMQ as well from a CloudAMQP customer’s point of view. We hope that this book takes you far on your message queuing journey with RabbitMQ.
Page
11
P A R T O N E INTRODUCTION TO RABBITMQ A MICROSERVICE ARCHITECTURE Modules with various functionalities operate together to form a complete software application via properly articulated operations and interfaces.
Page
12
11 Welcome to the wonderful world of message queuing! Many modern cloud architectures are a collection of loosely coupled microservices that communi- cate via a message queue. This microservice architecture is beneficial in that all the different parts of the system are easier to develop, deploy and maintain. One of the advantages of a message queue is that it makes your data temporarily persistent, reducing the chance of errors that may occur when different parts of the system are offline. If one part of the system is unreachable, the other part can continue to interact with the queue. This part of the book features basic information on micros- ervices and message queuing, and the benefits of RabbitMQ.
Page
13
P A R T O N E MICROSERVICES AND RABBITMQ
Page
14
13 Managing a complex, unified enterprise application can be a lot more difficult than it seems. Before making even a small change, hours of testing and analysis are required for possible impacts your change could have on the overall system. New developers must spend days getting familiar with the system before they are con- sidered ready to write a non-breaking line of code. Fortunately, the situation becomes far less complicated thanks to the use of microservices. This chapter talks about the benefits of a microservice architecture. In a microservice architecture, also called modular architecture, modules are decoupled from each other. These modules often offer various functionalities that operate together to form a complete software application via properly articulated oper- ations and interfaces. Unlike a monolithic application, where all chunks of functionality are present within a single module, a microservice application divides different func- tionalities across different modules to enhance maintainability and ease-of-use. More and more organizations are adopting microservice architecture because it allows for a more agile approach to software development and maintenance. Microservices also make it easy for businesses to scale and deliver updated versions of software weekly instead of having to wait through years of testing. BENEFITS OF A MICROSERVICE ARCHITECTURE • Easier development and maintenance Imagine building a huge, bulky billing application that will involve authentica- tion, authorization, charging and reporting of transactions. Dividing the applica- tion across multiple services (one for each functionality) will separate the respon- sibilities and give developers the freedom to write code for a specific service in any chosen language; something that a monolithic application can’t provide. Addi- tionally, it will also be easier to maintain written code and make changes to the system. For example, updating an authentication scheme will only require the addition of code to the authentication module and testing without having to worry about disrupting any other functionalities of the application. • Fault isolation Another obvious advantage offered by a microservice architecture is its ability to isolate the fault to a single module. For example, if a reporting service is down, authentication and billing services will still be running, ensuring that customers can perform important transactions even when they are not able to view reports. • Increased speed and productivity Microservice architecture is all about decoupling functions into manageable modules that are easy to maintain. Different developers can work on different
Page
15
14 GET STARTED FOR FREE WITH CLOUDAMQP Perfectly configured and optimized RabbitMQ clusters, ready in 2 minutes. www.cloudamqp.com Custom Alarms • Free Plan Available • Easy Monitoring • 24/7 support • 99.95% SLA modules at the same time. In addition to the development cycle, the testing phase is also sped up by the use of microservices, as each microservice can be tested independently to determine the readiness of the overall system. • Improved scalability Microservices also allow effortless system scaling whenever required. Adding new components to just one service can be completed without changing any of the other services. Along the same lines, resource-intensive services can also be deployed across multiple servers by using microservices. • Easy to understand Another advantage offered by a microservice architecture is the ease of under- standability. Because each module represents a single functionality, getting to know the relevant details becomes easier and faster. For example, a consultant that works on charging services does not have to understand the whole system to perform maintenance or enhancements to their part of the system. THE ROLE OF A MESSAGE QUEUE IN A MICROSERVICE ARCHITECTURE Think of a message broker like a delivery person who takes mail from a sender and delivers it to the correct destination. In a microservice architecture, there typically are cross-dependencies that mean no single service can perform without getting help from other services. This is where it is crucial for systems to have a mechanism in place, allowing services to keep in touch with each other with no blocked responses. Message queuing fulfils this purpose by providing a means for services to push messages to a queue asynchronously and ensure they are delivered to the correct destination. To implement message queuing, a message broker is required.
Page
16
15
Page
17
P A R T O N E WHAT IS RABBITMQ?
Page
18
17 Message queuing is an important facet of any microservice architecture; it is a way of exchanging data between processes, applications and servers. RabbitMQ is one of the most widely used message brokers in the world, boasting over 35,000 production deployments across both startups and big enterprises alike. This chapter gives a brief understanding of message queuing and defines essential RabbitMQ concepts. The chapter also explains the steps to go through when setting up connec- tions including how to publish as well as consume messages from a queue. Figure 1 - Messages sent from a sender to a receiver. RabbitMQ is a message queuing software also known as a message broker or queue manager. Simply put, it is a software where queues can be defined, and applications may connect and transfer a message onto it. Message queues enable asynchronous communication, which means that other applications (endpoints) that are producing and consuming messages interact with the queue instead of communicating directly with each other. A message can include any information. It could, for example, contain information about a process or job that should start on another application, possibly even on another server, or it might be a simple text message. The message broker stores the messages until a receiving application connects and consumes a message off the queue. The receiving application then appropriately processes the message. A message producer adds messages to a queue without having to wait for them to be processed.
Page
19
18 RABBITMQ EXAMPLE A message broker can act as a middleman for various services. A broker can be used to reduce loads and delivery times by web application servers. A task that would generally take a lot of time to process can be delegated to a third party whose only job is to perform them. This chapter is a case study of a web application, a PDF generator. The application allows users to upload information to a website. The website handles the information and generates a PDF, which is then emailed back to the user. The whole processing in this example takes several seconds to perform. When the user has entered user information into the web interface, the web applica- tion puts a “PDF processing” job into a message, including information about the job such as name and email. The message is then placed onto a queue defined in RabbitMQ using an exchange process. The underlying architecture of a message queue is simple - client applications called producers create messages and deliver them to the broker. The broker is, essentially, the message queue. Other applications, known as consumers, connect to the queue and subscribe to messages from the broker, processing those messages according to their function. The software interacting with the broker can be a producer, a consumer, or both. Messages placed in the queue are stored until the consumer retrieves them. If the PDF processing application crashes, or if many PDF requests are coming at the same time, messages would continue to pile up in the queue until the consumer starts again. It would then process all the messages, one by one. Figure 2 - A sketch of the RabbitMQ workflow. WHY AND WHEN TO USE RABBITMQ Message queuing allows web servers to respond to requests in their own time instead of being forced to perform resource-heavy procedures immediately. Message queuing is also useful for distributing a message to multiple recipients for consumption or when balancing the load between workers. The consumer application removes a message from the queue and processes the PDF at the same time as the producer pushes new task messages to the queue. The consumer can be on an entirely different server than the publisher or they can be located on the same server, it makes no difference. Requests can be created in one programming language and handled in another programming language, as the two applications only communicate through the messages they are sending to each other. The two services have what is known as ‘low coupling’ between the sender and the receiver.
Page
20
19 Figure 3 - Application architecture example with RabbitMQ. Example 1. The user sends a PDF create request to the web application. 2. The web application (the producer) sends a message to RabbitMQ that includes data from the request, such as name and email. 3. An exchange accepts the messages from a producer application and routes them to correct message queues. 4. The PDF processor (the consumer) receives the job message from the queue and starts the processing of the PDF.
The above is a preview of the first 20 pages. Register to read the complete e-book.
AI Reading Assistant
Whole-book reading guide from stratified index samples; jump to passages in the text
AI guide
# The Optimal RabbitMQ Guide
## 【One-Line Pitch】
A practical, hands-on guide to mastering RabbitMQ—from core concepts like exchanges and bindings to advanced best practices for high performance and high availability—written for developers and architects building microservices with message queuing.
## 【Book Arc】
- **Opening (~0%–4%)**: Introduces the book's three-part structure and the motivation behind message queuing—why monoliths are evolving into microservices and how RabbitMQ fits into modern, cloud-native architectures.
- **Early (~4%–28%)**: Covers RabbitMQ fundamentals: setting up an instance (including CloudAMQP), the four exchange types (direct, topic, fanout, headers), routing keys, bindings, and vhosts for environment isolation.
- **Early (~28%–36%)**: Walks through client library implementations—Ruby with Bunny, Node.js with Amqplib, and Python with Pika—showing publisher/consumer patterns, connection handling, and reconnection logic.
- **Middle (~36%–48%)**: Explores the Management Interface in depth: monitoring connections and channels, confirm mode, inspecting queues, managing bindings, and importing/exporting broker definitions as JSON for backup.
- **Middle (~48%–52%)**: Transitions to advanced topics: the tradeoff between performance and guaranteed delivery, and the start of best practices for high throughput and high availability scenarios.
- **Late (~52%–end)**: Delves into advanced best practices—lazy queues for predictable performance, queue size limits with TTL/max-length, queue federation for cluster migration, and real-world user stories from CloudAMQP customers.
## 【Key Takeaways】
- **Exchange types determine routing behavior** (Early): Direct exchanges match routing keys exactly, topic exchanges use wildcard patterns, fanout broadcasts to all bound queues, and headers route on message attributes. Choosing the right type is the foundation of any RabbitMQ design.
- **Vhosts provide logical isolation on a single broker** (Early): Like separate databases in PostgreSQL, vhosts let you isolate users, exchanges, and queues per project or environment (e.g., production vs. staging)—but note there's no resource isolation between vhosts on a shared instance.
- **Client libraries follow a common pattern** (Early): Ruby, Node.js, and Python implementations all involve creating a connection, opening a channel, declaring a queue, publishing with a routing key, and consuming via a callback—making it easy to mix languages across publishers and consumers.
- **Robust connection handling is critical** (Early): The Node.js example demonstrates reconnection logic, offline message queuing during disconnects, and prefetch limits—patterns essential for production-grade consumers that must survive broker restarts.
- **Confirm mode enables reliable delivery tracking** (Middle): When a channel is in confirm mode, both broker and client count messages, and the broker confirms as it handles them—giving publishers certainty about message acceptance.
- **Lazy queues trade throughput for predictability** (Middle): Lazy queues (RabbitMQ 3.6+) write messages to disk immediately, supporting millions of queued messages with smooth, predictable performance—but at the cost of higher latency. Use them for batch jobs or when consumers may lag behind publishers.
- **Queue size limits prevent unbounded growth** (Middle): Setting max-length or TTL on queues discards messages from the head when limits are hit, keeping queues short—recommended for spike-prone applications where throughput matters more than guaranteed delivery.
- **The Management Interface is a full operational tool** (Middle): Beyond monitoring, it supports importing/exporting broker definitions as JSON (a backup solution), manually inspecting messages with re-queue, and managing bindings—making it essential for day-to-day operations.
## 【Reading Tips】
- **Skim the client library chapters** (Ruby, Node.js, Python) if you're already familiar with one language—the patterns are nearly identical across all three, so read one deeply and skim the others for syntax differences.
- **Deep-read the exchanges chapter** (Early, ~16%–28%): This is the conceptual core of RabbitMQ. The direct/topic/fanout/headers distinctions with scenarios are worth studying carefully before writing any code.
- **Pay attention to the Node.js reconnection example** (Early, ~28%–32%): Even if you don't use Node.js, the connection lifecycle handling—reconnect on close, offline queue for failed publishes, prefetch settings—is the most production-relevant code in the book.
- **The Management Interface chapter** (Middle, ~36%–48%) is best read alongside a live instance—open your own broker and click through the tabs as you read to make the concepts stick.
- **For the best practices section** (Middle, ~48%+), focus on the decision framework: understand when lazy queues, TTL limits, and federation make sense for your specific workload rather than memorizing rules.
## 【Coverage Limits】
Excerpts cover the book's structure, core concepts, client examples, management interface, and early best practices, but do not include the full text of the advanced chapters on high availability, queue federation details, protocols, or the user stories in Part Three.
##
Passage locations
Excerpt 1
Event-based Communication .................................. 132 INTRODUCTION Times are changing; reliability and scalability are more important than ever. B...
View in text
Excerpt 2
“amq. direct”. 30 A message with routing key agreements.eu.berlin is sent to the exchange agreements. The message is routed to the queue berlin_agreements be...
View in text
Excerpt 3
.error("[AMQP] channel error", err.message); ch.on("close", function() { console.log("[AMQP] channel closed"); ch.prefetch(10); ch.assertQueue("jobs", { dura...
View in text
Excerpt 4
ng a message to the exchange or deleting the exchange. 70 Figure 23 - Specific information about a single queue. CONSUMERS The consumer’s field shows the con...
View in text
Recommended for You
{{#thumbnailUrl}}
{{/thumbnailUrl}}
{{^thumbnailUrl}}
{{/thumbnailUrl}}
Loading recommended books...
Failed to load, please try again later
Tip the Site
Scan the WeChat Pay or Alipay code to tip. No login required.
WeChat Pay
Alipay