The Optimal RabbitMQ Guide (Lovisa Johansson) (z-library.sk, 1lib.sk, z-lib.sk)

138

THE OPTIMAL RABBITMQ GUIDE From Beginner to Advanced I want to say a big thank you to everyone who has helped me, from the earliest draft of this book, up to this edition. A special thanks go out to my lovely colleagues at 84codes and my tech friends. Finally: A big thank you, to all CloudAMQP users for your feedback and continued support. It has been a great motivator to see our customers’ projects succeed! - Lovisa Johansson, CloudAMQP

The Optimal RabbitMQ Guide Book version: 2.1 Author: Lovisa Johansson Email: lovisa@cloudamqp.com Co-author: Elin Vinka Email: elin@cloudamqp.com Graphics: Daniel Marklund, Elin Vinka Published: 2020-03-10 ISBN: 978-91-519-3115-9 Copyright @2020 by 84codes AB

I’d love to hear from you Please e-mail me any comments that you might have about the book and let me know what you think should or shouldn’t be included in the next edition. If you have an application using RabbitMQ or a user story that you would like to share, please send me an e-mail!

This book is dedicated to all the frequent and future users of RabbitMQ, and to the Swedish mentality that makes us queue lovers by heart.

P A R T O N E Introduction to RabbitMQ ....................................................10 Microservices and RabbitMQ ..................................... 12 What is RabbitMQ? ....................................................... 16 Exchanges, Routing Keys and Bindings ................... 28 RabbitMQ and client libraries .................................... 38 Rabbitmq with Ruby and bunny ............................... 40 RabbitMQ and Node.js with Amqplib .......................46 RabbitMQ and Python with Pika ............................... 56 The Management Interface ....................................... 62 P A R T T W O P A R T T W O Advanced Message Queuing .............................................80 RabbitMQ ..................................................................... 82 Best Practice ................................................................. 82 Best Practices for High Performance ....................... 96 Best Practices for High Availability ..........................100 Queue Federation ....................................................... 104 RabbitMQ Protocols ....................................................110 P A R T T H R E E RabbitMQ User Stories ......................................................... 116 A Monolithic System into Microservices .................118 How RabbitMQ Transformed Farmbot ................... 122 Microservice Architecture Built Upon RabbitMQ . 128 Event-based Communication .................................. 132

INTRODUCTION Times are changing; reliability and scalability are more important than ever. Because of that, companies are rethinking their architecture. Monoliths are evolving into microservices and servers are moving into the cloud. Message Queues and RabbitMQ in particular, as one of the most widely deployed open-source message brokers, have come to play a significant role in the growing world of microservices. This book is divided into three parts: The first part is an introduction to microservices and RabbitMQ, including a section on how to use RabbitMQ on the CloudAMQP platform. This section includes the most important RabbitMQ concepts and how it allows users to create products that meet current and future industry demands. The second part of the book is for more advanced users, who will learn how to take full advantage of RabbitMQ. This section explores Best Practice recommendations for High Performance and High Availability and common RabbitMQ errors and mistakes. It is also a deep dive into some RabbitMQ concepts and features. The third part gives some real-world user stories from our own experiences with RabbitMQ as well from a CloudAMQP customer’s point of view. We hope that this book takes you far on your message queuing journey with RabbitMQ.

P A R T O N E INTRODUCTION TO RABBITMQ A MICROSERVICE ARCHITECTURE Modules with various functionalities operate together to form a complete software application via properly articulated operations and interfaces.

11 Welcome to the wonderful world of message queuing! Many modern cloud architectures are a collection of loosely coupled microservices that communi- cate via a message queue. This microservice architecture is beneficial in that all the different parts of the system are easier to develop, deploy and maintain. One of the advantages of a message queue is that it makes your data temporarily persistent, reducing the chance of errors that may occur when different parts of the system are offline. If one part of the system is unreachable, the other part can continue to interact with the queue. This part of the book features basic information on micros- ervices and message queuing, and the benefits of RabbitMQ.

P A R T O N E MICROSERVICES AND RABBITMQ

13 Managing a complex, unified enterprise application can be a lot more difficult than it seems. Before making even a small change, hours of testing and analysis are required for possible impacts your change could have on the overall system. New developers must spend days getting familiar with the system before they are con- sidered ready to write a non-breaking line of code. Fortunately, the situation becomes far less complicated thanks to the use of microservices. This chapter talks about the benefits of a microservice architecture. In a microservice architecture, also called modular architecture, modules are decoupled from each other. These modules often offer various functionalities that operate together to form a complete software application via properly articulated oper- ations and interfaces. Unlike a monolithic application, where all chunks of functionality are present within a single module, a microservice application divides different func- tionalities across different modules to enhance maintainability and ease-of-use. More and more organizations are adopting microservice architecture because it allows for a more agile approach to software development and maintenance. Microservices also make it easy for businesses to scale and deliver updated versions of software weekly instead of having to wait through years of testing. BENEFITS OF A MICROSERVICE ARCHITECTURE • Easier development and maintenance Imagine building a huge, bulky billing application that will involve authentica- tion, authorization, charging and reporting of transactions. Dividing the applica- tion across multiple services (one for each functionality) will separate the respon- sibilities and give developers the freedom to write code for a specific service in any chosen language; something that a monolithic application can’t provide. Addi- tionally, it will also be easier to maintain written code and make changes to the system. For example, updating an authentication scheme will only require the addition of code to the authentication module and testing without having to worry about disrupting any other functionalities of the application. • Fault isolation Another obvious advantage offered by a microservice architecture is its ability to isolate the fault to a single module. For example, if a reporting service is down, authentication and billing services will still be running, ensuring that customers can perform important transactions even when they are not able to view reports. • Increased speed and productivity Microservice architecture is all about decoupling functions into manageable modules that are easy to maintain. Different developers can work on different

14 GET STARTED FOR FREE WITH CLOUDAMQP Perfectly configured and optimized RabbitMQ clusters, ready in 2 minutes. www.cloudamqp.com Custom Alarms • Free Plan Available • Easy Monitoring • 24/7 support • 99.95% SLA modules at the same time. In addition to the development cycle, the testing phase is also sped up by the use of microservices, as each microservice can be tested independently to determine the readiness of the overall system. • Improved scalability Microservices also allow effortless system scaling whenever required. Adding new components to just one service can be completed without changing any of the other services. Along the same lines, resource-intensive services can also be deployed across multiple servers by using microservices. • Easy to understand Another advantage offered by a microservice architecture is the ease of under- standability. Because each module represents a single functionality, getting to know the relevant details becomes easier and faster. For example, a consultant that works on charging services does not have to understand the whole system to perform maintenance or enhancements to their part of the system. THE ROLE OF A MESSAGE QUEUE IN A MICROSERVICE ARCHITECTURE Think of a message broker like a delivery person who takes mail from a sender and delivers it to the correct destination. In a microservice architecture, there typically are cross-dependencies that mean no single service can perform without getting help from other services. This is where it is crucial for systems to have a mechanism in place, allowing services to keep in touch with each other with no blocked responses. Message queuing fulfils this purpose by providing a means for services to push messages to a queue asynchronously and ensure they are delivered to the correct destination. To implement message queuing, a message broker is required.

15

P A R T O N E WHAT IS RABBITMQ?

17 Message queuing is an important facet of any microservice architecture; it is a way of exchanging data between processes, applications and servers. RabbitMQ is one of the most widely used message brokers in the world, boasting over 35,000 production deployments across both startups and big enterprises alike. This chapter gives a brief understanding of message queuing and defines essential RabbitMQ concepts. The chapter also explains the steps to go through when setting up connec- tions including how to publish as well as consume messages from a queue. Figure 1 - Messages sent from a sender to a receiver. RabbitMQ is a message queuing software also known as a message broker or queue manager. Simply put, it is a software where queues can be defined, and applications may connect and transfer a message onto it. Message queues enable asynchronous communication, which means that other applications (endpoints) that are producing and consuming messages interact with the queue instead of communicating directly with each other. A message can include any information. It could, for example, contain information about a process or job that should start on another application, possibly even on another server, or it might be a simple text message. The message broker stores the messages until a receiving application connects and consumes a message off the queue. The receiving application then appropriately processes the message. A message producer adds messages to a queue without having to wait for them to be processed.

18 RABBITMQ EXAMPLE A message broker can act as a middleman for various services. A broker can be used to reduce loads and delivery times by web application servers. A task that would generally take a lot of time to process can be delegated to a third party whose only job is to perform them. This chapter is a case study of a web application, a PDF generator. The application allows users to upload information to a website. The website handles the information and generates a PDF, which is then emailed back to the user. The whole processing in this example takes several seconds to perform. When the user has entered user information into the web interface, the web applica- tion puts a “PDF processing” job into a message, including information about the job such as name and email. The message is then placed onto a queue defined in RabbitMQ using an exchange process. The underlying architecture of a message queue is simple - client applications called producers create messages and deliver them to the broker. The broker is, essentially, the message queue. Other applications, known as consumers, connect to the queue and subscribe to messages from the broker, processing those messages according to their function. The software interacting with the broker can be a producer, a consumer, or both. Messages placed in the queue are stored until the consumer retrieves them. If the PDF processing application crashes, or if many PDF requests are coming at the same time, messages would continue to pile up in the queue until the consumer starts again. It would then process all the messages, one by one. Figure 2 - A sketch of the RabbitMQ workflow. WHY AND WHEN TO USE RABBITMQ Message queuing allows web servers to respond to requests in their own time instead of being forced to perform resource-heavy procedures immediately. Message queuing is also useful for distributing a message to multiple recipients for consumption or when balancing the load between workers. The consumer application removes a message from the queue and processes the PDF at the same time as the producer pushes new task messages to the queue. The consumer can be on an entirely different server than the publisher or they can be located on the same server, it makes no difference. Requests can be created in one programming language and handled in another programming language, as the two applications only communicate through the messages they are sending to each other. The two services have what is known as ‘low coupling’ between the sender and the receiver.

19 Figure 3 - Application architecture example with RabbitMQ. Example 1. The user sends a PDF create request to the web application. 2. The web application (the producer) sends a message to RabbitMQ that includes data from the request, such as name and email. 3. An exchange accepts the messages from a producer application and routes them to correct message queues. 4. The PDF processor (the consumer) receives the job message from the queue and starts the processing of the PDF.