RabbitMQ in Depth (Gavin M. Roy) (Z-Library)

M A N N I N G Gavin M. Roy IN DEPTH

RabbitMQ in Depth

RabbitMQ in Depth GAVIN ROY M A N N I N G SHELTER ISLAND

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com ©2018 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. Manning Publications Co. Development editor: Karen Miller 20 Baldwin Road Technical editor: James Titcumb PO Box 761 Technical development editor: Phillip Warner Shelter Island, NY 11964 Copyeditor: Andy Carroll Proofreader: Alyson Brener Technical proofreader: Karsten Strøbæk Typesetter: Dennis Dalinnik Cover designer: Marija Tudor ISBN: 9781617291005 Printed in the United States of America 1 2 3 4 5 6 7 8 9 10 – EBM – 22 21 20 19 18 17

brief contents PART 1 RABBITMQ AND APPLICATION ARCHITECTURE ...............1 1 ■ Foundational RabbitMQ 3 2 ■ How to speak Rabbit: the AMQ Protocol 18 3 ■ An in-depth tour of message properties 38 4 ■ Performance trade-offs in publishing 58 5 ■ Don’t get messages; consume them 79 6 ■ Message patterns via exchange routing 101 PART 2 MANAGING RABBITMQ IN THE DATA CENTER OR THE CLOUD..........................................................133 7 ■ Scaling RabbitMQ with clusters 135 8 ■ Cross-cluster message distribution 148 PART 3 INTEGRATIONS AND CUSTOMIZATION.........................175 9 ■ Using alternative protocols 177 10 ■ Database integrations 205v

contents preface xiii acknowledgments xiv about this book xv PART 1 RABBITMQ AND APPLICATION ARCHITECTURE......1 1 Foundational RabbitMQ 3 1.1 RabbitMQ’s features and benefits 4 RabbitMQ and Erlang 5 ■ RabbitMQ and AMQP 6 1.2 Who’s using RabbitMQ, and how? 7 1.3 The advantages of loosely coupled architectures 8 Decoupling your application 10 ■ Decoupling database writes 11 ■ Seamlessly adding new functionality 12 Replication of data and events 12 ■ Multi-master federation of data and events 13 ■ The Advanced Message Queuing model 14 1.4 Summary 16vii

CONTENTSviii2 How to speak Rabbit: the AMQ Protocol 18 2.1 AMQP as an RPC transport 19 Kicking off the conversation 20 ■ Tuning in to the right channel 20 2.2 AMQP’s RPC frame structure 21 AMQP frame components 21 ■ Types of frames 22 Marshaling messages into frames 23 ■ The anatomy of a method frame 24 ■ The content header frame 26 The body frame 26 2.3 Putting the protocol to use 27 Declaring an exchange 27 ■ Declaring a queue 28 Binding a queue to an exchange 29 ■ Publishing a message to RabbitMQ 29 ■ Consuming messages from RabbitMQ 30 2.4 Writing a message publisher in Python 32 2.5 Getting messages from RabbitMQ 36 2.6 Summary 37 3 An in-depth tour of message properties 38 3.1 Using properties properly 39 3.2 Creating an explicit message contract with content-type 41 3.3 Reducing message size with gzip and content-encoding 43 3.4 Referencing messages with message-id and correlation-id 45 Message-id 45 ■ Correlation-id 45 3.5 Born-on dating: the timestamp property 46 3.6 Automatically expiring messages 47 3.7 Balancing speed with safety using delivery-mode 48 3.8 Validating message origin with app-id and user-id 49 app-id 50 ■ user-id 51 3.9 Getting specific with the message type property 51 3.10 Using reply-to for dynamic workflows 52 3.11 Custom properties using the headers property 53 3.12 The priority property 54 3.13 A property you can’t use: cluster-id/reserved 54 3.14 Summary 55

CONTENTS ix4 Performance trade-offs in publishing 58 4.1 Balancing delivery speed with guaranteed delivery 59 What to expect with no guarantees 60 ■ RabbitMQ won’t accept non-routable messages with mandatory set 62 ■ Publisher Confirms as a lightweight alternative to transactions 64 Using alternate exchanges for unroutable messages 66 Batch processing with transactions 68 ■ Surviving node failures with HA queues 70 ■ HA queues with transactions 72 Persisting messages to disk via delivery-mode 2 72 4.2 When RabbitMQ pushes back 75 Checking the connection status with rabbitpy 77 ■ Using the management API for connection status 77 4.3 Summary 78 5 Don’t get messages; consume them 79 5.1 Basic.Get vs. Basic.Consume 80 Basic.Get 80 ■ Basic.Consume 82 5.2 Performance-tuning consumers 84 Using no-ack mode for faster throughput 85 ■ Controlling consumer prefetching via quality of service settings 86 Using transactions with consumers 89 5.3 Rejecting messages 90 Basic.Reject 90 ■ Basic.Nack 91 ■ Dead letter exchanges 92 5.4 Controlling queues 94 Temporary queues 94 ■ Permanent queues 97 ■ Arbitrary queue settings 99 5.5 Summary 99 6 Message patterns via exchange routing 101 6.1 Simple message routing using the direct exchange 102 Creating the application architecture 103 ■ Creating the RPC worker 107 ■ Writing a simple RPC publisher 110 6.2 Broadcasting messages via the fanout exchange 115 Modifying the facial detection consumer 116 ■ Creating a simple image-hashing consumer 117 6.3 Selectively routing messages with the topic exchange 119 6.4 Selective routing with the headers exchange 122

CONTENTSx6.5 Exchange performance benchmarking 124 6.6 Going meta: exchange-to-exchange routing 125 6.7 Routing messages with the consistent-hashing exchange 127 6.8 Summary 131 PART 2 MANAGING RABBITMQ IN THE DATA CENTER OR THE CLOUD................................................133 7 Scaling RabbitMQ with clusters 135 7.1 About clusters 136 Clusters and the management UI 137 ■ Cluster node types 138 Clusters and queue behavior 139 7.2 Cluster setup 142 Virtual machine setup 143 ■ Adding nodes to the cluster 144 7.3 Summary 147 8 Cross-cluster message distribution 148 8.1 Federating exchanges and queues 149 Federated exchanges 149 ■ Federated queues 152 8.2 Creating the RabbitMQ virtual machines 153 Creating the first instance 153 ■ Duplicating the EC2 instance 159 8.3 Connecting upstream 162 Defining federation upstreams 162 ■ Defining a policy 164 Leveraging upstream sets 167 ■ Bidirectional federated exchanges 170 ■ Federation for cluster upgrades 171 8.4 Summary 173 PART 3 INTEGRATIONS AND CUSTOMIZATION ...............175 9 Using alternative protocols 177 9.1 MQTT and RabbitMQ 178 The MQTT protocol 178 ■ Publishing via MQTT 182 MQTT subscribers 184 ■ MQTT plugin configuration 187

CONTENTS xi9.2 STOMP and RabbitMQ 189 The STOMP protocol 190 ■ Publishing messages 191 Consuming messages 195 ■ Configuring the STOMP plugin 198 ■ Using STOMP in the web browser 199 9.3 Stateless publishing via HTTP 200 How statelessd came to be 200 ■ Using statelessd 201 Operational architecture 202 ■ Publishing messages via statelessd 203 9.4 Summary 203 10 Database integrations 205 10.1 The pg_amqp PostgreSQL extension 206 Installing the pg_amqp extension 207 ■ Configuring the pg_amqp extension 209 ■ Publishing a message via pg_amqp 210 10.2 Listening to PostgreSQL notifications 212 Installing the PostgreSQL LISTEN exchange 213 ■ Policy-based configuration 215 ■ Creating the exchange 217 ■ Creating and binding a test queue 217 ■ Publishing via NOTIFY 218 10.3 Storing messages in InfluxDB 219 InfluxDB installation and setup 220 ■ Installing the InfluxDB storage exchange 222 ■ Creating a test exchange 223 Testing the exchange 224 10.4 Summary 227 appendix Getting set up 228 index 237

preface When Manning Publications published RabbitMQ in Action back in April 2012, RabbitMQ was gaining popularity rapidly. As of today, it is one of the stalwart leaders in the world of message brokers and is ideal for a variety of application uses. Facilitating communi- cation through distributed applications, using micro-services in a service oriented architecture, and enabling logical separation of CQRS and Event Sourcing compo- nents are just some of the common uses of RabbitMQ. We now present a new, in-depth exploration of RabbitMQ itself, digging deep under the surface by examining things like how the Advanced Message Queuing Protocol is structured, progressive exploration into the various exchanges, and exami- nation of various performance aspects. RabbitMQ in Depth aims to take your under- standing of RabbitMQ to a new level, enabling you to further apply this knowledge in real-world applications today.xiii

acknowledgments This book has been some time in the making, and so first and foremost a big thank you goes out to all of our families and friends who tirelessly stood by us all, put up with us, and made those late evening coffees to keep us going through the many additional long hours of work required to write such a book – thank you! To Alvaro Videla and Jason J.W. Williams, authors of RabbitMQ in Action (also pub- lished by Manning Publications, in April 2012), for laying the foundation for count- less developers’ insight and interest in RabbitMQ itself. To Karen, our development editor, for her endless patience and understanding with us all over this entire period, plus the entire Manning team for the fantastic effort involved by all to finally get us to this point. It was hard work, and we ended up doing a fair few rounds, but we are grateful for the solid production effort which has emerged as a result! Thanks, too, to technical proofreader, Karsten Strøbæk, whose contribution helped the book immensely, as did the comments of the reviewers: Phillip Warner, Jerry Kuch, Nadia Saad Noori, Bruce Snyder, Robert Kielty, Milos Milivojevic, Arathi Maddula, Ian Dallas, George Harley, Dimitri Aivaliotis, Hechen Gao, Stefan Turalski, Andrew Meredith, Artem Dayneko, David Paccoud, Barry Alexander, Biju Kunjummen, Adolfo Pérez Álvarez, Brandon Wilhite, David Pull, and Ray Lugo. There are a great many others which have helped contribute in various ways to this book as well. We cannot mention everyone by name as this would just mean the acknowledgements roll on and on, but a big thank you goes out to everyone else who had a hand in helping make this possible!xiv

about this book RabbitMQ is an open source message broker written in Erlang, currently under the wing of Pivotal Software. It’s based around the AMQP open protocol, with official cli- ent libraries in Java, .NET, Erlang, as well as libraries for most other popular program- ming languages. This book is up-to-date with RabbitMQ 3.6.3, so with the erratic release schedule of RabbitMQ itself, by the time this book reaches you there may be newer versions released. Not to fret though, as in our experience RabbitMQ has rarely broken fea- tures with releases, only added new features and fixed issues! The code examples used throughout the book are written with Python, but if you don’t have a working setup with Python and RabbitMQ, or you’d just like to experi- ment without setting up the whole environment, we’ve included instructions on set- ting up a Vagrant box with everything pre-installed. Make sure you check out the appendix for instructions on how to get this up and running first. Road Map Chapter 1 looks at the foundation of RabbitMQ: the various features of RabbitMQ itself and the foundation of RabbitMQ, the Advanced Messaging Queuing model. Chapter 2 explores the AMQ protocol, looking at the frame structure, and the low- level process that occurs when a message is published or retrieved from RabbitMQ. Chapter 3 goes even further and looks at the message properties, including the headers that add important meta-data to messages, such as content-type and encod- ing, and how you can leverage these headers in your applications.xv

ABOUT THIS BOOKxvi Chapter 4 considers performance trade-offs which must be made. With each level of guarantee, your applications risk taking a hit on performance. This chapter explores what these options are and will help you balance your environments’ need for guaran- teed message assurance versus lightning fast delivery, the Goldilocks Principle. Chapter 5 explores the concept of consuming messages, looking at the fundamen- tal difference between Basic.Get and Basic.Consume at a low level (and why the latter is usually better), as well as pre-fetching and Quality of Service, message acknowledge- ments, dead letter exchanges, temporary queues, and message expiry. Chapter 6 takes an in-depth look into the four core exchange types in RabbitMQ and how they can benefit your application architecture. Chapter 7 looks at how you can scale up RabbitMQ by managing clusters, crash recovery in a cluster, and further performance considerations when working with a clustered environment. Chapter 8 builds on the core concepts of clustering by taking a look at federated exchanges and queues, integrating RabbitMQ clusters with Amazon Web Services, and applying policies. Chapter 9 looks at other ways of talking to RabbitMQ: using MQTT and STOMP as alternative protocols, or using statelessd-based HTTP messaging. Finally, Chapter 10 looks at database integration into both PostgreSQL and InfluxDB for further interesting integrations. Code Just about all of the code shown in the book can be found in various forms in the sam- ple source code which accompanies this book. The sample code can be downloaded free of charge from the Manning website (https://www.manning.com/books/rabbitmq- in-depth), as well as from this Github repository: https://github.com/gmr/RabbitMQ- in-Depth. Book forum Purchase of RabbitMQ in Depth includes free access to a private web forum run by Man- ning Publications where you can make comments about the book, ask technical ques- tions, and receive help from the author and from other users. To access the forum, go to https://forums.manning.com/forums/rabbitmq-in-depth. You can also learn more about Manning’s forums and the rules of conduct at https://forums.manning.com/ forums/about. Manning’s commitment to our readers is to provide a venue where a meaningful dialogue between individual readers and between readers and the author can take place. It is not a commitment to any specific amount of participation on the part of the author, whose contribution to the forum remains voluntary (and unpaid). We sug- gest you try asking the author some challenging questions lest his interest stray! The forum and the archives of previous discussions will be accessible from the publisher’s website as long as the book is in print.

ABOUT THIS BOOK xviiAbout the author GAVIN M. ROY is an active open-source evangelist and advocate who has been working with internet and Enterprise technologies since the mid-90’s. About the cover The figure on the cover of RabbitMQ in Depth is captioned “A man from Mikanovac, Srijem, Croatia.” The illustration is taken from a reproduction of an album of Croa- tian traditional costumes from the mid-nineteenth century by Nikola Arsenovic, pub- lished by the Ethnographic Museum in Split, Croatia, in 2003. The illustrations were obtained from a helpful librarian at the Ethnographic Museum in Split, itself situated in the Roman core of the medieval center of the town: the ruins of Emperor Diocle- tian’s retirement palace from around AD 304. The book includes finely colored illus- trations of figures from different regions of Croatia, accompanied by descriptions of the costumes and of everyday life. Dress codes and lifestyles have changed over the last 200 years, and the diversity by region, so rich at the time, has faded away. It’s now hard to tell apart the inhabitants of different continents, let alone of different hamlets or towns separated by only a few miles. Perhaps we have traded cultural diversity for a more varied personal life—cer- tainly for a more varied and fast-paced technological life. Manning celebrates the inven- tiveness and initiative of the computer business with book covers based on the rich diversity of regional life of two centuries ago, brought back to life by illustrations from old books and collections like this one.

Part 1 RabbitMQ and application architecture In this part of the book, we’ll explore the structure of the AMQ protocol, which is how your application communicates with RabbitMQ. We’ll also look at the messages themselves, leveraging features such as message headers, priority, and more, to enhance message interactions. We’ll explore performance trade-offs, balancing stability and transactional safety against high-performance throughput with no guarantees. Additionally, we’ll investigate the different exchange types and why they work the way they do.