RabbitMQ in Action Distributed Messaging for Everyone (Alvaro Videla, Jason J. W. Williams) (Z-Library)

📄 File Format: PDF

💾 File Size: 8.0 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

M A N N I N G Alvaro Videla Jason J.W. Williams FOREWORD BY ALEXIS RICHARDSON IN ACTION Distributed messaging for everyone

📄 Page 2

RabbitMQ in Action

📄 Page 3

(This page has no text content)

📄 Page 4

RabbitMQ in Action DISTRIBUTED MESSAGING FOR EVERYONE ALVARO VIDELA JASON J.W. WILLIAMS M A N N I N G SHELTER ISLAND

📄 Page 5

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 261 Shelter Island, NY 11964 Email: orders@manning.com ©2012 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. Manning Publications Co. Development editors: Maria Townsley, Cynthia Kane 20 Baldwin Road Technical proofreader: Jerry Kuch PO Box 261 Copyeditor: Benjamin Berg Shelter Island, NY 11964 Proofreader: Katie Tennant Typesetter: Dottie Marsico Cover designer: Marija Tudor ISBN 9781935182979 Printed in the United States of America 1 2 3 4 5 6 7 8 9 10 – MAL – 17 16 15 14 13 12

📄 Page 6

To my grandfather, Maximiliano Godoy, who showed me the ways of life. Gracias. —A.V. To Mama, Papa, and my sister J’aime. Your love, support, and faith in me has made it possible to climb mountains... and to God who always carries me to the other side. —J.W.

📄 Page 7

(This page has no text content)

📄 Page 8

brief contents 1 ■ Pulling RabbitMQ out of the hat 1 2 ■ Understanding messaging 12 3 ■ Running and administering Rabbit 37 4 ■ Solving problems with Rabbit: coding and patterns 60 5 ■ Clustering and dealing with failure 87 6 ■ Writing code that survives failure 107 7 ■ Warrens and Shovels: failover and replication 120 8 ■ Administering RabbitMQ from the Web 137 9 ■ Controlling Rabbit with the REST API 154 10 ■ Monitoring: Houston, we have a problem 167 11 ■ Supercharging and securing your Rabbit 195 12 ■ Smart Rabbits: extending RabbitMQ 216vii

📄 Page 9

(This page has no text content)

📄 Page 10

contents foreword xv preface xvii acknowledgments xix about this book xxi about the cover illustration xxiv 1 Pulling RabbitMQ out of the hat 1 1.1 Living in other people’s dungeons 3 1.2 AMQP to the rescue 5 1.3 A brief history of RabbitMQ 5 1.4 Picking RabbitMQ out of the hat (and other open options) 8 1.5 Installing RabbitMQ on Unix systems 8 Why environment matters—living la vida Erlang 8 ■ Getting the package 9 ■ Setting up the folder structure 9 ■ Firing Rabbit up for the first time 9 1.6 Summary 10 2 Understanding messaging 12 2.1 Consumers and producers (not an economics lesson) 13 2.2 Building from the bottom: queues 16 2.3 Getting together: exchanges and bindings 20ix

📄 Page 11

CONTENTSx2.4 Multiple tenants: virtual hosts and separation 24 2.5 Where’s my message? Durability and you 25 2.6 Putting it all together: a day in the life of a message 28 2.7 Using publisher confirms to verify delivery 33 2.8 Summary 35 3 Running and administering Rabbit 37 3.1 Server management 38 Starting nodes 38 ■ Stopping nodes 39 ■ Stopping and restarting the application: what’s the difference? 40 ■ Rabbit configuration files 41 3.2 Asking permission 43 Managing users 43 ■ Rabbit’s permissions system 44 3.3 Checking up 47 Viewing statistics 47 ■ Understanding RabbitMQ’s logs 52 3.4 Fixing a bad Rabbit: troubleshooting 55 badrpc,nodedown and other Erlang-induced problems 55 3.5 Summary 59 4 Solving problems with Rabbit: coding and patterns 60 4.1 A decoupling story: what pushes us to messaging 61 An asynchronous state of mind (separating requests and actions) 61 ■ Affording scale: a world without load balancers 63 Zero-effort APIs: why be locked into just one language? 64 4.2 Fire-and-forget models 65 Sending alerts 65 ■ Parallel processing 74 4.3 Remember me: RPC over RabbitMQ and waiting for answers 80 Private queues and sending acknowledgements 81 ■ Simple JSON RPC with reply_to 82 4.4 Summary 86 5 Clustering and dealing with failure 87 5.1 Batteries included: RabbitMQ clustering 88 5.2 Architecture of a cluster 89 Queues in a cluster 89 ■ Distributing exchanges 91 ■ Am I RAM or a disk? 92

📄 Page 12

CONTENTS xi5.3 Setting up a cluster on your laptop 94 5.4 Distributing the nodes to more machines 97 5.5 Upgrading cluster nodes 100 5.6 Mirrored queues and preserving messages 101 Declaring and using mirrored queues 101 ■ Under the hood with mirrored queues 104 5.7 Summary 105 6 Writing code that survives failure 107 6.1 Load balancing your Rabbits 108 Installing HAProxy 110 ■ Configuring HAProxy 110 6.2 Lost connections and failing clients between servers 112 6.3 Summary 119 7 Warrens and Shovels: failover and replication 120 7.1 Warrens: another way of clustering 121 7.2 Setting up load balancer–based master/slave clusters 123 7.3 Long-distance communication and replication 126 Shoveling your Rabbits: an introduction to the Shovel plugin 126 Installing Shovel 129 ■ Configuring and running Shovel 130 7.4 Summary 136 8 Administering RabbitMQ from the Web 137 8.1 Beyond rabbitmqctl: the RabbitMQ Management plugin 138 Why you need the Management plugin 138 ■ Management plugin features 138 ■ Enabling the Management plugin 139 8.2 Managing RabbitMQ from the web console 141 Monitoring the Erlang VM 141 ■ Importing configuration from JSON files 142 8.3 Managing users from the web console 143 Creating users 143 ■ Managing users’ permissions 145 8.4 Managing exchanges and queues from the web console 146 Listing queues 148 ■ Creating queues 149

📄 Page 13

CONTENTSxii8.5 Back to the command line 150 Why another CLI? 150 ■ CLI administration the easier way 151 ■ Installing rabbitmqadmin script 152 ■ Purging queues, creating exchanges, and more 152 8.6 Summary 153 9 Controlling Rabbit with the REST API 154 9.1 What can you do with the RabbitMQ REST API? 155 9.2 Granting your clients access 157 9.3 Accessing statistics 158 9.4 Automating vhost and user provisioning 161 9.5 Summary 165 10 Monitoring: Houston, we have a problem 167 10.1 RabbitMQ monitoring: keeping an eye on your warren 168 Writing health checks for Nagios 168 ■ Checking that RabbitMQ is alive with AMQP simulation checks 170 ■ Checking aliveness with the REST API 172 ■ Creating a watchdog for configuration changes 176 ■ Monitoring your cluster status 180 10.2 Making sure consumers are consuming 185 Monitoring queue levels through AMQP 186 ■ Using the REST API to watch queue levels 190 ■ Rules of thumb for establishing a queue count baseline 193 10.3 Summary 194 11 Supercharging and securing your Rabbit 195 11.1 The need for speed 196 Message durability 196 ■ Message acknowledgment 197 Routing algorithm and bindings 197 ■ Delivering messages 198 11.2 Memory usage and process limits 200 Memory usage 201 ■ Erlang process count 203 11.3 SSL connections 204 SSL certificates 204 ■ Setting up a certificate authority 206 Generating the root certificate 209 ■ Generating the server certificates 210 ■ Generating the client certificates 211 Enabling SSL listeners in RabbitMQ 211 ■ Testing your RabbitMQ SSL setup 213 11.4 Summary 215

📄 Page 14

CONTENTS xiii12 Smart Rabbits: extending RabbitMQ 216 12.1 RabbitMQ plugins 217 What can you do with plugins? 217 ■ Where do you find plugins? 218 ■ Installing plugins 218 ■ Removing plugins 220 12.2 Making your own plugins 221 Getting the RabbitMQ Public Umbrella 222 ■ Setting up the folder structure 223 ■ Including the plugin build system 223 Creating the Erlang application file 224 12.3 Creating your custom exchange module 225 Registering your exchange with RabbitMQ 227 ■ Implementing the exchange behaviour 230 ■ Compiling your custom exchange 236 ■ Taking your plugin for a test drive 239 12.4 Summary 243 appendix A Using Rabbit from Java and .NET 244 appendix B Online resources 270 appendix C Installing RabbitMQ on Windows 275 index 279

📄 Page 15

(This page has no text content)

📄 Page 16

foreword Welcome to RabbitMQ in Action. If you’re like me, possibly you’re thinking, “Should I read past page one?” Alas, too many technology books are written and published, and not all merit more than superficial attention. So let me invite you to read on, if you think this description fits you: ■ You want a practical way to learn about push technology, streaming data, and other messaging patterns. ■ You want to achieve professional-level expertise with RabbitMQ, including best practices for design and running in production. In other words, this book is not just a guide to RabbitMQ. It teaches fundamental design patterns across many use cases. It shows why more applications are using them—and what the “dos” and “don’ts” are. What are these patterns? If you’ve ever wanted to draw a picture of your system as an information flow or network, rather than as a stack, then you’re probably using messaging, or are ready to do so. You may be thinking of data delivery, nonblocking operations, or push notifications. Or you want to use publish/subscribe, asynchro- nous processing, or work queues. All of these are patterns, and they form part of the design canvas known as messaging. Messaging is a critical capability: it enables software applications to connect and scale. Applications can connect to each other as components of a larger application, or to user devices and data. Messaging is essentially asynchronous in that it decouples applications by separating the sending and receiving of data. The wonderful thing is that this connection pattern works in the same way at any scale. xv

📄 Page 17

FOREWORDxvi Scale is the point. The dominance of the internet as a basis for application delivery has made scale the critical factor in application design. Thinking small is no longer acceptable. Recently the term big data has become fashionable. But everything is big now, compared to only a few years ago. For example, the number of mobile-connected devices will exceed the number of people on earth soon, probably in 2012. As I write this, Facebook is about to IPO. CTO Bret Taylor said that “Facebook would have been a mobile application if the technol- ogy had been available when Mark Zuckerberg was building it in his dorm room.” Take a moment to think about that. Most applications used to look like this: you load a document or get data from a database, do some processing, and write the results to disk. Future applications will look more like Facebook: always on, cloud hosted, and accessible anywhere. Input and processing are continuous and automatic, and deliver a filtered stream of information that the user wants, as it happens. These levels of automation, reach, and scale are impossible without adopting a very specific set of design patterns. It is these patterns that you can learn in this book. Derek Collison, one of the originators of modern messaging technology, memorably described messaging as enabling “data in motion.” It’s hard to imagine an application that doesn’t need to move data. So messaging is everywhere. This book gets you started immediately. The patterns are presented as code exam- ples that you can run, and the authors take special care to help you operate your sys- tem as well. With Jason J. W. Williams and Alvaro Videla, you have access to experts who’ve been running large-scale RabbitMQ systems for years. This book is a natural culmination of their outstanding work sharing these experiences with the community. After you get a feel for RabbitMQ, it’s very easy to get help and find more examples via the extensive RabbitMQ user community, regardless of which languages you’re writing code in. This makes RabbitMQ an excellent choice for your messaging needs. I hope this has whetted your appetite to turn the page and read on. There will be messages, and there will be rabbits, and all will be revealed. ALEXIS RICHARDSON COFOUNDER AND FORMER CEO RABBIT TECHNOLOGIES, LTD. SENIOR DIRECTOR VMWARE CLOUD APPLICATION PLATFORM

📄 Page 18

preface Writing this book has been like discovering RabbitMQ itself—encountering a prob- lem that needed solving, but not knowing what the solution looked like. Until May 2010, we didn’t even know each other. We both had been active in the RabbitMQ com- munity for the past two years, but we’d never actually bumped into each other. Then one day a conversation with Alexis Richardson (Rabbit’s CEO at the time) introduced Alvaro and me to each other, and made what you hold in your hands possible. What we had in common was a desire to write down in a single place all the knowledge we had acquired about RabbitMQ the hard way. Back in 2010, that knowledge was (and today still largely is) scattered across the internet in a smattering of blog articles and terse technical tutorials. In other words, we both wanted to write the book we wished had existed when we started with RabbitMQ two years earlier. Neither of us came from a traditional messaging background, which made us fast friends and has largely informed the tone of RabbitMQ in Action; we wanted this book to be accessible for folks who’ve never heard of a queue or a binding before. In fact, when each of us discovered RabbitMQ, we didn’t even know what “messaging” was or that it was the solution to the problems we were having. My (Jason’s) situation was that my company needed a way to take the spam reportings we received from our custom- ers and process them out-of-band from our main stream of incoming messages. In Alvaro’s case, his company had a social network whose member communication sys- tem was creaking under the load of a 200 GB database. Like so many others who’ve come to messaging, both us had first tried to solve our queue-centric issues using data- base tables. Problems, like ensuring that only one application instance consumed any particular queue item, plagued our attempts at a database-driven solution and sent usxvii

📄 Page 19

PREFACExviiilooking for a better way. After all, we knew we couldn’t be the first people in the his- tory of software to have these issues. The solution for both of us came in a surprisingly similar way: a friend at Plaxo told me to check out this “RabbitMQ thing” as a way to solve my queue-centric problems, and an Erlang colleague of Alvaro’s in China gave him the same advice. Halfway around the world, both of us discovered RabbitMQ in the same way, and in response to trying to solve almost exactly the same problem! In fact, since you’re reading this book about RabbitMQ, it’s likely that similar challenges have led you to discover Rab- bitMQ in the same way. That speaks to the fact of why RabbitMQ is so popular: it eas- ily solves the basic problems of distributing data that each of us runs into again and again when trying to scale the software that we build. Our hope is that RabbitMQ in Action will help you design solutions to those chal- lenges more quickly and easily with RabbitMQ, so you can spend more time writing the software that will change the world and less time getting up to speed on the mes- saging broker that will help you do it. Perhaps, along the way, RabbitMQ will intro- duce you to an awesome coauthor who will become the lifelong friend you never expected.1 This book is a product of how much we love writing software, and our hope is that it will help you do the same in ways you never thought possible. ALVARO VIDELA DÜBENDORF, SWITZERLAND JASON J. W. WILLIAMS BOISE, IDAHO, UNITED STATES 1 They say that coauthor relationships have a worse “divorce” rate than marriage. It’s not a bad comparison, since writing a book together requires the constant give-and-take and mutual respect that it takes to make liv- ing in close quarters work. So it’s been an unexpected blessing to not only be able to write a book, but to dis- cover a friend whose ideas can live in close quarters with yours and make a whole far greater than you could achieve alone.

📄 Page 20

acknowledgments Only two names appear on the cover of this book, but there are many more without whom it would not exist. First and foremost, we’d like to thank Alexis Richardson, RabbitMQ’s CEO when we started writing. Without his recommendation, Manning would not have come knocking on our inboxes, and we would never have written a book together. We also thank him for providing the foreword to our book. In that vein, we need to express our utmost gratitude to the RabbitMQ team for continual help and answers to our incessant questions about the minutiae of Rabbit. In particular, we owe a thank you to Matthew Sackman and Matthias Radestock, without whom the chapters on clustering and RabbitMQ internals would not have been possible. Above all, we owe an incalculable debt of gratitude to Jerry Kuch from the RabbitMQ team. Jerry volunteered countless hours repeatedly reviewing drafts of each chapter for accuracy, including doing the “official” technical review of the completed book by himself. Every time we needed clarification or advice outside our experience, Jerry was always a quick IM away. He was never cranky and never complained about being our point person on the RabbitMQ team. If you find yourself discovering little picadillos you never knew about Rabbit’s operation, you likely have Jerry Kuch to thank. He truly made this a better book, and is a fantastic engineer. At Manning, we cannot thank our primary development editor Maria Townsley enough. Maria kept us writing and on track. She put up with our work schedules, and our feast-or-famine style of delivering material. Above all she was our advocate and fought for what was important to us. If you enjoy the style of RabbitMQ in Action, thank Maria as she carried the flag for it. We also need to thank Cynthia Kane tremendously for getting us through the final chapters and into print. Cynthia stepped in as ourxix