Kubernetes Patterns Reusable Elements for Designing Cloud-Native Applications (Bilgin Ibryam, Roland Huß)（Z-Library）

Bilgin Ibryam and Roland Huß Kubernetes Patterns Reusable Elements for Designing Cloud-Native Applications Boston Farnham Sebastopol TokyoBeijing

978-1-492-05028-5 [LSI] Kubernetes Patterns by Bilgin Ibryam and Roland Huß Copyright © 2019 Bilgin Ibryam and Roland Huß. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: John Devins Development Editor: Virginia Wilson Production Editor: Katherine Tozer Copyeditor: Christine Edwards Proofreader: Sharon Wilkey Indexer: Judith McConville Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Rebecca Demarest May 2019: First Edition Revision History for the First Edition 2019-04-04: First Release See https://www.oreilly.com/catalog/errata.csp?isbn=9781492050285 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Kubernetes Patterns, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

Table of Contents Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi 1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 The Path to Cloud Native 1 Distributed Primitives 3 Containers 4 Pods 5 Services 7 Labels 7 Annotations 9 Namespaces 9 Discussion 11 More Information 12 Part I. Foundational Patterns 2. Predictable Demands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Problem 15 Solution 16 Runtime Dependencies 16 Resource Profiles 18 Pod Priority 20 Project Resources 22 Capacity Planning 22 iii

Discussion 23 More Information 24 3. Declarative Deployment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Problem 25 Solution 25 Rolling Deployment 27 Fixed Deployment 29 Blue-Green Release 30 Canary Release 30 Discussion 31 More Information 33 4. Health Probe. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Problem 35 Solution 35 Process Health Checks 36 Liveness Probes 36 Readiness Probes 37 Discussion 38 More Information 40 5. Managed Lifecycle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Problem 41 Solution 41 SIGTERM Signal 42 SIGKILL Signal 42 Poststart Hook 43 Prestop Hook 44 Other Lifecycle Controls 45 Discussion 46 More Information 46 6. Automated Placement. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Problem 47 Solution 47 Available Node Resources 48 Container Resource Demands 49 Placement Policies 49 Scheduling Process 50 Node Affinity 51 iv | Table of Contents

Pod Affinity and Antiaffinity 52 Taints and Tolerations 54 Discussion 57 More Information 59 Part II. Behavioral Patterns 7. Batch Job. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Problem 63 Solution 64 Discussion 67 More Information 68 8. Periodic Job. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Problem 69 Solution 70 Discussion 71 More Information 72 9. Daemon Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Problem 73 Solution 74 Discussion 76 More Information 77 10. Singleton Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Problem 79 Solution 80 Out-of-Application Locking 80 In-Application Locking 82 Pod Disruption Budget 84 Discussion 85 More Information 86 11. Stateful Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 Problem 87 Storage 88 Networking 89 Identity 89 Ordinality 89 Table of Contents | v

Other Requirements 89 Solution 90 Storage 91 Networking 92 Identity 94 Ordinality 94 Other Features 95 Discussion 96 More information 97 12. Service Discovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Problem 99 Solution 100 Internal Service Discovery 101 Manual Service Discovery 104 Service Discovery from Outside the Cluster 107 Application Layer Service Discovery 111 Discussion 113 More Information 115 13. Self Awareness. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 Problem 117 Solution 117 Discussion 121 More Information 121 Part III. Structural Patterns 14. Init Container. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 Problem 125 Solution 126 Discussion 130 More Information 130 15. Sidecar. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Problem 131 Solution 132 Discussion 134 More Information 134 vi | Table of Contents

16. Adapter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 Problem 135 Solution 135 Discussion 138 More Information 138 17. Ambassador. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Problem 139 Solution 139 Discussion 141 More Information 142 Part IV. Configuration Patterns 18. EnvVar Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 Problem 145 Solution 145 Discussion 148 More Information 149 19. Configuration Resource. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 Problem 151 Solution 151 Discussion 156 More Information 156 20. Immutable Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 Problem 157 Solution 157 Docker Volumes 158 Kubernetes Init Containers 159 OpenShift Templates 162 Discussion 163 More Information 164 21. Configuration Template. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 Problem 165 Solution 165 Discussion 170 More Information 171 Table of Contents | vii

Part V. Advanced Patterns 22. Controller. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 Problem 175 Solution 176 Discussion 186 More Information 187 23. Operator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 Problem 189 Solution 190 Custom Resource Definitions 190 Controller and Operator Classification 192 Operator Development and Deployment 195 Example 197 Discussion 201 More Information 202 24. Elastic Scale. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 Problem 203 Solution 204 Manual Horizontal Scaling 204 Horizontal Pod Autoscaling 205 Vertical Pod Autoscaling 210 Cluster Autoscaling 213 Scaling Levels 216 Discussion 219 More Information 219 25. Image Builder. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 Problem 221 Solution 222 OpenShift Build 223 Knative Build 230 Discussion 234 More Information 235 Afterword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 viii | Table of Contents

Foreword When Craig, Joe, and I started Kubernetes nearly five years ago, I think we all recog‐ nized its power to transform the way the world developed and delivered software. I don’t think we knew, or even hoped to believe, how quickly this transformation would come. Kubernetes is now the foundation for the development of portable, reli‐ able systems spanning the major public clouds, private clouds, and bare-metal envi‐ ronments. However, even as Kubernetes has become ubiquitous to the point where you can spin up a cluster in the cloud in less than five minutes, it is still far less obvi‐ ous to determine where to go once you have created that cluster. It is fantastic that we have seen such significant strides forward in the operationalization of Kubernetes itself, but it is only a part of the solution. It is the foundation on which applications will be built, and it provides a large library of APIs and tools for building these appli‐ cations, but it does little to provide the application architect or developer with any hints or guidance for how these various pieces can be combined into a complete, reli‐ able system that satisfies their business needs and goals. Although the necessary perspective and experience for what to do with your Kuber‐ netes cluster can be achieved through past experience with similar systems, or via trial and error, this is expensive both in terms of time and the quality of systems delivered to our end users. When you are starting to deliver mission-critical services on top of a system like Kubernetes, learning your way via trial and error simply takes too much time and results in very real problems of downtime and disruption. This then is why Bilgin and Roland’s book is so valuable. Kubernetes Patterns enables you to learn from the previous experience that we have encoded into the APIs and tools that make up Kubernetes. Kubernetes is the by-product of the community’s experience building and delivering many different, reliable distributed systems in a variety of different environments. Each object and capability added to Kubernetes represents a foundational tool that has been designed and purpose-built to solve a specific need for the software designer. This book explains how the concepts in Kubernetes solve real-world problems and how to adapt and use these concepts to build the system that you are working on today. ix

In developing Kubernetes, we always said that our North Star was making the devel‐ opment of distributed systems a CS 101 exercise. If we have managed to achieve that goal successfully, it is books like this one that are the textbooks for such a class. Bilgin and Roland have captured the essential tools of the Kubernetes developer and distil‐ led them into segments that are easy to approach and consume. As you finish this book, you will become aware not just of the components available to you in Kuber‐ netes, but also the “why” and “how” of building systems with those components. — Brendan Burns, Cofounder, Kubernetes x | Foreword

Preface With the evolution of microservices and containers in recent years, the way we design, develop, and run software has changed significantly. Today’s applications are optimized for scalability, elasticity, failure, and speed of change. Driven by new prin‐ ciples, these modern architectures require a different set of patterns and practices. This book aims to help developers create cloud-native applications with Kubernetes as a runtime platform. First, let’s take a brief look at the two primary ingredients of this book: Kubernetes and design patterns. Kubernetes Kubernetes is a container orchestration platform. The origin of Kubernetes lies some‐ where in the Google data centers where Google’s internal container orchestration platform, Borg, was born. Google used Borg for many years to run its applications. In 2014, Google decided to transfer its experience with Borg into a new open source project called “Kubernetes” (Greek for “helmsman” or “pilot”), and in 2015, it became the first project donated to the newly founded Cloud Native Computing Foundation (CNCF). Right from the start, Kubernetes gained a whole community of users, and the number of contributors grew at an incredibly fast pace. Today, Kubernetes is considered one of the most active projects on GitHub. It is probably fair to claim that at the time of this writing, Kubernetes is the most commonly used and feature-rich container orchestration platform. Kubernetes also forms the foundation of other platforms built on top of it. The most prominent of those Platform-as-a-Service systems is Red Hat OpenShift, which provides various additional capabilities to Kubernetes, including ways to build applications within the platform. These are only some of the reasons we chose Kubernetes as the reference platform for the cloud-native patterns in this book. This book assumes you have some basic knowledge of Kubernetes. In Chapter 1, we recapitulate the core Kubernetes concepts and lay out the foundation for the follow‐ ing patterns. xi

1 Christopher Alexander and his team defined the original meaning in the context of architecture as follows: “Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice,” (A Pattern Language, Christopher Alexander et al., 1977, p. x). We think this definition works for the patterns we describe in this book, except that we probably don’t have as much variability in our solutions. Design Patterns The concept of design patterns dates back to the 1970s and from the field of architec‐ ture. Christopher Alexander, an architect and system theorist, and his team published the groundbreaking A Pattern Language (Oxford University Press) in 1977, which describes architectural patterns for creating towns, buildings, and other construction projects. Sometime later this idea was adopted by the newly formed software indus‐ try. The most famous book in this area is Design Patterns—Elements of Reusable Object-Oriented Software by Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides—the Gang of Four (Addison-Wesley). When we talk about the famous Sin‐ gleton, Factories, or Delegation patterns, it’s because of this defining work. Many other great pattern books have been written since then for various fields with differ‐ ent levels of granularity, like Enterprise Integration Patterns by Gregor Hohpe and Bobby Woolf (Addison-Wesley) or Patterns of Enterprise Application Architecture by Martin Fowler (Addison-Wesley). In short, a pattern describes a repeatable solution to a problem.1 It is different from a recipe because instead of giving step-by-step instructions to solving a problem, a pat‐ tern provides a blueprint for solving a whole class of similar problems. For example, the Alexandrian pattern “Beer Hall” describes how public drinking halls should be constructed where “strangers and friends are drinking companions” and not “anchors of the lonely.” All halls built after this pattern look different, but share common char‐ acteristics such as open alcoves for groups of four to eight and a place where a hun‐ dred people can meet with beverages, music, and other activities. However, a pattern does more than provide a solution. It is also about forming a lan‐ guage. The unique pattern names form a dense, noun-centric language in which each pattern carries a unique name. When this language is established, these names auto‐ matically evoke similar mental representations when people speak about these pat‐ terns. For example, when we talk about a table, anyone speaking English assumes we are talking about a piece of wood with four legs and a top on which you can put things. The same thing happens in software engineering when we talk about a “fac‐ tory.” In an object-oriented programming language context, we immediately associate with a “factory” an object that produces other objects. Because we immediately know the solution behind the pattern, we can move on to tackle yet unsolved problems. xii | Preface

There are also other characteristics of a pattern language. Patterns are interconnected and can overlap so that together, they cover most of the problem space. Also, as already laid out in the original A Pattern Language, patterns do not have the same level of granularity and scope. More general patterns cover an extensive problem space and provide rough guidance on how to solve the problem. Granular patterns have a very concrete solution proposal but are not as widely applicable. This book contains all sort of patterns, and many patterns reference other patterns or may even include other patterns as part of the solution. Another feature of patterns is that they follow a rigid format. However, each author defines a different format, and unfortunately there is no common standard for the way patterns should be laid out. Martin Fowler gives an excellent overview of the for‐ mats used for pattern languages in Writing Software Patterns. How This Book Is Structured We chose a simple pattern format for this book. We do not follow any particular pat‐ tern description language. For each pattern, we use the following structure: Name Each pattern carries a name, which is also the chapter’s title. The name is the cen‐ ter of the pattern’s language. Problem This section gives the broader context and describes the pattern space in detail. Solution This section is about how the pattern solves the problem in a Kubernetes-specific way. This section also contains cross-references to other patterns that are either related or part of the given pattern. Discussion A discussion about the advantages and disadvantages of the solution for the given context follows. More Information This final section contains additional information sources related to the pattern. We organized the patterns of this book as follows: • Part I, Foundational Patterns, covers the core concepts of Kubernetes. These are the underlying principles and practices for building container-based cloud-native applications. Preface | xiii

• Part II, Behavioral Patterns, describes patterns that sit on top of the foundational patterns and add finer-grained concepts for managing various types of container and platform interactions. • Part III, Structural Patterns, contains patterns related to organizing containers within a Pod, which is the atom of the Kubernetes platform. • Part IV, Configuration Patterns, gives insight into the various ways application configuration can be handled in Kubernetes. These are very granular patterns, including concrete recipes for connecting applications to their configuration. • Part V, Advanced Patterns, is a collection of advanced concepts, such as how the platform itself can be extended or how to build container images directly within the cluster. A pattern might not always fit into one category alone. Depending on the context, the same pattern might fit into several categories. Every pattern chapter is self-contained, and you can read chapters in isolation and in any order. Who This Book Is For This book is for developers who want to design and develop cloud-native applications for the Kubernetes platform. It is most suitable for readers who have some basic familiarity with containers and Kubernetes concepts, and want to take it to the next level. However, you don’t need to know the low-level details of Kubernetes to under‐ stand the use cases and patterns. Architects, technical consultants, and developers will all benefit from the repeatable patterns described here. This book is based on use cases and lessons learned from real-world projects. We want to help you create better cloud-native applications—not reinvent the wheel. What You Will Learn There’s a lot to discover in this book. Some of the patterns may read like excerpts from a Kubernetes manual at first glance, but upon closer look you’ll see the patterns are presented from a conceptual angle not found in other books on the topic. Other patterns are explained with a different approach, with detailed guidelines for very concrete problems, as in “Configuration Patterns” in Part IV. Regardless of the pattern granularity, you will learn everything Kubernetes offers for each particular pattern, with plenty of examples to illustrate the concepts. All these examples have been tested, and we tell you how to get the full source code in “Using Code Examples”. xiv | Preface

Before we start to dive in, let’s briefly look at what this book is not: • This book is not a guide on how to set up a Kubernetes cluster itself. Every pat‐ tern and every example assumes you have Kubernetes up and running. You have several options for trying out the examples. If you are interested in learning how to set up a Kubernetes cluster, we recommend Managing Kubernetes by Brendan Burns and Craig Tracey (O’Reilly). Also, the Kubernetes Cookbook by Michael Hausenblas and Sébastien Goasguen (O’Reilly) has recipes for setting up a Kubernetes cluster from scratch. • This book is not an introduction to Kubernetes, nor a reference manual. We touch on many Kubernetes features and explain them in some detail, but we are focusing on the concepts behind those features. Chapter 1, Introduction, offers a brief refresher on Kubernetes basics. If you are looking for a comprehensive book on how to use Kubernetes, we highly recommend Kubernetes in Action by Marko Lukša (Manning Publications). The book is written in a relaxed style, and is similar to a series of essays that can be read independently. Conventions As mentioned, patterns form a kind of simple, interconnected language. To empha‐ size this web of patterns, each pattern is capitalized in italics, (e.g., Sidecar). When a pattern is named like a Kubernetes core concept (like Init Container or Controller), we use this specific formatting only when we directly reference the pattern itself. Where it makes sense, we also interlink pattern chapters for ease of navigation. We also use the following conventions: • Everything you can type in a shell or editor is rendered in fixed font width. • Kubernetes resource names are always rendered in uppercase (e.g., Pod). If the resource is a combined name like ConfigMap, we keep it like this in favor of the more natural “config map” for clarity and to make it clear that it refers to a Kubernetes concept. • Sometimes a Kubernetes resource name is identical to a common concept like “service” or “node”. In these cases we use the resource name format only when referring to the resource itself. Preface | xv

Using Code Examples Every pattern is backed with fully executable examples, which you can find on the accompanying web page. You can find the link to each pattern’s example in the “More Information” section of each chapter. The “More Information” section also contains many links to further information related to the pattern. We keep these lists updated in the example repository. Changes to the link collections will also be posted on Twitter. The source code for all examples in this book is available at GitHub. The repository and the website also have pointers and instructions on how to get a Kubernetes clus‐ ter to try out the examples. When you go through the examples, please also have a look into the provided resource files. They contain many useful comments that help further in understanding the example code. Many examples use a REST service called random-generator that returns random numbers when called. It is uniquely crafted for playing well with the examples of this book. Its source can be found at GitHub as well, and its container image k8spat terns/random-generator is hosted on Docker Hub. For describing resource fields, we use a JSON path notation. For example, .spec.rep licas points to the replicas field of the resource’s spec section. If you find an issue in the example code or documentation or if you have a question, don’t hesitate to open a ticket in the GitHub issue tracker. We monitor these GitHub issues and are happy to answer any questions over there. All example code is distributed under the Creative Commons Attribution 4.0 (CC BY 4.0) license. The code is free to use, and you are free to share and adapt it for com‐ mercial and noncommercial projects. However, you should give attribution back to this book if you copy or redistribute the material. This attribution can be either a reference to the book including title, author, pub‐ lisher, and ISBN, as in “Kubernetes Patterns by Bilgin Ibryam and Roland Huß (O’Reilly). Copyright 2019 Bilgin Ibryam and Roland Huß, 978-1-492-05028-5.” Alternatively, add a link back to the accompanying website along with a copyright notice and link to the license. We love code contributions, too! If you think we can improve our examples, we are happy to hear from you. Just open a GitHub issue or create a pull request, and let’s start a conversation. xvi | Preface

O’Reilly Online Learning For almost 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help compa‐ nies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, conferences, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in- depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, please visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book where we list errata, examples, and additional information. You can access this page at https://oreil.ly/kubernetes_patterns. To comment or ask technical questions about this book, email bookques‐ tions@oreilly.com. For more information about our books, courses, conferences, and news, see our web‐ site at http://www.oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Follow the authors on Twitter: https://twitter.com/bibryam, https://twitter.com/ro14nd Find the authors on GitHub: https://github.com/bibryam, https://github.com/rhuss Follow their blogs: https://www.ofbizian.com, https://ro14nd.de Preface | xvii

Acknowledgments Creating this book was a long journey spanning over two years, and we want to thank all of our reviewers who kept us on the right track. Special kudos go out to Paolo Antinori and Andrea Tarocchi for helping us through the whole journey. Big thanks also to Marko Lukša, Brandon Philips, Michael Hüttermann, Brian Gracely, Andrew Block, Jiri Kremser, Tobias Schneck, and Rick Wagner, who supported us with their expertise and advices. Last, but not least, big thanks to our editors Virginia Wilson, John Devins, Katherine Tozer, Christina Edwards and all the awesome folks at O’Reilly for helping us push this book over the finish line. xviii | Preface