Monolith to Microservices Evolutionary Patterns to Transform Your Monolith (Sam Newman) (z-library.sk, 1lib.sk, z-lib.sk)

📄 File Format: PDF

💾 File Size: 11.0 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

Sam Newman Monolith to Microservices Evolutionary Patterns to Transform Your Monolith

📄 Page 2

Sam Newman Monolith to Microservices Evolutionary Patterns to Transform Your Monolith Boston Farnham Sebastopol TokyoBeijing

📄 Page 3

978-1-492-07554-7 [LSI] Monolith to Microservices by Sam Newman Copyright © 2020 Sam Newman. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Developmental Editor: Chris Guzikowski Developmental Editor: Alicia Young Production Editor: Nan Barber Copyeditor: Jasmine Kwityn Proofreader: Sharon Wilkey Indexer: Ellen Troutman-Zaig Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Rebecca Demarest October 2019: First Edition Revision History for the First Edition 2019-10-11: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492047841 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Monolith to Microservices, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. This work is part of a collaboration between O’Reilly and NGINX. See our statement of editorial inde‐ pendence.

📄 Page 4

Table of Contents Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi 1. Just Enough Microservices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 What Are Microservices? 1 Independent Deployability 2 Modeled Around a Business Domain 2 Own Their Own Data 5 What Advantages Can Microservices Bring? 6 What Problems Do They Create? 6 User Interfaces 7 Technology 8 Size 8 And Ownership 10 The Monolith 12 The Single Process Monolith 12 The Distributed Monolith 14 Third-Party Black-Box Systems 14 Challenges of Monoliths 15 Advantages of Monoliths 15 On Coupling and Cohesion 16 Cohesion 17 Coupling 17 Just Enough Domain-Driven Design 28 Aggregate 29 Bounded Context 31 Mapping Aggregates and Bounded Contexts to Microservices 31 iii

📄 Page 5

Further Reading 32 Summary 32 2. Planning a Migration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Understanding the Goal 33 Three Key Questions 35 Why Might You Choose Microservices? 35 Improve Team Autonomy 35 Reduce Time to Market 37 Scale Cost-Effectively for Load 37 Improve Robustness 38 Scale the Number of Developers 40 Embrace New Technology 41 When Might Microservices Be a Bad Idea? 42 Unclear Domain 43 Startups 43 Customer-Installed and Managed Software 44 Not Having a Good Reason! 45 Trade-Offs 45 Taking People on the Journey 47 Changing Organizations 47 Establishing a Sense of Urgency 48 Creating the Guiding Coalition 48 Developing a Vision and Strategy 49 Communicating the Change Vision 50 Empowering Employees for Broad-Based Action 51 Generating Short-Term Wins 51 Consolidating Gains and Producing More Change 52 Anchoring New Approaches in the Culture 52 Importance of Incremental Migration 53 It’s Production That Counts 53 Cost of Change 54 Reversible and Irreversible Decisions 54 Easier Places to Experiment 56 So Where Do We Start? 56 Domain-Driven Design 56 How Far Do You Have to Go? 57 Event Storming 58 Using a Domain Model for Prioritization 58 A Combined Model 60 Reorganizing Teams 62 Shifting Structures 62 iv | Table of Contents

📄 Page 6

It’s Not One Size Fits All 63 Making a Change 65 Changing Skills 68 How Will You Know if the Transition Is Working? 71 Having Regular Checkpoints 71 Quantitative Measures 72 Qualitative Measures 72 Avoiding the Sunk Cost Fallacy 73 Being Open to New Approaches 73 Summary 74 3. Splitting the Monolith. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 To Change the Monolith, or Not? 76 Cut, Copy, or Reimplement? 76 Refactoring the Monolith 77 Migration Patterns 78 Pattern: Strangler Fig Application 79 How It Works 79 Where to Use It 81 Example: HTTP Reverse Proxy 83 Data? 86 Proxy Options 86 Changing Protocols 90 Example: FTP 93 Example: Message Interception 94 Other Protocols 97 Other Examples of the Strangler Fig Pattern 97 Changing Behavior While Migrating Functionality 97 Pattern: UI Composition 98 Example: Page Composition 99 Example: Widget Composition 99 Example: Micro Frontends 103 Where to Use It 104 Pattern: Branch by Abstraction 104 How It Works 105 As a Fallback Mechanism 111 Where to Use It 112 Pattern: Parallel Run 113 Example: Comparing Credit Derivative Pricing 113 Example: Homegate Listings 115 Verification Techniques 116 Using Spies 116 Table of Contents | v

📄 Page 7

GitHub Scientist 117 Dark Launching and Canary Releasing 118 Where to Use It 118 Pattern: Decorating Collaborator 118 Example: Loyalty Program 119 Where to Use It 120 Pattern: Change Data Capture 120 Example: Issuing Loyalty Cards 120 Implementing Change Data Capture 121 Where to Use It 124 Summary 124 4. Decomposing the Database. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 Pattern: The Shared Database 125 Coping Patterns 127 Where to Use It 127 But It Can’t Be Done! 127 Pattern: Database View 128 The Database as a Public Contract 129 Views to Present 130 Limitations 131 Ownership 131 Where to Use It 132 Pattern: Database Wrapping Service 132 Where to Use It 134 Pattern: Database-as-a-Service Interface 135 Implementing a Mapping Engine 136 Compared to Views 137 Where to Use It 137 Transferring Ownership 137 Pattern: Aggregate Exposing Monolith 138 Pattern: Change Data Ownership 141 Data Synchronization 143 Pattern: Synchronize Data in Application 145 Step 1: Bulk Synchronize Data 145 Step 2: Synchronize on Write, Read from Old Schema 146 Step 3: Synchronize on Write, Read from New Schema 147 Where to Use This Pattern 148 Where to Use It 148 Pattern: Tracer Write 149 Data Synchronization 152 Example: Orders at Square 154 vi | Table of Contents

📄 Page 8

Where to Use It 158 Splitting Apart the Database 158 Physical Versus Logical Database Separation 158 Splitting the Database First, or the Code? 160 Split the Database First 161 Split the Code First 165 Split Database and Code Together 170 So, Which Should I Split First? 170 Schema Separation Examples 171 Pattern: Split Table 171 Where to Use It 173 Pattern: Move Foreign-Key Relationship to Code 173 Moving the Join 175 Data Consistency 176 Where to Use It 178 Example: Shared Static Data 178 Transactions 187 ACID Transactions 187 Still ACID, but Lacking Atomicity? 188 Two-Phase Commits 190 Distributed Transactions—Just Say No 193 Sagas 193 Saga Failure Modes 195 Implementing Sagas 199 Sagas Versus Distributed Transactions 205 Summary 205 5. Growing Pains. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 More Services, More Pain 207 Ownership at Scale 209 How Can This Problem Show Itself? 209 When Might This Problem Occur? 210 Potential Solutions 210 Breaking Changes 210 How Can This Problem Show Itself? 211 When Might This Problem Occur? 211 Potential Solutions 212 Reporting 215 When Might This Problem Occur? 216 Potential Solutions 216 Monitoring and Troubleshooting 217 When Might These Problems Occur? 218 Table of Contents | vii

📄 Page 9

How Can These Problems Occur? 218 Potential Solutions 218 Local Developer Experience 222 How Can This Problem Show Itself? 223 When Might This Occur? 223 Potential Solutions 223 Running Too Many Things 224 How Might This Problem Show Itself? 224 When Might This Problem Occur? 224 Potential Solutions 224 End-to-End Testing 226 How Can This Problem Show Itself? 226 When Might This Problem Occur? 226 Potential Solutions 227 Global Versus Local Optimization 229 How Can This Problem Show Itself? 229 When Might This Problem Occur? 229 Potential Solutions 230 Robustness and Resiliency 232 How Can This Problem Show Itself? 232 When Might This Problem Occur? 232 Potential Solutions 232 Orphaned Services 233 How Can This Problem Show Itself? 233 When Might This Problem Occur? 234 Potential Solutions 234 Summary 236 6. Closing Words. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 A. Bibliography. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 B. Pattern Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 viii | Table of Contents

📄 Page 10

Foreword We are on the cusp of a revolution in application architecture. According to IDC, by 2022, 90% of all new applications will be based on microservices architectures. As with DevOps, adopting microservices improves agility and flexibility, enabling enter‐ prises to bring their products and services to market faster. Many enterprises are thinking about migrating their existing apps to microservices- based architectures. But there are many critical questions to ask first. How do you re- architect an existing system without having to stop all other work on it? How big should a microservice be? When might microservices be a bad idea? What are some of the migration patterns you could adopt when splitting up a monolith? Monolith to Microservices answers those questions and more. It’s a must-have for application developers and architects, as well as DevOps engineers. Not only does this book provide details about the implementation of microservices, it also sheds light on the challenges associated with a microservices architecture, and helps you understand if starting this journey is even right for you. Sam Newman, the author of the O’Reilly bestseller Building Microservices, brings a wealth of experience to guide your path from monolith to microservices in a holistic fashion. He provides in-depth guidance in an incremental and logical manner, from planning to implementation and beyond—how best to manage the growing pains as you scale your microservices. NGINX can help enable you to achieve enterprise-grade traffic management for your microservices-based applications. NGINX is widely used to manage traffic among microservices, with more than 250 users running more than 3 million NGINX instances in production microservices environments. NGINX is also the most popu‐ lar Ingress controller for Kubernetes (a container orchestration platform used to deploy and operate microservices), running in 64% of all Kubernetes environments. Additionally, NGINX supports OpenShift, a container application platform from Red Hat. ix

📄 Page 11

As you embrace microservices, NGINX enables you to achieve enterprise-grade traf‐ fic management for your microservices-based applications. We sincerely hope you enjoy this book as you migrate your monolithic applications to distributed applica‐ tions based on microservices. — Karthik Krishnaswamy Director, Product Marketing, NGINX October 2019 x | Foreword

📄 Page 12

Preface A few years ago, some of us were chatting about microservices being an interesting idea. The next thing you know it’s become the default architecture for hundreds of companies around the world (many probably launched as startups aiming to solve the problems microservices cause), and has everyone running to jump on a band‐ wagon that they are worried is about to disappear over the horizon. I must admit, I’m partly to blame. Since I wrote my own book on this subject, Build‐ ing Microservices, back in 2015, I’ve made a living working with people to help them understand this type of architecture. What I’ve always tried to do is to cut through the hype, and help companies decide if microservices are right for them. For many of my clients with existing (non-microservice-oriented) systems, the challenge has been about how to adopt microservice architectures. How do you take an existing system and rearchitect it without having to stop all other work? That is where this book comes in. As importantly, I’ll aim to give you an honest appraisal of the challenges associated with microservice architecture, and help you understand whether starting this journey is even right for you. What You Will Learn This book is designed as a deep dive into how you think about, and execute, breaking apart existing systems into a microservice architecture. We will touch on many topics related to microservice architecture, but the focus is on the decomposition side of things. For a more general guide to microservice architectures, my previous book Building Microservices would be a good place to start. In fact, I strongly recommend that you consider that book to be a companion to this one. Chapter 1 contains an overview of what microservices are, and explores further the ideas that led us to these sorts of architectures. It should work well for people who are new to microservices, but I also strongly urge those of you with more experience to not skip this chapter. I feel that in the flurry of technology, some of the important xi

📄 Page 13

core ideas of microservices often get missed: these are concepts that the book will return to again and again. Understanding more about microservices is good, but knowing if they are right for you is something else. In Chapter 2, I walk you through how to go about assessing whether or not microservices are right for you, and also give you some really impor‐ tant guidelines for how to manage a transition from a monolith to a microservice architecture. Here we’ll touch on everything from domain-driven design to organiza‐ tional change models—vital underpinnings that will stand you in good stead even if you decide not to adopt a microservice architecture. In Chapters 3 and 4 we dive deeper into the technical aspects associated with decom‐ posing a monolith, exploring real-world examples and extracting out migration pat‐ terns. Chapter 3 focuses on application decomposition aspects, where Chapter 4 is a deep dive on data issues. If you really want to move from a monolithic system to a microservice architecture, we’ll need to pull some databases apart! Finally, Chapter 5 looks at the sorts of challenges you will face as your microservice architecture grows. These systems can offer huge benefits, but they come with a lot of complexity and problems you won’t have had to face before. This chapter is my attempt at helping you spot these problems as they start to occur, and at offering ways to deal with the growing pains associated with microservices. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. xii | Preface

📄 Page 14

This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. Using Code Examples This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require per‐ mission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Monolith to Microservi‐ ces by Sam Newman (O’Reilly). Copyright 2020 Sam Newman, 978-1-492-07554-7.” O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, conferences, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in- depth learning paths, interactive coding environments, and a vast collection of text Preface | xiii

📄 Page 15

and video from O’Reilly and 200+ other publishers. For more information, please visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/monolith-to-microservices. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For more information about our books, courses, conferences, and news, see our web‐ site at http://www.oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Acknowledgments Without the help and understanding of my wonderful wife, Lindy Stephens, this book would not have been possible. This book is for her. Lindy, sorry for being so grouchy when various deadlines came and went. I’d also like to thank the lovely Gillman Staynes clan for all their support—I’m lucky to have such a great family. This book benefited hugely from people who kindly volunteered their time and energy to read various drafts and provide valuable insights. I especially want to call out Chris O’Dell, Daniel Bryant, Pete Hodgson, Martin Fowler, Stefan Schrass, and Derek Hammer for their efforts in this area. There were also people who directly con‐ tributed in numerous ways, so I’d also like to thank Graham Tackley, Erik Doernen‐ berg, Marcin Zasepa, Michael Feathers, Randy Shoup, Kief Morris, Peter Gillard- Moss, Matt Heath, Steve Freeman, Rene Lengwinat, Sarah Wells, Rhys Evans, and Berke Sokhan. If you find errors in this book, the mistakes are mine, not theirs. xiv | Preface

📄 Page 16

The team at O’Reilly has also been incredibly supportive, and I would like to high‐ light my editors Eleanor Bru and Alicia Young, in addition to Christopher Guzikow‐ ski, Mary Treseler, and Rachel Roumeliotis. I also want to say a big thanks to Helen Codling and her colleagues elsewhere in the world for continuing to drag my books to various conferences, Susan Conant for keeping me sane while navigating the changing world of publishing, and Mike Loukides for initially getting me involved with O’Reilly. I know there are many more people behind the scenes who have hel‐ ped, so thanks as well to you all. Beyond those who directly contributed to this book, I also want to call out others who, whether they realized it or not, helped this book come about. So I would like to thank (in no particular order) Martin Kelppmann, Ben Stopford, Charity Majors, Alistair Cockburn, Gregor Hohpe, Bobby Woolf, Eric Evans, Larry Constantine, Leslie Lamport, Edward Yourdon, David Parnas, Mike Bland, David Woods, John All‐ spaw, Alberto Brandolini, Frederick Brooks, Cindy Sridharan, Dave Farley, Jez Hum‐ ble, Gene Kim, James Lewis, Nicole Forsgren, Hector Garcia-Molina, Sheep & Cheese, Kenneth Salem, Adrian Colyer, Pat Helland, Kresten Thorup, Henrik Kni‐ berg, Anders Ivarsson, Manuel Pais, Steve Smith, Bernd Rucker, Matthew Skelton, Alexis Richardson, James Governor, and Kane Stephens. As is always the case with these things, it seems highly likely that I’ve missed someone who has materially contributed to this book. To those people, all I can say is I’m very sorry for forgetting to thank you by name, and that I hope you can forgive me. Finally, some people ask me from time to time about the tools I used to write this book. I wrote in AsciiDoc using Visual Studio Code along with João Pinto’s AsciiDoc plug-in. The book was source controlled in Git, using O’Reilly’s Atlas system. I wrote mostly on my laptop, using an external Razer mechanical keyboard, but toward the end also made heavy use of an iPad Pro running Working Copy to finish off the last few things. This enabled me to write while traveling, allowing me on one memorable occasion to write about database refactoring on a ferry to the Orkneys. The resulting seasickness was totally worth it. Preface | xv

📄 Page 17

(This page has no text content)

📄 Page 18

CHAPTER 1 Just Enough Microservices Well, that escalated quickly, really got out of hand fast! —Ron Burgundy, Anchorman Before we dive into how to work with microservices, it is important that we have a common, shared understanding about what microservice architectures are. I’d like to address some common misconceptions I see on a regular basis, as well as nuances that are often missed. You’ll need this firm foundation of knowledge to get the most out of what follows in the rest of the book. As such, this chapter will provide an explanation of microservice architectures, look briefly at how microservices devel‐ oped (which means, naturally, taking a look at monoliths), and examine some of the advantages and challenges of working with microservices. What Are Microservices? Microservices are independently deployable services modeled around a business domain. They communicate with each other via networks, and as an architecture choice offer many options for solving the problems you may face. It follows that a microservice architecture is based on multiple collaborating microservices. They are a type of service-oriented architecture (SOA), albeit one that is opinionated about how service boundaries should be drawn, and that independent deployability is key. Microservices also have the advantage of being technology agnostic. From a technology viewpoint, microservices expose the business capabilities that they encapsulate via one or more network endpoints. Microservices communicate with each other via these networks—making them a form of distributed system. They also encapsulate data storage and retrieval, exposing data, via well-defined interfaces. So databases are hidden inside the service boundary. 1

📄 Page 19

There is a lot to unpack in all of that, so let’s dig a bit deeper into some of these ideas. Independent Deployability Independent deployability is the idea that we can make a change to a microservice and deploy it into a production environment without having to utilize any other services. More importantly, it’s not just that we can do this; it’s that this is actually how you manage deployments in your system. It’s a discipline you practice for the bulk of your releases. This is a simple idea that is nonetheless complex in execution. If there is only one thing you take out of this book, it should be this: ensure you embrace the concept of independent deployability of your microservices. Get into the habit of releasing changes to a single microservice into production without having to deploy any‐ thing else. From this, many good things will follow. To guarantee independent deployability, we need to ensure our services are loosely coupled—in other words, we need to be able to change one service without having to change anything else. This means we need explicit, well-defined, and stable contracts between services. Some implementation choices make this difficult—the sharing of databases, for example, is especially problematic. The desire for loosely coupled serv‐ ices with stable interfaces guides our thinking about how we find service boundaries in the first place. Modeled Around a Business Domain Making a change across a process boundary is expensive. If you need to make a change to two services to roll out a feature, and orchestrate the deployment of these two changes, that takes more work than making the same change inside a single ser‐ vice (or, for that matter, a monolith). It therefore follows that we want to find ways of ensuring we make cross-service changes as infrequently as possible. Following the same approach I used in Building Microservices, this book uses a fake domain and company to illustrate certain concepts when it isn’t possible to share real- world stories. The company in question is Music Corp, a large multi-national organi‐ zation that somehow remains in business, despite it focusing almost entirely on selling CDs. We’ve decided to move Music Corp kicking and screaming into the 21st century, and as part of that we’re assessing the existing system architecture. In Figure 1-1, we see a simple three-tiered architecture. We have a web-based user interface, a business logic layer in the form of a monolithic backend, and data storage in a traditional database. These layers, as is common, are owned by different teams. 2 | Chapter 1: Just Enough Microservices

📄 Page 20

Figure 1-1. Music Corp’s systems as a traditional three-tiered architecture We want to make a simple change to our functionality: we want to allow our custom‐ ers to specify their favorite genre of music. This change requires us to change the user interface to show the genre choice UI, the backend service to allow for the genre to be surfaced to the UI and for the value to be changed, and the database to accept this change. These changes will need to be managed by each team, as outlined in Figure 1-2, and those changes will need to be deployed in the correct order. Figure 1-2. Making a change across all three tiers is more involved What Are Microservices? | 3