Designing Machine Learning Systems An Iterative Process for Production-Ready (Chip Huyen) (Z-Library)

Chip Huyen Designing Machine Learning Systems An Iterative Process for Production-Ready Applications

MACHINE LE ARNING “This is, simply, the very best book you can read about how to build, deploy, and scale machine learning models at a company for maximum impact.” —Josh Wills Software Engineer at WeaveGrid and former Director of Data Engineering, Slack “In a blooming but chaotic ecosystem, this principled view on end-to-end ML is both your map and your compass: a must-read for practitioners inside and outside of Big Tech.” —Jacopo Tagliabue Director of AI, Coveo Designing Machine Learning Systems US $59.99 CAN $74.99 ISBN: 978-1-098-10796-3 Twitter: @oreillymedia linkedin.com/company/oreilly-media youtube.com/oreillymedia Machine learning systems are both complex and unique. Complex because they consist of many different components and involve many different stakeholders. Unique because they’re data dependent, with data varying wildly from one use case to the next. In this book, you’ll learn a holistic approach to designing ML systems that are reliable, scalable, maintainable, and adaptive to changing environments and business requirements. Author Chip Huyen, co-founder of Claypot AI, considers each design decision—such as how to process and create training data, which features to use, how often to retrain models, and what to monitor—in the context of how it can help your system as a whole achieve its objectives. The iterative framework in this book uses actual case studies backed by ample references. This book will help you tackle scenarios such as: • Engineering data and choosing the right metrics to solve a business problem • Automating the process for continually developing, evaluating, deploying, and updating models • Developing a monitoring system to quickly detect and address issues your models might encounter in production • Architecting an ML platform that serves across use cases • Developing responsible ML systems Chip Huyen is co-founder of Claypot AI, a platform for real-time machine learning. Through her work at NVIDIA, Netflix, and Snorkel AI, she’s helped some of the world’s largest organizations develop and deploy ML systems. Chip based this book on her lectures for CS 329S: Machine Learning Systems Design, a course she teaches at Stanford University.

Praise for Designing Machine Learning Systems There is so much information one needs to know to be an effective machine learning engineer. It’s hard to cut through the chaff to get the most relevant information, but Chip has done that admirably with this book. If you are serious about ML in production, and care about how to design and implement ML systems end to end, this book is essential. —Laurence Moroney, AI and ML Lead, Google One of the best resources that focuses on the first principles behind designing ML systems for production. A must-read to navigate the ephemeral landscape of tooling and platform options. —Goku Mohandas, Founder of Made With ML Chip’s manual is the book we deserve and the one we need right now. In a blooming but chaotic ecosystem, this principled view on end-to-end ML is both your map and your compass: a must-read for practitioners inside and outside of Big Tech—especially those working at “reasonable scale.” This book will also appeal to data leaders looking for best practices on how to deploy, manage, and monitor systems in the wild. —Jacopo Tagliabue, Director of AI, Coveo; Adj. Professor of MLSys, NYU This is, simply, the very best book you can read about how to build, deploy, and scale machine learning models at a company for maximum impact. Chip is a masterful teacher, and the breadth and depth of her knowledge is unparalleled. —Josh Wills, Software Engineer at WeaveGrid and former Director of Data Engineering, Slack This is the book I wish I had read when I started as an ML engineer. —Shreya Shankar, MLOps PhD Student

Designing Machine Learning Systems is a welcome addition to the field of applied machine learning. The book provides a detailed guide for people building end-to-end machine learning systems. Chip Huyen writes from her extensive, hands-on experience building real-world machine learning applications. —Brian Spiering, Data Science Instructor at Metis Chip is truly a world-class expert on machine learning systems, as well as a brilliant writer. Both are evident in this book, which is a fantastic resource for anyone looking to learn about this topic. —Andrey Kurenkov, PhD Candidate at the Stanford AI Lab Chip Huyen has produced an important addition to the canon of machine learning literature—one that is deeply literate in ML fundamentals, but has a much more concrete and practical approach than most. The focus on business requirements alone is uncommon and valuable. This book will resonate with engineers getting started with ML and with others in any part of the organization trying to understand how ML works. —Todd Underwood, Senior Engineering Director for ML SRE, Google, and Coauthor of Reliable Machine Learning

978-1-098-10796-3 [LSI] Designing Machine Learning Systems by Chip Huyen Copyright © 2022 Huyen Thi Khanh Nguyen. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Nicole Butterfield Development Editor: Jill Leonard Production Editor: Gregory Hyman Copyeditor: nSight, Inc. Proofreader: Piper Editorial Consulting, LLC Indexer: nSight, Inc. Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea May 2022: First Edition Revision History for the First Edition 2022-05-17: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781098107963 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Designing Machine Learning Systems, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix 1. Overview of Machine Learning Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 When to Use Machine Learning 3 Machine Learning Use Cases 9 Understanding Machine Learning Systems 12 Machine Learning in Research Versus in Production 12 Machine Learning Systems Versus Traditional Software 22 Summary 23 2. Introduction to Machine Learning Systems Design. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Business and ML Objectives 26 Requirements for ML Systems 29 Reliability 29 Scalability 30 Maintainability 31 Adaptability 31 Iterative Process 32 Framing ML Problems 35 Types of ML Tasks 36 Objective Functions 40 Mind Versus Data 43 Summary 46 3. Data Engineering Fundamentals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Data Sources 50 Data Formats 53 JSON 54 iii

Row-Major Versus Column-Major Format 54 Text Versus Binary Format 57 Data Models 58 Relational Model 59 NoSQL 63 Structured Versus Unstructured Data 66 Data Storage Engines and Processing 67 Transactional and Analytical Processing 67 ETL: Extract, Transform, and Load 70 Modes of Dataflow 72 Data Passing Through Databases 72 Data Passing Through Services 73 Data Passing Through Real-Time Transport 74 Batch Processing Versus Stream Processing 78 Summary 79 4. Training Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Sampling 82 Nonprobability Sampling 83 Simple Random Sampling 84 Stratified Sampling 84 Weighted Sampling 85 Reservoir Sampling 86 Importance Sampling 87 Labeling 88 Hand Labels 88 Natural Labels 91 Handling the Lack of Labels 94 Class Imbalance 102 Challenges of Class Imbalance 103 Handling Class Imbalance 105 Data Augmentation 113 Simple Label-Preserving Transformations 114 Perturbation 114 Data Synthesis 116 Summary 118 5. Feature Engineering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Learned Features Versus Engineered Features 120 Common Feature Engineering Operations 123 Handling Missing Values 123 Scaling 126 iv | Table of Contents

Discretization 128 Encoding Categorical Features 129 Feature Crossing 132 Discrete and Continuous Positional Embeddings 133 Data Leakage 135 Common Causes for Data Leakage 137 Detecting Data Leakage 140 Engineering Good Features 141 Feature Importance 142 Feature Generalization 144 Summary 146 6. Model Development and Offline Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 Model Development and Training 150 Evaluating ML Models 150 Ensembles 156 Experiment Tracking and Versioning 162 Distributed Training 168 AutoML 172 Model Offline Evaluation 178 Baselines 179 Evaluation Methods 181 Summary 188 7. Model Deployment and Prediction Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 Machine Learning Deployment Myths 194 Myth 1: You Only Deploy One or Two ML Models at a Time 194 Myth 2: If We Don’t Do Anything, Model Performance Remains the Same 195 Myth 3: You Won’t Need to Update Your Models as Much 196 Myth 4: Most ML Engineers Don’t Need to Worry About Scale 196 Batch Prediction Versus Online Prediction 197 From Batch Prediction to Online Prediction 201 Unifying Batch Pipeline and Streaming Pipeline 203 Model Compression 206 Low-Rank Factorization 206 Knowledge Distillation 208 Pruning 208 Quantization 209 ML on the Cloud and on the Edge 212 Compiling and Optimizing Models for Edge Devices 214 ML in Browsers 222 Summary 223 Table of Contents | v

8. Data Distribution Shifts and Monitoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 Causes of ML System Failures 226 Software System Failures 227 ML-Specific Failures 229 Data Distribution Shifts 237 Types of Data Distribution Shifts 237 General Data Distribution Shifts 241 Detecting Data Distribution Shifts 242 Addressing Data Distribution Shifts 248 Monitoring and Observability 250 ML-Specific Metrics 251 Monitoring Toolbox 256 Observability 259 Summary 261 9. Continual Learning and Test in Production. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 Continual Learning 264 Stateless Retraining Versus Stateful Training 265 Why Continual Learning? 268 Continual Learning Challenges 270 Four Stages of Continual Learning 274 How Often to Update Your Models 279 Test in Production 281 Shadow Deployment 282 A/B Testing 283 Canary Release 285 Interleaving Experiments 285 Bandits 287 Summary 291 10. Infrastructure and Tooling for MLOps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 Storage and Compute 297 Public Cloud Versus Private Data Centers 300 Development Environment 302 Dev Environment Setup 303 Standardizing Dev Environments 306 From Dev to Prod: Containers 308 Resource Management 311 Cron, Schedulers, and Orchestrators 311 Data Science Workflow Management 314 ML Platform 319 Model Deployment 320 vi | Table of Contents

Model Store 321 Feature Store 325 Build Versus Buy 327 Summary 329 11. The Human Side of Machine Learning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 User Experience 331 Ensuring User Experience Consistency 332 Combatting “Mostly Correct” Predictions 332 Smooth Failing 334 Team Structure 334 Cross-functional Teams Collaboration 335 End-to-End Data Scientists 335 Responsible AI 339 Irresponsible AI: Case Studies 341 A Framework for Responsible AI 347 Summary 353 Epilogue. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 Table of Contents | vii

Preface Ever since the first machine learning course I taught at Stanford in 2017, many people have asked me for advice on how to deploy ML models at their organizations. These questions can be generic, such as “What model should I use?” “How often should I retrain my model?” “How can I detect data distribution shifts?” “How do I ensure that the features used during training are consistent with the features used during inference?” These questions can also be specific, such as “I’m convinced that switching from batch prediction to online prediction will give our model a performance boost, but how do I convince my manager to let me do so?” or “I’m the most senior data scientist at my company and I’ve recently been tasked with setting up our first machine learning platform; where do I start?” My short answer to all these questions is always: “It depends.” My long answers often involve hours of discussion to understand where the questioner comes from, what they’re actually trying to achieve, and the pros and cons of different approaches for their specific use case. ML systems are both complex and unique. They are complex because they consist of many different components (ML algorithms, data, business logic, evaluation metrics, underlying infrastructure, etc.) and involve many different stakeholders (data scien‐ tists, ML engineers, business leaders, users, even society at large). ML systems are unique because they are data dependent, and data varies wildly from one use case to the next. For example, two companies might be in the same domain (ecommerce) and have the same problem that they want ML to solve (recommender system), but their resulting ML systems can have different model architecture, use different sets of features, be evaluated on different metrics, and bring different returns on investment. ix

Many blog posts and tutorials on ML production focus on answering one specific question. While the focus helps get the point across, they can create the impression that it’s possible to consider each of these questions in isolation. In reality, changes in one component will likely affect other components. Therefore, it’s necessary to consider the system as a whole while attempting to make any design decision. This book takes a holistic approach to ML systems. It takes into account different components of the system and the objectives of different stakeholders involved. The content in this book is illustrated using actual case studies, many of which I’ve personally worked on, backed by ample references, and reviewed by ML practitioners in both academia and industry. Sections that require in-depth knowledge of a certain topic—e.g., batch processing versus stream processing, infrastructure for storage and compute, and responsible AI—are further reviewed by experts whose work focuses on that one topic. In other words, this book is an attempt to give nuanced answers to the aforementioned questions and more. When I first wrote the lecture notes that laid the foundation for this book, I thought I wrote them for my students to prepare them for the demands of their future jobs as data scientists and ML engineers. However, I soon realized that I also learned tremendously through the process. The initial drafts I shared with early readers sparked many conversations that tested my assumptions, forced me to consider different perspectives, and introduced me to new problems and new approaches. I hope that this learning process will continue for me now that the book is in your hand, as you have experiences and perspectives that are unique to you. Please feel free to share with me any feedback you might have for this book, via the MLOps Discord server that I run (where you can also find other readers of this book), Twitter, LinkedIn, or other channels that you can find on my website. Who This Book Is For This book is for anyone who wants to leverage ML to solve real-world problems. ML in this book refers to both deep learning and classical algorithms, with a leaning toward ML systems at scale, such as those seen at medium to large enterprises and fast-growing startups. Systems at a smaller scale tend to be less complex and might benefit less from the comprehensive approach laid out in this book. Because my background is engineering, the language of this book is geared toward engineers, including ML engineers, data scientists, data engineers, ML platform engi‐ neers, and engineering managers. You might be able to relate to one of the following scenarios: x | Preface

• You have been given a business problem and a lot of raw data. You want to engineer this data and choose the right metrics to solve this problem. • Your initial models perform well in offline experiments and you want to deploy them. • You have little feedback on how your models are performing after your models are deployed, and you want to figure out a way to quickly detect, debug, and address any issue your models might run into in production. • The process of developing, evaluating, deploying, and updating models for your team has been mostly manual, slow, and error-prone. You want to automate and improve this process. • Each ML use case in your organization has been deployed using its own work‐ flow, and you want to lay down the foundation (e.g., model store, feature store, monitoring tools) that can be shared and reused across use cases. • You’re worried that there might be biases in your ML systems and you want to make your systems responsible! You can also benefit from the book if you belong to one of the following groups: • Tool developers who want to identify underserved areas in ML production and figure out how to position your tools in the ecosystem. • Individuals looking for ML-related roles in the industry. • Technical and business leaders who are considering adopting ML solutions to improve your products and/or business processes. Readers without strong tech‐ nical backgrounds might benefit the most from Chapters 1, 2, and 11. What This Book Is Not This book is not an introduction to ML. There are many books, courses, and resources available for ML theories, and therefore, this book shies away from these concepts to focus on the practical aspects of ML. To be specific, the book assumes that readers have a basic understanding of the following topics: • ML models such as clustering, logistic regression, decision trees, collaborative fil‐ tering, and various neural network architectures including feed-forward, recur‐ rent, convolutional, and transformer • ML techniques such as supervised versus unsupervised, gradient descent, objec‐ tive/loss function, regularization, generalization, and hyperparameter tuning • Metrics such as accuracy, F1, precision, recall, ROC, mean squared error, and log-likelihood Preface | xi

• Statistical concepts such as variance, probability, and normal/long-tail distribution • Common ML tasks such as language modeling, anomaly detection, object classifi‐ cation, and machine translation You don’t have to know these topics inside out—for concepts whose exact definitions can take some effort to remember, e.g., F1 score, we include short notes as refer‐ ences—but you should have a rough sense of what they mean going in. While this book mentions current tools to illustrate certain concepts and solutions, it’s not a tutorial book. Technologies evolve over time. Tools go in and out of style quickly, but fundamental approaches to problem solving should last a bit longer. This book provides a framework for you to evaluate the tool that works best for your use cases. When there’s a tool you want to use, it’s usually straightforward to find tutorials for it online. As a result, this book has few code snippets and instead focuses on providing a lot of discussion around trade-offs, pros and cons, and concrete examples. Navigating This Book The chapters in this book are organized to reflect the problems data scientists might encounter as they progress through the lifecycle of an ML project. The first two chapters lay down the groundwork to set an ML project up for success, starting from the most basic question: does your project need ML? It also covers choosing the objectives for your project and how to frame your problem in a way that makes for simpler solutions. If you’re already familiar with these considerations and impatient to get to the technical solutions, feel free to skip the first two chapters. Chapters 4 to 6 cover the pre-deployment phase of an ML project: from creating the training data and engineering features to developing and evaluating your models in a development environment. This is the phase where expertise in both ML and the problem domain are especially needed. Chapters 7 to 9 cover the deployment and post-deployment phase of an ML project. We’ll learn through a story many readers might be able to relate to that having a model deployed isn’t the end of the deployment process. The deployed model will need to be monitored and continually updated to changing environments and business requirements. Chapters 3 and 10 focus on the infrastructure needed to enable stakeholders from different backgrounds to work together to deliver successful ML systems. Chapter 3 focuses on data systems, whereas Chapter 10 focuses on compute infrastructure and ML platforms. I debated for a long time on how deep to go into data systems and where to introduce it in the book. Data systems, including databases, data formats, xii | Preface

data movements, and data processing engines, tend to be sparsely covered in ML coursework, and therefore many data scientists might think of them as low level or irrelevant. After consulting with many of my colleagues, I decided that because ML systems depend on data, covering the basics of data systems early will help us get on the same page to discuss data matters in the rest of the book. While we cover many technical aspects of an ML system in this book, ML systems are built by people, for people, and can have outsized impact on the life of many. It’d be remiss to write a book on ML production without a chapter on the human side of it, which is the focus of Chapter 11, the last chapter. Note that “data scientist” is a role that has evolved a lot in the last few years, and there have been many discussions to determine what this role should entail—we’ll go into some of these discussions in Chapter 10. In this book, we use “data scientist” as an umbrella term to include anyone who works developing and deploying ML models, including people whose job titles might be ML engineers, data engineers, data analysts, etc. GitHub Repository and Community This book is accompanied by a GitHub repository that contains: • A review of basic ML concepts • A list of references used in this book and other advanced, updated resources • Code snippets used in this book • A list of tools you can use for certain problems you might encounter in your workflows I also run a Discord server on MLOps where you’re encouraged to discuss and ask questions about the book. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Preface | xiii

This element signifies a general note. This element indicates a warning or caution. Using Code Examples As mentioned, supplemental material (code examples, exercises, etc.) is available for download at https://oreil.ly/designing-machine-learning-systems-code. If you have a technical question or a problem using the code examples, please send email to bookquestions@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Designing Machine Learning Systems by Chip Huyen (O’Reilly). Copyright 2022 Huyen Thi Khanh Nguyen, 978-1-098-10796-3.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. xiv | Preface

Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/designing-machine-learning- systems. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly-media Follow us on Twitter: https://twitter.com/oreillymedia Watch us on YouTube: https://youtube.com/oreillymedia Acknowledgments This book took two years to write, and many more years beforehand to prepare. Looking back, I’m equally amazed and grateful for the enormous amount of help I received in writing this book. I tried my best to include the names of everyone who has helped me here, but due to the inherent faultiness of human memory, I undoubtedly neglected to mention many. If I forgot to include your name, please know that it wasn’t because I don’t appreciate your contribution and please kindly remind me so that I can rectify as soon as possible! First and foremost, I’d like to thank the course staff who helped me develop the course and materials this book was based on: Michael Cooper, Xi Yin, Chloe He, Kinbert Chou, Megan Leszczynski, Karan Goel, and Michele Catasta. I’d like to Preface | xv

thank my professors, Christopher Ré and Mehran Sahami, without whom the course wouldn’t exist in the first place. I’d like to thank a long list of reviewers who not only gave encouragement but also improved the book by many orders of magnitude: Eugene Yan, Josh Wills, Han-chung Lee, Thomas Dietterich, Irene Tematelewo, Goku Mohandas, Jacopo Tagliabue, Andrey Kurenkov, Zach Nussbaum, Jay Chia, Laurens Geffert, Brian Spier‐ ing, Erin Ledell, Rosanne Liu, Chin Ling, Shreya Shankar, and Sara Hooker. I’d like to thank all the readers who read the early release version of the book and gave me ideas on how to improve the book, including Charles Frye, Xintong Yu, Jordan Zhang, Jonathon Belotti, and Cynthia Yu. Of course, the book wouldn’t have been possible with the team at O’Reilly, especially my development editor, Jill Leonard, and my production editors, Kristen Brown, Sharon Tripp, and Gregory Hyman. I’d like to thank Laurence Moroney, Hannes Hapke, and Rebecca Novack, who helped me get this book from an idea to a proposal. This book, after all, is an accumulation of invaluable lessons I learned throughout my career to date. I owe these lessons to my extremely competent and patient coworkers and former coworkers at Claypot AI, Primer AI, Netflix, NVIDIA, and Snorkel AI. Every person I’ve worked with has taught me something new about bringing ML into the world. A special thanks to my cofounder Zhenzhong Xu for putting out the fires at our startup and allowing me to spend time on this book. Thank you, Luke, for always being so supportive of everything that I want to do, no matter how ambitious it is. xvi | Preface

1 Mike Schuster, Melvin Johnson, and Nikhil Thorat, “Zero-Shot Translation with Google’s Multilingual Neural Machine Translation System,” Google AI Blog, November 22, 2016, https://oreil.ly/2R1CB. 2 Larry Hardesty, “A Method to Image Black Holes,” MIT News, June 6, 2016, https://oreil.ly/HpL2F. CHAPTER 1 Overview of Machine Learning Systems In November 2016, Google announced that it had incorporated its multilingual neural machine translation system into Google Translate, marking one of the first success stories of deep artificial neural networks in production at scale.1 According to Google, with this update, the quality of translation improved more in a single leap than they had seen in the previous 10 years combined. This success of deep learning renewed the interest in machine learning (ML) at large. Since then, more and more companies have turned toward ML for solutions to their most challenging problems. In just five years, ML has found its way into almost every aspect of our lives: how we access information, how we communicate, how we work, how we find love. The spread of ML has been so rapid that it’s already hard to imagine life without it. Yet there are still many more use cases for ML waiting to be explored in fields such as health care, transportation, farming, and even in helping us understand the universe.2 Many people, when they hear “machine learning system,” think of just the ML algo‐ rithms being used such as logistic regression or different types of neural networks. However, the algorithm is only a small part of an ML system in production. The system also includes the business requirements that gave birth to the ML project in the first place, the interface where users and developers interact with your system, the data stack, and the logic for developing, monitoring, and updating your models, as well as the infrastructure that enables the delivery of that logic. Figure 1-1 shows you the different components of an ML system and in which chapters of this book they will be covered. 1