Kubeflow for Machine Learning From Lab to Production (Trevor Grant, Holden Karau, Boris Lublinsky etc.) (z-library.sk, 1lib.sk, z-lib.sk)

Trevor Grant, Holden Karau, Boris Lublinsky, Richard Liu & Ilan Filonenko Foreword by Chris Albon Kubeflow for Machine Learning From Lab to Production

Trevor Grant, Holden Karau, Boris Lublinsky, Richard Liu, and Ilan Filonenko Kubeflow for Machine Learning From Lab to Production Boston Farnham Sebastopol TokyoBeijing

978-1-492-05012-4 [LSI] Kubeflow for Machine Learning by Trevor Grant, Holden Karau, Boris Lublinsky, Richard Liu, and Ilan Filonenko Copyright © 2021 Trevor Grant, Holden Karau, Boris Lublinsky, Richard Liu, and Ilan Filonenko. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Jonathan Hassell Development Editor: Amelia Blevins Production Editor: Deborah Baker Copyeditor: JM Olejarz Proofreader: Justin Billing Indexer: Sue Klefstad Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea November 2020: First Edition Revision History for the First Edition 2020-10-12: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492050124 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Kubeflow for Machine Learning, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

Table of Contents Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi 1. Kubeflow: What It Is and Who It Is For. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Model Development Life Cycle 1 Where Does Kubeflow Fit In? 2 Why Containerize? 2 Why Kubernetes? 3 Kubeflow’s Design and Core Components 4 Data Exploration with Notebooks 4 Data/Feature Preparation 5 Training 6 Hyperparameter Tuning 6 Model Validation 6 Inference/Prediction 7 Pipelines 7 Component Overview 8 Alternatives to Kubeflow 9 Clipper (RiseLabs) 9 MLflow (Databricks) 10 Others 10 Introducing Our Case Studies 10 Modified National Institute of Standards and Technology 11 Mailing List Data 11 Product Recommender 11 CT Scans 12 Conclusion 12 iii

2. Hello Kubeflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Getting Set Up with Kubeflow 13 Installing Kubeflow and Its Dependencies 14 Setting Up Local Kubernetes 15 Setting Up Your Kubeflow Development Environment 16 Creating Our First Kubeflow Project 18 Training and Deploying a Model 19 Training and Monitoring Progress 20 Test Query 21 Going Beyond a Local Deployment 23 Conclusion 24 3. Kubeflow Design: Beyond the Basics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Getting Around the Central Dashboard 26 Notebooks (JupyterHub) 27 Training Operators 28 Kubeflow Pipelines 28 Hyperparameter Tuning 30 Model Inference 31 Metadata 32 Component Summary 33 Support Components 33 MinIO 34 Istio 36 Knative 38 Apache Spark 40 Kubeflow Multiuser Isolation 40 Conclusion 42 4. Kubeflow Pipelines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Getting Started with Pipelines 44 Exploring the Prepackaged Sample Pipelines 44 Building a Simple Pipeline in Python 46 Storing Data Between Steps 52 Introduction to Kubeflow Pipelines Components 53 Argo: the Foundation of Pipelines 54 What Kubeflow Pipelines Adds to Argo Workflow 58 Building a Pipeline Using Existing Images 58 Kubeflow Pipeline Components 61 Advanced Topics in Pipelines 62 Conditional Execution of Pipeline Stages 63 Running Pipelines on Schedule 65 iv | Table of Contents

Conclusion 66 5. Data and Feature Preparation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Deciding on the Correct Tooling 68 Local Data and Feature Preparation 68 Fetching the Data 69 Data Cleaning: Filtering Out the Junk 70 Formatting the Data 71 Feature Preparation 71 Custom Containers 72 Distributed Tooling 73 TensorFlow Extended 73 Distributed Data Using Apache Spark 78 Distributed Feature Preparation Using Apache Spark 87 Putting It Together in a Pipeline 88 Using an Entire Notebook as a Data Preparation Pipeline Stage 89 Conclusion 90 6. Artifact and Metadata Store. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 Kubeflow ML Metadata 92 Programmatic Query 94 Kubeflow Metadata UI 96 Using MLflow’s Metadata Tools with Kubeflow 98 Creating and Deploying an MLflow Tracking Server 99 Logging Data on Runs 101 Using the MLflow UI 104 Conclusion 106 7. Training a Machine Learning Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Building a Recommender with TensorFlow 108 Getting Started 109 Starting a New Notebook Session 110 TensorFlow Training 110 Deploying a TensorFlow Training Job 113 Distributed Training 117 Using GPUs 121 Using Other Frameworks for Distributed Training 122 Training a Model Using Scikit-Learn 122 Starting a New Notebook Session 123 Data Preparation 124 Scikit-Learn Training 126 Explaining the Model 127 Table of Contents | v

Exporting Model 129 Integration into Pipelines 129 Conclusion 129 8. Model Inference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Model Serving 132 Model Serving Requirements 133 Model Monitoring 134 Model Accuracy, Drift, and Explainability 134 Model Monitoring Requirements 135 Model Updating 135 Model Updating Requirements 136 Summary of Inference Requirements 137 Model Inference in Kubeflow 137 TensorFlow Serving 138 Review 141 Seldon Core 142 Designing a Seldon Inference Graph 143 Testing Your Model 148 Serving Requests 150 Monitoring Your Models 151 Review 158 KFServing 159 Serverless and the Service Plane 159 Data Plane 160 Example Walkthrough 162 Peeling Back the Underlying Infrastructure 168 Review 175 Conclusion 176 9. Case Study Using Multiple Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 The Denoising CT Scans Example 180 Data Prep with Python 181 DS-SVD with Apache Spark 182 Visualization 183 The CT Scan Denoising Pipeline 186 Sharing the Pipeline 191 Conclusion 191 10. Hyperparameter Tuning and Automated Machine Learning. . . . . . . . . . . . . . . . . . . . . 193 AutoML: An Overview 194 Hyperparameter Tuning with Kubeflow Katib 195 vi | Table of Contents

Katib Concepts 196 Installing Katib 198 Running Your First Katib Experiment 198 Prepping Your Training Code 199 Configuring an Experiment 199 Running the Experiment 201 Katib User Interface 204 Tuning Distributed Training Jobs 208 Neural Architecture Search 210 Advantages of Katib over Other Frameworks 213 Conclusion 214 A. Argo Executor Configurations and Trade-Offs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 B. Cloud-Specific Tools and Configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 C. Using Model Serving in Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 Table of Contents | vii

Foreword Occasionally over the years people will ask me what skills are most in demand in tech. Ten years ago I would tell them to study machine learning, which can scale automated decision making in ways previously impossible. However, these days I have a different answer: machine learning engineering. Even just a few years ago if you knew machine learning and started at an organiza‐ tion, you would likely walk in the door as the only person with that skill set, allowing you to have an outsized impact. However, a side effect of the proliferation of books, tutorials, e-courses, and boot camps (some of which I have written myself) teaching an entire generation of technologists the skills required is that now machine learning is being used across tens of thousands of companies and organizations. These days a more likely scenario is that, walking into your new job, you find an organization using machine learning locally but unable to deploy it to production or able to deploy models but unable to manage them effectively. In this setting, the most valuable skill is not being able to train a model, but rather to manage all those models and deploy them in ways that maximize their impact. In this volume, Trevor Grant, Holden Karau, Boris Lublinsky, Richard Liu, and Ilan Filonenko have put together what I believe is an important cornerstone in the educa‐ tion of data scientists and machine learning engineers. For the foreseeable future the open source Kubeflow project will be a common tool in an organization’s toolkit for training, management, and deployment of machine learning models. This book rep‐ resents the codification of a lot of knowledge that previously existed scattered around internal documentation, conference presentations, and blog posts. ix

If you believe, as I do, that machine learning is only as powerful as how we use it, then this book is for you. — Chris Albon Director of Machine Learning, The Wikimedia Foundation https://chrisalbon.com x | Foreword

Preface We wrote this book for data engineers and data scientists who are building machine learning systems/models they want to move to production. If you’ve ever had the experience of training an excellent model only to ask yourself how to deploy it into production or keep it up to date once it gets there, this is the book for you. We hope this gives you the tools to replace Untitled_5.ipynb with something that works rela‐ tively reliably in production. This book is not intended to serve as your first introduction to machine learning. The next section points to some resources that may be useful if you are just getting started on your machine learning journey. Our Assumption About You This book assumes that you either understand how to train models locally, or are working with someone who does. If neither is true, there are many excellent intro‐ ductory books on machine learning to get you started, including Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition, by Aurélien Géron (O’Reilly). Our goal is to teach you how to do machine learning in a repeatable way, and how to automate the training and deployment of your models. A serious problem here is that this goal includes a wide range of topics, and it is more than reasonable that you may not be intimately familiar with all of them. Since we can’t delve deeply into every topic, we would like to provide you a short list of our favorite primers on several of the topics you will see covered here: • Python for Data Analysis, 2nd Edition, by Wes McKinney (O’Reilly) • Data Science from Scratch, 2nd Edition, by Joel Grus (O’Reilly) • Introduction to Machine Learning with Python by Andreas C. Müller and Sarah Guido (O’Reilly) xi

• Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edi‐ tion, by Aurélien Géron (O’Reilly) • Kubernetes: Up and Running by Brendan Burns et al. (O’Reilly) • Learning Spark by Holden Karau et al. (O’Reilly) • Feature Engineering for Machine Learning by Alice Zheng and Amanda Casari (O’Reilly) • Building Machine Learning Pipelines by Hannes Hapke and Catherine Nelson (O’Reilly) • Apache Mahout: Beyond MapReduce by Dmitriy Lyubimov and Andrew Palumbo (CreateSpace) • R Cookbook, 2nd Edition, by J. D. Long and Paul Teetor (O’Reilly) • Serving Machine Learning Models by Boris Lublinsky (O’Reilly) • “Continuous Delivery for Machine Learning” by Danilo Sato et al. • Interpretable Machine Learning by Christoph Molnar (self-published) • “A Gentle Introduction to Concept Drift in Machine Learning” by Jason Brown‐ lee • “Model Drift and Ensuring a Healthy Machine Learning Lifecycle” by A. Besir Kurtulmus • “The Rise of the Model Servers” by Alex Vikati • “An Overview of Model Explainability in Modern Machine Learning” by Rui Aguiar • Machine Learning with Python Cookbook by Chris Albon (O’Reilly) • Machine Learning Flashcards by Chris Albon Of course, there are many others, but those should get you started. Please don’t be overwhelmed by this list—you certainly don’t need to be an expert in each of these topics to effectively deploy and manage Kubeflow. In fact, Kubeflow exists to stream‐ line many of these tasks. However, there may be some topic into which you wish to delve deeper—and so this should be thought of as a “getting started” list. Containers and Kubernetes are a wide, rapidly evolving area of practice. If you want to deepen your knowledge of Kubernetes we recommend looking at the following: • Cloud Native Infrastructure by Justin Garrison and Kris Nova (O’Reilly) • Kubernetes: Up and Running by Brendan Burns et al. (O’Reilly) xii | Preface

Your Responsibility as a Practitioner This book helps you put your machine learning models into production to solve real- world problems. Solving real-world problems with machine learning is great, but as you go forth and apply your skills, remember to think about the impact. First, it’s important to make sure your models are sufficiently accurate, and there are great tools for this in Kubeflow, covered in “Training and Deploying a Model” on page 19. Even the best tools will not save you from all mistakes—for example, hyper‐ parameter tuning on the same dataset to report final cross-validation results. Even models with significant predictive power can have unintended effects and biases that may not show up during the regular training-evaluation phase. Unintended bia‐ ses can be hard to discover, but there are many stories (e.g., the Amazon machine learning–based recruiting engine that turned out to have intense biases and decided to hire only men) that demonstrate the profound potential implications of our work. Failing to address these issues early on can lead to having to abandon your entire work, as demonstrated by IBM’s decision to stop its facial recognition program and similar pauses across the industry after the implications of racial bias in facial recog‐ nition in the hands of law enforcement became clear. Even seemingly unbiased data, like raw purchase records, can turn out to have intense biases resulting in incorrect recommendations or worse. Just because a data‐ set is public and widely available does not mean it is unbiased. The well-known prac‐ tice of word embeddings has been shown to have many types of bias, including sexism, anti-LGBTQ, and anti-immigrant. When looking at a new dataset it is crucial to look for examples of bias in your data and attempt to mitigate it as much as possi‐ ble. With the most popular public datasets, various techniques are often discussed in the research, and you can use these to guide your own work. While this book does not have the tools to solve bias, we encourage you to think criti‐ cally about potential biases in your system and explore solutions before going into pro‐ duction. If you don’t know where to start, check out Katharine Jarmul’s excellent introductory talk. IBM has a collection of tools and examples in its AI Fairness 360 open source toolkit that can be a great place to start your exploration. A critical step to reducing bias in your models is to have a diverse team to notice potential issues early. As Jeff Dean said: “AI is full of promise, with the potential to revolutionize so many different areas of modern society. In order to realize its true potential, our field needs to be welcoming to all people. As it stands today, it is definitely not. Our field has a problem with inclusiveness.” Preface | xiii

1 Remember the Twitter bot that through reinforcement learning became a neo-Nazi in less than a weekend? It’s important to note that removing biases or validating accuracy in your results is not a “one and done”; model performance can degrade and biases can be introduced over time—even if you don’t personally change anything.1 Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. xiv | Preface

We will use warnings to indicate any situations where the resulting pipeline is likely to be nonportable and call out portable alternatives that you can use. Code Examples Supplemental material (code examples, etc.) is available for download at https:// oreil.ly/Kubeflow_for_ML. These code examples are available under an Apache 2 license, or as described in the next section. There are additional examples under their own respective licenses that you may find useful. The Kubeflow project has an example repo, which at the time of writing is available under an Apache 2 license. Canonical also has a set of resources that may be of special interest to MicroK8s users. Using Code Examples If you have a technical question or a problem using the code examples, please send email to bookquestions@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. Additional details on license can be found in the repos. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Kubeflow for Machine Learning by Holden Karau, Trevor Grant, Boris Lublinsky, Richard Liu, and Ilan Filo‐ nenko (O’Reilly). Copyright 2021 Holden Karau, Trevor Grant, Boris Lublinsky, Richard Liu, and Ilan Filonenko, 978-1-492-05012-4.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. Preface | xv

O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com. How to Contact the Authors For feedback, email us at intro-to-ml-kubeflow@googlegroups.com. For random ram‐ blings, occasionally about Kubeflow, follow us online: Trevor • Twitter • Blog • GitHub • Myspace Holden • Twitter • YouTube • Twitch • LinkedIn • Blog • GitHub • Facebook Boris • LinkedIn • GitHub Richard • GitHub xvi | Preface

Ilan • LinkedIn • GitHub How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) You can access the web page for this book, where we list errata, examples, and any additional information, at https://oreil.ly/Kubeflow_for_Machine_Learning. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For news and information about our books and courses, visit http://oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Acknowledgments The authors would like to thank everyone at O’Reilly Media, especially our editors Amelia Blevins and Deborah Baker, as well as the Kubeflow community for making this book possible. Clive Cox and Alejandro Saucedo from Seldon made amazing contributions to Chapter 8, without which this book would be missing key parts. We’d like to thank Google Cloud Platform for resources that allowed us to ensure examples worked on GCP. Perhaps most importantly, we’d like to thank our reviewers, without whom this book would not exist in its current form. This includes Taka Shinagawa, Pete MacKinnon, Kevin Haas, Chris Albon, Hannes Hapke, and more. To all early readers and reviewers of books, thank you for your contributions. Preface | xvii

Holden Would like to thank her girlfriend Kris Nóva for her help debugging her first Kubeflow PR, as well as the entire Kubeflow community for being so welcoming. She would also like to thank her wife Carolyn DeSimone, her puppy Timbit DeSimone-Karau (pictured in Figure P-1), and her stuffed animals for the sup‐ port needed to write. She would like to thank the doctors at SF General and UCSF for fixing up her hands so she could finish writing this book (although she does wish the hands did not hurt anymore) and everyone who came to visit her in the hospital and nursing home. A special thank you to Ann Spencer, the first editor who showed her how to have fun writing. Finally, she would like to thank her datefriend Els van Vessem for their support in recovering after her accident, especially reading stories and reminding her of her love of writing. Figure P-1. Timbit the dog Ilan Would like to thank all his colleagues at Bloomberg who took the time to review, mentor, and encourage him to write and contribute to open source. The list includes but is not limited to: Kimberly Stoddard, Dan Sun, Keith Laban, Steven Bower, and Sudarshan Kadambi. He would also like to thank his family—Galia, Yuriy, and Stan—for their unconditional love and support. Richard Would like to thank the Google Kubeflow team, including but not limited to: Jeremy Lewi, Abhishek Gupta, Thea Lamkin, Zhenghui Wang, Kunming Qu, Gabriel Wen, Michelle Casbon, and Sarah Maddox—without whose support none of this would have been possible. He would also like to thank his cat Tina (see Figure P-2) for her support and understanding during COVID-19. xviii | Preface