Software Engineering for Data Scientists From Notebooks to Scalable Systems (Catherine Nelson)（Z-Library）

Praise for Software Engineering for Data Scientists This book is the missing link data scientists have long sought, masterfully bridging the gap between data science and software engineering. It offers a clear, actionable guide that fills the crucial skill gap many data scientists face in software engineering, elevating their coding practices to new heights. Truly, this is the book we’ve been waiting for. —Gabriela de Queiroz, Director of AI, Microsoft; Startup Advisor and Angel Investor Catherine’s book demystifies how to scale your individual work to production capacity. Whether you are a data scientist, developer, or executive, she makes data services at scale accessible. From startup to massive corporate data, following her best practices will set your data projects up for success. —Carol Willing, Core Developer of Python; 2017 ACM Software System Award recipient for Jupyter’s lasting influence I love this book! It’s the missing piece on every data scientist’s shelf. For years, bootcamps, universities, and industry managers have been trying to get skilled scientists to function more like software engineers. No book bridges that gap, until this one. —Shawn Ling Ramirez, CEO, eloraHQ

Software Engineering for Data Scientists is a must read if you want to take your data science skills from ideas to fully implemented systems. It’s a terrific guide to help you through the most important engineering aspects of coding. I wish I’d had this book years ago, it would have saved me countless hours! I thoroughly recommend it. —Laurence Moroney, AI Advocacy Lead, Google Since its beginnings, data scientists have come from a wide variety of backgrounds in education and experience. While in many ways this has been a strength of the field, often data scientists lack the software engineering skills to work closely with peers from more traditional software development backgrounds. In this book, Catherine Nelson provides a much-needed bridge between the two disciplines, giving data scientists the knowledge to level up their own work and impact. —Chris Albon, Director of Machine Learning, The Wikimedia Foundation

Software Engineering for Data Scientists From Notebooks to Scalable Systems Catherine Nelson

Software Engineering for Data Scientists by Catherine Nelson Copyright © 2024 Catherine Nelson. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800- 998-9938 or corporate@oreilly.com. Acquisitions Editor: Nicole Butterfield Development Editor: Virginia Wilson Production Editor: Christopher Faucher Copyeditor: Piper Editorial Consulting, LLC Proofreader: Krsta Technology Solutions Indexer: WordCo Indexing Services, Inc. Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea April 2024: First Edition Revision History for the First Edition

2024-04-16: First Release See http://oreilly.com/catalog/errata.csp? isbn=9781098136208 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Software Engineering for Data Scientists, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-098-13620-8 [LSI]

Preface Data science happens in code. Whether you’re building a machine learning system, exploring your data for the first time, visualizing the distribution of your data, or running a statistical analysis, your coding and computation skills are what make it happen. If you are working on production code, these skills are essential for writing successful, maintainable code. Even if you aren’t working in a production software team, you’ll find it beneficial to write more robust, reproducible code that other data scientists can use easily. And if you’re working alone, good practices will accelerate your coding and help you pick up your code after a break. I didn’t always see the value of good engineering. Earlier in my data science career, I joined a team where I was the only data scientist. My teammates were software engineers and designers, and I was concerned that it would be hard to increase my skills with no other data scientists to learn from. I expressed my concern to my coworker, a developer. He said, “But learning to write better code will let you do more data science.” This comment stuck with me, and I’ve found since then that improving my software engineering skills has been incredibly beneficial in doing data science. It’s helped me write code that is easier for my coworkers to use and that is still easy to change when I go back to it many months later. My aim with this book is to guide you on your journey to writing better data science code. I’ll describe best practices for common tasks including testing, error handling, and logging. I’ll explain how to write code that is easier to maintain and that will remain robust as your projects grow.

I’ll show you how to make your code easy for other people to use, and by the end of this book you’ll be able to integrate your data science code with a larger codebase. You might think that software engineering skills are less useful in the age of generative AI. Can’t ChatGPT just write your code for you? I’d argue that the content in this book is still just as useful even when you can speed up your coding with an AI assistant. As I’ll show throughout this book, there are many choices available for every function you write, and it’s incredibly helpful to understand the principles for why you might pick one line of code over another. You’ll need to evaluate the output of any AI assistant and check that it has made a good choice for the problem you’re working on. This book will help you do that.

Who Is This Book For? This book is aimed at data scientists, but people working in closely related fields such as data analysts, machine learning (ML) engineers, and data engineers will also find it useful. I’ll explain well-established software engineering principles that will be useful to anyone who writes code, but the examples I’ll use to illustrate these principles will be most familiar to data scientists. I’ve aimed to make this book accessible to data scientists who are relatively new to the field. Maybe you’ve just finished a degree in data science or you’re starting your first job in industry. This book will cover the practical software engineering skills that are not always included in introductory data science courses. Or maybe you didn’t take a formal data science course. Maybe you’re self-taught or you’re moving into data science from math or another science. No matter which route you’re taking into data science, this book is for you. More experienced data scientists will also learn a great deal, and you’ll find this book especially useful if you’re in a job where you’ll often interact with software developers. You’ll learn the skills that will help you work effectively on a larger codebase and how to write Python code that will work efficiently in production. I’m assuming that you already know the fundamentals of data science, including data exploration, data visualization, data wrangling, basic ML, and the math skills that go along with these. I’m also assuming that you already know the basics of how to code in Python: how to write functions and control flow statements, and the basics of how to use modules including NumPy, Matplotlib, pandas, and scikit-learn. If these are new to you, I recommend the following books:

Python Data Science Handbook by Jake VanderPlas (O’Reilly, 2023) Data Science From Scratch by Joel Grus (O’Reilly, 2019) Learning Data Science by Sam Lau, Joseph Gonzalez, and Deborah Nolan (O’Reilly, 2023) This is not a book for software developers who are looking to learn data science and machine learning skills. If this is your situation, I recommend AI and Machine Learning for Coders by Laurence Moroney (O’Reilly, 2020). SOFTWARE ENGINEERING VERSUS DATA SCIENCE It’s useful at this point to define what I see as the distinction between data science and software engineering mindsets. Data scientists generally come from a background that emphasizes the scientific processes of exploration, discovery, and hypothesis testing. The end result of a project is not known at the beginning. Software engineering, in contrast, is a process that focuses on planning what to build, designing the best way to build, then writing the code to build what was planned. The expected outcome of the project is known at the start of the project. Software engineering practices emphasize standardization and automation. Data scientists can use aspects of the engineering mindset to improve the quality of their code, a subject I will discuss in detail in Chapter 1. Why Python?

All the code examples in this book are written in Python, and many of the chapters describe Python-specific tools. In recent years, Python has become the most popular programming language for data science. The following quote is from a 2021 survey of over 3,000 data scientists carried out by Anaconda: “63% of respondents said they always or frequently use Python, making it the most popular language included in this year’s survey. In addition, 71% of educators are teaching Python, and 88% of students reported being taught Python in preparation to enter the data science/ML field.” Python has an extremely solid set of open source libraries for data science, with good backing and a healthy community of maintainers. Large trend-setting companies have chosen Python for their main ML frameworks, including TensorFlow (Google) and PyTorch (Meta). Because of this, Python appears to be especially popular among data scientists working on production machine learning code, where good coding skills are particularly important. In my experience, the Python community has been friendly and welcoming, with many excellent events that have helped me improve my skills. It’s my preferred programming language, so it was an easy choice for this book. What Is Not in This Book As I mentioned in “Who Is This Book For?”, this is not an introduction to data science or an introduction to programming. Additionally, none of the following topics appears in this book:

Installing Python: I assume that you have already installed a recent version of Python (3.9 or later) and you have some form of IDE (integrated development environment) where you can write code, such as VS Code or PyCharm. I won’t describe how to install Python, but I will explain how to set up a virtual environment in Chapter 10. Other programming languages: This book covers only Python, for the reasons given in “Why Python?”. I haven’t included any examples in R, Julia, SQL, MATLAB or any other language. Command line scripting: Command line or shell scripting is a powerful way to work with files and text. I don’t include it here because other sources cover it in great detail, including Data Science at the Command Line by Jeroen Janssens (O’Reilly, 2021). Advanced Python: The examples in this book contain relatively simple code. For coverage of more advanced Python coding, I recommend Robust Python by Patrick Viafore (O’Reilly, 2021). Guide to This Book In this book, I start by walking through good practices at the level of writing individual functions and go into detail about how you can improve your coding. In later chapters, I’ll describe how you can take that code and make it easy for someone else to use, and I’ll explain some common techniques for deployment and best practices for working in software. This book is divided into 14 chapters. Here is an overview of their contents:

Chapter 1, “What Is Good Code?”, introduces the basics of how to write code that is simple, modular, readable, efficient, and robust. Chapter 2, “Analyzing Code Performance”, describes how to measure the performance of your code and discusses some options for making your data science code run more efficiently. Chapter 3, “Using Data Structures Effectively”, discusses the trade-offs involved in choosing the data structures you work with. The data structure you choose can make a huge difference to the efficiency of your code. Chapter 4, “Object-Oriented Programming and Functional Programming”, describes the basics of these styles of programming. Used correctly, they can help you write code that is well structured and efficient. Chapter 5, “Errors, Logging, and Debugging”, walks you through what to do when your code breaks, how to raise useful errors, and strategies to identify where those errors are coming from. Chapter 6, “Code Formatting, Linting, and Type Checking”, describes how to standardize your code using tools that can automate this process. Chapter 7, “Testing Your Code”, covers how to make your code robust to changes in inputs through testing. This is a vital step in writing code that is easy to maintain. Chapter 8, “Design and Refactoring”, discusses how to structure your projects in a standardized, consistent way and how to go from a notebook to a script. Chapter 9, “Documentation”, shows you how to make your code readable for other people, including best practices for naming and commenting on your code.

Chapter 10, “Sharing Your Code: Version Control, Dependencies, and Packaging”, covers the basics of version control using Git and how to manage your project’s dependencies in virtual environments. It also shows the steps involved in turning a script into a Python package. Chapter 11, “APIs”, introduces the concept of APIs, shows how you can use them, and includes a basic example using FastAPI. Chapter 12, “Automation and Deployment”, describes the basics of deploying code, how to automate your code deployments using CI/CD (Continuous Integration/Continuous Deployment or Delivery) and GitHub Actions, and how to deploy your code to a cloud environment in a Docker container. Chapter 13, “Security”, discusses common security risks, how these risks can be mitigated, and some of the security threats unique to machine learning. Chapter 14, “Working in Software”, introduces you to common practices in software development teams including Agile ways of working, describes common roles in software teams, and introduces the wider community. Chapter 15, “Next Steps”, wraps up with some thoughts on how coding might change in the future and some suggestions for what you can do next. Reading Order You don’t necessarily need to read the chapters in this book in order, but I recommend that you start by reading Chapter 1. In this chapter, I’ll explain the fundamentals of how to write good code, and I’ll introduce topics that I’ll cover in greater detail in the rest of the book. I’ll also

introduce several of the code examples that I’ll use throughout the book. Following Chapter 1, many of the chapters can be read on their own, with these exceptions: You should read Chapter 2 before reading Chapter 3. You should read Chapters 6, 7, 10, and 11 before you read Chapter 12. Some chapters include a section that goes deeper into a machine learning topic. These sections always include ML in the section name, and if your job doesn’t involve ML you can skip these sections without missing anything that you would need to understand the rest of the chapter. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user.

Constant width italic Shows text that should be replaced with user-supplied values or by values determined by context. TIP This element signifies a tip or suggestion. NOTE This element signifies a general note. WARNING This element indicates a warning or caution. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/catherinenelson1/SEforDS. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of

example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Software Engineering for Data Scientists by Catherine Nelson (O’Reilly). Copyright 2024 Catherine Nelson, 978-1-098-13620-8.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning NOTE For more than 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com. How to Contact Us Please address comments and questions to sefordatascientists@gmail.com or to the publisher:

O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-889-8969 (in the United States or Canada) 707-827-7019 (international or local) 707-829-0104 (fax) support@oreilly.com https://www.oreilly.com/about/contact.html We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/software-engineering-data-scientists. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly- media Watch us on YouTube: https://youtube.com/oreillymedia Acknowledgments Sending a huge thank you to everyone who has helped me with this book! Your comments, feedback, discussions, and support have been so valuable. It’s been an absolute pleasure working with the team at O’Reilly. Thank you to Virginia Wilson for being a superb, supportive editor. I really enjoyed working with you. Thank

you to Nicole Butterfield for valuable overall direction and your help with the book proposal process. Thank you to Jeff Bleiel for thorough reviews of several of the chapters and Chris Faucher for making the production process go smoothly. Thank you so much to my technical reviewers William Jamir Silva, Ganesh Harke, Jo Stichbury, Antony Milne, Jess Males, and Swetha Kommuri. Your feedback was super constructive, and it’s made the final book so much better. I really appreciated your attention to detail and your helpful suggestions. Thank you to Rob Masson for great feedback on the final draft and thoughtful discussions throughout the writing process. Thank you to Carol Willing, Ricardo Martín Brualla, Chris Trudeau, Michelle Liu, Maryam Ehsani, Shivani Patel, John Sweet, Andy Ross, and Abigail Mesrenyame Dogbe for valuable technical discussions and insightful conversations. I’ve also benefited hugely from being part of the wider Python and PyLadies community; thank you to all the volunteers who give their time to it. Finally, thank you to my amazing friends and family for all your support. Rob, Mum, Richard, Lina, Salomé, Ricardo, Chris, Kiana, and Katie—I appreciate you all so much.

Chapter 1. What Is Good Code? This book aims to help you write better code. But first, what makes code “good”? There are a number of ways to think about this: the best code could be the code that runs fastest. Or it could be easiest to read. Another possible definition is that good code is easy to maintain. That is, if the project changes, it should be easy to go back to the code and change it to reflect the new requirements. The requirements for your code will change frequently because of updates to the business problem you’re solving, new research directions, or updates elsewhere in the codebase. In addition, your code shouldn’t be complex, and it shouldn’t break if it gets an unexpected input. It should be easy to add a simple new feature to your code; if this is hard it suggests your code is not well written. In this chapter, I’ll introduce aspects of good code and show examples for each. I’ll divide these into five categories: simplicity, modularity, readability, performance, and robustness. Why Good Code Matters Good code is especially important when your data science code integrates with a larger system. This could be putting a machine learning model into production, writing packages for wider distribution, or building tools for other data scientists. It’s most useful for larger codebases that will be run repeatedly. As your project grows in size and complexity, the value of good code will increase. Sometimes, the code you write will be a one-off, a prototype that needs to be hacked together today for a demo tomorrow. And if you truly will run the code only once, then don’t spend the time making it beautiful: just write code to do the job it’s needed for. But in my experience, even the code you write for a one-off demo is almost always run again or reused for another purpose. I encourage you to take the time to go back to your code after the urgency has passed and tidy it up for future use. CODE AS CRAFT Many software engineers see code as something worth doing well for its own sake. There is inherent value in an efficient, elegant piece of code. They take pride in something well done in the same way that a carpenter takes pride in a beautiful wooden cabinet, where the doors open smoothly and the drawers fit exactly. They derive job satisfaction from building something that will last. This doesn’t mean you should spend endless hours polishing the details of your code, but there are a great many small decisions you will make every time you sit in front of a keyboard. Once you know what to look for, you can choose to write better code. It’s a good feeling to practice the craft of writing software and to make something you are proud of. Good code is also easier to maintain. There’s a phenomenon known as “bit-rot”: the need to update code that hasn’t been used in some time. This happens because things your code depends on also change (for example, third-party libraries or even the operating system you’re using). If you come back to code you haven’t used for a while, you’ll probably need to do some work to modernize it. This is much easier if your code is well structured and well documented.