Powerful Python Patterns and Strategies with Modern Python (Aaron Maxwell) (Z-Library)

Pow erful Python Aaron Maxwell Powerful Python Patterns and Strategies with Modern Python

ISBN: 978-1-098-17570-2 US $64.99 CAN $81.99 PY THON / PROGR AMMING Aaron Maxwell is a software engineer and Pythonista. Through a decade working in Silicon Valley engineering teams, he gained production experience in backend engineering at scale, data science and machine learning, test automation infrastructure, DevOps and SRE, cloud infrastructure, marketing automation, and coding in a variety of languages. He’s taught advanced Python to over 10,000 technology professionals worldwide. How do you become proficient at writing complex, powerful Python applications—without wasting time rehashing the basics you already know or getting bogged down in features that just don’t matter? In this unique book, author Aaron Maxwell focuses on the Python first principles that act to accelerate everything else: the 5% of programming knowledge that makes the remaining 95% fall like dominos. You’ll learn: • Higher-order function abstractions to create powerful, expressive code • How to make all your Python code more robust and scalable with generator design patterns • Cognitive benefits of Pythonic comprehensions, how to build more complex comprehension structures, and their surprising link with generators • Metaprogramming with decorators, for potent abstractions and code reuse patterns that cannot be captured any other way • Python’s exception model for “out of band” signaling of errors and other events • Advanced object-oriented programming techniques within Python’s object model • How to leverage test-driven development to write better software faster and get into “flow” coding states • Effective module organization, basic and advanced Python logging, and more Powerful Python “The concepts taught in Powerful Python are essential for anyone who takes their Python skills seriously, and the relatable prose and clear examples make it as simple as possible for the reader to learn those concepts.” Rodrigo Girão Serrão Author of mathspp.com

Aaron Maxwell Powerful Python Patterns and Strategies with Modern Python

978-1-098-17570-2 [LSI] Powerful Python by Aaron Maxwell Copyright © 2025 MigrateUp LLC. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Brian Guerin Development Editor: Virginia Wilson Production Editor: Aleeya Rahman Copyeditor: Helena Stirling Proofreader: Krsta Technology Solutions Indexer: Ellen Troutman-Zaig Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea November 2024: First Edition Revision History for the First Edition 2024-11-08: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781098175702 for release details. Powerful Python is a trademark of MigrateUp LLC. All rights reserved. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Powerful Python, the cover image, and related trade dress are trade‐ marks of O’Reilly Media, Inc. The views expressed in this work are those of the author and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii 1. Scaling with Generators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Iteration in Python 1 Generator Functions 4 Advancing next() 6 Converting to a Generator Function 7 Do You Need Generators? 8 Generator Patterns and Scalable Composability 9 Text Lines to Dicts 11 Composable Interfaces 12 Fanning Out 13 Fanning In 14 Python Is Filled with Iterators 16 The Iterator Protocol 17 Conclusion 21 2. Creating Collections with Comprehensions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 List Comprehensions 24 Formatting for Readability (and More) 26 Multiple Sources and Filters 27 Independent Clauses 28 Multiple Filters 29 Comprehensions and Generators 31 Dictionaries, Sets, and Tuples 34 Conclusion 36 iii

3. Advanced Functions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Accepting and Passing Variable Arguments 37 Argument Unpacking 39 Variable Keyword Arguments 40 Keyword Unpacking 41 Combining Positional and Keyword Arguments 42 Functions as Objects 43 Key Functions in Python 47 Conclusion 49 4. Decorators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 The Basic Decorator 52 Generic Decorators 54 Decorating Methods 55 Data in Decorators 57 Accessing Inner Data 60 Nonlocal Decorator State 61 Decorators That Take Arguments 64 Class-Based Decorators 67 Implementing Class-Based Decorators 67 Benefits of Class-Based Decorators 69 Decorators for Classes 71 Conclusion 73 5. Exceptions and Errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 The Basic Idea 75 Handling Exceptions 76 Exceptions for Flow Control 77 Finally Blocks 79 Dictionary Exceptions 81 Exceptions Are Objects 82 Raising Exceptions 84 Catching and Re-Raising 86 The Most Diabolical Python Antipattern 88 Conclusion 92 6. Classes and Objects: Beyond the Basics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Properties 93 Property Patterns 96 Validation 96 Properties and Refactoring 98 iv | Table of Contents

The Factory Patterns 100 Alternative Constructors: The Simple Factory 100 Dynamic Type: The Factory Method Pattern 104 The Observer Pattern 106 The Simple Observer 107 A Pythonic Refinement 108 Several Channels 112 Magic Methods 114 Rebelliously Misusing Magic Methods 119 Conclusion 121 7. Automated Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 What Is Test-Driven Development? 124 Unit Tests and Simple Assertions 125 Fixtures and Common Test Setup 130 Asserting Exceptions 132 Using Subtests 133 Conclusion 136 8. Module Organization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Spawning a Module 139 Creating Separate Libraries 143 Multifile Modules 145 Import Syntax and Version Control 148 Nested Submodule Structure 150 Antipattern Warning 152 Import Side Effects 155 Conclusion 158 9. Logging in Python. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 The Basic Interface 159 Log Levels 160 Why Do We Have Log Levels? 161 Configuring the Basic Interface 162 Passing Arguments 165 Beyond Basic: Loggers 166 Log Destinations: Handlers and Streams 167 Logging to Multiple Destinations 170 Record Layout with Formatters 172 Conclusion 173 Table of Contents | v

Parting Words. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 vi | Table of Contents

Preface Python has become the lingua franca of modern computing. The thesis of this book is that Python is the most important programming language in the world today…with outsized rewards for those who master it. This book is designed to teach you tech‐ niques, patterns, and tools to permanently catapult your skill with everything Python has to offer. If you write Python code at least part of the time, this book will vastly amplify what you can accomplish and increase the speed at which you do it. And slash the amount of time you spend debugging, too. Who This Book Is For This book is for you if you know the basics of Python and have mastered just about everything the beginner tutorials can teach you. It is also for those who want to learn more advanced techniques and strategies, so you can do more with Python, and more with coding, than you could before. This book is not for people who want just enough Python to get by. Like I said, Python is important, and rewards those who master it. And this book is not for the unambitious. In writing, I assume you want to build a career you are proud of, doing work with a high positive impact. Further, this book is not for the mentally rigid. The difference between elite engineers and “normal” coders lies in the distinctions they make, the mental models they lever‐ age, and their ability to perceive what others cannot. The Two Levels of Learning It is not enough to gather knowledge. What you really want is to develop new capabil‐ ities. Hence, this book recognizes two levels of learning. vii

The first is the information level. This is the level of learning where you read some‐ thing, or I tell you something, and you memorize it. This puts facts, opinions, and other information in your mind that you can recall later; parrot back to me; and use in logical reasoning. Which is great. We certainly need this, as a foundation. But there is a deeper level of learning, called the ability level. The ability to do things you could not do before, when you are writing code. Both are important. But the ability level is what truly matters. You see, the information level can be deceptive. It makes you feel like you understand something. But then you go to write code using it, staring at a blank editor…and you find yourself stuck. “Wait a second. How do I actually use this?” Know that feeling? Of course you do. Every coder does. That feeling means you have learned at the information level, but not yet at the ability level. Because when you do, what you need just comes out of you, as naturally as thought itself. For the most part, reading a book or watching a video can only teach you at the infor‐ mation level. But this book aims to break that trend in several ways. Our Strategy in This Book Modern Problem #1: You have too much to learn. Modern Problem #2: Society has evolved to reduce your time and energy for deep focused learning, due to changes in technology and culture. This seems like a recipe for misery. But there is a way out: what are called first principles. In any field of human activity—including Python coding—there are foundational concepts which everything builds on. These include powerful distinctions, abstrac‐ tions, and mental models. When you learn what these first principles are and how to work with them, you find yourself cutting through the noise and getting ahead much more easily. These first principles are accelerators, in that they give you the tools, inner resources, and capabilities to solve many problems. It effectively creates a “95/5” rule: there is a 5% you can focus on learning, which makes the remaining 95% fall like dominos. That 5% is what we mean by the first principles of Python. Which is what this book is really about. viii | Preface

Hence, this book is selective in what it covers. It is not a comprehensive “one stop shop” for everything Python. Further, this book contains practical guidance based on lessons learned when writing real-world software—often as part of a team of engineers. So factors like maintainability, robustness, and readability are considered more important than anything else. There is a balance between leveraging powerful abstractions, and writing code that is easy to work with by everyone on your team. This book aims to walk that line. Throughout, I give much attention to cognitive aspects of software development. How do you write code that you and others can reason about easily, quickly, and accu‐ rately? This is one reason variable and function naming is important. But it goes far beyond that syntax level… to intelligently choosing which language features and library resources to use, and which to avoid. This book is not large, as measured by number of pages. That’s a feature, not a bug: you already have too much to read. The focus is on what’s most valuable, so that—as much as possible—everything you learn will serve you for years. Convention for Callables This book employs a writing convention that purists will find controversial. In prose, when referring to the names of identifiers, I use a monospace format. So the variable “x” will be x, the class named “Point” will be Point, and so on. The impurity: when referring to a function or method, I will append a pair of paren‐ theses to the identifier name. So the function called “compute” is referred to as com pute(), not compute. I do this even when the function must be called with arguments; that "()" is essentially an annotation, declaring that this identifier is callable. In my experience teaching and writing about advanced Python, this improves reading com‐ prehension, so I maintain this convention throughout this book. What’s Not Covered Here are some topics I have chosen to omit: • I barely mention anything outside the standard library. We have plenty to cover just for Python and its included batteries. • Type annotations. As we go to press, the dust is still settling on this rich feature. And as dear as it is to some, it is far from universally used. • Dataclasses. There are endless tutorials on this tool, and Chapter 6, “Classes and Objects: Beyond the Basics” is already the largest in the book. Preface | ix

• Concurrency. The fact is, most Python is written as single-threaded programs. And doing justice to threading, multiprocessing, and asyncio could double the page count. • Anything depending on specific Python versions. Fortunately, the Python pat‐ terns and strategies that work best are surprisingly independent of version. It is these slow changing yet powerful principles we focus on. • Less commonly used features such as keyword-only and positional-only argu‐ ments, conditional (ternary) expressions, pattern matching, and so on. Not to say they are not useful; but better for them to be covered elsewhere. • And other topics people like, I am sure. What is present covers the important keys of Python, many of which are not new, but are criminally underused and misunderstood, and will be highly valuable for all Pythonistas. If you simply cannot bear the injustice of this book not covering your favorite Python topic, I can only refer you to what the French poet Paul Valéry said. Which—trans‐ lated, paraphrased, and shortened—boils down to: “A work of art is never completed, only abandoned.” Such is this book, which I have invested nearly a full decade of my life producing for you. At some point, if it is to be of value to anyone at all, I must ship this thing. Getting the Most Out of This Book It is ultimately up to you to transform the information in this book into ability-level learning. And you do that by putting what you read into practice. To help, I have created coding exercises for every chapter, plus other fun resources— exclusively for readers of this book. To get these along with email notifications of future book releases, go to https://powerfulpython.com/register and follow the instructions. For professional training options, go to https://powerfulpython.com and browse the resources there. If you have feedback on this book; corrections; or suggestions for the future, send them to aaron@powerfulpython.com. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. x | Preface

Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This element signifies a tip or suggestion. Using Code Examples This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Powerful Python by Aaron Maxwell (O’Reilly). Copyright 2025 MigrateUp LLC., 978-1-098-17570-2.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Preface | xi

Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-889-8969 (in the United States or Canada) 707-827-7019 (international or local) 707-829-0104 (fax) support@oreilly.com https://oreilly.com/about/contact.html We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/powerful_python. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly-media. Watch us on YouTube: https://youtube.com/oreillymedia. Acknowledgments This book was nearly a decade in the making. And I have many to thank. First, I want to thank the thousands of readers of the earlier, self-published editions of this book—including the hundreds of professional students in Powerful Python Boot‐ camp. Your many excellent questions and comments—and pointing out bugs!— helped me continually improve the book from day one. Speaking of which, the O’Reilly team is stellar. If you are an author considering pub‐ lishing with this amazing group of people, I cannot recommend them enough. I specifically want to thank my development editor, Virginia Wilson; my production editor, Aleeya Rahman; Sarah Grey and Helena Stirling, who together caught more errors than I thought possible; Brian Guerin, for ensuring the project got started in the first place; Yasmina Greco, for wrangling the live O’Reilly training sessions that formed fertile ground for researching this book; and others I am unfairly not men‐ tioning, or who worked behind the scenes. xii | Preface

But the greatest heroes are the technical reviewers. I want to thank Peter Norvig, whose deep feedback on the final self-published version stratospherically elevated this O’Reilly edition; Rodrigo Girão Serrão, whose exceptional expertise in the Python language prevented what would have been many terrible errors; Jess Males, who saved you all from a number of confusingly worded passages and pointed out how to make them comprehensible; and Han Qi, whose formidably sharp mind made it nearly impossible for any bug to escape detection. To all of you, I cannot express enough my gratitude for your help in creating this wonderful book, and making it the best it can be. Preface | xiii

CHAPTER 1 Scaling with Generators This for loop seems simple: for item in items: do_something_with(item) And yet, miracles hide here. As you probably know, going through a collection one element at a time is called iteration. Few understand how Python’s iteration system really works and appreciate how deep and well-thought-out it is. This chapter makes you one of those people. You gain the ability to write highly scalable Python applica‐ tions, which can handle ever-larger data sets in performant, memory-efficient ways. Iteration is also core to one of Python’s most powerful tools: the generator function. Generator functions are not just a convenient way to create useful iterators. They enable some exquisite patterns of code organization, in a way that—by their very nature—intrinsically encourage excellent coding habits. This chapter is special, because understanding it threatens to make you a perma‐ nently better programmer in every language. Mastering Python generators tends to do that, because of the distinctions and insights you gain along the way. Let’s dive in. Iteration in Python Python has a built-in function called iter(). When you pass it a collection, you get back an iterator object: >>> numbers = [7, 4, 11, 3] >>> iter(numbers) <list_iterator object at 0x10219dc50> An iterator is an object producing the values in a sequence, one at a time: 1

>>> numbers_iter = iter(numbers) >>> for num in numbers_iter: ... print(num) 7 4 11 3 You don’t normally need to use iter(). If you instead write for num in numbers, what Python effectively does under the hood is call iter() on that collection. This happens automatically. Whatever object it gets back is used as the iterator for that for loop: # This... for num in numbers: print(num) # ... is effectively just like this: numbers_iter = iter(numbers) for num in numbers_iter: print(num) An iterator over a collection is a separate object, with its own identity—which you can verify with id(): >>> # id() returns a unique number for each object. ... # Different objects will always have different IDs. >>> id(numbers) 4330133896 >>> id(numbers_iter) 4330216640 How does iter() actually get the iterator? It can do this in several ways, but one relies on a magic method called __iter__(). This is a method any class (including yours) may define, and it is called with no arguments. Each time, it produces a fresh new iterator object object. Lists have this method, for example: >>> numbers.__iter__ <method-wrapper '__iter__' of list object at 0x10130e4c8> >>> numbers.__iter__() <list_iterator object at 0x1013180f0> Python makes a distinction between objects which are iterators, and objects which are iterable. We say an object is iterable if and only if you can pass it to iter() and get back a ready-to-use iterator. If that object has an __iter__() method, iter() will call it to get the iterator. Python lists and tuples are iterable. So are strings, which is why you can write for char in my_str: to iterate over my_str ’s characters. Any con‐ tainer you might use in a for loop is iterable. A for loop is the most common way to step through a sequence. But sometimes your code needs to step through in a finer-grained way. For this, use the built-in function 2 | Chapter 1: Scaling with Generators

next(). You normally call it with a single argument, which is an iterator. Each time you call it, next(my_iterator) fetches and returns the next element: >>> names = ["Tom", "Shelly", "Garth"] >>> # Create a fresh iterator... >>> names_it = iter(names) >>> next(names_it) 'Tom' >>> next(names_it) 'Shelly' >>> next(names_it) 'Garth' What happens if you call next(names_it) again? next() will raise a special built-in exception, called StopIteration: >>> next(names_it) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration Raising this specific exception is how an iterator signals that the sequence is done. You rarely have to raise or catch this exception yourself, though we’ll see some pat‐ terns later where it is useful to do so. A good mental model for how a for loop works is to imagine it calling next() each time through the loop, exiting when StopIteration gets raised. When using next() yourself, you can provide a second argument, for the default value. If you do, next() will return that instead of raising StopIteration at the end: >>> names = ["Tom", "Shelly", "Garth"] >>> new_names_it = iter(names) >>> next(new_names_it, "Rick") 'Tom' >>> next(new_names_it, "Rick") 'Shelly' >>> next(new_names_it, "Rick") 'Garth' >>> next(new_names_it, "Rick") 'Rick' >>> next(new_names_it) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration >>> next(new_names_it, "Jane") 'Jane' Consider a different situation. What if you aren’t working with a simple sequence of numbers? What if you are calculating or reading or otherwise obtaining the sequence elements as you go along? Iteration in Python | 3

Let’s start with a simple example. Suppose you need to write a function creating a list of square numbers which will be processed by other code: def fetch_squares(max_root): squares = [] for n in range(max_root): squares.append(n**2) return squares MAX = 5 for square in fetch_squares(MAX): do_something_with(square) This works. But there are potential problems lurking here. Can you spot any? Here’s one: what if MAX is not 5, but 10,000,000? Or 10,000,000,000? Or more? Your memory footprint will be pointlessly dreadful: the code here creates a massive list, uses it once, then throws it away. On top of that, the consuming for loop cannot even start until the entire list of squares has been fully calculated. If some poor human is using this program, they’ll wonder if it got stuck. Even worse: What if you are not doing arithmetic to get each element—which is fast and cheap—but making a truly expensive calculation? Or making an API call over the network? Or reading from a database? Your program will be sluggish, perhaps unre‐ sponsive, and might even crash with an out-of-memory error. Its users will think you’re a terrible programmer. The solution is to create an iterator to start with, lazily computing each value only when needed. Then each cycle through the loop happens just in time. So how do you do that? It turns out there are several ways. But the best way is called a generator function. And you’re going to love it! Generator Functions Python provides a tool called the generator function, which…well, it’s hard to describe everything it gives you in one sentence. Of its many talents, I’ll first focus on how it is a very useful shortcut for creating iterators. A generator function looks a lot like a regular function. But instead of saying return, it uses a new and different keyword: yield. Here’s a simple example: def gen_nums(): n = 0 while n < 4: yield n n += 1 4 | Chapter 1: Scaling with Generators