Metaprogramming with Python A programmer’s guide to writing reusable code to build smarter applications (Sulekha Aloorravi) (z-library.sk, 1lib.sk, z-lib.sk)

📄 File Format: PDF

💾 File Size: 25.2 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

(This page has no text content)

📄 Page 2

Metaprogramming with Python A programmer’s guide to writing reusable code to build smarter applications Sulekha AloorRavi BIRMINGHAM—MUMBAI

📄 Page 3

Metaprogramming with Python Copyright © 2022 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Associate Group Product Manager: Gebin George Publishing Product Manager: Shweta Bairoliya Senior Editor: Nisha Cleetus Content Development Editor: Yashi Gupta Technical Editor: Pradeep Sahu Copy Editor: Safis Editing Project Coordinator: Deeksha Thakkar Proofreader: Safis Editing Indexer: Hemangini Bari Production Designer: Prashant Ghare Marketing Coordinator: Sonakshi Bubbar First published: August 2022 Production reference: 1110822 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-83855-465-1 www.packt.com

📄 Page 4

To my husband, Dileep V, and to all my family members, for their sacrifices and for exemplifying the power of determination during one of the toughest times of our lives. – Sulekha AloorRavi

📄 Page 5

C o n t r i b u t o r s About the author Sulekha AloorRavi is an engineer and data scientist with a wide technical breadth and deep understanding of many technologies and systems. Her background has led her to working on the advanced Python- based application development in the field of artificial intelligence. She enjoys solving real-world business problems with technology and working with data science and business intelligence teams to deliver real value. She has 15+ years of experience in software engineering and has worked with major IT solution providers and international banks. She graduated with an engineering degree in information technology and later completed a postgraduate program in big data and machine learning. She also enjoys teaching artificial intelligence and machine learning. I want to thank the people who have been close to me and supported me, especially my husband, Dileep, my nephew, Sathvik, and all my family members.

📄 Page 6

About the reviewers Florian Dahlitz has worked in the IT industry together with companies in the insurance, banking, and public industries to realize digitalization and automation as well as AI projects. He received a BSc in applied computer science from the Baden-Württemberg Cooperative State University and will shortly receive his MSc in information systems engineering and management from the Karlsruhe Institute of Technology (KIT). Florian enjoys teaching others programming in Python and helps them raise their Python skills to the next level. He spends his free time in nature and likes to capture landscapes with his camera.  Sri Manikanta Palakollu is a full-stack web developer with experience in Java, Python, C, C++, databases, AEM, machine learning, and data science. He is a tech reviewer for various tech book publishers. He has published many articles in various fields, such as data science, programming, and cybersecurity, in publications such as HackerNoon, freeCodeCamp, and DDI. He also wrote a book named Practical System Programming with C, Apress Publications. Sri Manikanta has won a national-level hackathon and regularly contributes to various open source projects. He has mentored more than 5,000 students in many national- and international-level coding hackathons hosted by multiple organizations, colleges, and universities. Dr. Madhavi Vaidya is an experienced and qualified academician and researcher with a demonstrated history of working in the education management industry, skilled in various programming languages.  Dr. Madhavi has an understanding and knowledge of various programming and database technologies, data analytics, information retrieval, software engineering, and project management. She is a strong education professional with a Master of Computer Applications and Doctor of Philosophy in the subject of computer science and engineering. One of the key areas of her research is big data analytics using Hadoop MapReduce and various big data technologies.

📄 Page 7

(This page has no text content)

📄 Page 8

Table of Contents Preface Part 1: Fundamentals – Introduction to Object-Oriented Python and Metaprogramming 1 The Need for and Applications of Metaprogramming Technical requirements 4 An overview of metaprogramming 4 Metaprogramming – a practical introduction 5 Metadata of the add function 7 Resolving type errors using metaprogramming 9 Understanding why we need metaprogramming 13 Don’t Repeat Yourself 13 Exploring the applications of metaprogramming 21 Summary 22 2 Refresher of OOP Concepts in Python Technical requirements 24 Introducing our core example 24 Creating classes 25 Understanding objects 27 Applying methods 30 Implementing inheritance 34 Extending to multiple inheritance 40 Understanding polymorphism 42 Polymorphism within inheritance 42 Polymorphism in independent classes 45 Hiding details with abstraction 46 Protecting information with encapsulation 50 Private members 50 Protected members 54 Summary 57

📄 Page 9

Table of Contentsviii Part 2: Deep Dive – Building Blocks of Metaprogramming I 3 Understanding Decorators and their Applications Technical requirements 62 Looking into simple function decorators 62 Understanding function decorators with an application 64 Exchanging decorators from one function to another 68 Applying multiple decorators to one function 75 Exploring class decorators 77 Understanding class decorators with an application 80 Getting to know built-in decorators 84 The static method 85 The class method 87 Summary 88 4 Working with Metaclasses Technical requirements 89 Overview of metaclasses 90 The structure of a metaclass 91 Analyzing the arguments 92 The application of metaclasses 94 Inheriting the metaclass 99 Inheriting as a parent and metaclass 100 Switching metaclasses 101 Inheritance in metaclasses 105 Manipulating class variables 109 Summary 114 5 Understanding Introspection Technical requirements 116 Introducing built-in functions 116 Using the built-in id function 116 Debugging unintentional assignments using id 119

📄 Page 10

Table of Contents ix Finding out whether an object is callable 121 Checking whether an object has an attribute 123 Checking whether an object is an instance 128 Checking whether an object is a subclass 131 Understanding the usage of property 135 Using property as a decorator 137 Summary 139 6 Implementing Reflection on Python Objects Technical requirements 142 Introducing built-in functions used in reflection 142 Using id to delete duplicates 142 Using callable to dynamically check and generate methods 149 Using hasattr to set values 152 Using isinstance to modify an object 156 Using issubclass to modify a class 159 Applying property on a class 162 Summary 165 7 Understanding Generics and Typing Technical requirements 168 What are generics? 168 How are generics connected to metaprogramming? 168 How are generics handled in Python? 168 What happens when data types are specified? 170 Type hints as annotations 170 Typing with explicit type checks – approach 1 171 Creating a class to implement type checking 172 Creating a class to test type checking 174 Typing with explicit type checks – approach 2 175 Creating a class to implement type checking 175 Creating a class to test type checking 177 Adding data types with constraints 180 Creating a simple custom data type 182 Creating a domain-specific data type 186 Summary 191

📄 Page 11

Table of Contentsx 8 Defining Templates for Algorithms Technical requirements 194 Explaining a sequence of operations 194 Back to our core example 194 The vegetables and dairy counter 195 Less than 10 items counter 196 The greater than 10 items counter 197 Electronics counter 198 Defining the sequence of methods 200 The vegetable counter 200 Less than 10 items counter 204 Greater than 10 items counter 205 The electronics counter 207 Identifying the common functionalities 208 Designing templates 210 Summary 216 Part 3: Deep Dive – Building Blocks of Metaprogramming II 9 Understanding Code through Abstract Syntax Tree Technical requirements 220 Exploring the ast library 220 Inspecting Python code with abstract syntax trees 221 Reviewing simple code using ast 221 Modifying simple code using ast 224 Understanding abstract syntax trees with applications 227 Understanding the ast of a class 227 Modifying the ast of a code block by parsing 229 Modifying the ast of a code block by transforming nodes 234 Summary 236 10 Understanding Method Resolution Order of Inheritance Technical requirements 238 Understanding the MRO of a class 238 Understanding MRO in single inheritance 239 Understanding MRO in multiple inheritances 243 Reviewing MRO in multilevel inheritance 245

📄 Page 12

Table of Contents xi Understanding the importance of modifying the order of inheritance 247 Impact of unintended change of order in inheritance 251 Summary 254 11 Creating Dynamic Objects Technical requirements 256 Exploring type for dynamic objects 256 Creating multiple instances of a class dynamically 260 Creating dynamic classes 263 Creating multiple dynamic classes 264 Creating dynamic attributes and methods 265 Defining attributes dynamically 266 Defining methods dynamically 267 Summary 271 12 Applying GOF Design Patterns – Part 1 Technical requirements 273 An overview of design patterns 274 Exploring behavioral design patterns 274 Understanding the chain of responsibility 274 Learning about the command design pattern 279 The strategy design pattern 286 Summary 291 13 Applying GOF Design Patterns – Part 2 Technical requirements 293 Exploring structural design patterns 293 Understanding the bridge pattern 294 Understanding the facade pattern 298 Understanding the proxy pattern 305 Exploring creational design patterns 309 Understanding the factory method 310 Understanding the prototype method 314 Understanding the singleton pattern 317 Summary 318

📄 Page 13

Table of Contentsxii 14 Generating Code from AST Technical requirements 319 Generating a simple class with a template 320 Generating multiple classes from a list 322 Generating a class with attributes 326 Generating a class with methods 329 Generating a class with an init method 329 Generating a class with a user-defined method 330 Defining a custom class factory 332 Developing a code generator to generate a simple library 333 Summary 335 15 Implementing a Case Study Technical requirements 337 Explaining the case study 337 Defining base classes 340 Developing a code generator library 343 Generating code 349 Designing an execution framework 353 Summary 356 16 Following Best Practices Technical requirements 357 Following PEP 8 standards 358 Indentation 358 Neat representation 359 Writing clear comments for debugging and reusability 361 Adding documentation strings 363 Documentation string for metaprogramming 364 Naming conventions 365 Class names 366 Variables 366 Functions and methods 367 Avoiding the reuse of names 368 Avoiding metaprogramming where not required 371 Summary 371 Index Other Books You May Enjoy

📄 Page 14

Preface Effective and reusable code makes your application development process seamless and easily maintainable. With Python, you have access to advanced metaprogramming features that you can use to build high-performing applications. This book starts by introducing you to the need for and applications of metaprogramming, before navigating the fundamentals of object-oriented programming. As you progress, you will learn about simple decorators, then work with meta classes, and later focus on introspection and reflection. You will also delve into generics and typing, before defining templates for algorithms. After that, you will understand your code using abstract syntax trees and explore method resolution order. This book also shows you how to create your own dynamic objects before structuring the objects through design patterns. Finally, you will learn about simple code-generation techniques along with best practices and eventually build your own applications. By the end of this learning journey, you will have the skills and confidence you need to design and build reusable high-performing applications that can solve real-world problems. Who this book is for If you are an intermediate-level Python programmer looking to enhance your coding skills by developing reusable and advanced frameworks, this book is for you. Basic knowledge of Python programming will help you get the most out of this learning journey. What this book covers Chapter 1, The Need for and Applications of Metaprogramming, explains the need for one of the most advanced features in Python and its practical applications.  Chapter 2, Refresher of OOP Concepts in Python, gives an overview of the existing OOP concepts, such as classes, methods, and objects, along with examples.  Chapter 3, Understanding Decorators and Their Applications, covers the concept of decorators on functions and classes with the intent to provide you with a detailed overview of decorators, how to code them, and where to use them. This chapter also covers a detailed code walkthrough of the examples. 

📄 Page 15

Prefacexiv Chapter 4, Working with Metaclasses, covers the concept of base classes and metaclasses with the intent to provide you with a detailed overview of metaclasses, how to code them, and where to use them. This chapter also covers a detailed code walkthrough of the examples.  Chapter 5, Understanding Introspection, covers the concept of introspection in Python with the intent to provide you with a detailed overview of introspection, how to code it, and where to use it. This chapter also covers a detailed code walkthrough of the examples.  Chapter 6, Implementing Reflection on Python Objects, covers the concept of reflection in Python with the intent to provide you with a detailed overview of reflection, how to code it, and where to use it. This chapter also covers a detailed code walkthrough of the examples.  Chapter 7, Understanding Generics and Typing, covers the concept of generics in Python with the intent to provide you with a detailed overview of generics, how to code them, and where to use them. This chapter also covers a detailed code walkthrough of the examples. Chapter 8, Defining Templates for Algorithms, covers the concept of templates in Python with the intent to provide you with a detailed overview of templates, how to code them, and where to use them. This chapter also covers a detailed code walkthrough of the examples.  Chapter 9, Understanding Code through Abstract Syntax Trees, covers the concept of abstract syntax trees in Python with the intent to provide you with a detailed overview of what abstract syntax trees are, how to code them, and where to use them. This chapter also covers a detailed code walkthrough of the examples.  Chapter 10, Understanding Method Resolution Order of Inheritance, covers the concept of method resolution order in Python with the intent to provide you with a detailed overview of method resolution order, how to code it, and where to use it. This chapter also covers a detailed code walkthrough of the examples.  Chapter 11, Creating Dynamic Objects, covers the concept of dynamic objects in Python with the intent to provide you with a detailed overview of dynamic objects, how to code them, and where to use them. This chapter also covers a detailed code walkthrough of the examples.  Chapter 12, Applying GOF Design Patterns – Part 1, covers the concept of behavioral design patterns in Python with the intent to provide you with a detailed overview of behavioral design patterns and apply them in different applications. This chapter also covers a detailed code walkthrough of the examples.  Chapter 13, Applying GOF Design Patterns – Part 2, covers the concept of structural and creational design patterns in Python with the intent to provide you with a detailed overview of structural and creational design patterns and apply them in different applications. This chapter also covers a detailed code walkthrough of the examples.  Chapter 14, Code Generation, covers the concept of code generation in Python with the intent to provide you with a detailed overview of code generation, how to develop a code generator that generates reusable code, and where to use it. This chapter also covers a detailed code walkthrough of the examples. 

📄 Page 16

To get the most out of this book xv Chapter 15, Development of an End-to-End Case Study-Based Application, covers the implementation of all the concepts we have learned so far by developing a case study-based application and a framework to test it. Detailed code with classes and methods along with an explanation of the code is covered in this chapter. Additionally, the steps on how to package and deploy the developed application into a Python library are also covered.  Chapter 16, Following Best Practices, covers the best practices that can be followed while implementing the concepts of metaprogramming and answers questions such as where to use and where not to use these concepts in your Python application development life cycle.  To get the most out of this book Please install the latest version of Python, preferably Python 3.0 or above, and install the latest version of Anaconda from https://www.anaconda.com/products/distribution. Once installed, open Jupyter Notebook to run the examples provided in this book. If you are using the digital version of this book, we advise you to type the code yourself or access the code from the book’s GitHub repository (a link is available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code. Download the example code files You can download the example code files for this book from GitHub at https://github.com/ PacktPublishing/Metaprogramming-with-Python. If there’s an update to the code, it will be updated in the GitHub repository. We also have other code bundles from our rich catalog of books and videos available at https:// github.com/PacktPublishing/. Check them out! Download the color images We also provide a PDF file that has color images of the screenshots and diagrams used in this book. You can download it here: https://packt.link/LTQbb.

📄 Page 17

Prefacexvi Conventions used There are a number of text conventions used throughout this book. Code in text: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: “To explain this further, let us look at an example where we will generate a class named VegCounter by parsing a series of strings using the ast module.” A block of code is set as follows: actualclass = compile(class_tree, 'vegctr_tree', 'exec') actualclass When we wish to draw your attention to a particular part of a code block or show the output of a code, the relevant lines or items are set in bold: <code object <module> at 0x0000028AAB0D2A80, file "vegctr_ tree", line 1> Tips or Important Notes Appear like this. Get in touch Feedback from our readers is always welcome. General feedback: If you have questions about any aspect of this book, email us at customercare@ packtpub.com and mention the book title in the subject of your message. Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata and fill in the form. Piracy: If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at copyright@packt.com with a link to the material. If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com.

📄 Page 18

Share Your Thoughts xvii Share Your Thoughts Once you’ve read Metaprogramming with Python, we’d love to hear your thoughts! Please click here to go straight to the Amazon review page for this book and share your feedback. Your review is important to us and the tech community and will help us make sure we’re delivering excellent quality content.

📄 Page 19

(This page has no text content)

📄 Page 20

P a r t 1 : F u n d a m e n t a l s – I n t r o d u c t i o n t o O b j e c t - O r i e n t e d P y t h o n a n d M e t a p r o g r a m m i n g The objective of this section is to give you an overview of the concept of metaprogramming, its usage, and its advantages in building Python-based applications. This section also covers the basics of object-oriented programming in Python, such as the usage of classes, functions, and objects, to help you familiarize yourself with the basic concepts, before deep diving into the complex properties of metaprogramming.  This part contains the following chapters: • Chapter 1, The Need for and Applications of Metaprogramming • Chapter 2, Refresher of OOP Concepts in Python