Share E-Book

Expert Python Programming - Fourth Edition Master Python by learning the best coding practices and advanced programming… (Michal Jaworski, Tarek Ziadé)(Z-Library)

Author

,

Rating 4.0 AI initial rating

Log in to rate

Programming
Language English

Attain a deep understanding of building, maintaining, packaging, and shipping robust Python applications Key Features • Discover the new features of Python, such as dictionary merge, the zoneinfo module, and structural pattern matching • Create manageable code to run in various environments with different sets of dependencies • Implement effective Python data structures and algorithms to write, test, and optimize code Book Description Python is used in a wide range of domains owing to its simple yet powerful nature. Although writing Python code is easy, making it readable, reusable, and easy to maintain can be challenging. Complete with best practices, useful tools, and standards implemented by professional Python developers, this fourth edition will help you in not only overcoming such challenges but also learning Python’s latest features and advanced concepts. The book begins with a warm-up, where you will catch-up with the latest Python improvements, syntax elements, and interesting tools to boost your development efficiency. Further, the initial few chapters should allow experienced programmers coming from different languages to safely land in the Python ecosystem. As you progress, you will explore common software design patterns and various programming methodologies, such as event-driven programming, concurrency, and metaprogramming. You will also go through complex code examples and try to solve meaningful problems by bridging Python with C and C++, writing extensions that benefit from the strengths of multiple languages. Finally, you will understand the complete lifetime of any application after it goes live. By the end of the book, you should be proficient in writing efficient and maintainable Python code. Who This Book Is For The Python programming book is intended for expert programmers who want to learn Python’s advanced-level concepts and latest features. Anyone who has basic Python skills should be able to follow the content…

Format PDF
Size 4.4 MB
5
Views
(First 20 pages)

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

Page 1
(This page has no text content)
Page 2
Expert Python Programming Fourth Edition Master Python by learning the best coding practices and advanced programming concepts Michał Jaworski Tarek Ziadé BIRMINGHAM—MUMBAI
Page 3
Expert Python Programming Fourth Edition Copyright © 2021 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Producer: Shailesh Jain Acquisition Editor – Peer Reviews: Saby D'silva Project Editor: Rianna Rodrigues Content Development Editor: Edward Doxey Copy Editor: Safis Editor Technical Editor: Aditya Sawant Proofreader: Safis Editor Indexer: Pratik Shirodkar Presentation Designer: Pranit Padwal First published: September 2008 Second edition: May 2016 Third edition: April 2019 Fourth edition: May 2021 Production reference: 1260521 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-80107-110-9 www.packt.com
Page 4
Contributors About the authors Michał Jaworski has more than 10 years of professional experience in writing software using various programming languages. Michał has spent most of his career writing high-performance and distributed backend services for web applications. He has served in various roles at multiple companies: from an ordinary software engineer to lead software architect. His beloved language of choice has always been Python. I want to thank my wife for giving me constant support. Oliwia, you're the only person who knew from the very beginning that I was lying to myself when I said that this will be a light project that won't take much of my (our) time. I don't know why, but you encouraged me to do it anyway. Tarek Ziadé is a software engineer, located in Burgundy, France. He works at Elastic, building tools for developers. Before Elastic, he worked at Mozilla for 10 years, and he founded a French Python User group, called AFPy. Tarek has also written several articles about Python for various magazines, and a few books in French and English. I would like to thank Freya, Suki, Milo, and Amina for being so supportive of all my book projects.
Page 5
About the reviewer Tal Einat has been developing software for nearly 20 years, of which Python has always been a major element. He's been a core developer of the Python language since 2010. Tal holds a B.Sc. in Math & Physics from Tel Aviv University. He likes hiking, computer games, philosophical sci-fi, and spending time with his family. For the past eight years, Tal has been developing educational technology, first at Compedia where he built a group developing VR and AR education apps, and later at the startup FullProof of which he was a co-founder. Tal currently works at Rhino Health, a startup working to the enable development and use of medical AI models with patient data from across the globe while preserving patient privacy. I dedicate my part of the work on this book to my grandfather Jacob "Yanek" Friedman who passed away recently — you live on in our thoughts, making us strong and putting a smile on our faces. I'd like to thank my wife, children, brother, parents and extended family, for being ever supportive and filling my life with so much joy.
Page 6
[ i ] Table of Contents Preface ix Chapter 1: Current Status of Python 1 Where are we now and where are we going? 2 What to do with Python 2 3 Keeping up to date 5 PEP documents 6 Active communities 8 Other resources 11 Summary 12 Chapter 2: Modern Python Development Environments 15 Technical requirements 16 Python's packaging ecosystem 17 Installing Python packages using pip 17 Isolating the runtime environment 19 Application-level isolation versus system-level isolation 23 Application-level environment isolation 24 Poetry as a dependency management system 27 System-level environment isolation 32 Containerization versus virtualization 34 Virtual environments using Docker 36 Writing your first Dockerfile 37 Running containers 41 Setting up complex environments 43 Useful Docker and Docker Compose recipes for Python 46 Virtual development environments using Vagrant 56 Popular productivity tools 59 Custom Python shells 59
Page 7
Table of Contents [ ii ] Using IPython 61 Incorporating shells in your own scripts and programs 65 Interactive debuggers 66 Other productivity tools 68 Summary 70 Chapter 3: New Things in Python 71 Technical requirements 72 Recent language additions 72 Dictionary merge and update operators 73 Alternative – Dictionary unpacking 76 Alternative – ChainMap from the collections module 76 Assignment expressions 79 Type-hinting generics 83 Positional-only parameters 84 zoneinfo module 87 graphlib module 88 Not that new, but still shiny 93 breakpoint() function 93 Development mode 94 Module-level __getattr__() and __dir__() functions 97 Formatting strings with f-strings 98 Underscores in numeric literals 100 secrets module 100 What may come in the future? 101 Union types with the | operator 102 Structural pattern matching 103 Summary 108 Chapter 4: Python in Comparison with Other Languages 109 Technical requirements 110 Class model and object-oriented programming 110 Accessing super-classes 112 Multiple inheritance and Method Resolution Order 114 Class instance initialization 120 Attribute access patterns 124 Descriptors 125 Real-life example – lazily evaluated attributes 128 Properties 132 Dynamic polymorphism 138 Operator overloading 140 Dunder methods (language protocols) 141 Comparison to C++ 145
Page 8
Table of Contents [ iii ] Function and method overloading 147 Single-dispatch functions 149 Data classes 151 Functional programming 155 Lambda functions 157 The map(), filter(), and reduce() functions 159 Partial objects and partial functions 162 Generators 163 Generator expressions 165 Decorators 166 Enumerations 168 Summary 171 Chapter 5: Interfaces, Patterns, and Modularity 173 Technical requirements 174 Interfaces 175 A bit of history: zope.interface 177 Using function annotations and abstract base classes 186 Using collections.abc 191 Interfaces through type annotations 192 Inversion of control and dependency injection 195 Inversion of control in applications 197 Using dependency injection frameworks 206 Summary 212 Chapter 6: Concurrency 213 Technical requirements 214 What is concurrency? 214 Multithreading 216 What is multithreading? 217 How Python deals with threads 221 When should we use multithreading? 223 Application responsiveness 223 Multiuser applications 224 Work delegation and background processing 225 An example of a multithreaded application 226 Using one thread per item 229 Using a thread pool 231 Using two-way queues 236 Dealing with errors in threads 238 Throttling 241 Multiprocessing 245 The built-in multiprocessing module 247 Using process pools 251
Page 9
Table of Contents [ iv ] Using multiprocessing.dummy as the multithreading interface 254 Asynchronous programming 255 Cooperative multitasking and asynchronous I/O 256 Python async and await keywords 257 A practical example of asynchronous programming 262 Integrating non-asynchronous code with async using futures 265 Executors and futures 267 Using executors in an event loop 268 Summary 269 Chapter 7: Event-Driven Programming 271 Technical requirements 272 What exactly is event-driven programming? 272 Event-driven != asynchronous 273 Event-driven programming in GUIs 274 Event-driven communication 277 Various styles of event-driven programming 279 Callback-based style 280 Subject-based style 281 Topic-based style 286 Event-driven architectures 288 Event and message queues 290 Summary 293 Chapter 8: Elements of Metaprogramming 295 Technical requirements 296 What is metaprogramming? 296 Using decorators to modify function behavior before use 297 One step deeper: class decorators 299 Intercepting the class instance creation process 304 Metaclasses 307 The general syntax 309 Metaclass usage 312 Metaclass pitfalls 315 Using the __init__subclass__() method as an alternative to metaclasses 317 Code generation 319 exec, eval, and compile 319 The abstract syntax tree 321 Import hooks 323 Notable examples of code generation in Python 323 Falcon's compiled router 324 Hy 325 Summary 326
Page 10
Table of Contents [ v ] Chapter 9: Bridging Python with C and C++ 327 Technical requirements 329 C and C++ as the core of Python extensibility 329 Compiling and loading Python C extensions 330 The need to use extensions 332 Improving performance in critical code sections 333 Integrating existing code written in different languages 334 Integrating third-party dynamic libraries 335 Creating efficient custom datatypes 335 Writing extensions 336 Pure C extensions 337 A closer look at the Python/C API 341 Calling and binding conventions 345 Exception handling 349 Releasing GIL 351 Reference counting 353 Writing extensions with Cython 356 Cython as a source-to-source compiler 356 Cython as a language 360 Downsides of using extensions 362 Additional complexity 363 Harder debugging 364 Interfacing with dynamic libraries without extensions 365 The ctypes module 365 Loading libraries 365 Calling C functions using ctypes 367 Passing Python functions as C callbacks 369 CFFI 372 Summary 374 Chapter 10: Testing and Quality Automation 377 Technical requirements 378 The principles of test-driven development 379 Writing tests with pytest 381 Test parameterization 389 pytest's fixtures 392 Using fakes 402 Mocks and the unittest.mock module 405 Quality automation 410 Test coverage 411 Style fixers and code linters 415 Static type analysis 419 Mutation testing 420
Page 11
Table of Contents [ vi ] Useful testing utilities 427 Faking realistic data values 427 Faking time values 429 Summary 430 Chapter 11: Packaging and Distributing Python Code 433 Technical requirements 434 Packaging and distributing libraries 434 The anatomy of a Python package 435 setup.py 438 setup.cfg 440 MANIFEST.in 440 Essential package metadata 442 Trove classifiers 443 Types of package distributions 445 sdist distributions 445 bdist and wheel distributions 447 Registering and publishing packages 450 Package versioning and dependency management 453 The SemVer standard for semantic versioning 455 CalVer for calendar versioning 456 Installing your own packages 457 Installing packages directly from sources 457 Installing packages in editable mode 458 Namespace packages 459 Package scripts and entry points 461 Packaging applications and services for the web 465 The Twelve-Factor App manifesto 466 Leveraging Docker 467 Handling environment variables 470 The role of environment variables in application frameworks 475 Creating standalone executables 480 When standalone executables are useful 481 Popular tools 481 PyInstaller 482 cx_Freeze 486 py2exe and py2app 488 Security of Python code in executable packages 490 Summary 491 Chapter 12: Observing Application Behavior and Performance 493 Technical requirements 494 Capturing errors and logs 494 Python logging essentials 495 Logging system components 497
Page 12
Table of Contents [ vii ] Logging configuration 505 Good logging practices 509 Distributed logging 511 Capturing errors for later review 514 Instrumenting code with custom metrics 518 Using Prometheus 520 Distributed application tracing 530 Distributed tracing with Jaeger 534 Summary 540 Chapter 13: Code Optimization 541 Technical requirements 542 Common culprits for bad performance 542 Code complexity 543 Cyclomatic complexity 544 The big O notation 545 Excessive resource allocation and leaks 548 Excessive I/O and blocking operations 549 Code profiling 549 Profiling CPU usage 551 Macro-profiling 551 Micro-profiling 557 Profiling memory usage 560 Using the objgraph module 562 C code memory leaks 570 Reducing complexity by choosing appropriate data structures 571 Searching in a list 571 Using sets 573 Using the collections module 574 deque 574 defaultdict 576 namedtuple 578 Leveraging architectural trade-offs 580 Using heuristics and approximation algorithms 580 Using task queues and delayed processing 581 Using probabilistic data structures 585 Caching 586 Deterministic caching 587 Non-deterministic caching 590 Summary 595 Why subscribe? 597 Other Books You May Enjoy 599 Index 601
Page 13
(This page has no text content)
Page 14
[ ix ] Preface Python rocks! From the earliest version in the late 1980s to the current 3.9 version, Python has evolved with the same philosophy: providing a multi-paradigm programming language with readability and productivity in mind. Initially, people used to see Python as yet another scripting language. Many of them didn't believe it could be used to build large and complex systems. But over the years, and thanks to some pioneer companies, it became obvious that Python could be used to build almost any kind of a software. Although writing Python code is easy, making it readable, reusable, and easy to maintain is challenging. You can achieve those qualities only through good software artistry and technique, which you will build gradually by constantly learning and gaining experience. This book was written to express many years of professional experience in building all kinds of applications with Python, from small system scripts done in a couple of hours to very large applications written by dozens of developers over several years. This book is divided into three parts: 1. Knowing your tools: Chapters 1 to 4 focus on basic elements of Python programmer's toolbelt. From productivity tools, through modern environments, to the newest syntax elements introduced in the latest Python releases. It also offers a safe landing zone for programmers who have experience with other programming languages and are just starting to learn more advanced Python. 2. Building applications with Python: Chapters 5 to 9 are all about design patterns, programming paradigms, and metaprogramming techniques. We will try to build some small but useful programs and will be often taking a deeper look into application architecture. We will also go a bit beyond Python and see how we can integrate code written using other programming languages.
Page 15
Preface [ x ] 3. Maintaining Python applications: Chapters 10 to 13 will be discussing all the things that usually happen after the application "goes live". We will showcase tools and techniques that help to keep applications easily maintainable and show how to approach common problems with packaging, deployment, monitoring, and performance optimization. Who this book is for The Python programming book is intended for expert programmers who want to learn about Python's advanced-level concepts and latest features. This book is written for Python developers who wish to go further in mastering Python. And by developers, I mean mostly professional programmers who write Python software for a living. This is because it focuses mostly on tools and practices that are crucial for creating performant, reliable, and maintainable software in Python. However, this does not mean that hobbyists won't find anything interesting. This book is great for anyone who is interested in learning advanced-level concepts with Python. Anyone who has basic Python skills should be able to follow the content of the book, although it might require some additional effort from less experienced programmers. It should also be a good introduction to the newest releases of Python for those who are still a bit behind and continue to use older versions of Python. What this book covers Chapter 1, Current Status of Python, showcases the current state of the Python language and its community. We will see how Python is constantly changing and why it is changing. We will learn what to do with old Python 2 code and how to be constantly up to date with what is currently happening in the Python community. Chapter 2, Modern Python Development Environments, describes modern ways of setting up repeatable and consistent development environments for Python programmers. We will learn differences between application-level and system- level isolation. We will concentrate on two popular tools for environment isolation, virtualenv-type environments and Docker containers, but will also review other alternatives. At the end of the chapter, we will discuss common productivity tools that are extremely useful during development. Chapter 3, New Things in Python, showcases recent Python language additions. We will review the most important Python syntax changes that happened in the last four releases of Python. We will also take a look at exciting changes that are scheduled for the next major Python release—Python 3.10.
Page 16
Preface [ xi ] Chapter 4, Python in Comparison with Other Languages, shows how Python compares to other languages. We will learn what programming idioms are and how to recognize them in code. We will take a deeper look into key elements of Python's object-oriented programming model and how it is different from other object- oriented programming languages but will also discuss other popular programming language features like descriptors, decorators, and dataclasses. This chapter should allow programmers with experience in other languages to safely land in the Python ecosystem. Chapter 5, Interfaces, Patterns, and Modularity, discusses elements of Python that allow for implementing various reusable design patterns. If focuses on the concept of class interfaces and how they can be implemented in Python. It also discusses inversion of control and dependency injection—two extremely useful but not necessarily popular programming techniques. Chapter 6, Concurrency, explains how to implement concurrency in Python using different approaches and libraries. It features three major concurrency models: multithreading, multiprocessing and asynchronous programming. In this chapter we will learn key differences between those models and how to use them effectively. Chapter 7, Event-Driven Programming, describes what event-driven programming is and how it relates to asynchronous programming and different concurrency models. We will present various approaches to event-driven programming along with useful libraries. Chapter 8, Elements of Metaprogramming, presents an overview of common approaches to metaprogramming available to Python programmers. We will learn about common metaprogramming techniques like decorators, as well as metaclasses and code generation patterns. Chapter 9, Bridging Python with C and C++, explains how to integrate code written in different languages in your Python application. We will learn when extensions in C can be useful and how to create them. Chapter 10, Testing and Quality Automation, is about providing automated testing and quality processes. We will learn about a popular testing framework—Pytest—and many useful testing techniques. We will also cover tools that can be used to assess code quality metrics and improve code style in fully automated way. Chapter 11, Packaging and Distributing Python Code, describes the current state of Python packaging and best practices for creating packages that are to be distributed as open source code in the Python Package Index (PyPI). We will also cover the topics of packaging applications for web development and creating standalone Python executables for desktop applications.
Page 17
Preface [ xii ] Chapter 12, Observing Application Behavior and Performance, discusses the topic of application observability. We will learn about Python logging systems, how to monitor application metrics and perform distributed transaction tracing. We will also learn how to scale simple observability practices to large-scale distributed systems. Chapter 13, Code Optimization, discusses the basic rules of optimization that every developer should be aware of. We will learn how to identify application performance bottlenecks and how to use common profiling tools. We will also learn common optimization techniques and strategies that can be easily applied in many situations once you know where the bottleneck is. To get the most out of this book This book is written for developers who work under any operating system for which Python 3 is available. This is not a book for beginners, so I assume you have Python installed in your environment or know how to install it. Anyway, this book takes into account the fact that not everyone needs to be fully aware of the latest Python features or officially recommended tools. This is why Chapter 2, Modern Python Development Environments provides an overview of recommended techniques and tools (such as virtual environments and pip) for setting up development environments. Download the example code files The code bundle for the book is also hosted on GitHub at https://github.com/ PacktPublishing/Expert-Python-Programming-Fourth-Edition. In case there's an update to the code, it will be updated on the existing GitHub repository. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out! Download the color images We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://static.packt-cdn.com/ downloads/9781801071109_ColorImages.pdf.
Page 18
Preface [ xiii ] Conventions used There are a number of text conventions used throughout this book. CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, and user input. Here is an example: "Any attempt to run the code that has such issues will immediately cause the interpreter to fail, raising a SyntaxError exception." A block of code is set as follows: print("hello world") Any command-line input or output is written as follows: $ python3 script.py Some code examples will be representing input of shells. You can recognize them by specific prompt characters: • >>> for interactive Python shell • $ for Bash shell (macOS and Linux) • > for CMD or PowerShell (Windows) Some code or command-line examples will require providing your own name or values in provided placeholders. Placeholders will be surrounded with <> characters as in following example: $ python <my-module-name> Warnings or important notes appear like this. Tips and tricks appear like this.
Page 19
Preface [ xiv ] Get in touch Feedback from our readers is always welcome. General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at customercare@packtpub.com. Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packt.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details. Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at copyright@packt.com with a link to the material. If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com. Reviews Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you! For more information about Packt, please visit packt.com.
Page 20
[ 1 ] 1 Current Status of Python Python is amazing. It is amazing because it has managed to stay relevant and keep growing for so many years. For a very long time, one of the most important virtues of Python was interoperability. Thanks to that feature it didn't matter what operating system you or your customers were using. If the Python interpreter was available for a given operating system, your software written in Python would work there. And, most importantly, your software would always work the same way regardless of that operating system. However, this is now a common trait. Many modern programming languages provide similar interoperability capabilities. Also, with the advent of cloud computing, web-based applications, and reliable virtualization software, it isn't that important to have a programming language that works in many operating systems. What seems to be increasingly more important for programmers nowadays is productivity. With a constant search for innovation, it is often important to build something that you can test in the field with real users and then iterate quickly from that point until you build a really valuable product. And Python allows programmers to iterate really fast. The Official Python Package Index is a tremendous library of software libraries and frameworks that can be easily reused in your software. It greatly reduces the amount of time and effort required to get your job done. This vast availability of community libraries together with clear and concise syntax that favors readability greatly limits the time and effort that has to be spent on creating and maintaining the software. That makes Python really shine in the area of programming productivity.
The above is a preview of the first 20 pages. Register to read the complete e-book.

Recommended for You

Loading recommended books...
Failed to load, please try again later

Tip the Site

Scan the WeChat Pay or Alipay code to tip. No login required.

WeChat Pay
Alipay
← Back to List