Share E-Book

CPython A Complete Guide to CPython’s Architecture and Performance (Chien-Lung Kao)(Z-Library)

Author

Rating No ratings yet

Log in to rate

C
Language English

Deep Dive CPython explores the internal mechanics of CPython, the widely used Python interpreter written in C. Starting with a practical guide on downloading and compiling the CPython source, this book is perfect for developers eager to understand Python’s behaviour at a fundamental level. The book takes readers from basic concepts to complex details with a systematic breakdown of core components. It covers everything from CPython’s data structures like PyObject and PyTypeObject to object lifecycle management, giving insight into memory allocation and object reference counting. Each chapter illustrates CPython's architecture, such as Python's "everything is an object" philosophy, list handling, string manipulation, and dictionary operations. Readers will explore Python’s REPL modifications, string internals, and custom type creation with practical examples, like crafting a "backflipping" PyKitty_Type. Detailed sections on Python’s virtual machine operations, bytecode generation, and exception handling enrich readers’ understanding of how Python code is parsed, compiled, and executed. This book is a thorough guide for readers who want to go beyond basic Python use and understand how it works internally. Covering complex concepts like generators, iterators, descriptors, and metaclasses, this book equips readers with a thorough grasp of Python's performance optimization and design complexities. What you will learn How to download, compile, and modify CPython's source code Gain insight into fundamental structures like PyObject and PyTypeObjectUnderstand Python's detailed handling of lists, strings, dictionaries, and the REPL environment What are bytecode generation, custom types, and the inner workings of Python’s virtual machine Who this book is for Python programmers aiming to gain a deeper understanding of Python’s internals and move beyond standard usage, as well as software professionals interested in CPython’s C-based implementation and core architecture.

Format PDF
Size 10.0 MB
7
Views
(First 20 pages)

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

Page 1
(This page has no text content)
Page 2
CPython A Complete Guide to CPython’s Architecture and Performance Chien-Lung Kao
Page 3
CPython: A Complete Guide to CPython’s Architecture and Performance ISBN-13 (pbk): 979-8-8688-1768-7 ISBN-13 (electronic): 979-8-8688-1769-4 https://doi.org/10.1007/979-8-8688-1769-4 Copyright © 2025 by Chien-Lung Kao This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Managing Director, Apress Media LLC: Welmoed Spahr Acquisitions Editor: Celestin Suresh John Development Editor: James Markham Editorial Assistant: Gryffin Winkler Cover designed by eStudioCalamar Cover image designed by kuritafsheen77 on freepik Distributed to the book trade worldwide by Springer Science+Business Media New York, 1 New York Plaza, New York, NY 10004. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com, or visit www.springeronline.com. Apress Media, LLC is a Delaware LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please e-mail booktranslations@springernature.com; for reprint, paperback, or audio rights, please e-mail bookpermissions@springernature.com. Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales web page at http://www.apress.com/bulk-sales. Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub. For more detailed information, please visit https://www.apress.com/gp/services/ source-code. If disposing of this product, please recycle the paper Chien-Lung Kao Taipei, T´ai-pei, Taiwan
Page 4
(This page has no text content)
Page 5
(This page has no text content)
Page 6
(This page has no text content)
Page 7
(This page has no text content)
Page 8
(This page has no text content)
Page 9
(This page has no text content)
Page 10
(This page has no text content)
Page 11
(This page has no text content)
Page 12
xi About the Author Chien-Lung Kao is a seasoned programmer with over 30 years of hands-on experience and 15 years as an educator, specializing in JavaScript, Ruby, and Python. He is a respected author of several books on Ruby on Rails and Git published in Taiwan, sharing his in-depth knowledge and insights with a global audience. Kao has also been a featured speaker at conferences and tech events across Asia, including Taiwan and Japan, where he connects with developers and tech enthusiasts on advanced programming techniques and industry trends.
Page 13
1 © Chien-Lung Kao 2025 C. Kao, CPython, https://doi.org/10.1007/979-8-8688-1769-4_1 CHAPTER 1 Reading the CPython Source Code What Is CPython? If this is your first time hearing the name “CPython,” you might think it’s a new implementation of Python. But no, CPython isn’t a new toy or a new programming language—it’s the very same Python interpreter that most people use every day. What some may not know is that Python was originally developed in the C programming language, so when people mention “Python” without any further clarification, they are typically referring to CPython. Why Read the Source Code? Why? For no particular reason—just for fun. And having fun is important! If your goal in reading this book is to dramatically improve your Python programming skills by exploring the CPython source code, you might be a little disappointed. While reading the CPython source can enhance your understanding of Python to some degree, the primary benefit is the opportunity to improve your C programming skills, since most of the CPython code base is written in C. Therefore, if you simply want to learn Python syntax, an introductory Python book would be a better choice. The main purpose of delving into the CPython source code is to uncover some of the “black magic” behind the language. For example, what exactly are “objects” in Python? How are modules loaded? How does Python manage memory? These are the kinds of topics this book will focus on.
Page 14
2 Where to Begin? Obtaining the Source Code Although the entire CPython project is available to browse directly on GitHub, it’s more efficient to download the CPython source code onto your local computer for in-depth exploration. The next chapter will guide you through where and how to download it. While the latest version of Python is currently 3.13, this book references the CPython source code for version 3.12.6. Source code can vary between different versions, especially for changes beyond minor releases. If you want to follow along exactly, it’s recommended to use the same version as I do to ensure that we’re viewing the same content. Additionally, because the CPython source code can be quite complex at times, I may occasionally omit sections for clarity. For example, the original code might look like this: // file: Include/object.h struct _object { _PyObject_HEAD_EXTRA #if (defined(__GNUC__) || defined(__clang__)) \ && !(defined __STDC_VERSION__ && __STDC_VERSION__ >= 201112L) // On C99 and older, anonymous union is a GCC and clang extension __extension__ #endif #ifdef _MSC_VER // Ignore MSC warning C4201: "nonstandard extension used: // nameless struct/union" __pragma(warning(push)) __pragma(warning(disable: 4201)) #endif union { Py_ssize_t ob_refcnt; #if SIZEOF_VOID_P > 4 PY_UINT32_T ob_refcnt_split[2]; #endif Chapter 1 reading the Cpython SourCe Code
Page 15
3 #ifdef _MSC_VER __pragma(warning(pop)) #endif PyTypeObject *ob_type; There are several #ifdef conditional compilation directives, some related to the operating system, others to debugging information. Most of the time, these may not be directly relevant to the core topic at hand, so for the sake of focus, I might simplify the code like this: // file: Include/object.h struct _object { _PyObject_HEAD_EXTRA union { Py_ssize_t ob_refcnt; PyTypeObject *ob_type; To be clear, it’s not that the omitted portions are unimportant, but removing them helps us concentrate on the essentials. Development Tools There are many tools you can use to read source code, such as Vim, Visual Studio Code (hereafter referred to as VS Code), or even full-featured IDEs like Visual Studio. These editors provide helpful features for source code navigation, such as jumping quickly to definitions of functions, macros, constants, or searching for keywords—all of which make source code exploration more efficient. In this book, I will use VS Code for source code exploration. If you haven’t installed it yet, you can download it from the official VS Code website. Chapter 1 reading the Cpython SourCe Code
Page 16
4 Besides VS Code, I have also started using a new editor called Zed , which is developed in Rust. While it doesn’t have as many plugins as VS Code, its performance is significantly better, especially for larger files or projects with numerous lines of code— this difference is noticeable. I highly recommend trying it out. • VS Code: https://code.visualstudio.com/ • Zed: https://zed.dev/ Can I Understand This Without Knowing C? Here’s the bottom line: Even if you don’t grasp everything 100%, it’s perfectly fine. Skimming through the code can still be a valuable learning experience. Learning is not always linear; you don’t need to understand every single line to gain insights. Sometimes, just examining code structure, function calls, or variable names can be informative. It’s like piecing together a puzzle—you don’t have to complete every piece to get a sense of the overall picture. To be honest, I’m not a C expert either. Fortunately, we live in an age of advanced AI tools such as ChatGPT that can help explain what’s going on when things get confusing. As I go through the source code, I’m learning C along the way too. Even if it feels like “the blind leading the blind,” you’ll eventually get a sense of the bigger picture. Also, not every part of the CPython project is written in C—some modules are written in Python. If you’re already comfortable with Python, you might want to start by looking at those modules, which tend to be more approachable. However, if you’re a complete beginner and have never written any kind of code before, you might find it challenging to keep up. For this reason, I assume readers have some basic programming knowledge— it doesn’t have to be Python; any language will do—as long as you’re familiar with concepts like variables, functions, loops, and control flow. Ready to get started? In the next chapter, we’ll take a look at the CPython project structure and walk through compiling the source code on your own machine, so you can build your very own version of Python! Chapter 1 reading the Cpython SourCe Code
Page 17
5 © Chien-Lung Kao 2025 C. Kao, CPython, https://doi.org/10.1007/979-8-8688-1769-4_2 CHAPTER 2 Overview of the CPython Project Structure In this chapter, we’ll take a look at the overall structure of the CPython project and try building the project ourselves. We’ll even make some small modifications to the CPython source code to get a taste of what it’s like to be a Python Core Developer (just for fun, of course)! Project Structure First, let’s grab a copy of the project from GitHub: $ git clone git@github.com:python/cpython.git -b v3.12.6 --depth=1 In this book, I am using version 3.12.6 of CPython. If you’re using a different version, some source code details might vary. Excluding some less important files, the directory structure of the freshly cloned CPython project from GitHub looks like this: CPython ├── Doc ├── Grammar ├── Include ├── Lib ├── Mac ├── Misc ├── Modules ├── Objects ├── PC ├── PCbuild
Page 18
6 ├── Parser ├── Programs ├── Python └── Tools Here’s a brief introduction to what each directory contains: • Doc: As the name suggests, this directory contains all the documentation, written in reStructuredText (.rst) format. It’s actually pretty good bedtime reading if you’re struggling to fall asleep! • Grammar: Files for defining how Python’s grammar and parser rules work. • Include: C language header files used throughout the project. If you plan to write extensions for CPython, you’ll probably need files from here. • Lib: The standard library—this directory contains modules written in Python. If you’re familiar with Python, you’ll find this folder relatively approachable. • Modules: Similar to the Lib directory, but the modules here are written in C. • Mac: Tools and files specific to macOS. • Misc: A collection of miscellaneous files. I personally use such folders to store files that I can’t categorize elsewhere. • Objects: The source code for all Python built-in object types, such as str or list. You’ll see files from this directory quite often in this book. • PCbuild: Files for building Python on Windows, especially with Visual Studio. There are project files you can open directly. • PC: Similar to PCbuild, but targeted at older Windows versions. Most files are obsolete, but some are retained for compatibility. • Parser: Code for converting .py files into tokens that Python can understand. This part is a bit more advanced. Chapter 2 Overview Of the CpythOn prOjeCt StruCture
Page 19
7 • Programs: Contains source code related to CPython executables. • Python: Contains the interpreter source code itself. This is a more advanced area, but if you’re interested in how the interpreter works, it’s worth checking out. • Tools: Utilities for Python development and maintenance. For the purposes of this book, you’ll most often see the Include, Lib, Modules, Objects, and Python directories. These are the core source files of the CPython interpreter, and if you want to understand the inner workings of Python, you’ll spend most of your time here. Building the Project After downloading the project, switch into the project directory and run the ./configure command: $ cd cpython $ ./configure The ./configure command will print out a lot of information that might look intimidating at first. What it’s really doing is checking your system environment and dependencies to make sure nothing important is missing. Once everything checks out, a Makefile will be generated. This file describes how the make command should go about compiling the whole project. If you add the --prefix parameter to ./configure, like so: $ ./configure --prefix=/tmp/my-python With the --prefix option, when you run make install later, the compiled Python (and related files) will be installed under /tmp/my-python. However, I’m not planning to run make install just yet, so you can leave off the --prefix for now. We’ll revisit it if we need to run other tools (like pip) later. Next, run the make command. This command will compile the whole project according to the rules defined in the generated Makefile: $ make Chapter 2 Overview Of the CpythOn prOjeCt StruCture
Page 20
8 The compilation process may take a while. If it finishes without any errors, you should find a new executable named python.exe in the root directory—even on macOS, the filename is python.exe. I know it might seem odd to see .exe on macOS, but it’s intentional: since there’s already a Python/ directory in the CPython project, using python.exe helps avoid a naming conflict. If you want to “install” the Python version you just built, you can run: $ make install If you passed a --prefix argument to ./configure earlier, the make install command will install Python to that location. But even if you skip installation, you can still directly run the newly built python.exe. Try running it to launch the familiar REPL environment: $ ./python.exe Python 3.12.6+ (heads/3.12:b2a7d718e3b, Sep 15 2024, 23:31:57) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.version '3.12.6+ (heads/3.12:b2a7d718e3b, Sep 15 2024, 23:31:57) [Clang 15.0.0 (clang-1500.3.9.4)]' A + after the version number indicates that this is not an official release, but rather a “development” or custom-compiled version. Greet CPython! Now, let’s make a small change to the CPython source code to create a custom effect. For example, I’d like to print a “Hello” message upon entering the REPL and a “Bye” message when exiting, just to make things a bit friendlier! To achieve this, I need to locate the code responsible for entering the REPL. This is in Python/pythonrun.c, and by looking up the _PyRun_InteractiveLoopObject() function, you’ll find a do...while... loop— the very loop that implements the “Read-Eval-Print Loop” (REPL): Chapter 2 Overview Of the CpythOn prOjeCt StruCture
The above is a preview of the first 20 pages. Register to read the complete e-book.

Recommended for You

Loading recommended books...
Failed to load, please try again later

Tip the Site

Scan the WeChat Pay or Alipay code to tip. No login required.

WeChat Pay
Alipay
← Back to List