Share E-Book
Scan to open this page

Scan with your phone to open this page

Author: Micha Gorelick, Ian Ozsvald

Rating No ratings yet

Your Python code may run correctly, but what if you need it to run faster? This practical book shows you how to locate performance bottlenecks and significantly speed up your code in high-data-volume programs. By explaining the fundamental theory behind design choices, this expanded edition of High Performance Python helps experienced Python programmers gain a deeper understanding of Python’s implementation. How do you take advantage of multicore architectures or compilation? Or build a system that scales up beyond RAM limits or with a GPU? Authors Micha Gorelick and Ian Ozsvald reveal concrete solutions to many issues and include war stories from companies that use high-performance Python for GenAI data extraction, productionized machine learning, and more. Get a better grasp of NumPy, Cython, and profilers Learn how Python abstracts the underlying computer architecture Use profiling to find bottlenecks in CPU time and memory usage Write efficient programs by choosing appropriate data structures Speed up matrix and vector computations Process DataFrames quickly with Pandas, Dask, and Polars Speed up your neural networks and GPU computations Use tools to compile Python down to machine code Manage multiple I/O and computational operations concurrently Convert multiprocessing code to run on local or remote clusters

AI Reading Assistant

Whole-book reading guide from stratified index samples; jump to passages in the text

AI guide
【One-Line Pitch】 A practical field guide to making Python code run faster, from profiling bottlenecks to compiling, parallelizing, and scaling beyond a single machine. Best for experienced Python programmers who already write working code and now need it to handle high data volumes, multicore hardware, and GPUs. 【Book Arc】 - **Opening (~0%–10%)**: Frames the core problem—correct code that is too slow—and introduces the machine model underneath Python: CPU, memory hierarchy, buses, and why data movement, not just computation, dominates performance. - **Early (~10%–35%)**: Teaches profiling as the entry point to optimization, using CPU and memory profilers (cProfile, line profilers, Scalene, Specialist) to find real bottlenecks instead of guessing, with the Julia set as a running example. - **Middle (~35%–55%)**: Builds data-structure intuition—lists, tuples, dictionaries, and sets—explaining their complexity guarantees, resizing behavior, and when each answers your data questions fastest. - **Late (~55%–85%)**: Moves into acceleration and scale: NumPy and vectorized computation, compilation tools like Cython, DataFrame processing with Pandas, Dask, and Polars, GPU and neural-network speedups, and concurrency for I/O and computation. - **Ending (~85%–100%)**: Closes with distributed execution—converting multiprocessing code to local or remote clusters—plus "Lessons from the Field" war stories from companies applying high-performance Python in production. 【Key Takeaways】 - **Profile before you optimize** (Early): The book's central discipline is stating a testable hypothesis, changing one thing at a time, and gathering evidence—never optimizing on intuition alone. - **Performance is often about memory movement, not raw computation** (Opening): Understanding caches, buses, and data layout explains why some "fast" code stalls and why GPU transfers can be costly. - **Data structure choice is a performance decision** (Middle): Lists and tuples give O(1) positional access; dictionaries and sets trade memory for near-constant lookups, with resizing behavior you should understand. - **The GIL shapes Python's concurrency story** (Early): Multithreaded code can run at single-thread speed, which is why multiprocessing and other standard solutions matter—and why PEP 703's GIL-free future is significant. - **Compilation and vectorization unlock large gains** (Late): NumPy, Cython, and JIT developments let you push hot loops closer to machine speed when pure Python plateaus. - **DataFrame tooling has diversified** (Late): Pandas, Dask, and Polars offer different trade-offs for processing large tabular data quickly. - **Scaling out is a distinct skill from speeding up** (Ending): The book treats moving multiprocessing code to clusters as its own practical challenge, not an afterthought. - **Real-world lessons complement theory** (Ending): Field stories from GenAI data extraction and productionized machine learning ground the techniques in actual constraints. 【Reading Tips】 - Deep-read the profiling chapters early; they set up every later optimization and prevent wasted effort. - Skim the hardware/bus material on a first pass, then return to it when a specific bottleneck puzzles you. - Treat the data-structure chapters as reference—revisit them when choosing containers in your own code. - For the acceleration chapters, pick the tool matching your workload (NumPy/Cython for CPU loops, Dask/Polars for DataFrames, GPU tooling for neural networks) rather than reading linearly. - Read "Lessons from the Field" last, as a reality check on the trade-offs the technical chapters describe. 【Coverage Limits】 The excerpts cover the book's framing, profiling, data structures, and high-level roadmap, but do not detail the specific APIs, benchmarks, or code in the compilation, GPU, and distributed chapters. Chapter-level specifics beyond those shown are not covered here.
Excerpt 1
405 Tips for Using Less RAM 408 Probabilistic Data Structures 409 Very Approximate Counting with a 1-Byte Morris Counter 410 K-Minimum Values 413 Bloom Filte...
View in text
Excerpt 2
switching to the standard solutions (e.g., multiprocessing) described in this book, a significant developer overhead and communications over‐ head can be int...
View in text
Excerpt 3
t function. The list-creation steps are minor in comparison. 52 | Chapter 2: Profiling to Find Bottlenecks Example 2-12. Creating complex coordinates on the...
View in text
Excerpt 4
ash table are deleted, the table can be scaled down in size. However, resizing happens only during an insert. So, counterintuitively, if you have a dictionar...
View in text
Excerpt 5
d lists simply because there will be fewer branch misses).7 There are a lot more metrics that perf can keep track of, many of which are very spe‐ cific to th...
View in text
Excerpt 6
numerical result warrants this. For example, running torch.sum on a float16 in an autocast region will result in a float32! As a result, autocast is a fantas...
View in text
Excerpt 7
e significant speedups in some situations when you use exec. You can test for the presence of both in your environment with import bottleneck and import nume...
View in text
Excerpt 8
u’ll now have a folder structure containing a bin directory. Run it as shown in Example 8-16 to start PyPy. Example 8-16. Running PyPy to see that it impleme...
View in text
Tags
AI categories
PythonProgrammingData
ISBN: 1098165969
Publisher: O'Reilly Media
Publish Year: 2025
Language: English
Pages: 524
File Format: PDF
File Size: 9.8 MB
Text Preview (First 20 pages)
Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

Generating text preview…