AI Systems Performance Engineering Optimizing Model Training and Inference Workloads with GPUs, CUDA, and PyTorch (Chris Fregly)（Z-Library） (1)

# AI Systems Performance Engineering: Optimizing Model Training and Inference Workloads with GPUs, CUDA, and PyTorch ## 【One-Line Pitch】 A comprehensive, hands-on guide for engineers and researchers who want to squeeze maximum performance from AI systems by co-optimizing GPU hardware, CUDA kernels, PyTorch code, and distributed infrastructure—from single-GPU tuning to multi-node cluster scaling. ## 【Book Arc】 - **Opening (~0%–9%)**: Introduces the book's mission—co-optimizing hardware, software, and algorithms for AI training and inference—and previews advanced inference topics like dynamic/continuous batching, prompt compression, prefix caching, and disaggregated prefill-decode architectures, plus a real-world case study of DeepSeek training a ~680B-parameter MoE model on export-restricted H800 GPUs. - **Early (~9%–25%)**: Dives into GPU hardware fundamentals, comparing NVIDIA architectures (Hopper H100 vs. Blackwell B200) in terms of transistors, memory, and die-to-die interconnects, then covers system-level concerns like liquid cooling for high-density racks (NVL72) and thermal throttling prevention. - **Early (~25%–34%)**: Moves to CPU/OS configuration for GPU environments: MIG partitioning, NUMA-aware CPU affinity, GPU driver settings (persistence mode, MPS), data prefetching, pinned memory, and network stack tuning for RDMA/InfiniBand. - **Middle (~34%–47%)**: Focuses on distributed training patterns—comparing PyTorch DataParallel (with its GPU 0 bottleneck) vs. DistributedDataParallel (one process per GPU)—and introduces NVIDIA's NIXL for high-throughput point-to-point transfers, NCCL for collectives, and GPUDirect Storage (GDS) for bypassing host memory in storage I/O. - **Middle (~47%–53%)**: Covers CUDA kernel launch configuration (blocks per grid, threads per block), emphasizing warp-size multiples, latency hiding, and occupancy tuning for modern GPUs like Blackwell. ## 【Key Takeaways】 - **Co-optimization across the full stack is essential** (Opening): Peak AI performance comes from aligning GPU kernels, network transfers, CPU threads, and storage—a weakness in any layer bottlenecks the whole system. (Middle) - **Hardware architecture drives optimization decisions** (Early): Understanding GPU internals—like Blackwell's dual-die design with 10 TB/s NV-HBI interconnect and 192 GB HBM3e—helps you reason about memory bandwidth and compute limits before writing code. - **Liquid cooling is mandatory for high-density AI racks** (Early): The NVL72's 130 kW power density requires liquid cooling with cold plates and Coolant Distribution Units (CDUs), keeping GPUs at 50–70°C to prevent thermal throttling and maintain max clocks. - **CPU/OS configuration matters as much as GPU code** (Early): NUMA-aware pinning (numactl, taskset), persistence mode, MPS, and MIG partitioning are low-overhead ways to reduce latency and improve throughput in multi-tenant environments. - **Data loading is a common hidden bottleneck** (Early): PyTorch's DataLoader with prefetch_factor, num_workers, and pin_memory=True enables async transfers that overlap with computation—neglecting this means GPUs sit idle waiting for data. - **Prefer DistributedDataParallel over DataParallel** (Middle): DataParallel serializes kernel launches and makes GPU 0 a gradient-aggregation bottleneck; DDP with one process per GPU overlaps communication with computation for better scaling. - **Use purpose-built libraries instead of reinventing** (Middle): NCCL for collectives (all-reduce), NIXL for point-to-point/streaming transfers (KV cache offloading), and GPUDirect RDMA/GDS to eliminate host memory bounce buffers—each is heavily tuned for its workload. - **CUDA launch configuration is a first-order performance lever** (Middle): Threads per block should be multiples of 32 (256–512 for Blackwell), balancing occupancy, register usage, and latency hiding to avoid underfilled warps. ## 【Reading Tips】 - **Skim the hardware deep-dives** (Early chapters) if you're already familiar with GPU architectures; focus instead on the practical CPU/OS configuration checklists and networking guidance. - **Deep-read the distributed training sections** (Middle) if you're scaling beyond a single GPU—the DataParallel vs. DDP comparison and NIXL/NCCL usage patterns are directly actionable. - **Pay attention to the profiling guidance** (Middle): Use Nsight Systems to identify communication-bound vs. compute-bound workloads, and Nsight Compute/PyTorch profiler for kernel-level memory and compute efficiency. - **The CUDA kernel launch configuration section** (Late Middle) is a quick win—even small adjustments to block sizes can yield measurable improvements. - **Excerpts don't cover the final chapters** on advanced prefill-decode tuning (FlashMLA, DeepSeek) or the 175-item optimization checklist in detail—consider those as reference material for later. ## 【Coverage Limits】 This guide synthesizes the first ~53% of the book (hardware, CPU/OS config, distributed training, and CUDA basics). The later chapters on disaggregated inference, KV cache tuning, and the final optimization checklist are only previewed in the table of contents and not covered in depth here. ##