Distributed AI Systems A practical guide to building scalable training, inference, and serving systems for production AI (Fuheng Wu)（Z-Library）

【One-Line Pitch】 A production-minded bridge from single-GPU experiments to cluster-scale AI systems, covering the full path from distributed training through inference and serving. Best for ML engineers, infrastructure engineers, and technical leads who already know deep learning basics and now need to make large models actually run at scale. 【Book Arc】 - **Opening (~0%–20%)**: Establishes the physical and quantitative foundations—GPU architecture, interconnects, memory estimation, and data preparation—so later scaling decisions rest on real hardware constraints rather than guesswork. - **Early (~20%–40%)**: Introduces core parallelism strategies and data parallelism across single- and multi-node setups, giving readers the first working mental model of how training spreads across devices. - **Middle (~40%–60%)**: Moves into memory-efficient scaling—parameter sharding and techniques to reduce memory usage in large models—solving the central bottleneck that blocks trillion-parameter training. - **Late (~60%–80%)**: Shifts from training to inference and deployment: optimized attention, caching, operator fusion, router-based designs, then GPU-aware orchestration on schedulers and container platforms. - **Ending (~80%–100%)**: Closes with benchmarking, performance tuning, and forward-looking trends such as MoE models, edge-cloud coordination, and advanced parallelism—turning the stack into something measurable and improvable. 【Key Takeaways】 - **Memory is the first constraint, not compute** (Opening): The book front-loads GPU and memory estimation so you can size models and clusters before writing distributed code—avoiding the most common early planning failure. - **Parallelism is a toolkit, not a single switch** (Early): Data parallelism for single and multi-node setups is presented as one option among several, with the trade-offs that determine when each applies. - **Parameter sharding unlocks large-model training** (Middle): Sharding is framed as the memory-efficient path to scaling beyond what replication allows, paired with broader methods to cut memory usage. - **Inference is its own engineering discipline** (Late): Optimized attention, caching, operator fusion, and router-based designs are treated as distinct levers for high-performance serving, not afterthoughts to training. - **Deployment requires GPU-aware orchestration** (Late): Schedulers and container platforms are covered with GPU awareness in mind, and production stacks are assembled around reliability, scalability, and observability. - **You cannot tune what you do not measure** (Ending): Benchmarking and performance tuning close the loop, making performance work empirical rather than speculative. - **The field's direction is built into the ending** (Ending): MoE models, edge-cloud coordination, and advanced parallelism signal where practitioners should expect the next round of complexity. 【Reading Tips】 - **Read the opening deeply, not quickly.** The GPU architecture, interconnect, and memory-estimation material is the vocabulary for everything later; skimming it makes the parallelism chapters harder than they need to be. - **Treat training and inference as two passes.** If you are serving-focused, still read the sharding chapter—memory reasoning carries directly into inference sizing—then deep-read the inference and deployment section. - **Do the estimation math yourself.** Where the book gives memory or sizing methods, apply them to a model you actually work with; this is where the practical value compounds. - **Skim the trends chapter on a first read.** MoE, edge-cloud, and advanced parallelism are orientation, not prerequisites; return to them once the core stack is solid. - **Keep the benchmarking material as a checklist.** Use it to audit your own stack's observability and tuning loop after finishing. 【Coverage Limits】 This guide is based on the book's descriptive overview only; chapter-level detail, specific code examples, and exact techniques are not covered in the excerpts, so treat section boundaries as approximate.