Build a DeepSeek Model (From Scratch) - MEAP V02 (Raj Abhijit Dandekar, Rajat Dandekar etc.)（Z-Library）
, , ,
Artificial Intelligence
Learn how to build the features that set DeepSeek apart from other top LLMs! When DeepSeek started making waves in January 2025, it sounded too good to be true. How could a generative AI model get such incredible performance with such low training and operation costs? By creatively blending a variety of strategies and innovations like Mixture of Experts, Latent Attention, Multi-token Prediction, model distillation, and efficient parallelization, DeepSeek set a new standard for what’s possible in an open LLM. Now, in Build a DeepSeek Model (From Scratch) you can recreate a laptop-scale version of this cutting-edge model yourself! In Build a DeepSeek Model (From Scratch) you will learn how to: Implement DeepSeek’s core architectural innovations, including Multi-Head Latent Attention and Mixture-of-Experts layers Build a production-ready training pipeline with Multi-Token Prediction and FP8 quantization for efficiency and speed Maximize hardware utilization with parallelism strategies like DualPipe Apply post-training methods such as supervised fine-tuning and reinforcement learning to unlock reasoning capabilities Compress and distill large models into smaller, deployable versions for real-world use In Build a DeepSeek Model (From Scratch) you’ll build your own DeepSeek clone from the ground up. First, you’ll quickly review LLM fundamentals, with an eye to where DeepSeek’s innovations address the common problems and limitations of standard models. Then, you’ll learn everything you need to create your own DeepSeek-inspired model, including the innovations that put DeepSeek on the map: Multihead Latent Attention (MLA), Multi-Token Prediction (MTP), Mixture of Experts (MoE), model distillation, and reasoning.
8
Views
Registered users can read the full content for free
Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.
Page
1
(This page has no text content)
Page
2
MEAP Edition Manning Early Access Program Build a DeepSeek Model (From Scratch) Version 2 Copyright 2025 Manning Publications For more information on this and other Manning titles go to manning.com. © Manning Publications Co. To comment go to liveBook
Page
3
welcome Thank you for purchasing the MEAP for Build a DeepSeek Model (From Scratch). The ideas for this book grew out of our YouTube series, "Vizuara's Build DeepSeek from Scratch," which launched in February 2025. The series showed a clear demand for hands- on, first-principles material, encouraging us to create this more structured and detailed written guide. To get the most from this book, a foundation in machine learning and deep learning concepts is required. You should be comfortable with Python, be familiar with the basic operations in a framework like PyTorch, and have some exposure to the transformer architecture, even if you haven't implemented one yourself. This book is a hands-on guide to the technical innovations that make the DeepSeek model family work. We chose DeepSeek because it marked a significant moment in open- source AI, demonstrating that an open model could achieve performance comparable to leading proprietary systems. Our approach is to build the model's key components from scratch. The book is structured around a four-stage roadmap, covering the innovations in a logical order: 1. The foundational Key-Value (KV) Cache for efficient inference. 2. The core architectural components: Multi-Head Latent Attention (MLA) and Deepseek Mixture-of-Experts (MoE). 3. Advanced training techniques, including Multi-Token Prediction (MTP) and FP8 quantization. 4. Post-training methods like Reinforcement Learning (RL) and Knowledge Distillation. The implementations are designed to be accessible. We will work with scaled-down versions that run on consumer hardware, so a supercomputer is not necessary. The goal is for you to gain both a theoretical understanding and the practical skills to implement these modern AI techniques. Your feedback during the MEAP process is valuable for improving the final book. Please use the liveBook discussion forum to post any questions, comments, or suggestions. Let's begin. —Raj Abhijit Dandekar, Rajat Dandekar, Sreedath Panat, Naman Dwivedi © Manning Publications Co. To comment go to liveBook
Page
4
brief contents CHAPTERS 1 Introduction to DeepSeek 2 Solving the inference bottleneck with the key-value cache 3 The DeepSeek breakthrough: Multi-Head Latent Attention (MLA) 4 Mixture-of-Experts (MoE) in DeepSeek: Scaling intelligence eciently 5 Multi-token prediction and FP8 quantization 6 The DeepSeek training pipeline: Building a foundation model 7 Post-training: Supervised ne-tuning and reinforcement learning 8 Knowledge distillation: Making powerful models practical Appendix A. DeepSeek in context: A comparison with other LLMs © Manning Publications Co. To comment go to liveBook
Page
5
1 Introduction to DeepSeek This chapter covers Why DeepSeek represents a turning point in open-source AI A high-level roadmap of the key innovations we will build throughout this book The book's structure, scope, and prerequisites Large Language Models (LLMs) have transformed the technology landscape in recent years. We now live in a world where AI systems can carry on conversations, write code, draft essays, and even solve complex problems in ways that feel almost human. But what if you, a technically curious reader, could build one of these powerful AI models from scratch? What if you could understand the inner workings of a state-of-the-art LLM by constructing it step by step with code and theory hand-in-hand? That is what we plan to teach you in this book. We will understand the layers of a cutting-edge open-source LLM named DeepSeek, recreating its key innovations from the ground up. By the end, you will not only understand what makes DeepSeek unique but also how to implement those innovations yourself, gaining invaluable insights into modern AI development along the way. We will begin by understanding why DeepSeek matters: how it emerged as a turning point in open-source AI and why we chose it as the focal point of our book. Following that, we will present a roadmap of DeepSeek’s core innovations – terms like Multi-Head Latent Attention, Mixture-of-Experts, Multi-Token Prediction, and 8-bit floating-point format (FP8) quantization may sound scary now, but we will introduce them in an accessible way and explain why they are so important. We will then clarify the book structure, what you can expect to learn in this book (and what is outside our scope), ensuring our goals align with yours. Next, we will outline what you need to follow along, from background knowledge to hardware and software requirements. Don't worry, you won’t need a supercomputer! 1 © Manning Publications Co. To comment go to liveBook
Page
6
Before we dive in, a bit of context: this book was inspired by Vizuara’s YouTube series “Build DeepSeek from Scratch.” In that series, thousands of learners watched as we walked through coding each component of an LLM inspired by DeepSeek. Through this book, we have distilled our experience from academia and industry, as well as the hands-on lessons from our popular DeepSeek YouTube series, into a structured journey. 1.1 Why DeepSeek? A turning point in open-source AI With so many language models out there, you might wonder: why focus on DeepSeek? What makes this model so special that we decided to write an entire book about building it? The short answer is that DeepSeek represents a turning point in open-source AI, demonstrating for the first time that an openly available model can rival the performance of the best proprietary models. Figure 1.1 shows a simple interaction with the DeepSeek chat interface. Figure 1.1 A simple interaction with the DeepSeek chat interface. Let us step back for a moment and consider the state of AI before DeepSeek arrived. In the early 2020s, large language models were dominated by a few tech giants and research labs with vast resources. OpenAI’s GPT series, Google’s models like PaLM, and similar closed- source models led the field in capability, but they were (and remain) proprietary, often accessible only via paid APIs. The open-source community had successes like BERT and smaller GPT-style models, but there was a gap in performance. Open models tended to lag a generation behind closed models. Then, around 2023-2024, we saw a shift: Meta released LLaMA and later LLaMA-2 openly to researchers, and other organizations started emphasizing open science in AI. 2 © Manning Publications Co. To comment go to liveBook
Page
7
DeepSeek emerged in this context, but it took openness to a new level, both in terms of releasing weights freely to the public and in terms of pushing technical boundaries. DeepSeek was founded in 2023 (as an AI lab based in China, led by researcher Liang Wenfeng), and within a short span, it made waves by open-sourcing extremely large LLMs with performance on par with the very best. The significance of DeepSeek became clear with the release of its first major model, often referred to as DeepSeek-R1. A screenshot of the paper that introduced DeepSeek-R1 is shown in figure 1.2 (https://arxiv. org/pdf/2501. 12948). Figure 1.2 The title and abstract of the DeepSeek-R1 research paper. This model immediately stunned the AI community. Despite being openly available, R1 demonstrated an intelligence level that rivaled top-tier models from giants like OpenAI and Google, effectively narrowing the gap between open-source and closed-source AI to the smallest it had ever been. At the time of its release in early 2025, it was on par with or outperformed leading models like OpenAI's o1-1217 across a range of demanding reasoning benchmarks, including mathematical problem-solving (AIME 2024) and competitive coding (Codeforces). This achievement effectively narrowed the gap between open-source and closed-source AI to the smallest it had ever been. Another shocking announcement was that DeepSeek was trained at a fraction of the cost of leading OpenAI models. For us as learners and builders, DeepSeek-R1 is a perfect case study because its success comes from technical breakthroughs that we can understand. This raises several key questions that we will answer throughout this book: How did DeepSeek-R1 achieve state-of-the-art results with a fraction of the training cost? What was so novel in the DeepSeek-R1 architecture? What was so novel in the DeepSeek-R1 pre-training and post-training? 3 © Manning Publications Co. To comment go to liveBook
Page
8
You will learn answers to all the above questions in the subsequent chapters. DeepSeek’s team openly published many of their methods, and we will leverage those insights in this book. By reproducing key elements of DeepSeek, we get to explore state-of-the-art techniques in practice. This includes the topics we previewed earlier: new forms of attention, new training objectives, massive model scaling strategies, and novel ways to compress models. From a historical perspective, we can say that DeepSeek marked the moment when open-source AI truly went head-to-head with the tech giants and held its own. By replicating parts of DeepSeek, we effectively retrace the steps of some of the most advanced AI research of the day. This is immensely valuable if you aspire to work in AI research. Finally, there is a philosophical reason: DeepSeek embodies the spirit of democratization of AI. Knowledge that was once confined is now shared. In writing this book, we align with that spirit. The authors of DeepSeek published technical reports detailing their methods, and we take it a step further by translating those into an approachable tutorial. When you build something yourself, you own that knowledge in a way that reading a paper or using an API can’t provide. The hope is that by empowering more people to understand and build advanced models, we accelerate innovation and broaden the base of who can contribute to AI. Today it’s DeepSeek. Tomorrow, perhaps it will be you inventing the next big idea after having learned from this experience. 1.2 The key innovations we will build Now, let’s lay out the architectural roadmap for building DeepSeek from scratch, identifying the key innovations that differentiate DeepSeek from a standard Transformer-based language model. Understanding these specific components is crucial, as they are targeted solutions to fundamental bottlenecks in scaling language models. Each innovation from Multi-Head Latent Attention to Mixture-of-Experts addresses a distinct challenge related to computational complexity, memory bandwidth, or parameter scaling. By deconstructing and implementing these systems, we gain a first-principles understanding of the engineering trade-offs involved in modern LLM design. Our approach in this book will be to treat each innovation as a case study, first analyzing the limitations of the standard approach and then building its advanced replacement from the ground up 1.2.1 Architecture DeepSeek's architecture builds upon the well-established Transformer foundation that powers models like GPT-3 and ChatGPT. However, it introduces significant innovations to overcome key performance bottlenecks. To understand what makes DeepSeek unique, we must first look at the standard Transformer building block. 4 © Manning Publications Co. To comment go to liveBook
Page
9
The core of most modern LLMs is a stack of identical layers. Each layer consists of two primary sub-components: a multi-head self-attention mechanism, which allows the model to weigh the importance of different tokens in the input, and a feed-forward neural network, which processes the information further. Figure 1.4 shows a detailed view of this standard architecture.. Figure 1.3 A detailed view of a standard Transformer block, the foundational architecture used in models like LLaMA and the GPT series. It is composed of a multi-head attention block and a feed-forward network (NN). DeepSeek's key architectural innovation lies in replacing both of these standard sub- components with more efficient and powerful alternatives. As illustrated in Figure 1.4, the standard multi-head attention is replaced by Multi-Head Latent Attention (MLA), and the feed-forward network is replaced by a DeepSeek-Mixture-of-Experts (MoE) structure. 5 © Manning Publications Co. To comment go to liveBook
Page
10
Figure 1.4 A simplified view of the DeepSeek model architecture. It modifies the standard Transformer by replacing the core components with Multi-Head Latent Attention (MLA) and a Mixture-of-Experts (MoE) layer. This design also utilizes RMS Norm (Root Mean Square Normalization) and a specialized Decoupled RoPE (Rotary Position Embedding). 6 © Manning Publications Co. To comment go to liveBook
Page
11
These two architectural changes, MLA and DeepSeek-MoE, are targeted solutions to major challenges in scaling LLMs. On top of these architectural changes, DeepSeek introduces advanced training and efficiency techniques. A new training objective called Multi-Token Prediction (MTP) improves learning and inference speed, while FP8 quantization (an 8-bit floating-point format) addresses computational efficiency and resource utilization. Together, these four innovations MLA, MoE, MTP, and FP8 quantization form the pillars of DeepSeek’s technical advancement. Each one is a targeted solution to a different fundamental challenge in scaling language models:quantization for inference) to push efficiency to the limit.. Each of these addresses a different problem area: MLA tackles the speed and memory bottleneck in attention for long sequences. MoE tackles the scaling and model capacity issue MTP improves learning and inference speed by predicting more than one token at a time FP8 quantization addresses the computational efficiency and resource utilization problem. 1.2.2 Training Beyond the core architecture, DeepSeek also innovates in how the model is trained and refined. The training pipeline is carefully designed to make large-scale training as efficient as possible. For example, DeepSeek employs an optimized scheduling strategy (internally nicknamed DualPipe) that overlaps different training tasks to keep hardware utilization high. In practice, this means data loading, preprocessing, and neural network computations are coordinated so that the GPU is never idle. When one batch is being processed by the model, the next batch is being prepared in parallel. 7 © Manning Publications Co. To comment go to liveBook
Page
12
Figure 1.5 An illustration of the DualPipe training pipeline on a single device. By overlapping the forward pass (the initial blocks), backward pass (the hatched blocks), and combined computations, this scheduling strategy minimizes GPU idle time and maximizes hardware utilization during large-scale training. Figure 1.5 shows a timeline of Device 1 running tasks in a dual-pipe pipeline. The process begins with the forward pass, followed by the backward pass. Crucially, the pipeline then enters a steady state where the forward pass of a new batch is performed concurrently with the backward pass of the previous one (represented by the cross-hatched blocks). This overlapping ensures the device remains fully utilized throughout the training process. 1.2.3 Post-training The base model, which was trained by DeepSeek, was called DeepSeek-v3. DeepSeek-v3 went through several post-training steps, ultimately resulting in DeepSeek-R1. These steps are shown in figure 1.6. 8 © Manning Publications Co. To comment go to liveBook
Page
13
Figure 1.6 The multi-step post-training pipeline used to create DeepSeek-R1 from the DeepSeek-V3 base model. This process involves a combination of reinforcement learning (Pure RL), data generation (Rejection sampling), and fine-tuning to instill advanced reasoning capabilities. Below is a very simplified explanation of the five steps involved in DeepSeek R1 post- training: Step 1 (Foundation): Start with a lightly fine-tuned base (DeepSeek- V3) using a relatively small “cold-start” dataset. Step 2 (Pure RL): Implement reinforcement learning algorithms to allow the model to explore and develop reasoning patterns through trial-and- error learning. This unsupervised approach enabled the model to discover effective problem-solving strategies without explicit human guidance. Step 3 (Self-Labeling): Introduce rejection sampling techniques. The model generated multiple candidate responses and selected the highest- quality outputs to create its own synthetic training data. Step 4 (Blending Data): Merge this synthetic data with supervised examples to balance quality with domain breadth. Step 5 (Final RL): The training concluded with a comprehensive reinforcement learning phase using diverse prompt distributions. This final optimization step enhanced the model's robustness and generalization across various task categories and input formats. Figure 1.7 is taken from the DeepSeek-R1 paper, which was released in January 2025. Here we can see that DeepSeek-R1 is on par with or performs better than OpenAI reasoning models on several benchmarks that were tested. 9 © Manning Publications Co. To comment go to liveBook
Page
14
Figure 1.7 Benchmark performance of DeepSeek-R1 against other leading models (as of January 2025). Another important post-training technique is knowledge distillation and model compression. The idea is to take the large “teacher” model (the full DeepSeek) and compress its knowledge into one or several smaller, more practical “student” models. This has been shown in figure 1.8. 10 © Manning Publications Co. To comment go to liveBook
Page
15
Figure 1.8 The concept of knowledge distillation. A large, powerful "teacher" model (like DeepSeek-R1) is used to generate training data to teach a much smaller, more efficient "student" model, transferring its capabilities without the high computational cost. The DeepSeek R1 model was based on DeepSeek v3, which had around 671 billion parameters. When the DeepSeek paper was released, the team also released distilled models that were as low as 1.5 billion parameters in size. In particular, DeepSeek open- sourced 1.5B, 7B, 8B, 14B, 32B, and 70B checkpoints based on Qwen2.5 and Llama3 series to the community. These smaller models are highly performant and efficient. In summary, DeepSeek’s roadmap consists of innovations at multiple levels: Novel architectural components (MLA and MoE) A smarter training objective (MTP) with cutting-edge precision techniques (FP8) An efficient large-scale training pipeline (overlapping computations and communications, etc.) Post-training (RL-based reasoning skills and model distillation). In the coming chapters, we will build each of these components step by step and demonstrate how they come together into a cohesive mini-DeepSeek model. 11 © Manning Publications Co. To comment go to liveBook
Page
16
1.3 Book structure and scope We have organized the book into a clear, four-stage roadmap. This structure is designed to be progressive, with each stage building directly upon the knowledge and code from the previous one. We will begin with the essential building blocks of modern LLM inference, move through the core architectural innovations of DeepSeek, and then explore the advanced training and post-training techniques that give the model its power. Figure 1.9 provides a high-level overview of this entire process. It visualizes the four distinct stages and lists the key technical concepts we will implement within each one. Think of this as the master blueprint for our project and the table of contents for your learning journey. Figure 1.9 The four-stage roadmap for building a mini-DeepSeek model in this book. We will progress from foundational concepts (Stage 1) and core architecture (Stage 2) to advanced training (Stage 3) and post- training techniques (Stage 4), implementing each key innovation along the way. Stages 1 and 2 are related to the architectural innovations in DeepSeek. Stage 3 is related to the training pipeline, and Stage 4 is related to the post-training pipeline. 12 © Manning Publications Co. To comment go to liveBook
Page
17
In stage 1, we will understand what is meant by the Key-Value Cache (KV Cache) and why it is the foundational building block for ultimately understanding multi-head latent attention (MLA), which is one of the key innovations in the DeepSeek architecture. In stage 2, we will look at multi-head latent attention (MLA) and mixture of experts (MoE). We will visually understand how MLA and MoE work and also code them in practice. In Stage 3, we will implement the DeepSeek training pipeline. In this stage, we will learn about: 1. Multi-token prediction (MTP) 2. FP8 quantization 3. Dual pipe parallelism Finally, in stage 4, we will look at post-training techniques implemented by DeepSeek: 1. Supervised Fine-tuning 2. Reinforcement Learning (RL) 3. Model distillation 1.4 What this book will teach you and what it won’t This book is designed as a hands-on journey through the architectural innovations that make DeepSeek possible. We believe that the best way to understand these complex systems is to build them yourself. What you will learn in this book spans both theoretical understanding and practical implementation. You will discover how Multi-Head Latent Attention (MLA) dramatically reduces memory requirements while maintaining model quality, implementing the mechanism that allows DeepSeek to run efficiently on hardware that would struggle with traditional transformers. You will master the intricacies of Mixture-of-Experts (MoE) architectures, understanding how to route different tokens to specialized sub-networks and balance computational load across experts. Through Multi-Token Prediction (MTP), you'll see how predicting multiple future tokens simultaneously can accelerate both training and inference. And with FP8 quantization, you'll learn to compress model weights and activations to just eight bits while preserving the model's capabilities. You will also learn about how DeepSeek pre-trained their model. Finally, you will learn the details regarding the post-training techniques that DeepSeek implemented, which include Reinforcement Learning (RL) and Distillation. 13 © Manning Publications Co. To comment go to liveBook
Page
18
What this book won't do is reproduce DeepSeek's proprietary training data or attempt to replicate their exact model weights. We will not dive into the massive-scale distributed training infrastructure required to train models with hundreds of billions of parameters— that would require resources beyond what most readers have access to. We also will not cover production deployment concerns like serving models to millions of users or implementing safety filters and content moderation systems. Instead, we focus on clarity and comprehension. Every concept is introduced with minimal assumptions about prior knowledge, building up from first principles. For example, when we implement MLA, we will start with standard attention, understand its limitations, and then gradually transform it into the latent version. This approach means that you will not only know how to implement these techniques but also understand them deeply enough to modify and improve upon them. 1.5 What you will need to follow along Following along with this book requires a foundation in machine learning concepts, but not expertise. If you are comfortable with Python and have worked through introductory deep learning materials, you are ready to begin. Specifically, you should understand how neural networks learn through backpropagation, be familiar with the basic operations in PyTorch or a similar framework, and have some exposure to the transformer architecture, even if you haven't implemented one yourself. In terms of hardware, we have designed all the implementations to be accessible. While training large language models typically requires massive computational resources, we will work with scaled-down versions that capture the essential ideas while remaining tractable on consumer hardware. A laptop with a decent CPU can run most of the examples, though they will train slowly. A single consumer GPU with 8-12GB of VRAM will make the experience much smoother, allowing you to experiment more freely and see results faster. For the more ambitious experiments, particularly when working with the MoE architectures, having 24-48GB of VRAM opens up additional possibilities, though it's not required. We will provide complete environment specifications for each chapter, ensuring you can reproduce our results exactly. We will include configurations for Google Colab and similar platforms, making it possible to follow along without any local setup. While DeepSeek was trained on trillions of tokens, we will use smaller datasets that still allow us to observe the key phenomena while keeping training times reasonable. Let us build something amazing together! 1.6 Summary Large Language Models (LLMs) have become a dominant force in technology, but the knowledge to build them has often been confined to a few large labs. 14 © Manning Publications Co. To comment go to liveBook
Page
19
DeepSeek marked a pivotal moment by releasing open-source models with performance that rivaled the best proprietary systems, demonstrating that cutting-edge AI could be developed and shared openly. This book will guide you through a hands-on process of building a mini- DeepSeek model, focusing on its key technical innovations to provide a deep, practical understanding of modern LLM architecture and training. The core innovations we will implement are divided into four stages: (1) KV Cache Foundation, (2) Core Architecture (MLA & MoE), (3) Advanced Training Techniques (MTP & FP8), and (4) Post-training (RL & Distillation). By building these components yourself, you will gain not just theoretical knowledge but also the practical skills to implement and adapt state-of- the-art AI techniques. 15 © Manning Publications Co. To comment go to liveBook
Page
20
2 Solving the inference bottleneck with the key-value cache This chapter covers The inefficiency of autoregressive LLM inference The Key-Value Cache: a solution with a cost MQA and GQA: First-Gen Solutions to KV Cache Memory Limits To understand the key innovations in the DeepSeek architecture, we must begin with the technical problem they were designed to address. Our journey follows the four-stage roadmap outlined at the start of this book, and this chapter is dedicated entirely to Stage 1: The Key-Value Cache Foundation. This stage addresses the most fundamental bottleneck in modern LLM inference. Before we can appreciate advanced architectural choices like DeepSeek's Multi-Head Latent Attention (MLA) in Stage 2, we must first master the mechanisms it evolved from and the problems it was designed to solve. 16 © Manning Publications Co. To comment go to liveBook
The above is a preview of the first 20 pages. Register to read the complete e-book.
AI Reading Assistant
Whole-book reading guide from stratified index samples; jump to passages in the text
AI guide
【One-Line Pitch】
A hands-on guide to rebuilding the architectural and training innovations that made DeepSeek cheap and powerful, scaled down to laptop size. Best for practitioners who already know basic deep learning and want to understand modern LLM efficiency tricks by implementing them in PyTorch.
【Book Arc】
- **Opening (~0%–10%)**: Frames the puzzle — how DeepSeek achieved strong results at unusually low cost — and contrasts a standard Transformer block (multi-head attention + feed-forward network) with DeepSeek's replacements (MLA + MoE). Introduces the book's four-stage roadmap: architecture, core innovations, training, post-training.
- **Early (~10%–35%)**: Builds the inference fundamentals. Walks through attention mechanics step by step, exposes the quadratic cost of naive autoregressive generation, then derives the KV cache and its memory bottleneck. Surveys the MHA → MQA → GQA trade-off curve as motivation for what comes next.
- **Middle (~35%–50%)**: The conceptual pivot to Multi-Head Latent Attention: instead of reducing heads, compress the Key/Value information itself. Covers positional awareness via RoPE and DeepSeek's Decoupled RoPE, combining content and position paths into a working PyTorch module.
- **Late (beyond ~50%, per roadmap)**: Moves into the remaining DeepSeek innovations — Mixture-of-Experts layers, Multi-Token Prediction, FP8 quantization, and parallelism strategies like DualPipe — plus the training pipeline that ties them together.
- **Ending (final stage)**: Post-training and deployment: supervised fine-tuning, reinforcement learning for reasoning (the R1-style multi-step pipeline), and distillation/compression into smaller deployable models.
【Key Takeaways】
- **The book's spine is a trade-off curve, not a feature list** (Early–Middle): MHA gives expressivity at high memory cost, MQA cuts memory but sacrifices per-head diversity, GQA compromises. Understanding this curve is the prerequisite for appreciating why MLA matters.
- **MLA reframes the problem from "fewer heads" to "more compact information"** (Middle): rather than sharing Key/Value projections across heads, it compresses them via down- and up-projections, aiming to keep MHA-level expressivity at MQA-level cache size.
- **KV caching is the single highest-leverage inference optimization** (Early): the excerpts report a >5x speedup generating 100 tokens from simply enabling it, with larger gains for bigger models and longer sequences — a concrete reason to internalize the mechanism, not just call a library.
- **Positional encoding is treated as a first-class design problem** (Middle): the book builds RoPE from scratch after examining simpler approaches' flaws, then implements Decoupled RoPE with separate content and position paths.
- **DeepSeek's advantage is compositional** (Opening): MoE, MLA, MTP, distillation, and efficient parallelization are presented as a blend, not a single trick — the book's value is showing how they fit together.
- **Post-training is where reasoning emerges** (Opening): the R1 pipeline runs foundation fine-tuning → pure RL → rejection-sampling self-labeling → data blending → final RL, an iterative loop rather than a one-shot fine-tune.
- **Everything is meant to be built, not just read** (throughout): code listings accompany the concepts, with an official GitHub repository referenced for the full project.
【Reading Tips】
- **Deep-read the KV cache derivation (Early)**: the step-by-step "what to cache and why only Keys and Values" reasoning is the foundation for every later attention variant; skimming it will make MLA feel arbitrary.
- **Skim the attention-score walkthrough if you already know Transformers**: the Query/Key/Value matrix arithmetic is review; the payoff is the inefficiency argument that follows.
- **Treat the MHA/MQA/GQA comparison as a decision framework**: note where each sits on the performance-vs-memory curve, since MLA is defined by how it tries to escape that curve.
- **Code along with the PyTorch modules**: the MQA and MLA implementations are where the abstractions become concrete; reading them passively undersells the book.
- **Expect the later stages to be roadmap-level in these excerpts**: use the four-stage structure as your map and verify depth against the actual chapters.
【Coverage Limits】
These excerpts concentrate on the introduction and the attention/KV-cache chapters; MoE, MTP, FP8, DualPipe, and the full post-training pipeline are described at roadmap level only, so this guide cannot assess their implementation depth.
Passage locations
Excerpt 1
istillation, and reasoning. 1 1 Introduction to DeepSeek This chapter covers Why DeepSeek represents a turning point in open-source AI A high-level roadmap...
View in text
Excerpt 2
sure tokens can only gather context from previous tokens in the sequence, preventing it from "cheating" by looking ahead at tokens it is not supposed to know...
View in text
Excerpt 3
even more dramatic, often reaching 6x or more. This is the incredible advantage of the KV cache: it makes real-time, interactive generation feasible by elimi...
View in text
Excerpt 4
hile preserving the expressive power of standard Multi-Head Attention. Second, we will tackle positional awareness by building the modern solution, Rotary Po...
View in text
Recommended for You
{{#thumbnailUrl}}
{{/thumbnailUrl}}
{{^thumbnailUrl}}
{{/thumbnailUrl}}
Loading recommended books...
Failed to load, please try again later
Tip the Site
Scan the WeChat Pay or Alipay code to tip. No login required.
WeChat Pay
Alipay