Build a DeepSeek Model (From Scratch) (MEAP) (Raj Abhijit Dandekar, Rajat Dandekar etc.) (z-library.sk, 1lib.sk, z-lib.sk)

Page 1

(This page has no text content)

Page 2

(This page has no text content)

Page 3

Build a DeepSeek Model (From Scratch) 1. welcome 2. 1_Introduction_to_DeepSeek 3. 2_Solving_the_inference_bottleneck_with_the_key-value_cache 4. 3_The_DeepSeek_breakthrough:_Multi-Head_Latent_Attention_(MLA) 5. 4_Mixture-of- Experts_(MoE)_in_DeepSeek:_Scaling_intelligence_efficiently

Page 4

welcome Thank you for purchasing the MEAP for Build a DeepSeek Model (From Scratch). The ideas for this book grew out of our YouTube series, "Vizuara's Build DeepSeek from Scratch," which launched in February 2025. The series showed a clear demand for hands-on, first-principles material, encouraging us to create this more structured and detailed written guide. To get the most from this book, a foundation in machine learning and deep learning concepts is required. You should be comfortable with Python, be familiar with the basic operations in a framework like PyTorch, and have some exposure to the transformer architecture, even if you haven't implemented one yourself. This book is a hands-on guide to the technical innovations that make the DeepSeek model family work. We chose DeepSeek because it marked a significant moment in open-source AI, demonstrating that an open model could achieve performance comparable to leading proprietary systems. Our approach is to build the model's key components from scratch. The book is structured around a four-stage roadmap, covering the innovations in a logical order: 1. The foundational Key-Value (KV) Cache for efficient inference. 2. The core architectural components: Multi-Head Latent Attention (MLA) and Deepseek Mixture-of-Experts (MoE). 3. Advanced training techniques, including Multi-Token Prediction (MTP) and FP8 quantization. 4. Post-training methods like Reinforcement Learning (RL) and Knowledge Distillation. The implementations are designed to be accessible. We will work with scaled-down versions that run on consumer hardware, so a supercomputer is

Page 5

not necessary. The goal is for you to gain both a theoretical understanding and the practical skills to implement these modern AI techniques. Your feedback during the MEAP process is valuable for improving the final book. Please use the liveBook discussion forum to post any questions, comments, or suggestions. Let's begin. —Raj Abhijit Dandekar, Rajat Dandekar, Sreedath Panat, Naman Dwivedi In this book welcome 1 Introduction to DeepSeek 2 Solving the inference bottleneck with the key-value cache 3 The DeepSeek breakthrough: Multi-Head Latent Attention (MLA) 4 Mixture-of-Experts (MoE) in DeepSeek: Scaling intelligence efficiently

Page 6

1 Introduction to DeepSeek This chapter covers Why DeepSeek represents a turning point in open-source AI A high-level roadmap of the key innovations we will build throughout this book The book's structure, scope, and prerequisites Large Language Models (LLMs) have transformed the technology landscape in recent years. We now live in a world where AI systems can carry on conversations, write code, draft essays, and even solve complex problems in ways that feel almost human. But what if you, a technically curious reader, could build one of these powerful AI models from scratch? What if you could understand the inner workings of a state-of-the-art LLM by constructing it step by step with code and theory hand-in-hand? That is what we plan to teach you in this book. We will understand the layers of a cutting-edge open-source LLM named DeepSeek, recreating its key innovations from the ground up. By the end, you will not only understand what makes DeepSeek unique but also how to implement those innovations yourself, gaining invaluable insights into modern AI development along the way. We will begin by understanding why DeepSeek matters: how it emerged as a turning point in open-source AI and why we chose it as the focal point of our book. Following that, we will present a roadmap of DeepSeek’s core innovations – terms like Multi-Head Latent Attention, Mixture-of-Experts, Multi-Token Prediction, and 8-bit floating-point format (FP8) quantization may sound scary now, but we will introduce them in an accessible way and explain why they are so important. We will then clarify the book structure, what you can expect to learn in this book (and what is outside our scope), ensuring our goals align with yours.

Page 7

Next, we will outline what you need to follow along, from background knowledge to hardware and software requirements. Don't worry, you won’t need a supercomputer! Before we dive in, a bit of context: this book was inspired by Vizuara’s YouTube series “Build DeepSeek from Scratch.” In that series, thousands of learners watched as we walked through coding each component of an LLM inspired by DeepSeek. Through this book, we have distilled our experience from academia and industry, as well as the hands-on lessons from our popular DeepSeek YouTube series, into a structured journey. 1.1 Why DeepSeek? A turning point in open-source AI With so many language models out there, you might wonder: why focus on DeepSeek? What makes this model so special that we decided to write an entire book about building it? The short answer is that DeepSeek represents a turning point in open-source AI, demonstrating for the first time that an openly available model can rival the performance of the best proprietary models. Figure 1.1 shows a simple interaction with the DeepSeek chat interface. Figure 1.1 A simple interaction with the DeepSeek chat interface.

Page 8

Let us step back for a moment and consider the state of AI before DeepSeek arrived. In the early 2020s, large language models were dominated by a few tech giants and research labs with vast resources. OpenAI’s GPT series, Google’s models like PaLM, and similar closed-source models led the field in capability, but they were (and remain) proprietary, often accessible only via paid APIs. The open-source community had successes like BERT and smaller GPT-style models, but there was a gap in performance. Open models tended to lag a generation behind closed models. Then, around 2023-2024, we saw a shift: Meta released LLaMA and later LLaMA-2 openly to researchers, and other organizations started emphasizing open science in AI.

Page 9

DeepSeek emerged in this context, but it took openness to a new level, both in terms of releasing weights freely to the public and in terms of pushing technical boundaries. DeepSeek was founded in 2023 (as an AI lab based in China, led by researcher Liang Wenfeng), and within a short span, it made waves by open-sourcing extremely large LLMs with performance on par with the very best. The significance of DeepSeek became clear with the release of its first major model, often referred to as DeepSeek-R1. A screenshot of the paper that introduced DeepSeek-R1 is shown in figure 1.2 (https://arxiv.org/pdf/2501.12948). Figure 1.2 The title and abstract of the DeepSeek-R1 research paper. This model immediately stunned the AI community. Despite being openly available, R1 demonstrated an intelligence level that rivaled top-tier models from giants like OpenAI and Google, effectively narrowing the gap between open-source and closed-source AI to the smallest it had ever been. At the

Page 10

time of its release in early 2025, it was on par with or outperformed leading models like OpenAI's o1-1217 across a range of demanding reasoning benchmarks, including mathematical problem-solving (AIME 2024) and competitive coding (Codeforces). This achievement effectively narrowed the gap between open-source and closed-source AI to the smallest it had ever been. Another shocking announcement was that DeepSeek was trained at a fraction of the cost of leading OpenAI models. For us as learners and builders, DeepSeek-R1 is a perfect case study because its success comes from technical breakthroughs that we can understand. This raises several key questions that we will answer throughout this book: How did DeepSeek-R1 achieve state-of-the-art results with a fraction of the training cost? What was so novel in the DeepSeek-R1 architecture? What was so novel in the DeepSeek-R1 pre-training and post-training? You will learn answers to all the above questions in the subsequent chapters. DeepSeek’s team openly published many of their methods, and we will leverage those insights in this book. By reproducing key elements of DeepSeek, we get to explore state-of-the-art techniques in practice. This includes the topics we previewed earlier: new forms of attention, new training objectives, massive model scaling strategies, and novel ways to compress models. From a historical perspective, we can say that DeepSeek marked the moment when open-source AI truly went head-to-head with the tech giants and held its own. By replicating parts of DeepSeek, we effectively retrace the steps of some of the most advanced AI research of the day. This is immensely valuable if you aspire to work in AI research. Finally, there is a philosophical reason: DeepSeek embodies the spirit of democratization of AI. Knowledge that was once confined is now shared. In writing this book, we align with that spirit. The authors of DeepSeek published technical reports detailing their methods, and we take it a step further by translating those into an approachable tutorial.

Page 11

When you build something yourself, you own that knowledge in a way that reading a paper or using an API can’t provide. The hope is that by empowering more people to understand and build advanced models, we accelerate innovation and broaden the base of who can contribute to AI. Today it’s DeepSeek. Tomorrow, perhaps it will be you inventing the next big idea after having learned from this experience. 1.2 The key innovations we will build Now, let’s lay out the architectural roadmap for building DeepSeek from scratch, identifying the key innovations that differentiate DeepSeek from a standard Transformer-based language model. Understanding these specific components is crucial, as they are targeted solutions to fundamental bottlenecks in scaling language models. Each innovation from Multi-Head Latent Attention to Mixture-of-Experts addresses a distinct challenge related to computational complexity, memory bandwidth, or parameter scaling. By deconstructing and implementing these systems, we gain a first-principles understanding of the engineering trade-offs involved in modern LLM design. Our approach in this book will be to treat each innovation as a case study, first analyzing the limitations of the standard approach and then building its advanced replacement from the ground up 1.2.1 Architecture DeepSeek's architecture builds upon the well-established Transformer foundation that powers models like GPT-3 and ChatGPT. However, it introduces significant innovations to overcome key performance bottlenecks. To understand what makes DeepSeek unique, we must first look at the standard Transformer building block. The core of most modern LLMs is a stack of identical layers. Each layer consists of two primary sub-components: a multi-head self-attention mechanism, which allows the model to weigh the importance of different tokens in the input, and a feed-forward neural network, which processes the information further. Figure 1.4 shows a detailed view of this standard architecture..

Page 12

Figure 1.3 A detailed view of a standard Transformer block, the foundational architecture used in models like LLaMA and the GPT series. It is composed of a multi-head attention block and a feed-forward network (NN). DeepSeek's key architectural innovation lies in replacing both of these standard sub-components with more efficient and powerful alternatives. As illustrated in Figure 1.4, the standard multi-head attention is replaced by Multi-Head Latent Attention (MLA), and the feed-forward network is replaced by a DeepSeek-Mixture-of-Experts (MoE) structure. Figure 1.4 A simplified view of the DeepSeek model architecture. It modifies the standard Transformer by replacing the core components with Multi-Head Latent Attention (MLA) and a

Page 13

Mixture-of-Experts (MoE) layer. This design also utilizes RMS Norm (Root Mean Square Normalization) and a specialized Decoupled RoPE (Rotary Position Embedding).

Page 14

(This page has no text content)

Page 15

These two architectural changes, MLA and DeepSeek-MoE, are targeted solutions to major challenges in scaling LLMs. On top of these architectural changes, DeepSeek introduces advanced training and efficiency techniques. A new training objective called Multi-Token Prediction (MTP) improves learning and inference speed, while FP8 quantization (an 8-bit floating-point format) addresses computational efficiency and resource utilization. Together, these four innovations MLA, MoE, MTP, and FP8 quantization form the pillars of DeepSeek’s technical advancement. Each one is a targeted solution to a different fundamental challenge in scaling language models:quantization for inference) to push efficiency to the limit.. Each of these addresses a different problem area: MLA tackles the speed and memory bottleneck in attention for long sequences. MoE tackles the scaling and model capacity issue MTP improves learning and inference speed by predicting more than one token at a time FP8 quantization addresses the computational efficiency and resource utilization problem. 1.2.2 Training Beyond the core architecture, DeepSeek also innovates in how the model is trained and refined. The training pipeline is carefully designed to make large- scale training as efficient as possible. For example, DeepSeek employs an optimized scheduling strategy (internally nicknamed DualPipe) that overlaps different training tasks to keep hardware utilization high. In practice, this means data loading, preprocessing, and neural network computations are coordinated so that the GPU is never idle. When one batch is being processed by the model, the next batch is being prepared in parallel. Figure 1.5 An illustration of the DualPipe training pipeline on a single device. By overlapping the forward pass (the initial blocks), backward pass (the hatched blocks), and combined

Page 16

computations, this scheduling strategy minimizes GPU idle time and maximizes hardware utilization during large-scale training. Figure 1.5 shows a timeline of Device 1 running tasks in a dual-pipe pipeline. The process begins with the forward pass, followed by the backward pass. Crucially, the pipeline then enters a steady state where the forward pass of a new batch is performed concurrently with the backward pass of the previous one (represented by the cross-hatched blocks). This overlapping ensures the device remains fully utilized throughout the training process. 1.2.3 Post-training The base model, which was trained by DeepSeek, was called DeepSeek-v3. DeepSeek-v3 went through several post-training steps, ultimately resulting in DeepSeek-R1. These steps are shown in figure 1.6. Figure 1.6 The multi-step post-training pipeline used to create DeepSeek-R1 from the DeepSeek- V3 base model. This process involves a combination of reinforcement learning (Pure RL), data generation (Rejection sampling), and fine-tuning to instill advanced reasoning capabilities.

Page 17

Below is a very simplified explanation of the five steps involved in DeepSeek R1 post-training: Step 1 (Foundation): Start with a lightly fine-tuned base (DeepSeek- V3) using a relatively small “cold-start” dataset. Step 2 (Pure RL): Implement reinforcement learning algorithms to allow the model to explore and develop reasoning patterns through trial- and-error learning. This unsupervised approach enabled the model to discover effective problem-solving strategies without explicit human guidance. Step 3 (Self-Labeling): Introduce rejection sampling techniques. The model generated multiple candidate responses and selected the highest-

Page 18

quality outputs to create its own synthetic training data. Step 4 (Blending Data): Merge this synthetic data with supervised examples to balance quality with domain breadth. Step 5 (Final RL): The training concluded with a comprehensive reinforcement learning phase using diverse prompt distributions. This final optimization step enhanced the model's robustness and generalization across various task categories and input formats. Figure 1.7 is taken from the DeepSeek-R1 paper, which was released in January 2025. Here we can see that DeepSeek-R1 is on par with or performs better than OpenAI reasoning models on several benchmarks that were tested. Figure 1.7 Benchmark performance of DeepSeek-R1 against other leading models (as of January 2025).

Page 19

Another important post-training technique is knowledge distillation and model compression. The idea is to take the large “teacher” model (the full DeepSeek) and compress its knowledge into one or several smaller, more practical “student” models. This has been shown in figure 1.8. Figure 1.8 The concept of knowledge distillation. A large, powerful "teacher" model (like DeepSeek-R1) is used to generate training data to teach a much smaller, more efficient "student" model, transferring its capabilities without the high computational cost.

Page 20

The DeepSeek R1 model was based on DeepSeek v3, which had around 671 billion parameters. When the DeepSeek paper was released, the team also released distilled models that were as low as 1.5 billion parameters in size. In particular, DeepSeek open-sourced 1.5B, 7B, 8B, 14B, 32B, and 70B checkpoints based on Qwen2.5 and Llama3 series to the community. These smaller models are highly performant and efficient. In summary, DeepSeek’s roadmap consists of innovations at multiple levels: Novel architectural components (MLA and MoE)