Build a Large Language Model (From Scratch) 从零开始构建大型语言模型 (Sebastian Raschka)（Z-Library）

【One-Line Pitch】 A hands-on guide that walks you through building a GPT-style large language model from scratch in PyTorch, covering data preparation, attention, pretraining, and fine-tuning. Best for Python-proficient learners, engineers, and researchers who want to understand LLMs by writing the code themselves rather than treating them as black boxes. 【Book Arc】 - **Opening (~0%–14%)**: Introduces what LLMs are, where they sit within AI/ML/deep learning, and the three-stage build plan (architecture + data prep → pretraining → fine-tuning). Frames the Transformer and the encoder-vs-decoder distinction (BERT vs GPT). - **Early (~14%–29%)**: Text processing fundamentals — tokenization, token IDs, special tokens, byte pair encoding, sliding-window sampling, token embeddings, and positional encoding. This stage solves the "how do we turn raw text into model-ready tensors" problem. - **Early–Middle (~29%–43%)**: The attention mechanism, built up step by step: simple self-attention, trainable self-attention, causal masking, dropout, and multi-head attention. This is the conceptual core of the Transformer. - **Middle (~43%–57%)**: Assembling a full GPT model — layer normalization, GELU feed-forward networks, shortcut connections, Transformer blocks, and text generation. Also covers parameter counts and storage requirements. - **Late (~57%–86%)**: Pretraining on unlabeled data — loss evaluation, training loops, decoding strategies (temperature, top-k), saving/loading weights, and loading OpenAI pretrained weights. Then fine-tuning for classification (e.g., spam detection). - **Ending (~86%–100%)**: Instruction fine-tuning — preparing supervised instruction datasets, batching, fine-tuning to follow human instructions, extracting and evaluating responses, plus closing thoughts on next steps and keeping up with the field. 【Key Takeaways】 - **The book follows a three-stage LLM build** (Opening): architecture and data preparation, pretraining to create a base model, then fine-tuning into a classifier or instruction-follower. This structure mirrors how real LLMs are actually developed. - **Tokenization is the foundation of everything downstream** (Early): byte pair encoding, special context tokens, and sliding-window sampling determine what the model actually sees. Getting this wrong undermines all later stages. - **Attention is built incrementally, not assumed** (Early–Middle): the book starts from a training-free self-attention example and progressively adds trainable weights, causal masks, dropout, and multi-head stacking. This makes the hardest concept approachable. - **A GPT model is assembled from understandable parts** (Middle): layer normalization, GELU activations, shortcut connections, and Transformer blocks are each implemented separately before being combined. This demystifies the architecture. - **Pretraining uses self-supervised next-word prediction** (Late): no labels are needed — the model generates its own training signal from raw text. Loss on training and validation sets measures generation quality. - **Decoding strategy controls output randomness** (Late): temperature scaling and top-k sampling let you trade off between deterministic and creative text generation. - **Fine-tuning splits into two practical paths** (Late–Ending): classification fine-tuning (e.g., spam detection with a classification head) and instruction fine-tuning (supervised instruction-answer pairs). Both reuse pretrained weights. - **Models are small but conceptually faithful** (Throughout): the book emphasizes that the small models you build use the same mechanisms as state-of-the-art LLMs, making them powerful educational tools. 【Reading Tips】 - **Read sequentially** — each chapter builds directly on the previous one, and skipping ahead will leave gaps in the implementation chain. - **Deep-read Chapters 2–4** (text processing, attention, GPT assembly) — these are the conceptual backbone; skim Chapter 1 if you already know the AI/ML landscape. - **Code along in Jupyter notebooks** — the book is designed for hands-on execution on a regular laptop, with GPU tips for scaling up. Don't just read the code; run and modify it. - **Use Appendix A if PyTorch is new to you** — it provides a concise introduction sufficient for following the book's examples. - **Treat fine-tuning chapters (6–7) as applied practice** — they show how the same base model adapts to classification and instruction-following, which is where the practical payoff lies. 【Coverage Limits】 The excerpts cover the book's structure, chapter topics, and stated goals in detail, but do not include actual code listings, mathematical derivations, or exercise content. Specific implementation details and results are not covered here.