Reinforcement Learning from Human Feedback Alignment and post-training of LLMs (Nathan Lambert) (z-library.sk, 1lib.sk, z-lib.sk)

Page 1

M A N N I N G LLM alignment and post-training Nathan Lambert Foreword by Thomas Wolf

Page 2

Reinforcement Learning from Human Feedback

Page 3

(This page has no text content)

Page 4

Reinforcement Learning from Human Feedback LLM ALIGNMENT AND POST-TRAINING Nathan Lambert Foreword by Thomas Wolf M A N N I N G SHELTER ISLAND

Page 5

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com © 2026 Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. ∞ Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. The author and publisher have made every effort to ensure that the information in this book was correct at press time. The author and publisher do not assume and hereby disclaim any liability to any party for any loss, damage, or disruption caused by errors or omissions, whether such errors or omissions result from negligence, accident, or any other cause, or from any usage of the information herein. Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Development editor: Elesha Hyde Review editor: Kishor Rit Technical editor: Kamron Mehradian Production editor: Keri Hales Copy editor: Tiffany Taylor Proofreader: Jason Everett Typesetter: Ammar Taha Mohamedy Cover designer: Marija Tudor ISBN 9781633434301 Printed in the United States of America

Page 6

brief contents Part 1 Overview .......................................................... 1 1 Introduction 3 2 A tiny history of RLHF 14 3 Training overview 18 Part 2 Core training methods ...................................29 4 Instruction fine-tuning 31 5 Reward modeling 38 6 Reinforcement learning 56 7 Reasoning and inference-time scaling 99 8 Direct-alignment algorithms 112 9 Rejection sampling 124 Part 3 Data and preferences .................................. 133 10 The nature of preferences 135 11 Preference data 144 12 Synthetic data 158 vii

Page 7

viii BRIEF CONTENTS Part 4 Applications and advanced topics ............... 175 13 Tool use and function calling 177 14 Over-optimization 188 15 Regularization 196 16 Evaluation 207 17 Crafting model character and products 218

Page 8

contents foreword xiv preface xvi acknowledgments xviii about this book xx about the author xxvi about the cover illustration xxvii Part 1 Overview ...........................................................1 1 Introduction 3 1.1 RLHF in three steps 4 1.2 What does RLHF do? 5 1.3 Walkthrough of an RLHF recipe 8 1.4 An intuition for post-training 9 1.5 How we got here 11 1.6 Future of RLHF 12 2 A tiny history of RLHF 14 2.1 Origins to 2018: RL onpreferences 15 2.2 2019 to 2022: RL from human preferences on language models 15 2.3 2023 to the present: TheChatGPT era 16 ix

Page 9

x CONTENTS 3 Training overview 18 3.1 Problem formulation 19 A simple example: The thermostat 20 Classic RL example: CartPole 20 Manipulating the standard RL setup 22 Fine-tuning and regularization 23 Optimization tools 24 Subtle advantages of RL in post-training language models 24 3.2 Canonical training recipes 25 InstructGPT 25 Tülu 3 26 DeepSeek R1 27 Part 2 Core training methods ................................... 29 4 Instruction fine-tuning 31 4.1 Chat templates and the structure of instructions 32 4.2 Best practices for instruction tuning 35 4.3 Implementation details 36 5 Reward modeling 38 5.1 Training a Bradley–Terry rewardmodel 39 5.2 Implementation example 42 5.3 Reward model variants 44 Preference margin loss 44 Balancing multiple comparisons per prompt 44 K-wise loss function 45 5.4 Outcome reward models 45 5.5 Process reward models 48 5.6 Comparing reward model types (and value functions) 51 Inference across reward model types 52 5.7 Generative reward modeling (a.k.a. LLM-as-a-judge) 53 5.8 Further reading 54 6 Reinforcement learning 56 6.1 The role of reinforcement learning in RLHF 57 6.2 Policy gradient algorithms 58 Deriving the policy gradient 60 Vanilla policy gradient 63 REINFORCE 63 REINFORCE leave one out 65 Proximal policy optimization 66 Understanding the PPO objective 68 Value functions and PPO 71 Group relative policy optimization 73 Group sequence policy optimization 75 Clipped importance sampling policy optimization 77 Comparing algorithms 78

Page 10

CONTENTS xi 6.3 Implementation 80 Policy-gradient basics 81 Loss aggregation tradeoffs 81 Asynchronous RL systems 86 Truncated importance sampling 88 Example: PPO 89 Example: GRPO 92 RLOO vs. GRPO 93 6.4 Auxiliary topics 94 Generalized advantage estimation 94 Double regularization 96 6.5 Further reading 97 7 Reasoning and inference-time scaling 99 7.1 The role of RLVR 100 7.2 The origins of new reasoningmodels 104 Why does RL work now? 104 RL training vs. inference-time scaling 104 The future (beyond reasoning) of RLVR 105 7.3 Understanding reasoning trainingmethods 105 Reasoning research before OpenAI o1 or DeepSeek R1 106 Early reasoning models 106 Common practices in training reasoning models 108 7.4 Looking ahead 110 8 Direct-alignment algorithms 112 8.1 Direct preference optimization 113 How DPO works 113 DPO derivation 115 8.2 Numerical concerns, weaknesses, and alternatives 119 8.3 Implementation details 121 8.4 DAAs with synthetic preference data 122 8.5 DAAs vs. RL:Online vs. offline data 122 9 Rejection sampling 124 9.1 Training process, step by step 125 Generating completions 126 Scoring completions 126 Fine-tuning 129 9.2 Implementation details 129 9.3 Related: Best-of-N sampling 130 Part 3 Data and preferences ................................... 133 10 The nature of preferences 135 10.1 When preference replaces correctness 136 10.2 The origins of RLHF and preferences 137

Page 11

xii CONTENTS 10.3 Specifying objectives: From logic of utility to reward functions 138 10.4 Tools for optimizing utility 139 10.5 Complexity of optimizing preferences 141 11 Preference data 144 11.1 Why we need preference data 145 11.2 Collecting preference data 145 Interfaces 146 Rankings vs. ratings 149 Multiturn data 151 Structured preference data 152 Sourcing and contracts 153 11.3 Bias: Things to watch out for in data collection 155 11.4 Open questions in RLHF preference data 156 12 Synthetic data 158 12.1 The roles of synthetic data 159 12.2 Distillation with synthetic data 160 12.3 The path to on-policy, teacher–student distillation 161 Adapting knowledge-distillation for LMs 162 From offline to on-policy distillation 163 Modern OPD variants 164 12.4 AI feedback 165 Balancing AI and human feedback data 166 Building specific LLMs for judgment 167 12.5 Constitutional AI 168 Further reading on CAI 168 12.6 Rubrics: Prompt-specific AI feedback for training 169 Part 4 Applications and advanced topics ................175 13 Tool use and function calling 177 13.1 Tool-use overview 178 13.2 Interweaving tool calls in generation 180 13.3 Multistep tool reasoning 183 13.4 Model Context Protocol 183 13.5 Implementation details 184 14 Over-optimization 188 14.1 Qualitative over-optimization 190 Managing proxy objectives 190 Over-refusal and “too much RLHF” 191

Page 12

CONTENTS xiii 14.2 Quantitative over-optimization 193 14.3 Misalignment and the role of RLHF 194 15 Regularization 196 15.1 KL divergence in RL optimization 197 Reference model to generations 198 Implementation example 198 15.2 Implicit regularization 199 SFT memorizes, RL generalizes 199 Retaining by doing: On-policy data mitigates forgetting 200 RL’s razor: Why online RL forgets less 203 15.3 Other types of regularization 205 Pretraining gradients 205 Margin-based regularization 206 16 Evaluation 207 16.1 Prompting formatting 209 Few-shot prompting and log-likelihood scoring 209 Chain-of-thought prompting 211 Zero-shot instruction following 212 Reasoning-era evaluation prompts 212 16.2 Why many external evaluation comparisons are unreliable 213 16.3 How labs actually use evaluations internally to improve models 214 16.4 Contamination 216 16.5 Tooling 217 17 Crafting model character and products 218 17.1 Character training 219 Persona vectors 221 The assistant axis 223 Persona subnetworks 225 17.2 Model specifications 226 17.3 Product cycles andwhat’s next for RLHF 228 appendix A Definitions 229 appendix B Beyond “just style” 234 appendix C Practical issues 240 References 245 index 273

Page 13

foreword There is a gap in AI that almost nobody talks about openly: between what the leading, closed labs know and what is written down in public. A technique becomes standard practice at a few for-profit organizations. It gets a paragraph in the “Methods” section of a paper. The practitioners who implemented it move on. A year later, a team somewhere else spends four months reproducing what could have been explained in an afternoon. Nobody wrote about it. It wasn’t anyone’s job. Nathan Lambert made it his role to communicate and document this science, and this book is the summary of all the attempts he has made to do so regarding large language models. Nathan joined Hugging Face in May 2022. We had just finished BLOOM—the biggest fully open language model run to date—with a coalition of hundreds of researchers across dozens of institutions; the science team was new, and nobody quite knew what came next. Nathan came to Paris that summer for the first science off-site. Over the year and a half that followed, across the eight- or nine-hour time zone gap between Europe and the U.S. West Coast, we had a lot of conversations about running, about research, about where the field was heading, and about what it meant to do good science when everything was moving faster than anyone could write it down. RLHF itself was not new. Christiano and collaborators had formulated preference- based deep RL back in 2017, Ziegler had applied it to language models in 2019, and InstructGPT put the full pipeline on paper earlier in 2022. What was new was the sudden, borderline-intellectually violent arrival of ChatGPT in November 2022. That shock pulled in the rest of the field. And once everyone wanted to do RLHF, the hard reality surfaced: almost none of the practitioner knowledge required to actually do it was written down anywhere public. xiv

Page 14

FOREWORD xv Chapter 9 is a perfect example of unexpected gaps in the literature: rejection sampling, one of the most widely used techniques in post-training (e.g., used in WebGPT, Llama, etc.). No canonical implementation anywhere. Little explanation of why it works. Everyone doing it, nobody writing it down. Until now. When Nathan and I caught up earlier this year, he described his time at Hugging Face as having “crystallized” many ideas he had only partially thought through during grad school. The word stuck with me. This book is also a crystallization—not only of Nathan’s own thinking, tutorials, and writing, but of a field that built itself by doing and never quite got around to writing down what it had built. Open source doesn’t just need code. It needs the knowledge that makes the code make sense. The “open science.” Publishing a library matters because doing so shares the understanding locked inside people’s heads, making it available to everyone, not only to the people who happened to be in the right room at the right time. What Nathan has done here—from reward models through the philosophical question of what a preference even is—is open the doors of these rooms to everyone. He writes with the clarity of someone who has had to explain the same concept many different ways, from live tutorials to blog posts, until he found the explanation that clicked. A reasonable worry about any book written over several years in this fast-paced field is that it will age out before it ships. This one didn’t and likely will not. The early chapters on the RLHF pipeline, the reward model, and policy gradients are the clean version of a story that is now rather well understood. But the book continues to all the topics that matter right now, in all their openness: chapter 7 on reasoning models and inference-time scaling, chapter 10 on the nature of preferences, chapter 11 on the opaque reality of collecting preference data at scale, chapter 12 on synthetic data and AI feedback, and chapters 16 and 17 on evaluation and the emerging craft of shaping model character. These are the frontiers today—and the book treats them with the same care as the foundations, even as they approach the frontier of our knowledge. That has always been what open research and open science are for. Nathan left Hugging Face for Ai2 in September 2023. He finished the book across winter breaks, in the margins of other jobs. The field kept moving. He wrote it anyway, and we are all lucky for his persistence. —Thomas Wolf Co-founder & Chief Science Officer, Hugging Face

Page 15

preface I became interested in AI during graduate school at UC Berkeley, where I arrived in the summer of 2017 and learned how deep learning was taking the computer science world by storm. I had no idea it was the same summer that the famous “Attention Is All You Need” paper, which introduced the Transformer, was released. Years later, after earning my Ph.D., I was working at the machine learning platform Hugging Face when ChatGPT captured the world’s attention. This brings us to the topic of the book: reinforcement learning from human feedback (RLHF). RLHF burst onto the scene following the release of ChatGPT, serving as the crucial added technique that transformed GPT-3.5 into the ChatGPT we fell in love with. Over the last few years, I’ve been doing open research building models like ChatGPT, and I’m consistently shocked by how little of the information about how to do this is public—even basic definitions for common training algorithms. The goal of this book is to be the canonical reference for RLHF as it matures into an established area of research. This is the book I wish I’d had when I started learning about RLHF for language models almost three years ago. In this time, I’ve watched first-hand as RLHF—and post-training, which has emerged as the term of art for related methods—has moved from a niche research topic to one of the central mechanisms shaping how modern AI systems behave. As models have scaled, the role of human data has evolved in the process. The result is a fast-developing field where research labs, open source communities, and companies of every size are continually experimenting, iterating, and publishing new methods. Getting started in RLHF can feel unnecessarily complicated. The practice blends ideas from reinforcement learning, supervised learning, human–computer inter- xvi

Page 16

PREFACE xvii action, philosophy, economics, and even practical software engineering. Most expla- nations in the wild assume long-standing familiarity with these ingredients. This book was created to make RLHF more approachable, providing a clear map of how RLHF works in practice and how it continues to evolve. This book has entry points for many different readers, from those who like a simple, text-only experience discussing the trade-offs of ML algorithms, to those who need code examples to ground their thinking, to those looking for a more theoretical approach. The hope is that this book becomes a reliable reference as you explore a field that is evolving quickly around it. Thank you for joining this exploration of RLHF. I’m excited to get a polished encapsulation of much of my hard work over the last few years into the hands of the growing AI community. May it earn a lasting place on both your digital and physical shelves.

Page 17

acknowledgments First and foremost, thank you to my partner, Helena, for her unending support of this crazy project of mine during the intense early language modeling era post-ChatGPT. Without Helena and my immediate family, I would lack the confidence to take on these seemingly impossible goals. Helena had great help from Phoebe, our dog, to distract me from rabbit holes that went a little too deep. Thank you to Caithrin Rintoul for pushing me through the process of professional- izing this book and getting the physical edition to exist. Thank you to Mia Ocolisanu for keeping me on top of the logistics and managing the multiparty communications and coordination process that is the book publishing process. Thank you to Florian Brand for helpingme read the pulse of the community when iterating on and evolving the book. Thank you to my core colleagues at Ai2 who helped me learn all this material and put it to good use. This phase of my learning about language models and sharing it with the world wouldn’t be close to a success without you. This core group includes Luca Soldaini, Kyle Lo, Dirk Groeneveld, Hanna Hajishirzi, Saumya Malik, and Ali Farhadi, but there are many, many others. Thank you to the following friends who helped me directly with the technical side of this project: Costa Huang, Ross Taylor, Hamish Ivison, John Schulman, Lewis Tunstall, Ed Beeching, Thomas Wolf, Sasha Rush, Valentina Pyatkin, Jacob Morrison, Daniel Han, Shane Gu, Joanne Jang, LJ Miranda, Andrew Carr, Cameron Wolfe, Sharan Maiya, Mikhail Noukhovitch, and others in my online network. Additionally, thank you to the contributors on GitHub who helped improve this project (https://github.com/natolambert/rlhf-book/graphs/contributors). Particu- larly, Zafir Stojanovski made substantial contributions to the book without prompting xviii

Page 18

ACKNOWLEDGMENTS xix from me, completely in theme with the style, including minimal implementations of the reinforcement learning algorithms covered in chapter 6. Numerous other people found small typos and problems and massively accelerated the editing process. Thanks as well to all the additional reviewers: Alex Zalesov, Arun Prakash A, Dmitrii Volkov, Edoardo Cecchinato, Ekkehard Schnoor, Fabio Montagna, Felipe Almei- da, Hamza Farooq, Hongming Zheng, Ignacio Cases, Karthik Ravichandran, Lesia Yanytska, Lindo William Khoza, Maziyar Panahi, Muhammad Ali Shafique, Salvatore Raieli, Saurabh Arjun Sawant, Sergey Polyashov, Sky Hester, Sophia Willows, Stepan Plotytsia, Syed Baqir Ali, and Thomas Viehmann. Your suggestions helped make this a better book. Thank you to the Manning editorial team, including Elesha Hyde, for always bringing a positive attitude in response to my weird requests in general content edit- ing, and to Kamron Mehradian for technical feedback. Thanks also to the Manning production team, who helped shepherd this book into its final format. I am incredibly lucky to be given the full credit for this project, which is built so heavily on the effort and feedback of so many. I hope to repay these efforts by making the materials accessible, extensible, and a joy to use.

Page 19

about this book This book is a comprehensive overview of what RLHF is, how it is done, and how it fits into the broader methods of modern post-training. RLHF is both complicated to implement and intensely interdisciplinary. The book is a horizontal cut through all the fields and pieces that go into RLHF: reinforcement learning, theory, code, philosophy, data, UX, product, and so on, with a few key chapters that go into great detail on the core methods—the places where complexity is part of the process. To be truthful about what is done today, the focus is on the methods used, their fundamental motivations and implementations, and the intuitions behind them. The reality of building actually useful AI models is that doing so in practice is messy and challenging, and this book was written largely because there were no canonical references for important topics in the RLHF workflow. Given the pace of progress on LLMs overall, combined with the complex nature of collecting and using human data, RLHF is an unusually academic field in which published results are often noisy and hard to reproduce across multiple settings. To develop strong intuitions, you’re encouraged to read multiple papers on each topic rather than taking any single result as definitive. To facilitate this, the book includes numerous academic-style citations to the canonical reference for a claim. In some cases, multiple opinions on how RLHF should be done are presented: the future may bring more clarity about how to build certain models, but for now, the debate is the truth. The goal is to give you theminimumknowledge needed to try a toy implementation or dive into the literature. This is not a comprehensive textbook, which would be impossible to write with the state of the art in flux and often undocumented, but rather a quick book for reference and getting started in the field. xx

Page 20

ABOUT THIS BOOK xxi Who should read this book This book is intended for audiences with entry-level experience with language model- ing, reinforcement learning, and general machine learning. It does not provide exhaustive documentation for all the techniques, only those crucial to understanding RLHF. How this book is organized: A road map This book is organized around four major themes: an orientation to RLHF, core training methods, the data that drives them, and practical challenges that span the training pipeline. It begins with introductory material: chapter 1 provides a high- level overview of why people use RLHF and how to form an intuition for its effect on models. This section continues with key related works in chapter 2 to provide a minimal history of how scientific progress led to ChatGPT. Part 1 of the book concludes with chapter 3’s review of RL fundamentals and how RLHF changes things slightly to make these techniques work on language models. Chapters 4, 5, and 6 constitute the core of the classical RLHF portion. The “three- step training process” of instruction fine-tuning to make the model respond to the question–answer format (chapter 4) builds the foundation, then a reward model is trained to capture human preferences (chapter 5), and finally, themodel is optimized with RL against that reward (chapter 6). Part 2 also includes methods that make up the broader literature on post-training, including a chapter on reasoning and reinforcement learning with verifiable re- wards (RLVR; chapter 7); the simple, direct alignment algorithms (DAAs) like direct preference optimization that made RLHF accessible to a much broader audience (chapter 8); and rejection sampling, another simple method for RLHF used in some key models, such as Meta’s Llama 2 and 3 (chapter 9). Part 3 turns to the data that drives these methods. It opens with the question of what preferences actually are and why they matter (chapter 10), linking RLHF to many other fields of inquiry beyond technical AI research, and then covers how preference data is collected and curated in practice (chapter 11). As the field has matured, synthetic data generated by AI models has become increasingly central to training pipelines. Chapter 12 covers synthetic data generation and Anthropic’s constitutional AI (CAI), where models provide their own feedback guided by a set of principles. The final chapters address practical challenges that cut across the entire train- ing process. Chapter 13 (tool-use and function-calling) introduces how models are trained to interact with external tools and APIs to solve fundamental limitations of stochastic, static model weights. Chapter 14 (over-optimization) examines why reward hacking is a problem that can never be fully solved when teaching reward models to approximate human preferences (a proxy reward). Chapter 15 (regularization) presents the techniques—most notably, Kullback–Leibler penalties—used to keep that optimization in check. Chapter 16 (evaluation) surveys the evolving landscape