Page
1
(This page has no text content)
Page
2
Praise for AI Agents: The Definitive Guide This book brilliantly consolidates fragmented AI agent research into an exhaustive blueprint for building production-ready agents. Chapters on cost optimization and threat modeling deliver the definitive manual for enterprise developers building secure agents at scale. —Vineeth Kalluru, enterprise AI architect and former engineer at Google and SambaNova Systems AI Agents is an excellent, comprehensive guide for anyone wishing to deploy AI agents. Nicole does a fantastic job explaining high-level concepts while also diving deep into open source frameworks, tools, and real example code you can actually use. —Joe Papa, author of PyTorch Pocket Reference This book treats AI agents as engineered systems, away from the hype. It offers rare clarity on architecture, orchestration, and evaluation, making it especially valuable for practitioners moving agent prototypes into real- world systems. —Ammar Mohanna, AI consultant and lecturer This is the part of AI agents that needs more attention: not just building demos, but actually operating them reliably in production. —Ana Pedra, Cloud DevSecOps engineer and AI engineer Love the book’s focus on what becomes critical once agents move into real products. That’s the gap most agent content refuses to cross. —Paolo Perrone, ML engineer with a plot twist The deployment and evaluation side is where things start getting real very quickly. Getting an agent demo working is one thing. Keeping it reliable once users, edge cases, fallback logic, and long-running workflows all interact together is a completely different challenge. —Matúš Paško, AI consultant and solution architect designing enterprise AI solutions
Page
3
Not just demos...real products. Nicole demonstrates the exact mindset shift the industry desperately needs right now. —Ali Karimian, clinical AI engineer, medical AI and clinical decision support systems, RAG systems, LLMs, and multi-agent AI architect I’ve been enjoying this book. It’s an excellent follow-on to books such as Hands-On Large Language Models and AI Engineering. —Chris Hughes, securing agentic AI, Zenity, and founder, Resilient Cyber AI Agents: The Definitive Guide is the best book on the subject. And Nicole’s live course is an excellent follow-up. —Chris Villanueva, cofounder and technical solutions architect
Page
4
AI Agents: The Definitive Guide Design, Deployment, and Evaluation for Production Nicole Koenigstein Foreword by Ajit Jaokar
Page
5
AI Agents: The Definitive Guide by Nicole Koenigstein Copyright © 2026 Nicole Koenigstein. All rights reserved. Published by O’Reilly Media, Inc., 141 Stony Circle, Suite 195, Santa Rosa, CA 95401. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (https://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Nicole Butterfield Development Editor: Michele Cronin Production Editor: Elizabeth Faerm Copyeditor: Emily Wydeven Proofreader: Dave Awl Indexer: BIM Creatives, LLC Cover Designer: Susan Brown Cover Illustrator: José Marzan Jr. Interior Designer: David Futato Interior Illustrator: Kate Dullea September 2026: First Edition
Page
6
Revision History for the First Edition 2026-08-26: First Release See https://oreilly.com/catalog/errata.csp?isbn=9798341666931 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. AI Agents: The Definitive Guide, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 979-8-341-66693-1 [LSI]
Page
7
Foreword I am pleased to write this foreword for Nicole’s book. We are using this book as course material in our University of Oxford Agentic AI workflows course, and it’s incredibly gratifying to see this work finally out in the world. The history of software is, in many ways, the history of abstraction. Each major shift in computing has introduced a new layer that changed not only what we could build, but how we thought about building it. From this perspective, agentic AI is just the latest profound abstraction in our industry. With agentic AI, AI has moved beyond raw foundation models. By incorporating goal-driven behavior, reasoning, memory, evaluation, governance, and context, we are now able to automate entire end-to-end workflows across enterprises. Organizations are rapidly moving beyond simple demos to build production-grade systems. In other words, agentic AI has officially become an engineering problem. This is why AI Agents: The Definitive Guide arrives at such an important moment. At an architectural level, builders grapple with a complex web of execution challenges: State and persistence: how should an agent maintain state across a long-running task? Memory architecture: how should memory be structured, indexed, and retrieved? Multi-agent collaboration: how do multiple specialized agents collaborate text here effectively? Holistic evaluation: how do we evaluate behavior that unfolds across dozens of autonomous decisions, rather than a single static response? Security and sandboxing: how do we secure systems that have agency to invoke tools, access sensitive data, and execute actions in external environments?
Page
8
Day-two operations: how do we monitor, debug, govern, and continuously improve these systems over time? One of the book’s strengths is its insistence on treating capability and reliability as equal partners. The post-deployment phase is where true engineering begins. This is where we must answer for how these systems scale, how they fail, how risks are mitigated, and ultimately, how trust is established. These considerations are especially urgent today for the enterprise. As agentic AI scales, the core challenge shifts from building a single agent to governing a rapidly evolving ecosystem of agents continuously and in real time. Because agents are inherently nondeterministic, organizations will inevitably struggle not just with their unpredictable nature, but with the sheer scale and proliferation of “nonengineer-led” agents emerging across business units. To survive this sprawl, agents must be managed across their entire lifecycle, typically via a governance layer that sits above individual development platforms. Governance is no longer a static, one-off approval process; it must become a continuous, close-looped system: Telemetry answers the vital question, “What is actually happening in production right now?” Evaluations act as the dials, establishing the boundaries of acceptable behavior. Feedback loops (drawing on classic control theory) use this telemetry data to dynamically adapt and improve the system. Implementing these complex, autonomous systems requires a rigorous approach. Because this landscape moves incredibly fast, tools will inevitably change. However, the fundamental principles Nicole outlines in this book are timeless. AI Agents: The Definitive Guide is the map you need to navigate this next frontier of software engineering. Ajit Jaokar
Page
9
Course director Artificial Intelligence – AI Engineering University of Oxford
Page
10
Preface When I first started working with transformers in 2019, AI agents, at least in the form we understand them today, were not yet the central topic they have since become. I was immediately drawn to the transformer architecture itself: the elegance of attention, its versatility, and the way the same underlying principles could extend across language, vision, audio, time series, and other modalities. Two years later, I built my own attention-based architecture around specialized modules that activate selectively, operate independently, and communicate only when the context requires it. Although I designed the model for dynamic prediction, the questions behind it now feel strikingly familiar: which components should act, what information should they share, and how should context shape what happens next? What continues to fascinate me is how those same ideas have since moved from individual model architectures into complete agentic systems, where models reason, use tools, maintain state, interact with their environments, and coordinate with other agents. The level of abstraction has changed, but many of the underlying principles remain the same. My own work continues to evolve alongside that shift, from attention-based and multimodal architectures toward coordination, adaptive systems, and AI agents. Today, my research and development focus on how these systems can learn and improve through reinforcement learning, search, memory, and adaptive coordination, a direction that also shapes my PhD research on self- improving AI agents. What makes agents so compelling to me is also what makes them difficult: a capable model does not automatically become a capable agent. Once a model is expected to act, harder questions emerge. What should it remember? How should it use tools safely? How do we evaluate behavior across an entire trajectory and the wider system? What happens when tools fail, costs multiply, or several agents begin working at
Page
11
cross-purposes? The production realities behind those questions shaped this book. As you read through this book, I’ll share the mental models I use to reason about those questions, together with the lessons that only become visible once systems leave the controlled environment of a demo. My goal is not only to show you what works, but to help you understand why it works, where it can fail, and how to recognize those problems early, so you don’t have to learn every lesson the hard way. Although the book you’re holding is titled AI Agents: The Definitive Guide, nothing in AI agents remains truly definitive for long. Models, frameworks, and interfaces will continue to change. What remains far more durable is a deep understanding of how agentic systems work beneath the surface and how their individual parts shape the behavior of the whole. That’s why I wrote this book around the inner workings of AI agents. By understanding how state, reasoning, tools, memory, evaluation, infrastructure, and governance interact, you gain the depth of understanding needed to carry your own use cases from initial design into reliable production systems.
Page
12
What This Book Is About This book is a practical systems guide to designing, building, deploying, and evaluating AI agents for production. It begins with the architectural foundations that make agentic behavior possible: state, control flow, planning, reasoning, reflection, and action. From there, it moves into increasingly complex systems, including human- in-the-loop workflows, hierarchical agents, swarms, search-based reasoning, reinforcement learning, and test-time compute. The book also looks at the models behind the agents and the trade-offs involved in choosing between decoder-only, encoder-only, mixture-of- experts, reasoning, open-weight, and closed models. You’ll see how model choice, quantization, adapters, and inference constraints shape the behavior and feasibility of the larger system. A substantial part of the book focuses on the transition from prototype to production. You’ll learn how to structure data contracts, validate outputs, integrate tools, use the Model Context Protocol, isolate execution, and govern programmatic tool calls. The later chapters address deployment, monitoring, fallbacks, inference backends, caching, evaluation, memory, infrastructure, cost, and threat modeling. Evaluation is treated as part of the full agent lifecycle. This includes stress testing before deployment, observing behavior in production, turning traces into custom benchmarks, and assessing long-horizon reasoning, tool use, and system reliability. Throughout the book, theory is paired with concrete implementations, architectural patterns, production considerations, and the trade-offs that become visible only when agentic systems are expected to operate under real conditions.
Page
13
What This Book Is Not This is not an introduction to large language models (LLMs). I assume that you already understand the basics of LLMs, embeddings, inference, and modern language models. Relevant concepts are explained where necessary, but the focus is on how those models become part of larger agentic systems. This is also not a prompt engineering guide or a collection of API recipes. Prompts and instructions matter, but they’re only one part of an agent. This book provides the foundation for designing, evaluating, securing, and deploying production agents, including state, planning, tools, contracts, memory, execution boundaries, infrastructure, and governance. Nor is this a framework manual. LangGraph, Pydantic, MCP, inference backends, and other tools appear throughout the book to make the examples concrete, but my goal is not to teach you one implementation stack. Frameworks and interfaces will change. My emphasis is on the architectural and engineering principles that allow you to evaluate new tools and apply them to your own systems. Finally, this book is also not a collection of polished demonstrations that stop once an agent successfully calls a tool. It focuses on what happens or fails after your prototype works: reliability, observability, failure recovery, security, cost, deployment, and long-term maintainability.
Page
14
Who This Book Is For This book is written for intermediate to advanced software engineers, data scientists, machine learning engineers, MLOps engineers, AI and DevOps specialists, and AI infrastructure engineers who are responsible for designing, building, deploying, and maintaining LLM-powered agents in production. It’s for readers dealing with the realities that appear once an agent moves beyond a controlled prototype. Specifically, I wrote this for: The engineer who needs to turn an agent prototype into a reliable production system and troubleshoot what happens when state breaks, tools fail, latency increases, or costs begin to scale The AI infrastructure engineers who are directly responsible for designing, developing, deploying, and maintaining AI agents powered by LLMs in production environments The architect who needs to decide how models, tools, memory, control flow, evaluation, security, and infrastructure should work together across the full system The technical leader who needs to assess feasibility, estimate cost, guide implementation teams, and understand the trade-offs between performance, reliability, security, and autonomy The deep-tech founder or CTO who needs enough technical depth to evaluate solutions, set realistic expectations, and make decisions about how an agentic product can be built and operated sustainably A working understanding of Python and large language models is assumed. Experience with cloud platforms can be helpful, but isn’t required. It’s also useful if you understand how transformers process sequences and have at least a basic sense of how embeddings work or vector stores support retrieval.
Page
15
Throughout these chapters, I don’t shy away from the complexity that production agent systems demand. Technical depth, however, will always serve a practical purpose and I pair it with clear implementations and concrete examples. The book focuses on the details that emerge when agents are expected to operate under real conditions, because those are the details that determine whether a system merely works in a demo or can be maintained in production. Navigating This Book The book is organized to move from the foundations of agent behavior to the challenges of operating agentic systems in production. Chapter 1, “From LLMs to Agents: The Foundational Blueprint”, introduces the transition from static language models to stateful agents, using finite and hierarchical state machines to establish the foundations of control flow and agent behavior. Chapter 2, “Architectures and Patterns: Planning, Reactivity, and Multi- Agent Systems”, explores common agent architectures and patterns, including structured reasoning, reflection, human-in-the-loop workflows, hierarchical systems, and swarms. Chapter 3, “Advanced Planning, Reasoning, and Scalable Execution in Agents”, moves into more advanced reasoning and learning, covering rewards, relative and absolute evaluation, reinforcement learning, test-time compute, search, and adaptive Monte Carlo Tree Search. Chapter 4, “Models Behind the Agents: Capabilities and Optimization”, examines the models behind agentic systems, including decoder-only, encoder-only, mixture-of-experts, and reasoning models, together with open and closed models, adapters, and quantization. Chapter 5, “From Prototypes to Production: Contracts, Tools, and Reliable Execution”, focuses on the move from prototypes to reliable execution through structured contracts, output validation, tool integration, the Model Context Protocol, server discovery, and planning-oriented deep agents.
Page
16
Chapter 6, “Secure Execution and Tool Governance”, addresses secure execution and tool governance, including sandboxing, runtime isolation, cloud coding environments, and safeguards for programmatic tool calling. Chapter 7, “Deploying Agents in Real Products”, covers the transition from MVPs to production systems, with a focus on monitoring, resilience, fallbacks, inference backends, cold starts, and caching. Chapter 8, “Foundational Evaluation and Operational Observation of Agentic Systems”, and Chapter 9, “Customized and Advanced Evaluation of Agentic Systems”, focus on evaluation across the agent lifecycle, from pre-deployment stress testing and post-deployment observation to custom benchmarks, production traces, and long-horizon reasoning across visual inputs. Chapter 10, “Agent Memory: How Persistence Turns Agents into Evolving Systems”, examines short-term, long-term, episodic, semantic, and procedural memory, together with memory hygiene and memory design in multi-agent systems. Chapter 11, “From Compute to Cost: Designing Efficient Agentic Systems”, turns to infrastructure and cost, including model memory, FlashAttention, inference economics, and the reuse of agentic state transitions. Chapter 12, “Threat Modeling for AI Agents”, concludes with threat modeling, layered defenses, red teaming, and system-level protection across the agent lifecycle. Each chapter is largely self-contained, so you can read selectively depending on your interests. However, the material builds conceptually, and reading sequentially provides the clearest view of how the different layers of an agentic system fit together. The chapters ahead take you from the structures that make responsible agency possible to the details that determine whether an agentic system can operate reliably in production. The difference between a convincing demonstration and a dependable system lies in how state is managed,
Page
17
actions are constrained, failures are observed, costs are controlled, and behavior is evaluated over time. Now it’s your turn. From state and reasoning to tools, execution, evaluation, memory, infrastructure, and security, the system is yours to design. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values determined by context. TIP This element signifies a tip or suggestion. NOTE This element signifies a general note.
Page
18
WARNING This element indicates a warning or caution. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://oreil.ly/github-ai-agents. If you have a technical question or a problem using the code examples, please send email to support@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “AI Agents: The Definitive Guide by Nicole Koenigstein (O’Reilly). Copyright 2026 Nicole Koenigstein, 979-8-341-66693-1.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com.
Page
19
O’Reilly Online Learning NOTE For more than 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 141 Stony Circle, Suite 195 Santa Rosa, CA 95401 800-889-8969 (in the United States or Canada) 707-827-7019 (international or local) 707-829-0104 (fax) support@oreilly.com
Page
20
https://oreilly.com/about/contact.html We have a web page for this book, where we list errata and any additional information. You can access this page at https://oreil.ly/ai-agents-the- definitive-guide. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly. Watch us on YouTube: https://youtube.com/oreillymedia. Acknowledgments A book about AI agents may carry a single author’s name, but the thinking behind it is shaped by far more than one person. That is particularly true in a field evolving as quickly as AI, where ideas are constantly refined through discussion, experimentation, teaching, and exchange. At the same time, much of the work still takes place in long stretches of solitary research, coding, and writing, which makes the support, questions, and outside perspectives of others all the more important. For that reason, I would like to dedicate these acknowledgments to the people whose ideas, questions, and contributions helped shape this book along the way. I am grateful to the colleagues, peers, students, readers, and course participants who have engaged with my work. Their questions, perspectives, and willingness to explore emerging ideas have repeatedly challenged me to explain concepts more clearly, examine assumptions more carefully, and develop the ideas in this book further. I would especially like to thank Nicole Butterfield, who first planted the idea for this book. Working on this second book with O’Reilly showed me how much I enjoy writing about AI and how much I value this collaboration. I would like to thank Michele Cronin and Liz Faerm for helping turn that idea into the book you are now reading. My thanks also go