Building Reliable AI Systems Applications and agents you can trust (Rush Shahani)（Z-Library）
Artificial Intelligence
This book shows you how to grow a promising prototype into a production product you can deliver, maintain, and scale. In 11 sharply-focused chapters, it walks you through a unique reliability process that author Rush Shahani refined while building an AI-powered sales intelligence platform used by over 15,000 go-to-market teams. Timely and practical, this book provides concrete steps to eliminate costly hallucinations, streamline computational resources, and confidently deploy secure, trustworthy, and compliant AI solutions. You’ll immediately appreciate how the book defines “AI reliability” along six critical dimensions—accuracy and grounding, safe agency, graceful failure, consistency, fairness, and operational efficiency—and provides a specific three-layer framework to achieve these goals. The first layer, with a focus on outputs, shows you how to get consistent and accurate responses using well-engineered prompts, RAG, and model customization. The second layer, about agents, establishes parameters for memory, tool usage, and orchestration in agentic systems. The book concludes with the third layer—Reliable Operations—covering deployment, monitoring, and responsible AI. The book’s reliability framework emphasizes the symbiotic relationship between evaluation and reliability, proving that you cannot improve what you do not measure. You’ll learn to measure your applications using fine-grained LLM-native evaluation rubrics like the Grounding Defect Rate, Hallucination Severity Score, and FActScore that audit everything from RAG outputs to the entire step-by-step reasoning trajectory of autonomous agents. The book stays rooted in reality from start to finish. As you go, you will build several projects, including a multi-agent travel planner and a medical assistant, and use industry standard tools and specifications like LangGraph and MCP. In the helpful appendices you’ll find a handy reference architecture and decision-making checklists…
8
Views
Registered users can read the full content for free
Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.
Page
1
M A N N I N G Rush Shahani Applications and agents you can trust
Page
2
Building Reliable AI Systems APPLICATIONS AND AGENTS YOU CAN TRUST RUSH SHAHANI M A N N I N G SHELTER ISLAND Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
3
For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com ©2026 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. The author and publisher have made every effort to ensure that the information in this book was correct at press time. The author and publisher do not assume and hereby disclaim any liability to any party for any loss, damage, or disruption caused by errors or omissions, whether such errors or omissions result from negligence, accident, or any other cause, or from any usage of the information herein. Manning Publications Co. Development editor: Elesha Hyde 20 Baldwin Road Technical editor: Michael Wang PO Box 761 Review editor: Radmila Ercegovac Shelter Island, NY 11964 Production editor: Keri Hales Copy editor: Keir Simpson Proofreader: Jason Everett Technical proofreader: Siddharth Shroff Typesetter and cover designer: Marija TudorISBN 9781633436732 Printed in the United States of America Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
4
For everyone building AI systems that have to work in the real world, and for my family and friends, who supported me every step of the way Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
5
contents preface xii acknowledgments xiv about this book xvi about the author xix about the cover illustration xx 1 AI reliability: Building LLMs for the real world 1 1.1 From benchmarks to the real world 2 1.2 The tangible impact of LLMs in the real world 3 Legal industry 4 ■ Customer service 4 ■ Programming and development 5 ■ Enterprise AI 5 1.3 Understanding hallucinations and reliable AI 6 When AI hallucinates convincingly 6 ■ What is a hallucination? 7 ■ What is reliable AI? 8 1.4 The AI reliability framework 8 Layer 1: Reliable outputs 8 ■ Layer 2: Reliable agents 10 Layer 3: Reliable operations 10 1.5 The reliability toolbox 11 1.6 Why reliable AI systems matter now 13 1.7 Requirements for following along 13iv Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
6
CONTENTS v PART 1 RELIABLE OUTPUTS ....................................... 15 2 Generating trustworthy responses with prompt engineering 17 2.1 Choosing the right model 18 Reasoning vs. nonreasoning models 18 ■ The cost-capability spectrum 18 ■ Model selection as a reliability decision 19 2.2 Tailoring LLM settings for maximum reliability 19 Optimizing temperature for predictable outputs 19 ■ Top-P Sampling technique 23 2.3 Limiting output and turns to reduce hallucination risk 25 2.4 Applying frequency and presence penalties for balanced content 26 2.5 Minimizing intrinsic randomness for stable performance 27 2.6 Foundations of prompt engineering for reliable LLMs 29 Designing components of a prompt for reliability 29 ■ Crafting basic prompts with a focus on dependability 31 2.7 Prompt engineering techniques for preventing hallucinations 32 Zero-shot prompting 32 ■ Few-shot prompting for contextual stability 33 ■ Chain-of-thought prompting for transparent reasoning 34 ■ Enhanced CoT prompt 37 ■ Automatic CoT for scalable reasoning 37 ■ Self-consistency for cross- verification 38 ■ Tree-of-thought prompting for structured decision-making 39 2.8 Project: Creating a reliable weather assistant with OpenAI’s function calls 41 Building a weather assistant 42 ■ Getting started and defining the weather function 43 ■ Integrating the chatbot 44 3 Grounding outputs with RAG 47 3.1 What is RAG? 49 3.2 RAG system architecture 50 The role of the retriever 51 ■ The factual generators 51 RAG system flow 51 ■ Example: E-commerce shopping assistant 51 3.3 Reducing hallucinations with RAG 52 Grounding the language model 52 ■ Grounding an e-commerce product query 52 ■ Advantages of RAG in reducing Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
7
CONTENTSvi hallucinations 53 ■ Critical applications 53 ■ Enhancing transparency 54 3.4 Data preparation and reliable indexing for RAG systems 54 Introducing LangChain 55 ■ Structuring product catalogs 56 Creating the searchable vector index 57 ■ Implementing an index 57 3.5 Building an effective RAG system 57 Building an e-commerce chatbot powered by RAG 58 Processing user queries 59 ■ Crafting an accurate response using an LLM 60 3.6 Evaluating and optimizing RAG systems 61 The role of evaluator LLMs in assessing hallucinations 61 Crafting evaluation datasets 63 ■ Practical metrics for RAG evaluation 65 ■ Implementing RAGAS evaluation 66 Reducing hallucinations by filtering retrievals and using metadata 67 ■ Metadata-based filtering in a telecom chatbot 69 Embedding chunk sizes and model choice 73 3.7 Advanced techniques for improving RAG 74 Fine-tuning embeddings for in-domain relevance 74 Incorporating metadata to reestablish context 74 ■ Implementing hybrid retrieval 74 ■ Enhancing queries via rephrasing and augmentation 75 ■ Implementing query routing 75 GraphRAG: Knowledge graph-enhanced retrieval 75 ■ When to use GraphRAG vs. traditional RAG 75 ■ Agentic RAG: Agent-controlled retrieval 77 3.8 Building an RAG system for an e-commerce chatbot 77 Step 1: Install the required libraries 78 ■ Step 2: Import libraries 78 ■ Step 3: Load the documents 79 ■ Step 4: Document transformers 79 ■ Step 5: Embed text 79 ■ Step 6: Choose a vector store 79 ■ Step 7: Prepare the LLM model 80 ■ Step 8: Add a retriever 80 ■ Step 9: Create a retrieval QA chain 80 ■ Step 10: Test the chatbot 80 Step 11: Evaluate and prevent hallucinations with LangChain 81 ■ Challenges and adaptations of the chatbot 82 4 Embeddings and vector search 85 4.1 What are embeddings? 88A mental model: Embeddings as coordinates in meaning space 88 Why the model matters 89 Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
8
CONTENTS vii 4.2 Embedding models in production 89 Commercial models: Powerful but opaque 90 ■ Open source models: Flexible and transparent 91 ■ Domain-specific models: Purpose- built precision 91 ■ How to choose the right model 91 Embeddings across industries 93 ■ Case study: Learning from Airbnb’s embedding journey 93 4.3 Beyond simple vectors: Hybrid and multistage retrieval 97 The limitations of pure vector search 97 ■ Hybrid retrieval: Combining dense and sparse approaches 97 ■ Multistage retrieval: Building precision through layers 98 ■ Building a hybrid retriever: Hands-on implementation for an employee chatbot 100 4.4 Essential retrieval optimizations with embeddings and RAG 104 Metadata filtering: Beyond pure semantic search 104 ■ Chunking strategies: Finding the right granularity 107 4.5 Exploring vector storage and databases 110 HNSW: The algorithm powering modern vector search 111 FAISS: Industry-standard vector indexing 112 ■ Vector databases: Complete vector storage solutions 113 ■ Vector database project: Building a health-care policy assistant with Pinecone 115 4.6 Practical challenges: Drift and compression 122 Embedding drift explained 123 ■ Diving deeper into vector compression 123 5 Fine-tuning LLMs for improved performance 126 5.1 Choosing the right approach: Prompting, RAG, or fine-tuning 127 When to use each approach: A practical decision framework 129 Hybrid approach: Combining RAG and fine-tuning 129 5.2 Case studies: Fine-tuning in the real world 130 Medicine: Med-PaLM 2 130 ■ Finance: BloombergGPT and FinGPT 130 ■ Code generation: Codex 130 5.3 Understanding the fine-tuning process 131 Data preparation 132 ■ Fine-tuning closed-source models (OpenAI) 135 ■ Understanding key fine-tuning approaches 138 Building a customer-support assistant 1405.4 Knowledge distillation for LLMs 148 How knowledge distillation works in LLMs 148 ■ Best practices 151 Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
9
CONTENTSviii PART 2 RELIABLE AGENTS ........................................ 153 6 Creating effective AI agents 155 6.1 Why do we need agents? 156 Static knowledge 157 ■ Inability to act 157 ■ No workflow management 158 ■ How AI agents solve these challenges 158 Broader applications of AI agents 160 6.2 Core components of reliable agents 160 Memory: Context that sticks 161 ■ Balancing memory systems with LangChain 164 ■ Active tool use vs. passive retrieval 167 Strategies for reducing hallucinations with tools 168 Decision-making: Making AI agents intelligent 170 6.3 Agentic RAG for reliable agents 172 Project: Building an agentic RAG system with LangChain 173 Setting up dependencies 174 ■ Building the knowledge base 174 Creating interactive tools 174 ■ Configuring the agent’s behavior 175 ■ Implementing transparent decision-making 175 Managing conversation context 176 ■ Making the system work in production 177 6.4 Advanced techniques 177 Cross-referencing 178 ■ Guardrails for security and preventing hallucinations 178 ■ Reinforcement Learning from AI Feedback 179 ■ Transparency in multistep reasoning 179 Case studies for agents in production 180 7 Tool integration and MCP 184 7.1 What is MCP? 185 7.2 The hidden complexity: The N×M problem 185 7.3 The solution: MCP 186 7.4 Building your first real MCP tool with a CSV-powered product catalog 188 Loading your product catalog 188 ■ Setting up the MCP server 188 7.5 Running and testing your server 190 Why this matters 190 ■ Next steps 191 7.6 Teaching your AI to use MCP tools 191 How models learn what tools they can use 191 ■ A complete example: Model > MCP > answer 192 ■ What you didn’t have to write 193 ■ Tool design becomes interface design 194 Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
10
CONTENTS ix 7.7 Adding a second tool: Inventory status 195 How models chain tools 196 ■ Tool descriptions guide model behavior 196 7.8 Handling failures gracefully 196 Catching and communicating errors 197 ■ Why structured responses matter 198 ■ Handling empty results vs. errors 199 Timeout and rate-limiting 200 7.9 Beyond tools: Production agent harnesses and OpenClaw 201 Why harnesses matter 201 ■ Designing agent-legible environments 202 ■ The security gap 203 8 Multi-agent systems 205 8.1 Multi-agent systems 206 Why multi-agent architecture matters 206 ■ Introducing LangGraph: The multi-agent framework 207 ■ Building intelligent agent coordination 207 ■ Real-world example: ShopBot multi-agent system 209 ■ Running your multi-agent system 218 Adding vision: Image-based product search with VisionAgent 219 What you built and why it matters 220 8.2 Testing multi-agent workflows 221 Category 1: Blended queries and multimodal interactions 223 Category 2: Ambiguous intent and robust routing decisions 224 Category 3: Agent failure and resilience verification 225 Category 4: State flow and information preservation across the workflow 226 ■ Regression: Moving forward without breaking what worked 228 8.3 Alternative framework: CrewAI 228 LangGraph vs. CrewAI: Different philosophies 228 ■ ShopBot with CrewAI 229 ■ When to use which multi-agent framework 232 The framework landscape 232 PART 3 RELIABLE OPERATIONS ................................ 235 9 Evaluation and performance for LLMs and agents 237 9.1 Identifying and measuring hallucinations 238 Four steps to identify and measure hallucinations 239 FActScore: fine-grained factual evaluation 240 ■ ROUGE for summarization evaluation 242 ■ LLM-as-a-judge: A holistic approach 243 ■ Red teaming and stress testing 244 Using monitoring frameworks to detect hallucinations 245 Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
11
CONTENTSx 9.2 Essential architectural patterns for performance 246 Token streaming: Presenting answers incrementally or at the same time 247 ■ Handling surges with batching: System-level vs. OpenAI’s Batch API 248 ■ Caching for efficiency 251 Multimodel fallback: Matching each query to the right model 257 Project: Building an e-commerce LLM service with batching, caching, and model fallback 261 ■ Further improvements 266 9.3 Evaluating agent performance 267 Core metrics for evaluating LLMs 268 ■ Evaluating agents: Beyond traditional LLM metrics 268 10 Deploying and monitoring 273 10.1 Introducing large language model operations 274 10.2 Serving LLMs: Hosted APIs vs. open source models 276 Using hosted APIs 276 ■ The open source alternative 277 The hybrid solution: Best of both worlds 279 10.3 Building LLM-native monitoring systems 281 What really matters: The four questions 281 ■ Logging what matters 282 ■ Setting up alerts that help 284 ■ Catching cost explosions before they hurt 285 ■ Building dashboards that drive action 286 ■ Output-quality monitoring 287 10.4 User experience and feedback monitoring 288 Explicit feedback collection 289 ■ Implicit feedback signals 290 Building actionable feedback loops 291 10.5 Ensuring high-quality outputs in production 292 The three-pillar quality framework 292 ■ Prompt engineering for consistent quality 292 ■ Continuous quality monitoring with automated testing 293 10.6 Observability in practice: Introducing Langfuse 296 11 Bias, privacy, and responsible AI 299 11.1 The responsible AI imperative 300 Regulatory pressure is accelerating 301 ■ User expectations have shifted 301 ■ Business risks have multiplied 301 ■ Real examples of AI bias in production 301 ■ The four failure modes 302 The responsible AI defense system 303 11.2 Data layer: Where bias begins 304 The fine-tuning bias trap 304 ■ Detecting bias in chat logs 305 The name experiment 307 ■ Three proven bias-mitigation strategies 307 Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
12
CONTENTS xi 11.3 Model layer: Where bias evolves 308 Why this matters for open source models 309 ■ Example: Adding fairness to a LoRA fine-tuning loop 309 ■ Anthropic’s constitutional AI: LLM-as-a-judge at training scale 309 11.4 Safety layer: Your last line of defense 310 Multilayered safety architecture 310 ■ Layer 3: Enhanced safety with commercial APIs 312 11.5 Privacy layer: Protecting personal data 313 Why LLM privacy failures are uniquely dangerous 313 ■ Building sensitive data detection 315 ■ Health-care privacy protection 317 European data protection 318 11.6 Real-world project: SafeMedAssist 320 Why build a medical AI assistant? 320 ■ Professional testing with LangTest 323 ■ Production-deployment considerations 326 The business case for responsible AI 327 ■ Completing the reliability framework 327 ■ The six principles of reliable AI 328 Final thoughts 329 references 331 bibliography 335 index 339Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
13
preface Modern language models feel like magic in a demo. You type a prompt, and within seconds, you get an answer that would have seemed like science fiction a few years ago. Then you try to ship that same capability to real users, and the magic starts to leak. The model invents a citation. It follows an instruction it should have refused. It works flawlessly for a week and then quietly degrades on a Monday morning when nobody is watching. I have lived on both sides of that gap. Over the years, I’ve built AI, machine learn- ing, and search systems at LinkedIn; worked on natural-language processing (NLP) and generative AI at Element AI; worked on backend and infrastructure at Shopify; and co-founded Persana AI, where our systems powered go-to-market work for thou- sands of teams. Across all those years, the same lesson kept repeating: the hard part is almost never getting a model to do something impressive once; the hard part is get- ting it to do the right thing reliably, for every user, on the thousandth ordinary day after launch. The numbers back this up. Industry studies have found that the vast majority of generative AI pilots never deliver real return on investment. Again and again, teams hit the same walls: hallucinations, flaky outputs, brittle tool calls, weak evaluations, and safety and fairness problems that surface only when real people are involved. The technology is extraordinary; the engineering discipline around it is still catching up. That gap is why I wrote this book. Plenty of resources explain how transformers work or how to call an API; far fewer treat reliability as a first-class engineering con- cern, something you design for, measure, and defend the same way you would anyxii other production system. This book is my attempt to fill that space with techniques you can apply immediately, drawn from what holds up when models meet real users. Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
14
PREFACE xiii The approach is deliberately practical. Every part of the book is organized around a working project, and every technique is presented in the context of a problem it solves. You start by making individual outputs trustworthy, move on to agents that take actions safely, and finish with the operational disciplines that keep a system reliable long after launch. Models will keep changing, and new capabilities and new failure modes will keep arriving. The principles here were chosen to outlast any particular model because they address fundamental challenges rather than temporary limitations. If you’re trying to close the distance between an impressive prototype and a system you can trust in production, this book is for you. Let’s build something reliable. Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
15
acknowledgments A book is a reliability project in its own right, and this one wouldn’t have shipped with- out a great many people catching the failures I couldn’t see myself. Thank you to Brian Sawyer for reaching out to me and bringing this book to Man- ning in the first place. This project wouldn’t exist without that first conversation. Thank you as well to the entire team at Manning for their patience and craft in turn- ing a rough manuscript into a finished book, and especially to Elesha Hyde, my devel- opment editor, whose steady focus on the reader shaped these chapters far more than any single technique I wrote about. Her instinct for where a reader would stumble made the book clearer on nearly every page. Siddharth Shroff, Yifan Wang, and Jonathan Reeves shaped the technical heart of this book. Siddharth reviewed the code and technical content with a rigor that saved readers from more than a few sharp edges, and I’m thankful for his care in making sure that the examples do what the prose claims they do. Yifan and Jonathan pushed on the arguments and the technical reviews until they held up. Thank you to all the reviewers who read early drafts and returned detailed, candid, and occasionally bruising feedback: Alex Zalesov, Amar Chheda, Andrew Ferlitsch, Andrew R Freed, Angelica Lo Duca, Ankit Virmani, Ankush Rastogi, Anshuman Guha, Astha Puri, Attila Bagossy, Bex Tuychiev, Chandra Parashara, Clyde Kallahan, Corey Gallon, Dan Sheikh, Eleni Tsiligianni, Felipe Coutinho, Georg Sommer, Gil De Grove, Giovanni Alzetta, Guilherme Augusto, Guillermo Alcantara, Harini Shankar, Heng Zhang, Hobson Lane, Ike Okonkwo, Jasmine Alkin, Jiri Pik, Jonathan Reeves, Karrtik Iyer, Kim Falk, Krzysztof Jędrzejewski, Laurence Giglio, Lokeshwar Vangala, Manishxiv Jain, Matteo Mazzola, Matthias Lein, Maxim Volgin, Mikael Dautrey, Natapong Sorn- prom, Nehaa Bansal, Oscar Rolon, Paul Soh, Pavan Emani, Pundi Sridhar, Raghu Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
16
ACKNOWLEDGMENTS xv Para, Raj Kumar, Richard Vaughan, Sachin Gagwani, Sam Nasr, Samer Hamad, Shailja Gupta, Shivam Vaishampayam, Siddharth Shroff, Srinath Chandramohan, Sriram Macharla, Sumit Pal, Tomáš Šipka, Ulina, Vaibhav Verdhan, Vaijanath Rao, Walter Alexander Mata López, Yash Chaturvedi, and Zohir Koufi. This book deals with a gen- uinely hard subject in a fast-moving field, and your comments sharpened the struc- ture, corrected the framing, and kept the book honest about what reliability requires. The book is better for every one of your notes. Finally, thank you to my colleagues at LinkedIn, Element AI, Shopify, and Persana AI, who taught me most of what I know about shipping machine learning into the real world. Special thanks to Sriya Maram, my co-founder at Persana AI, for building alongside me through the work that taught me many of the lessons in these pages. My deepest thanks go to my family. To my grandparents Gul and Saroj Shahani, who have championed me since I was young and who, even in their nineties, stay curi- ous about AI and still send me things worth thinking about, and in loving memory of my grandparents Indru and Mona Lalwani. To my parents, Nitin and Rekha Shahani; my brother, Naish, and my sister-in-law, Mini; my uncles, Naveen and Nitin; my aunt, Aanchal; and my cousins, Rhea, Jahaan, and Sureena, whose support reached far beyond the hours this book demanded and helped shape the person who wrote it. And to my friends, who kept me going the whole way.Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
17
about this book Building Reliable AI Systems is a practical guide to closing the gap between an AI proto- type that impresses and a production system you can trust. It focuses on the nonfunc- tional requirements of large language model (LLM) applications: accuracy, grounding, safety, fairness, privacy, evaluation, and operability. Rather than surveying model architectures in the abstract, it walks through the engineering techniques that make real systems dependable, using working projects to anchor every idea. Who should read this book This book is written for software engineers, machine-learning engineers, AI engi- neers, and technical leaders who are building or preparing to build LLM-powered applications and agents for production. If you’ve written a prompt, called a model API, or wired up an agentic workflow and felt the distance between a promising demo and an AI system you’d put in front of real users, you are the intended reader. To follow the examples, you should be comfortable reading Python and have a basic familiarity with machine learning and NLP concepts. You don’t need a research background. Every reliability technique is introduced from the problem it solves, so you can start applying the material without mastering the underlying theory. How this book is organized: A road map The book has 11 chapters grouped into three parts, moving from the reliability of individual outputs to the reliability of systems that take actions to the reliability of everything when it’s running in production.xvi Part 1—This part covers the foundation: getting the model to produce accu- rate, grounded, trustworthy responses in the first place. Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
18
ABOUT THIS BOOK xvii – Chapter 1 defines a reliable AI system, motivates why reliability matters for production deployments, and introduces the reliability framework and tool- box that structure the rest of the book. – Chapter 2 focuses on generating trustworthy responses through prompt engineering and techniques that steer models toward accurate output. – Chapter 3 grounds outputs in verified information by using retrieval-aug- mented generation (RAG). – Chapter 4 covers embeddings and vector search, the retrieval machinery that make grounding work at scale. – Chapter 5 turns to fine-tuning for domain expertise, adapting models when prompting and retrieval aren’t enough. Part 2—This part moves from generating answers to taking actions, when both the stakes and failure modes rise sharply. – Chapter 6 covers the architectures and patterns behind effective, controlla- ble AI agents. – Chapter 7 connects agents to external systems through tool integration and Model Context Protocol (MCP). – Chapter 8 covers multi-agent systems and the coordination and safeguards required to keep them from failing together. Part 3—This part covers the operational disciplines that keep a working system reliable over time at scale for every user. – Chapter 9 covers evaluation and performance, showing how to measure whether your system is reliable and fast enough to ship. – Chapter 10 covers deployment and monitoring, the infrastructure that keeps quality from silently drifting when real traffic arrives. – Chapter 11 closes the framework with bias, privacy, and responsible AI, ensuring that systems treat users fairly, protect their data, and operate trans- parently. The parts build on one another, so reading start to finish gives the most complete pic- ture. That said, each chapter is written to stand on its own, so you can jump to the technique you need most and follow the cross-references back to any foundations you want to fill in. About the code This book contains many examples of source code both in numbered listings and inline with normal text. In both cases, source code is formatted in a fixed-width font like this to separate it from ordinary text. Sometimes, code is also in bold to highlight code that has changed from earlier steps in the chapter, such as when a new feature adds to an existing line of code. Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
19
ABOUT THIS BOOKxviii In many cases, the original source code has been reformatted; we’ve added line breaks and reworked indentation to accommodate the available page space in the book. In rare cases, even this was not enough, and listings include line-continuation markers (➥). Also, comments in the source code were removed from the listings when the code was described in the text. Code annotations accompany many of the listings, highlighting important concepts. The in-book listings are written for clarity. To keep the focus on the reliability tech- nique being taught, some snippets omit boilerplate such as imports and setup that you’d need to run them as is. Complete, runnable versions of the examples, including that scaffolding, are provided as downloadable companion code so you can execute and adapt every project. Because the underlying tools and model APIs evolve quickly, the companion code is the canonical, maintained version of the examples. Where a printed snippet and the downloadable code differ, treat the downloadable code as authoritative. You can get executable snippets of code from the liveBook (online) version of this book at https://livebook.manning.com/book/building-reliable-ai-systems. The com- plete code for the examples in the book is available for download from the Manning website at https://www.manning.com/books/building-reliable-ai-systems and from GitHub at https://github.com/rusheel/manning-building-reliable-ai-systems. liveBook discussion forum Purchase of Building Reliable AI Systems includes free access to a private web forum run by Manning Publications where you can make comments about the book, ask techni- cal questions, and receive help from the author and other users. To access the forum and to subscribe to it, go to https://livebook.manning.com/#!/book/building-reli able-ai-systems/discussion. Manning’s commitment to our readers is to provide a venue where meaningful dialogue between individual readers and between readers and the author can take place. It isn’t a commitment to any specific amount of participation on the part of the author, whose contribution to the forum remains voluntary (and unpaid). We suggest that you try asking the author some challenging questions lest their interest stray. The forum and the archives of previous discussions will be accessible on the publisher’s website as long as the book is in print.Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
Page
20
about the author RUSH SHAHANI is an AI leader who has spent his career shipping machine learning sys- tems that hold up under real-world conditions. He co-founded Persana AI, a Y Combi- nator-backed sales intelligence platform that powered more than 15,000 go-to-market teams before joining forces with Rox. Named one of Y Combinator’s top generative AI startups, Persana helped companies turn AI into real revenue by combining agentic go-to-market workflows, data enrichment, and autonomous execution at scale. Before Persana, Rush built AI, machine learning, and search systems at LinkedIn and worked on NLP and generative AI at Element AI (acquired by ServiceNow). Ear- lier, he worked on backend, infrastructure, and payments at Shopify. He holds a com- puter science honors degree from the University of Waterloo. His work has been featured in Forbes and TechCrunch, and he speaks regularly on AI engineering at indus- try events. Building Reliable AI Systems draws on those experiences and the hard lessons learned when models leave the demo and meet real users. xix Licensed to THIAGO BANDEIRA <thiago@lar.ifce.edu.br>
The above is a preview of the first 20 pages. Register to read the complete e-book.
AI Reading Assistant
Whole-book reading guide from stratified index samples; jump to passages in the text
AI guide
【One-Line Pitch】
A practical field guide for turning an impressive AI demo into a dependable product, built around a six-dimension definition of reliability and a three-layer framework for outputs, agents, and operations. Best for engineers, ML practitioners, and technical leads who already know how to call a model and now need to ship one.
【Book Arc】
- **Opening (~0%–10%)**: Frames the core problem — demos feel like magic, but production leaks through hallucinations, ignored instructions, and silent degradation — and introduces the six reliability dimensions plus the three-layer framework (outputs, agents, operations).
- **Early (~10%–30%)**: Establishes the reliability toolbox: model selection, evaluation methods (reference-based metrics, LLM-as-judge, red-teaming, human review), observability, and monitoring. Then moves into output-layer craft — prompt engineering, sampling controls like top-p, few-shot patterns, and function calling to ground responses in live data.
- **Middle (~30%–55%)**: Builds grounding with RAG — the load/split/embed/store pipeline, embeddings as "coordinates in meaning space," LangChain as the orchestration layer, and a retail chatbot case that shows why retrieval beats prompting alone for stock and policy questions.
- **Late (~55%–85%)**: Shifts from outputs to agency — memory, tool usage, and orchestration parameters for agentic systems, with worked projects such as a multi-agent travel planner and a medical assistant, using LangGraph and MCP. Evaluation extends from single answers to full reasoning trajectories.
- **Ending (~85%–100%)**: Closes with Reliable Operations — deployment, drift detection, quality alerts, model updates, feedback loops, and responsible AI (bias, privacy, governance), plus appendices with a reference architecture and decision checklists.
【Key Takeaways】
- **Reliability is a first-class engineering concern, not a post-launch patch** (Opening): the book's central argument is that you design, measure, and defend reliability the way you would any production system — the hard part is doing the right thing on the thousandth ordinary day, not the first demo.
- **Six dimensions define "reliable"** (Opening): accuracy and grounding, safe agency, graceful failure, consistency, fairness, and operational efficiency. Fairness and privacy are treated as reliability dimensions, not separate compliance chores — a system that discriminates or leaks data is unreliable regardless of its accuracy scores.
- **You cannot improve what you do not measure** (Early): evaluation and reliability are symbiotic. The book favors LLM-native rubrics — Grounding Defect Rate, Hallucination Severity Score, FActScore — and stresses that reference metrics, LLM-as-judge, red-teaming, and human review catch different failure modes, so production systems combine them.
- **Grounding beats clever prompting for factual queries** (Middle): RAG's retrieve-then-generate loop, and function calling to authoritative APIs, both outsource facts to external sources rather than trusting pretrained memory — the weather-assistant and retail-chatbot examples make the hallucination reduction concrete.
- **Retrieval quality starts in data preparation** (Middle): the load/split/embed/store pipeline and chunking choices determine what the model can ground on; embeddings capture meaning so related terms cluster, which is why indexing deserves engineering attention rather than being an afterthought.
- **Agency changes the risk profile** (Early/Late): once systems act, reliability includes permissions, reversibility, tool robustness, and failure containment — not just answer quality. Agent design therefore needs explicit parameters for memory, tool use, and orchestration.
- **Production monitoring for AI is semantic, not just operational** (Late): uptime and latency are insufficient; teams must track drift, hallucination rates, and user satisfaction, and manage model updates without breaking existing behavior.
- **Sampling parameters are reliability levers** (Early): top-p tuning trades consistency against creativity — lower values for support and policy answers, higher for exploratory output — a small but recurring example of matching configuration to task stakes.
【Reading Tips】
- **Skim the conceptual opening, deep-read the framework chapter.** The first chapter's market statistics and adoption figures are context; the six-dimension definition and three-layer framework are the spine you will reuse throughout.
- **Treat the projects as the real curriculum.** The retail RAG chatbot, weather assistant, multi-agent travel planner, and medical assistant are where the abstractions become engineering judgment — build along rather than reading passively.
- **Expect the hardest material in the agent section.** Memory, tool orchestration, and trajectory-level evaluation are conceptually denser than prompt engineering; budget extra time and revisit the evaluation rubrics after finishing.
- **Keep the appendices as working references.** The reference architecture and decision checklists are designed for lookup during real design reviews, not for linear reading.
- **Carry one question throughout:** for each technique, ask which of the six reliability dimensions it defends and what failure it prevents.
【Coverage Limits】
The excerpts cover the framing, output layer, and RAG material in reasonable depth, but the agent and operations chapters appear only in summary form — specific implementation details, code, and the full evaluation rubrics are not visible here. Claims about later chapters rest on the book's own descriptions rather than excerpted content.
Passage locations
Excerpt 1
ts. The book stays rooted in reality from start to finish. As you go, you will build several projects, including a multi-agent travel planner and a medical a...
View in text
Excerpt 2
des, and production systems typically combine several modes. You’ll also learn about observability platforms such as Arize and Phoenix that help you understa...
View in text
Excerpt 3
e, and avoidance of speculation or unfounded claims. Empha- size the importance of staying grounded in the provided context and input data. The following exa...
View in text
Excerpt 4
ng of the text. The embeddings for dog and puppy, for exam- ple, would be more like each other than like the embedding for cat. Although the technical detail...
View in text
Recommended for You
{{#thumbnailUrl}}
{{/thumbnailUrl}}
{{^thumbnailUrl}}
{{/thumbnailUrl}}
Loading recommended books...
Failed to load, please try again later
Tip the Site
Scan the WeChat Pay or Alipay code to tip. No login required.
WeChat Pay
Alipay