Building Reliable AI Systems Applications and agents you can trust (Rush Shahani)（Z-Library）

【One-Line Pitch】 A practical field guide for turning an impressive AI demo into a dependable product, built around a six-dimension definition of reliability and a three-layer framework for outputs, agents, and operations. Best for engineers, ML practitioners, and technical leads who already know how to call a model and now need to ship one. 【Book Arc】 - **Opening (~0%–10%)**: Frames the core problem — demos feel like magic, but production leaks through hallucinations, ignored instructions, and silent degradation — and introduces the six reliability dimensions plus the three-layer framework (outputs, agents, operations). - **Early (~10%–30%)**: Establishes the reliability toolbox: model selection, evaluation methods (reference-based metrics, LLM-as-judge, red-teaming, human review), observability, and monitoring. Then moves into output-layer craft — prompt engineering, sampling controls like top-p, few-shot patterns, and function calling to ground responses in live data. - **Middle (~30%–55%)**: Builds grounding with RAG — the load/split/embed/store pipeline, embeddings as "coordinates in meaning space," LangChain as the orchestration layer, and a retail chatbot case that shows why retrieval beats prompting alone for stock and policy questions. - **Late (~55%–85%)**: Shifts from outputs to agency — memory, tool usage, and orchestration parameters for agentic systems, with worked projects such as a multi-agent travel planner and a medical assistant, using LangGraph and MCP. Evaluation extends from single answers to full reasoning trajectories. - **Ending (~85%–100%)**: Closes with Reliable Operations — deployment, drift detection, quality alerts, model updates, feedback loops, and responsible AI (bias, privacy, governance), plus appendices with a reference architecture and decision checklists. 【Key Takeaways】 - **Reliability is a first-class engineering concern, not a post-launch patch** (Opening): the book's central argument is that you design, measure, and defend reliability the way you would any production system — the hard part is doing the right thing on the thousandth ordinary day, not the first demo. - **Six dimensions define "reliable"** (Opening): accuracy and grounding, safe agency, graceful failure, consistency, fairness, and operational efficiency. Fairness and privacy are treated as reliability dimensions, not separate compliance chores — a system that discriminates or leaks data is unreliable regardless of its accuracy scores. - **You cannot improve what you do not measure** (Early): evaluation and reliability are symbiotic. The book favors LLM-native rubrics — Grounding Defect Rate, Hallucination Severity Score, FActScore — and stresses that reference metrics, LLM-as-judge, red-teaming, and human review catch different failure modes, so production systems combine them. - **Grounding beats clever prompting for factual queries** (Middle): RAG's retrieve-then-generate loop, and function calling to authoritative APIs, both outsource facts to external sources rather than trusting pretrained memory — the weather-assistant and retail-chatbot examples make the hallucination reduction concrete. - **Retrieval quality starts in data preparation** (Middle): the load/split/embed/store pipeline and chunking choices determine what the model can ground on; embeddings capture meaning so related terms cluster, which is why indexing deserves engineering attention rather than being an afterthought. - **Agency changes the risk profile** (Early/Late): once systems act, reliability includes permissions, reversibility, tool robustness, and failure containment — not just answer quality. Agent design therefore needs explicit parameters for memory, tool use, and orchestration. - **Production monitoring for AI is semantic, not just operational** (Late): uptime and latency are insufficient; teams must track drift, hallucination rates, and user satisfaction, and manage model updates without breaking existing behavior. - **Sampling parameters are reliability levers** (Early): top-p tuning trades consistency against creativity — lower values for support and policy answers, higher for exploratory output — a small but recurring example of matching configuration to task stakes. 【Reading Tips】 - **Skim the conceptual opening, deep-read the framework chapter.** The first chapter's market statistics and adoption figures are context; the six-dimension definition and three-layer framework are the spine you will reuse throughout. - **Treat the projects as the real curriculum.** The retail RAG chatbot, weather assistant, multi-agent travel planner, and medical assistant are where the abstractions become engineering judgment — build along rather than reading passively. - **Expect the hardest material in the agent section.** Memory, tool orchestration, and trajectory-level evaluation are conceptually denser than prompt engineering; budget extra time and revisit the evaluation rubrics after finishing. - **Keep the appendices as working references.** The reference architecture and decision checklists are designed for lookup during real design reviews, not for linear reading. - **Carry one question throughout:** for each technique, ask which of the six reliability dimensions it defends and what failure it prevents. 【Coverage Limits】 The excerpts cover the framing, output layer, and RAG material in reasonable depth, but the agent and operations chapters appear only in summary form — specific implementation details, code, and the full evaluation rubrics are not visible here. Claims about later chapters rest on the book's own descriptions rather than excerpted content.