Building Machine Learning Systems with a Feature Store Batch, Real-Time, and LLM Systems (Jim Dowling)（Z-Library）

# Building Machine Learning Systems with a Feature Store ## 【One-Line Pitch】 A practical, architecture-first guide for data scientists and ML engineers who want to move beyond notebook experiments and build production-grade batch, real-time, and LLM-powered ML systems using the feature-training-inference (FTI) architecture and feature stores. ## 【Book Arc】 - **Opening (~0%–9%)**: Introduces the core FTI architecture—decomposing any AI system into independent feature, training, and inference pipelines connected by a shared data layer. Establishes the taxonomy of data transformations (model-independent, model-dependent, on-demand) and frames the journey from static datasets to dynamic batch and real-time data. - **Early (~9%–28%)**: Dives into ML pipeline design and modular code practices. Covers the anatomy of batch systems (e.g., Spotify's weekly recommendations) versus real-time systems (e.g., TikTok's adaptive feed), explains why feature stores emerged (Uber's Michelangelo, 2017), and walks through the first end-to-end example—a Titanic survival prediction system with both batch and interactive UIs. - **Early–Middle (~28%–44%)**: Builds a complete air quality forecasting service as a second worked example. Demonstrates data exploration, missing-data handling, feature engineering with Pandas, and the full pipeline sequence: feature pipeline → training pipeline → inference pipeline, all orchestrated through Hopsworks. - **Middle (~44%–53%)**: Shifts to feature stores in depth. Explains point-in-time correct training data, temporal joins, and the critical problem of data leakage (both future leakage and stale features). Compares two feature store architectures—batch-synced offline-to-online versus stream-enabled online-first designs. - **Middle–Late (~53%–100%)**: Covers data transformations at scale using frameworks like Pandas, Polars, Apache Spark, and Apache Flink. Details model-independent transformations (row/column size–preserving, reducing, increasing, joins), lazy DataFrames, vectorized compute with Arrow, and builds toward the real-time credit card fraud detection system that threads through the book. ## 【Key Takeaways】 - **The FTI architecture is the unifying pattern for all AI systems** (Early): Every ML system—batch, real-time, or agentic—can be decomposed into feature, training, and inference pipelines connected by a shared data layer. This modularity lets teams develop each pipeline independently as long as they respect the data contract (schemas, validation rules, SLOs). - **Feature stores solve the stateful online inference problem** (Early): When clients can't send all context in a request, precomputed features stored in a feature store give online models low-latency access to history and personalization data. This was the insight behind Uber's Michelangelo and remains central to modern ML platforms. - **Data leakage has two time-based forms** (Middle): Future data leakage (using feature values from after the prediction time) and stale features (using values older than the actual observation time) both corrupt training data. Feature stores address this with temporal joins that create point-in-time correct training snapshots. - **Data transformations follow a three-way taxonomy** (Early): Model-independent transformations (MITs) run only in feature pipelines and create reusable features; model-dependent transformations (MDTs) run in training and inference pipelines for model-specific features; on-demand transformations (ODTs) compute real-time features at inference time and can also be applied historically. - **Modular code beats spaghetti notebooks** (Early): Refactor ML pipelines into functions and classes, keep code DRY, and test before deploying. This is the difference between a demo and a maintainable production system that can evolve as features, models, and requirements change. - **Feature store architecture choices matter** (Middle): Two patterns exist—batch-synced (clients write to offline store, periodically synced to online/vector index) and stream-enabled (stream API writes to online store, periodically synced to offline). The choice depends on freshness requirements and whether you need real-time feature updates. - **Start with data completeness checks** (Middle): Before feature engineering, examine data in notebooks—check for missing values with `isna().sum()`, drop incomplete rows, and validate that your target column and date/time columns are clean. This groundwork prevents downstream pipeline failures. ## 【Reading Tips】 - **Skim the Titanic and air quality examples if you're experienced** (~34%–44%): These are pedagogical walkthroughs with code. If you understand the FTI pattern, focus on the architecture decisions rather than the step-by-step notebook instructions. - **Deep-read the feature store chapters** (~44%–53%): The temporal join and data leakage material is conceptually dense and is the book's core intellectual contribution. Understanding point-in-time correctness here will pay off in every subsequent chapter. - **Pay attention to the fraud detection case study** (Middle–Late): This is the recurring example that ties batch, real-time, and streaming concepts together. It's worth tracking how features flow from the data mart and event-streaming platform into the feature store. - **Treat the transformation taxonomy as a mental model** (Early): The MIT/MDT/ODT distinction is the book's organizing principle for data work. Internalize it early—it will help you classify every transformation you encounter later. - **Skip the Hopsworks setup details if you're not using that platform** (~34%): The installation and API key instructions are specific to the book's chosen tooling. The architectural lessons transfer regardless of your feature store choice. ## 【Coverage Limits】 Excerpts cover roughly the first half of the book (through ~53%), including the FTI architecture, feature store fundamentals, and early data transformation material. Later chapters on real-time systems, agentic LLM workflows, and MLOps practices are not covered in this guide. ##