Streaming Databases Unifying Batch and Stream Processing (Hubert Dulay, Ralph M. Debusmann) (Z-Library)

# Streaming Databases: Unifying Batch and Stream Processing ## 【One-Line Pitch】 A practical guide for data engineers, architects, and analysts who want to build real-time analytics solutions by understanding how streaming databases unify batch and stream processing—covering everything from foundational concepts to serving data with push and pull queries. ## 【Book Arc】 - **Opening (~0%–9%)**: Establishes the historical evolution from big data batch processing (Hadoop, MapReduce) to distributed streaming (Kafka), introducing the write-ahead log (WAL) as the conceptual bridge between OLTP databases and streaming platforms. - **Early (~9%–25%)**: Explores how database features like materialized views and change feeds are externalized into streaming systems, using a running e-commerce use case (products, customers, click events) to illustrate real-world architecture. - **Early (~25%–34%)**: Dives into data pipelines, contrasting ELT limitations with stream processing approaches, and introduces the two fundamental stream types—append-only streams (e.g., click events) and change streams (from CDC). - **Middle (~34%–44%)**: Focuses on serving real-time data, defining key SLA metrics (latency, throughput, data freshness, accuracy, availability) and explaining how RTOLAP databases handle ingestion transformations and indexing. - **Middle (~44%–47%)**: Distinguishes synchronous pull queries (request-response) from asynchronous push queries (event-driven), covering technologies like SSEs and WebSockets for real-time client communication. - **Late (~47%+)**: Begins the deep dive into materialized views, using the "smart chipmunk Simon" analogy to contrast traditional views (recomputed on demand) with materialized views (precomputed and continuously updated). ## 【Key Takeaways】 - **Streaming databases emerged from the WAL concept** (Early): The write-ahead log in OLTP databases naturally supports streaming semantics—clients can subscribe to transactions in real time, and streaming platforms like Kafka essentially externalize this pattern for broader system use. - **Materialized views are the bridge between batch and stream** (Early): Traditional materialized views require manual refreshes (e.g., `REFRESH MATERIALIZED VIEW CONCURRENTLY` in Postgres), but when updated continuously, they become inherently real-time and fit naturally into streaming frameworks. - **Native change feeds reduce architectural complexity** (Early): OLTP databases like CockroachDB can push change events directly to Kafka via `CREATE CHANGEFEED`, eliminating the need for separate connectors and simplifying the streaming pipeline significantly. - **Append-only vs. change streams require different handling** (Early): Click events are unique, timestamped, append-only streams best kept out of OLTP databases (which would run out of space), while change streams from CDC represent updates to existing records—understanding this distinction is critical for pipeline design. - **ELT can support streaming, but only with the right target** (Early): Traditional ELT fails for real-time use cases when persisting before transformation, but using a stream processing platform (Flink, Kafka Streams) as the target enables extract-load-transform in a streaming context. - **SLA metrics define real-time success** (Middle): Latency, throughput/concurrency, data freshness, and accuracy are the primary metrics for real-time analytics, while availability, capacity, resilience, and security are secondary but essential considerations. - **Pull vs. push queries serve different consumption patterns** (Middle): Pull queries (request-response) are measured in QPS and strain OLAP systems under high refresh rates, while push queries eliminate polling by notifying clients of new data—served via SSEs or WebSockets for asynchronous delivery. - **Ingestion transformations preprocess data in flight** (Middle): RTOLAP databases can apply stateless transformations (like timestamp format conversion) before data hits persistent storage, forcing consumers to treat the topic data as a base format. ## 【Reading Tips】 - **Read Chapters 1–4 before Chapter 5**: The authors explicitly warn against skipping ahead—the foundational material on WALs, streams, and serving patterns is essential for understanding streaming databases introduced later. - **Follow the running e-commerce use case**: The book builds a consistent example (products, customers, click events) throughout; tracking this use case across chapters will help you connect abstract concepts to concrete architecture. - **Pay special attention to the stream types section**: The distinction between append-only and change streams (around 28% of the book) is a conceptual cornerstone that affects everything from storage to query design. - **Skim the historical background if you're experienced**: The early evolution narrative (Hadoop → Kafka) is useful context but can be skimmed if you're already familiar with big data history; focus instead on the WAL and materialized view discussions. - **Take notes on the SLA metrics framework**: The metrics discussion (around 34%) provides a practical vocabulary for evaluating real-time systems—useful for both designing and communicating requirements. ## 【Coverage Limits】 This guide covers the book's opening through approximately 47% of the content, focusing on foundations, pipelines, serving patterns, and the introduction to materialized views. The excerpts do not cover the later chapters on streaming database implementation details, advanced materialized view construction, or hands-on setup examples. ##