Data Engineering Design Patterns (Bartosz Konieczny)（Z-Library）

# Data Engineering Design Patterns ## 【One-Line Pitch】 A practical recipe book for experienced data engineers who want to stop reinventing the wheel, offering battle-tested design patterns for ingestion, error management, and data quality that solve recurring business problems with technology-agnostic solutions and open-source implementations. ## 【Book Arc】 - **Opening (~0%–9%)**: Introduces the concept of design patterns in data engineering, establishes the target audience (minimum six months commercial experience), and sets expectations for the book's structure—each pattern includes a problem statement, solution, and consequences. The author emphasizes that patterns provide a common language for teams and save time by avoiding repeated problem-solving. - **Early (~9%–28%)**: Covers data ingestion design patterns, starting with the Full Loader for slowly evolving reference datasets, then moving through Incremental Loader with partition-based implementations using Apache Airflow and Spark, Replication for cross-environment data copying, and the Compactor pattern to solve the small files problem that plagues object store–based lakehouses. - **Early (~28%–34%)**: Concludes the ingestion chapter with the External Trigger pattern for event-driven ingestion, explaining pull versus push semantics and how to handle unpredictable data arrival. The section emphasizes that ingestion is not always a predictable process and requires flexible architectural responses. - **Middle (~34%–47%)**: Shifts to error management design patterns, beginning with the Dead-Letter pattern for handling unprocessable records (poison pill messages) in streaming jobs, distinguishing between transient and nontransient errors, and showing how to keep pipelines running while preserving bad records for investigation. - **Middle (~47%–end of excerpts)**: Continues error management with the Windowed Deduplicator for handling at-least-once delivery semantics, the Dynamic Late Data Integrator for managing late-arriving data with state tables and lookback windows, and the Filter Interceptor for tracking filtering statistics—noting the challenges of implementing these patterns in declarative languages like SQL versus programmatic APIs. ## 【Key Takeaways】 - **Design patterns give data teams a shared vocabulary** (Early): Just as cooking recipes prevent reinventing the wheel, patterns like Dead-Letter and Compactor let engineers discuss solutions concisely with teammates they've just met, saving time and reducing miscommunication. - **The Full Loader pattern is deceptively simple but has hidden pitfalls** (Early): While it's just extract-and-load in two steps, you must avoid data transformations that introduce quality issues like type conversions or floating-point rounding—use native copy commands or raw text APIs instead of JSON I/O when possible. - **The small files problem is still alive in modern lakehouses** (Early): Even with virtually unlimited object storage, metadata operations like listing files can consume 70% of execution time. The Compactor pattern merges small files into larger ones, with Apache Iceberg's rewrite data file action and Delta Lake's OPTIMIZE command as implementations. - **Dead-Letter pattern keeps pipelines alive without losing bad data** (Middle): Instead of failing the entire job on poison pill messages, route unprocessable records to a separate table for investigation. However, implementing this in declarative SQL requires verbose custom try-catch logic that's hard to maintain. - **Exactly-once delivery is rare; deduplication is the practical answer** (Middle): The Windowed Deduplicator pattern treats data as limited within time boundaries, enabling both batch and streaming pipelines to process each occurrence only once despite at-least-once delivery semantics. - **Late data integration requires stateful thinking** (Middle): The Dynamic Late Data Integrator uses a state table to track last processed times per partition, with BigQuery's INFORMATION_SCHEMA.PARTITIONS and Iceberg's metadata tables providing this out of the box—but concurrency can cause duplicate integration runs. - **Declarative languages struggle with complex error handling** (Middle): Patterns like Filter Interceptor are easier to implement and maintain with programmatic APIs than SQL, and streaming implementations may require converting stateless jobs to stateful ones with additional overhead. ## 【Reading Tips】 - **Skim the opening chapter** if you're already familiar with design pattern concepts—the real value starts with Chapter 2's ingestion patterns, but do read the "What Should I Know" section to verify you have the prerequisites (ETL/ELT, cloud basics, and production experience with Java, Scala, Python, or SQL). - **Deep-read the Compactor and Dead-Letter patterns**—these address the most common real-world pain points (small files and poison pill messages) and include concrete implementation details across multiple technologies like Iceberg, Delta Lake, and Hudi. - **Pay attention to the "Consequences" sections**—each pattern includes trade-offs (like added complexity from Dead-Letter logic or concurrency issues in Dynamic Late Data Integrator) that are crucial for deciding when NOT to use a pattern. - **Use the GitHub repository alongside the book**—code examples are organized by chapter with README files and require Docker Compose, making it easy to run demos and see patterns in action rather than just reading about them. - **Skip ahead if you're only interested in batch processing**—the streaming-specific content (External Trigger, Dead-Letter for Kafka) is clearly marked, and you can focus on the ingestion and compaction patterns without losing context. ## 【Coverage Limits】 The excerpts cover the introduction, data ingestion patterns (Full Loader, Incremental Loader, Replication, Compactor, External Trigger), and the beginning of error management patterns (Dead-Letter, Windowed Deduplicator, Dynamic Late Data Integrator, Filter Interceptor). Later chapters on data quality, idempotency, and observability mentioned in the blurb are not covered in this guide. ##