Delta Lake The Definitive Guide (Denny Lee, Tristen Wentling etc.)（Z-Library）

Delta Lake The Definitive Guide Modern Data Lakehouse Architectures with Data Lakes Denny Lee, Tristen Wentling, Scott Haines & Prashanth Babu Forewords by Michael Armbrust & Dominique Brezinski

ISBN: 978-1-098-15194-2 US $79.99 CAN $99.99 DATA Denny Lee is a Delta Lake maintainer and a Unity Catalog, Apache Spark, and MLflow contributor. Tristen Wentling is a Delta practitioner who works at Databricks. Scott Haines is an Apache Spark and Delta Lake contributor. Prashanth Babu is a Delta practitioner who works at Databricks. Ready to simplify the process of building data lakehouses and data pipelines at scale? In this practical guide, learn how Delta Lake is helping data engineers, data scientists, and data analysts overcome key data reliability challenges with modern data engineering and management techniques. Authors Denny Lee, Tristen Wentling, Scott Haines, and Prashanth Babu (with contributions from Delta Lake maintainer R. Tyler Croy) share expert insights on all things Delta Lake—including how to run batch and streaming jobs concurrently and accelerate the usability of your data. You’ll also uncover how ACID transactions bring reliability to data lakehouses at scale. This book helps you: • Understand key data reliability challenges and how Delta Lake solves them • Explain the critical role of Delta transaction logs as a single source of truth • Learn the Delta Lake ecosystem with technologies like Apache Flink, Kafka, and Trino • Architect data lakehouses with the medallion architecture • Optimize Delta Lake performance with features like deletion vectors and liquid clustering Delta Lake: The Definitive Guide “Open table formats are the future. If you are invested in Delta Lake, this book will take you from zero to 100.” Adi Polak Author of Scaling Machine Learning with Spark

Praise for Delta Lake: The Definitive Guide Delta Lake has revolutionized data architectures by combining the best of data lakes and warehouses into the lakehouse architecture. This definitive guide by O’Reilly is an essential resource for anyone looking to harness the full potential of Delta Lake. It offers deep insights into building scalable, reliable, high-performance data architectures. Whether you’re a data engineer, scientist, or practitioner, this book will empower you to tackle your toughest data challenges with confidence and precision. —Matei Zaharia, associate professor of computer science at UC Berkeley and cofounder and chief technologist at Databricks This book not only provides excellent code examples for Delta Lake but also explains what happens behind the scenes. It’s a resource I’ll continue to rely on as a practical reference for Delta Lake APIs. Furthermore, it covers the latest exciting innovations within the Delta Lake ecosystem. —Ryan Zhu, founding developer of Delta Lake, cocreator of Delta Sharing, Apache Spark PMC member, Delta Lake maintainer The authors of this book fuse deep technical knowledge with pragmatism and clear exposition to allow readers to bring their Spark data lakehouse aspirations to life with the Delta Lake framework. —Matt Housley, CTO and coauthor of Fundamentals of Data Engineering Open table formats are the future. If you are invested in Delta Lake, this book will take you from zero to 100, including use cases, integrations, and how to overcome hiccups. —Adi Polak, author of Scaling Machine Learning with Spark

There are two types of people in data: those who believe they understand what Delta Lake is and those who read this book. —Andy Petrella, part of the second group, author of Fundamentals of Data Observability, and founder of Kensu Look no further if you want to master all things Delta Lake. Denny, Tristen, Scott, and Prashanth have gone above and beyond to give you more experience than you could ever imagine. —Jacek Laskowski, freelance Data(bricks) engineer Delta Lake is much more than Apache Parquet with a commit log. Delta Lake: The Definitive Guide takes the mystery out of streaming, data governance, and design patterns. —Bartosz Konieczny, waitingforcode.com

Denny Lee, Tristen Wentling, Scott Haines, and Prashanth Babu Forewords by Michael Armbrust and Dominique Brezinski Delta Lake: The Definitive Guide Modern Data Lakehouse Architectures with Data Lakes

978-1-098-15194-2 [LSI] Delta Lake: The Definitive Guide by Denny Lee, Tristen Wentling, Scott Haines, and Prashanth Babu Copyright © 2025 O’Reilly Media, Inc. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (https://oreilly.com). For more information, contact our corporate/institu‐ tional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Aaron Black Development Editor: Gary O’Brien Production Editor: Gregory Hyman Copyeditor: Arthur Johnson Proofreader: Emily Wydeven Indexer: BIM Creatives, LLC Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea November 2024: First Edition Revision History for the First Edition 2024-10-29: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781098151942 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Delta Lake: The Definitive Guide, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. This work is part of a collaboration between O’Reilly and Databricks. See our statement of editorial independence.

Table of Contents Foreword by Michael Armbrust. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Foreword by Dominique Brezinski. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv 1. Introduction to the Delta Lake Lakehouse Format. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 The Genesis of Delta Lake 1 Data Warehousing, Data Lakes, and Data Lakehouses 1 Project Tahoe to Delta Lake: The Early Years Months 5 What Is Delta Lake? 6 Common Use Cases 7 Key Features 8 Anatomy of a Delta Lake Table 10 Delta Transaction Protocol 11 Understanding the Delta Lake Transaction Log at the File Level 12 The Single Source of Truth 12 The Relationship Between Metadata and Data 13 Multiversion Concurrency Control (MVCC) File and Data Observations 13 Observing the Interaction Between the Metadata and Data 14 Table Features 16 Delta Kernel 18 Delta UniForm 19 Conclusion 20 2. Installing Delta Lake. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Delta Lake Docker Image 21 Delta Lake for Python 23 iii

PySpark Shell 24 JupyterLab Notebook 25 Scala Shell 25 Delta Rust API 26 ROAPI 27 Native Delta Lake Libraries 28 Multiple Bindings Available 29 Installing the Delta Lake Python Package 29 Apache Spark with Delta Lake 29 Setting Up Delta Lake with Apache Spark 30 Prerequisite: Set Up Java 30 Setting Up an Interactive Shell 31 PySpark Declarative API 33 Databricks Community Edition 33 Create a Cluster with Databricks Runtime 33 Importing Notebooks 35 Attaching Notebooks 36 Conclusion 37 3. Essential Delta Lake Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Create 40 Creating a Delta Lake Table 41 Loading Data into a Delta Lake Table 42 The Transaction Log 45 Read 46 Querying Data from a Delta Lake Table 46 Reading with Time Travel 47 Update 49 Delete 49 Deleting Data from a Delta Lake Table 50 Overwriting Data in a Delta Lake Table 51 Merge 53 Other Useful Actions 55 Parquet Conversions 55 Delta Lake Metadata and History 57 Conclusion 57 4. Diving into the Delta Lake Ecosystem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 Connectors 60 Apache Flink 61 Flink DataStream Connector 61 iv | Table of Contents

Installing the Connector 62 DeltaSource API 62 DeltaSink API 66 End-to-End Example 69 Kafka Delta Ingest 71 Install Rust 72 Build the Project 72 Run the Ingestion Flow 73 Trino 75 Getting Started 75 Configuring and Using the Trino Connector 79 Using Show Catalogs 79 Creating a Schema 80 Show Schemas 80 Working with Tables 81 Table Operations 84 Conclusion 88 5. Maintaining Your Delta Lake. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Using Delta Lake Table Properties 89 Delta Lake Table Properties Reference 90 Create an Empty Table with Properties 92 Populate the Table 92 Evolve the Table Schema 94 Add or Modify Table Properties 96 Remove Table Properties 97 Delta Lake Table Optimization 99 The Problem with Big Tables and Small Files 99 Using OPTIMIZE to Fix the Small File Problem 101 Table Tuning and Management 104 Partitioning Your Tables 104 Defining Partitions on Table Creation 105 Migrating from a Nonpartitioned to a Partitioned Table 106 Repairing, Restoring, and Replacing Table Data 108 Recovering and Replacing Tables 108 Deleting Data and Removing Partitions 109 The Life Cycle of a Delta Lake Table 110 Restoring Your Table 110 Cleaning Up 111 Conclusion 113 Table of Contents | v

6. Building Native Applications with Delta Lake. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 Getting Started 116 Python 116 Rust 127 Building a Lambda 131 What’s Next 137 7. Streaming In and Out of Your Delta Lake. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Streaming and Delta Lake 139 Streaming Versus Batch Processing 140 Delta as Source 146 Delta as Sink 147 Delta Streaming Options 149 Limit the Input Rate 149 Ignore Updates or Deletes 150 Initial Processing Position 152 Initial Snapshot with withEventTimeOrder 154 Advanced Usage with Apache Spark 156 Idempotent Stream Writes 156 Delta Lake Performance Metrics 161 Auto Loader and Delta Live Tables 162 Auto Loader 162 Delta Live Tables 163 Change Data Feed 164 Using Change Data Feed 165 Schema 169 Conclusion 171 8. Advanced Features. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 Generated Columns, Keys, and IDs 173 Comments and Constraints 175 Comments 176 Delta Table Constraints 178 Deletion Vectors 179 Merge-on-Read 180 Stepping Through Deletion Vectors 181 Conclusion 186 vi | Table of Contents

9. Architecting Your Lakehouse. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 The Lakehouse Architecture 188 What Is a Lakehouse? 188 Learning from Data Warehouses 188 Learning from Data Lakes 189 The Dual-Tier Data Architecture 190 Lakehouse Architecture 192 Foundations with Delta Lake 193 Open Source on Open Standards in an Open Ecosystem 193 Transaction Support 195 Schema Enforcement and Governance 197 The Medallion Architecture 201 Exploring the Bronze Layer 202 Exploring the Silver Layer 205 Exploring the Gold Layer 208 Streaming Medallion Architecture 210 Conclusion 212 10. Performance Tuning: Optimizing Your Data Pipelines with Delta Lake. . . . . . . . . . . . . 213 Performance Objectives 214 Maximizing Read Performance 214 Maximizing Write Performance 216 Performance Considerations 217 Partitioning 218 Table Utilities 220 Table Statistics 226 Cluster By 236 Bloom Filter Index 240 Conclusion 242 11. Successful Design Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 Slashing Compute Costs 243 High-Speed Solutions 244 Smart Device Integration 245 Efficient Streaming Ingestion 252 Streaming Ingestion 252 The Inception of Delta Rust 254 The Evolution of Ingestion 255 Coordinating Complex Systems 257 Combining Operational Data Stores at DoorDash 258 Change Data Capture 259 Table of Contents | vii

Delta and Flink in Harmony 261 Conclusion 262 12. Foundations of Lakehouse Governance and Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 Lakehouse Governance 264 The Emergence of Data Governance 270 Data Products and Their Relationship to Data Assets 273 Data Products in the Lakehouse 274 Maintaining High Trust 274 Data Assets and Access 275 The Data Asset Model 275 Unifying Governance Between Data Warehouses and Lakes 278 Permissions Management 279 Filesystem Permissions 280 Cloud Object Store Access Controls 281 Identity and Access Management 282 Data Security 283 Fine-Grained Access Controls for the Lakehouse 295 Conclusion 296 13. Metadata Management, Data Flow, and Lineage. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 Metadata Management 297 What Is Metadata Management? 298 Data Catalogs 298 Data Reliability, Stewards, and Permissions Management 299 Why the Metastore Matters 300 Unity Catalog 302 Data Flow and Lineage 304 Data Lineage 305 Data Sharing 311 Automating Data Life Cycles 311 Audit Logging 314 Monitoring and Alerting 315 What Is Data Discovery? 317 Conclusion 318 14. Data Sharing with the Delta Sharing Protocol. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319 The Basics of Delta Sharing 320 Data Providers 321 Data Recipients 322 Delta Sharing Server 323 Using the REST APIs 324 viii | Table of Contents

Anatomy of the REST URI 324 List Shares 325 Get Share 327 List Schemas in Share 328 List All Tables in Share 331 Delta Sharing Clients 332 Delta Sharing with Apache Spark 332 Stream Processing with Delta Shares 336 Delta Sharing Community Connectors 338 Conclusion 338 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 Table of Contents | ix

Foreword by Michael Armbrust The Delta protocol was first conceived when I met Dominique Brezinski at Spark Summit 2017. As he described to me the scale of data processing that he was envi‐ sioning, I knew that, through our collaborative approach to running Apache Spark, Databricks had already laid down the building blocks of the cloud-scale computing environment necessary to make him successful. Yet I also knew that these fundamen‐ tals would inevitably prove to be insufficient without us introducing a novel system to manage the complexities of transactional access to the ever-growing lake of data that Dom had been collecting in his private cloud. Recognizing that Apache Spark itself could serve as the engine of scalable transaction consistency enforcement was the key insight that underpins the ongoing success of Delta Lake. That is, to simplify and scale, we treated the metadata like how we processed and queried the data. Translating this single insight and the resulting protocol into Delta Lake, a compre‐ hensive toolset for developers to use in any streaming data management solution, has been a long road, with many collaborations along the way. Becoming an open source project allowed Delta Lake to evolve through community input and contributions. The robust ecosystem that has resulted now includes multiple implementations of the Delta protocol, in multiple frameworks, such as Flink, Trino, Presto, and Pulsar, and in multiple languages, including Rust, Go, Java, Scala, Hive, and Python. To celebrate and further build on this vibrant open source community, I’m now excited to present Delta Lake: The Definitive Guide. This guide details Delta Lake’s architecture, use cases, and best practices, catering to data engineers, scientists, and analysts alike. It encapsulates years of innovation in data management, offering a comprehensive resource for unlocking Delta Lake’s full potential. As you explore this book, you’ll gain the knowledge to leverage Delta Lake’s capabilities in your projects. I’m eager to see how you’ll use it to drive innovation and achieve your data goals. xi

Welcome to the shore of the Delta Lake. The water is great—let’s take a swim! — Michael Armbrust Creator of Delta Lake, Spark PMC Member, Delta Lake TSC and Maintainer xii | Foreword by Michael Armbrust

Foreword by Dominique Brezinski Delta Lake emerged from Michael and my discussions about the challenges I encoun‐ tered when building a high-scale streaming ETL system using Apache Spark, EC2, and S3. We faced the same challenges at Apple in processing vast amounts of data for intrusion monitoring and threat response. We needed to build a system that could do not only streaming ingestion but also streaming detection and support performant queries over a long retention window of large datasets. From these requirements Delta Lake was created to support ACID transactions and seamless integration of batch and streaming processes, allowing us to handle petabytes of daily data efficiently. This guide reveals Delta Lake’s architectural fundamentals, practical applications, and best practices. Whether you’re a data engineer, scientist, or business leader, you’ll find valuable insights to leverage Delta Lake effectively. I’m excited for you to explore this guide and witness how Delta Lake can propel your own innovations. Together, we’re shaping the future of data management, enabling the construction of reliable and performant data lakehouses. — Dominique Brezinski Distinguished Engineer, Apple Delta Lake Technical Steering Committee Member xiii

Preface Welcome to Delta Lake: The Definitive Guide! Since it became an open source project in 2019, Delta Lake has revolutionized how organizations manage and process their data. Designed to bring reliability, performance, and scalability to data lakes, Delta Lake addresses many of the inherent challenges traditional data lake architectures face. Over the past five years, Delta Lake has undergone significant transformation. Origi‐ nally focused on enhancing Apache Spark, Delta Lake now boasts a rich ecosystem with integrations across various platforms, including Apache Flink, Trino, and many more. This evolution has enabled Delta Lake to become a versatile and integral component of modern data engineering and data science workflows. Who This Book Is For As a team of production users and maintainers of the Delta Lake project, we’re thrilled to share our collective knowledge and experience with you. Our journey with Delta Lake spans from small-scale implementations to internet-scale production lakehouses, giving us a unique perspective on its capabilities and how to work around any complexities. The primary goal of this book is to provide a comprehensive resource for both newcomers and experts in data lakehouse architectures. For those just starting with Delta Lake, we aim to elucidate its core principles and help you avoid the common mistakes we encountered in our early days. If you’re already well versed in Delta Lake, you’ll find valuable insights into the underlying codebase, advanced features, and optimization techniques to enhance your lakehouse environment. Throughout these pages, we celebrate the vibrant Delta Lake community and its col‐ laborative spirit! We’re particularly proud to highlight the development of the Delta Rust API and its widely adopted Python bindings, which exemplify the community’s innovative approach to expanding Delta Lake’s capabilities. Delta Lake has evolved xv

significantly since its inception, growing beyond its initial focus on Apache Spark to embrace a wide array of integrations with multiple languages and frameworks. To reflect this diversity, we’ve included code examples featuring Flink, Kafka, Python, Rust, Spark, Trino, and more. This broad coverage ensures that you’ll find relevant examples regardless of your preferred tools and languages. While we cover the fundamental concepts, we’ve also included our personal experien‐ ces and lessons learned. More importantly, we go beyond theory to offer practical guidance on running a production lakehouse successfully. We’ve included best prac‐ tices, optimization techniques, and real-world scenarios to help you navigate the challenges of implementing and maintaining a Delta Lake–based system at scale. Whether you’re a data engineer, architect, or scientist, our goal is to equip you with the knowledge and tools to leverage Delta Lake effectively in your data projects. We hope this guide serves as your companion in building robust, efficient, and scalable lakehouse architectures. How This Book Is Organized We organized the book so that you can move from chapter to chapter—introducing concepts, demonstrating key concepts via example code snippets, and providing full code examples or notebooks in the book’s GitHub repository. The earlier chapters provide the fundamentals on how to install Delta Lake, its essential operations, understanding its ecosystem, building native Delta Lake applications, and maintain‐ ing your Delta Lake; the later chapters expand on these fundamentals and dive deeper into the features before coming back up to review how you can architect this all together for your production workloads: Chapter 1, “Introduction to the Delta Lake Lakehouse Format” We explain Delta Lake’s origins, what it is and what it does, its anatomy, and the transaction protocol. We impress upon you that the Delta transaction log is the single source of truth and is subsequently the single source of the relationship between its metadata and data. Chapter 2, “Installing Delta Lake” We discuss the various ways to install Delta Lake, whether through pip or through Docker implementations for Rust, Python, and Apache Spark. Chapter 3, “Essential Delta Lake Operations” In this chapter we look at CRUD operations, merge operations, conversion from Parquet to Delta, and management of Delta Lake metadata. xvi | Preface