The Data Lakehouse For Dummies®, 2nd Databricks Special Edition (Ari Kaplan Amit Kara)（Z-Library）

Page 1

(This page has no text content)

Page 2

(This page has no text content)

Page 3

The Data Lakehouse 2nd Databricks Special Edition by Ari Kaplan and Amit Kara

Page 4

The Data Lakehouse For Dummies®, 2nd Databricks Special Edition Published by John Wiley & Sons, Inc. 111 River St. Hoboken, NJ 07030-5774 www.wiley.com Copyright © 2026 by John Wiley & Sons, Inc., Hoboken, New Jersey. All rights, including for text and data mining, AI training, and similar technologies, are reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without the prior written permission of the Publisher. Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 111 River Street, Hoboken, NJ 07030, (201) 748-6011, fax (201) 748-6008, or online at http://www.wiley.com/go/ permissions. Trademarks: Wiley, For Dummies, the Dummies Man logo, The Dummies Way, Dummies.com, Making Everything Easier, and related trade dress are trademarks or registered trademarks of John Wiley & Sons, Inc. and/or its affiliates in the United States and other countries, and may not be used without written permission. Databricks and the Databricks logo are registered trademarks of Databricks. All other trademarks are the property of their respective owners. John Wiley & Sons, Inc., is not associated with any product or vendor mentioned in this book. LIMIT OF LIABILITY/DISCLAIMER OF WARRANTY: THE PUBLISHER AND THE AUTHOR MAKE NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE ACCURACY OR COMPLETENESS OF THE CONTENTS OF THIS WORK AND SPECIFICALLY DISCLAIM ALL WARRANTIES, INCLUDING WITHOUT LIMITATION WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE. NO WARRANTY MAY BE CREATED OR EXTENDED BY SALES OR PROMOTIONAL MATERIALS.  THE ADVICE AND STRATEGIES CONTAINED HEREIN MAY NOT BE SUITABLE FOR EVERY SITUATION. THIS WORK IS SOLD WITH THE UNDERSTANDING THAT THE PUBLISHER IS NOT ENGAGED IN RENDERING LEGAL, ACCOUNTING, OR OTHER PROFESSIONAL SERVICES.  IF PROFESSIONAL ASSISTANCE IS REQUIRED, THE SERVICES OF A COMPETENT PROFESSIONAL PERSON SHOULD BE SOUGHT. NEITHER THE PUBLISHER NOR THE AUTHOR SHALL BE LIABLE FOR DAMAGES ARISING HEREFROM. THE FACT THAT AN ORGANIZATION OR WEBSITE IS REFERRED TO IN THIS WORK AS A CITATION AND/OR A POTENTIAL SOURCE OF FURTHER INFORMATION DOES NOT MEAN THAT THE AUTHOR OR THE PUBLISHER ENDORSES THE INFORMATION THE ORGANIZATION OR WEBSITE MAY PROVIDE OR RECOMMENDATIONS IT MAY MAKE.  FURTHER, READERS SHOULD BE AWARE THAT INTERNET WEBSITES LISTED IN THIS WORK MAY HAVE CHANGED OR DISAPPEARED BETWEEN WHEN THIS WORK WAS WRITTEN AND WHEN IT IS READ. For general information on our other products and services, or how to create a custom For Dummies book for your business or organization, please contact our Business Development Department in the U.S. at 877-409-4177, contact info@dummies.biz, or visit www.dummies.com/ custom-solutions. For information about licensing the For Dummies brand for products or services, contact BrandedRights&Licenses@Wiley.com. ISBN 978-1-394-39663-4 (pbk); ISBN 978-1-394-39664-1 (ebk); ISBN 978-1-394-39665-8 (ebk) Publisher’s Acknowledgments Acquisitions Editor: Traci Martin Senior Managing Editor: Rev Mengle Development Editor: Jen Bingham Sales Manager: Molly Daugherty Content Refinement Specialist: Umeshkumar Rajasekhar

Page 5

Table of Contents iii Table of Contents INTRODUCTION ............................................................................................... 1 About This Book ................................................................................... 1 Icons Used in This Book ....................................................................... 2 Beyond the Book .................................................................................. 2 CHAPTER 1: Making the Case for Data Lakehouses ...................... 3 Exploring Traditional Data Warehouses ............................................ 4 Sorting out Data Warehouse Limitations .......................................... 4 Diving into Data Lakes ......................................................................... 5 Listing the Technical Reasons Why a Traditional Data Lake Isn’t Enough ........................................................................................... 6 The Advent of the Data Lakehouse .................................................... 7 What Data Lakehouses Solve for Enterprises ................................... 7 CHAPTER 2: Explaining Data Lakehouses .............................................. 9 Following the Data and AI Maturity Curve ...................................... 10 Sorting Out the Technical Concepts of a Lakehouse ..................... 11 Knowing What Data Lakehouses Bring to the Table ...................... 12 Multimodal support of a variety of data types .......................... 13 Lowers overall costs and avoids vendor lock-in ........................ 13 Ability to scale and manage all types of workloads .................. 13 Solving Problems with a Lakehouse ................................................ 14 CHAPTER 3: Understanding the Underlying Technology ......... 15 Looking into the Data and AI Benefits ............................................. 15 Data Reliability and Governance with Lakehouses ........................ 16 Seeing Why Lakehouses Are Best for BI and DW Workloads ....... 17 Building Transactional Applications on the Lakehouse ................. 19 Describing the Payoff for AI .............................................................. 19 CHAPTER 4: Bringing Data Intelligence to the Data Lakehouse ....................................................................................... 21 Introducing Data Intelligence ............................................................ 22 The Databricks Data Intelligence Platform ...................................... 23 CHAPTER 5: Ten Reasons Why You Need a Data Lakehouse ....................................................................................... 27

Page 6

Introduction 1 Introduction The data lakehouse enables companies to deliver faster on their data and artificial intelligence (AI) initiatives. It sim- plifies your data estate, eliminating data silos by combining the best of two worlds: the flexibility and cost-effectiveness of data lakes and the analytic capabilities of data warehouses. Lakehouses are built on open source and open standards, which unify and simplify your data management. They support all your data needs — including business intelligence (BI), data ware- housing (DW), online transaction processing (OLTP), machine learning (ML), and AI — enabling you to quickly build secure data and AI apps. This enables businesses to collaboratively build more intelligent applications on all of their data and workloads. And throughout it all, governance is central for providing end-to-end visibility and control of all your data estate. The data lakehouse reduces costs, unifies all types of data, simpli- fies workflows, produces faster analytical and AI insights, scales to trillions of records, and democratizes data to everyone. Histor- ically, many individual point solutions addressed each individual need: a database or data warehouse to store structured historical data, and a data lake to store unstructured data such as docu- ments, images, streaming social feeds, and videos. The lakehouse radically simplifies the enterprise data and AI infrastructure and accelerates innovation in an age when ML and generative AI (GenAI) are disrupting every industry. This archi- tecture supports structured, semistructured, unstructured, and streaming data in one unified and governed architecture, provid- ing the fuel for the full spectrum of data-driven use cases. About This Book This book introduces the data lakehouse to manage and govern all of your organization’s data assets, and explains the limitations of legacy solutions. It explains why a lakehouse is a foundation for solving data challenges and forms the basis for data intelligence platforms. You also discover how Databricks specifically builds on the open source architecture and what it all actually means for your company.

Page 7

2 The Data Lakehouse For Dummies, 2nd Databricks Special Edition Icons Used in This Book This book occasionally uses special icons to focus attention on important items. Here’s what you find: This icon reminds you of information that’s worth recalling. Expect to find something useful or helpful by way of suggestions, advice, or observations here, leveraging experiences from other implementations. Warning icons are meant to get your attention and to steer you clear of potholes, money pits, and other hazards. Paying extra attention to these parts of the book helps you avoid unnecessary roadblocks. This icon may be interpreted in one of two ways: Techies zero in on the juicy and significant details that follow; others will happily skip ahead to the next paragraph. Beyond the Book This book helps you understand how the lakehouse makes your data management efforts more effective and efficient in your company. However, because this is a relatively short book on data lakehouses, we also recommend checking out the following: » Lakehouse overview: databricks.com/product/ data-lakehouse » Video of the lakehouse architecture: youtube.com/ watch?v=13TownvHT7w » The Data Intelligence Platform For Dummies, Databricks Special Edition: databricks.com/resources/ebook/ maximize-your-organizations-potential-data-and-ai

Page 8

CHAPTER 1 Making the Case for Data Lakehouses 3 Chapter 1 IN THIS CHAPTER » Explaining data warehouses » Positioning data warehouse and data lake limitations » Describing the concept of data lakehouses Making the Case for Data Lakehouses Every company today aims to be a data and artificial intelli- gence (AI)-driven organization. This was once a contentious idea, but now it’s widely accepted. This approach is no longer just about data — AI is essential to success. But finding this type of success at scale is difficult for most organizations that need high-quality data that is both secure and consumable across the organization. The current state of data and AI systems is highly fragmented and extremely complex — a nightmare of high costs and proprietary formats. Organizations often have multiple data warehouses due to acquisitions, independent business units, and legacy systems. Many companies have accumulated a patchwork of data environ- ments over time. These silos create inefficiencies and prevent organizations from leveraging data effectively. Consolidation is critical for cost reduction and agility. That’s where data lake- houses come in. This chapter describes how managing data has evolved over time, how traditional solutions fall short, and why the data lakehouse architecture has emerged as the modern standard for data man- agement (DM) and data warehousing.

Page 9

4 The Data Lakehouse For Dummies, 2nd Databricks Special Edition Exploring Traditional Data Warehouses DM enables companies to corral their data across the whole com- pany by using consistent methods, techniques, and tools. The purpose of DM on an enterprise-wide scale is to fulfill all types of requirements for use cases, applications, and business processes. Simply put, DM supports the effective use of data, encompassing governance, quality, integration, and security. Data warehousing is one component of DM that focuses specifically on storing and analyzing structured data in rows and columns. Data lakes are another type of DM that supports unstructured data through file formats. The need for performing analytics on all types of data across multiple data sources, as well as run- ning end-to-end AI and business intelligence (BI), puts high demands on DM. In the early days of DM, the relational database was the primary method that companies used to collect and analyze data. Rela- tional databases offer a way for companies to store and analyze highly structured data, such as numbers, dates, and text, by using Structured Query Language (SQL). For many years, relational databases were simple and reliable ways to meet a company’s data needs — until the sheer volume of data increased so much that traditional databases could no longer handle it all. Data grew from billions of records to hundreds of billions and even trillions. Costs spiraled out of control, and insights struggled to be gener- ated in near real time. The rise of social media, mobile, the Internet of Things (IoT), and more led to companies drowning in data. To store all these new types and amounts of data, traditional databases were no longer sufficient. Companies, therefore, often had to build multiple dis- connected databases organized by lines of business to attempt to hold all the different data, users, and use cases, often failing. Sorting out Data Warehouse Limitations Without a way to centralize and efficiently use their data, com- panies ended up with decentralized, fragmented stores of data, called data silos, across the organization. With so much data stored

Page 10

CHAPTER 1 Making the Case for Data Lakehouses 5 across different silos, companies needed a way to unify them. Data warehouses were born to meet this need and to unite dispa- rate structured databases across the organization. The concept of data warehousing dates back to the late 1980s and, in essence, was intended to provide an architectural model for the flow of structured data from operational systems to decision-support environments. Early data warehouses were also on-premises, running on hardware fully managed by the com- pany itself. A shift toward cloud data warehousing in the early 2010s had external companies such as Amazon, Google, and Microsoft hosting and managing the hardware that data ware- houses ran on. The shift to cloud-based solutions offered several advantages over traditional on-premises data warehouses. It lowered upfront costs (operating expenses [OpEx] versus capital expenditures [CapEx]), set up and deployed faster, scaled larger, and improved access across the globe. Traditional data warehouses have inherent limitations that became more prohibitive as data volumes grew significantly larger, and a need arose to manage unstructured data cost effectively. These limitations greatly challenged enterprises, which started the push for better, faster, and more flexible DM solutions. The ability to store, manage, and govern a variety of data in a variety of formats had finally arrived. Diving into Data Lakes To make analytics possible on a variety of data formats and to address concerns about the cost and vendor lock-in of data ware- houses, Apache Spark emerged as the leading open-source dis- tributed data processing technology, replacing Hadoop, which was more limited and cumbersome to manage. These technolo- gies allowed large data sets to be processed with clusters of com- puters working in parallel.

Page 11

6 The Data Lakehouse For Dummies, 2nd Databricks Special Edition Listing the Technical Reasons Why a Traditional Data Lake Isn’t Enough Although suitable for storing data, data lakes lack some critical features that data warehouses are better for: » They don’t support atomic, consistent, isolated, and durable (ACID) transactions, which risk corrupting files and causing data inconsistencies. » They don’t enforce schema or data quality. » They’re inefficient, having to store multiple copies of data, and modifying existing data causes the rewriting of a lot of data when you just want to make short updates. » Their lack of data consistency and isolation makes it almost impossible to simultaneously write and append new data. » Jobs that fail midway lead to data quality issues, are hard to detect, and need to restart from scratch. » They make it difficult and inefficient to handle large volumes of unstructured data. As the number and size of files increase, performance can degrade, and it gets complex to understand the relationship among your sets of data without predefined schemas. » Data can proliferate into millions of tiny files or a few gigantic files, often negatively impacting performance. As the volume and variety of data kept surging, the need for a flexible, high-performance DM architecture kept increasing. More than ever, companies require systems for diverse data applications, including SQL analytics, real-time monitoring, data science (DS), machine learning (ML), and AI. Most of the recent advances in generative (Gen)AI incorporate better models to pro- cess unstructured data (text, images, video, audio, and social streaming). Still, these types of data are precisely the types that a data warehouse doesn’t support. Without a data lakehouse, multiple solutions must be patched together: several data lakes, data warehouses, ML, and GenAI tools. This introduces additional complexity and cost: Data pro- fessionals need to constantly move and copy data among the sys- tems, costing two to three times more to store and maintain all that redundant data. In addition, having all these multiple vendor

Page 12

CHAPTER 1 Making the Case for Data Lakehouses 7 solutions introduces a lack of unified access control, a lack of a single auditing log, and the cost of multiple vendor contracts. The Advent of the Data Lakehouse When data lakehouses came onto the scene, they were a water- shed technology because they enabled companies to analyze mas- sive amounts of both structured and unstructured data together for the first time, which before was simply too costly, too big, too slow, or too complex. One of the fundamental aspects of a lakehouse is unified data governance that eliminates data silos. Lakehouses unify data warehousing and AI use cases in a single architecture, simplifying the modern data stack for engineering, analytics, BI, data science, ML, and GenAI. Open source software (including Apache Spark, MLflow, Delta Lake, Apache Iceberg, and Unity Catalog) is the lakehouse’s underlying technology — it offers many advantages over tradi- tional data lakes and data warehouses: » Speed through in-memory processing, often 100 times faster » Ease of use through the support of Python, R, SQL, and Scala » Versatility for handling a variety of data processing methods such as batch and real-time streaming » GenAI and advanced analytics » Fault tolerance to avoid crashing and restarting lengthy processes What Data Lakehouses Solve for Enterprises Most organizations struggle to realize a vision that unifies all their data needs. There are so many systems: » Data warehousing for your structured data and data lakes for unstructured data » BI platforms to visualize your business insights

Page 13

8 The Data Lakehouse For Dummies, 2nd Databricks Special Edition » Orchestration and Extract, Transform, Load (ETL) solutions to prepare, merge, filter, and move data » Real-time systems for streaming use cases » Data science and ML platforms for advanced use cases such as predictions and classifications » GenAI for creating AI-driven applications and productivity agents Having all these divergent solutions without a unified data lake- house leads to many problems, as shown in Figure 1-1: » Enterprises are struggling with the massive sprawl across all these data silos. For each vendor, there are access and security controls, audit trails to track activity, monitoring dashboards, and governance frameworks. This sprawl adds risks, costs, and operational inefficiencies. » Data privacy and control are massive issues when attempting to apply them across data silos. With GenAI, a bright light is being trained on the ability to transparently understand and manage both the data inputs and the outputs from AI. Having one architecture unifies governance, reducing risks. » There’s a lack of technically skilled employees who can make sense of all these disparate solutions, and this lack can become a bottleneck. Your company relies on them to derive business insights. Having one architecture that democratizes managing data and getting business insights improves your business like never before. Even if you solve the prior problems, most of your company relies on your technical team to create data products. FIGURE 1-1: Age-old challenges that data lakehouses solve.

Page 14

CHAPTER 2 Explaining Data Lakehouses 9 Chapter 2 IN THIS CHAPTER » Looking at the data and AI maturity curve » Delineating the technical concepts » Understanding what lakehouses give you » Resolving challenges by adopting a lakehouse Explaining Data Lakehouses Data lakehouses are unified, open, and scalable. They com- bine the best of data lakes and data warehouses to remove data silos, bring all types of data together in one platform, provide a single unified governance, and simplify it all. This enables your business to deliver data and AI initiatives much more quickly, with more intelligence, transparency, and trust. At the same time, lakehouses reduce operational costs, enable col- laboration among all personas, and improve business intelli- gence, streaming, data science (DS), AI, data warehouse, and orchestration. Built on open source and open standards, a lake- house simplifies your data estate by eliminating the silos that historically complicate data and AI. Open data lakehouses are underpinned by widely adopted open source projects such as Apache Spark for processing; lakehouse storage such as Delta Lake and Iceberg; MLflow to manage the machine learning (ML) lifecycle; Delta Sharing to securely share live data from your lakehouse to any computing platform without replication; and Spark Declarative Pipelines to simplify compli- cated Extract, Transform, Load (ETL) processes using a declara- tive approach.

Page 15

10 The Data Lakehouse For Dummies, 2nd Databricks Special Edition The best lakehouses are flexible to run on all major cloud provid- ers. They allow Python, SQL, R, and Scala to run on all your uni- fied data. Lakehouses form the foundations for data intelligence platforms, which open up a whole new world of possibilities for democratizing data and AI across an organization. Data intel- ligence platforms use generative (Gen)AI to better understand the semantics of your data and use that across the platform (see Chapter 4). In this chapter, you will discover all you need to know about lake- houses, including what types of problems this architecture helps to overcome and why this is significantly different from other data warehousing solutions. Following the Data and AI Maturity Curve Enabling data intelligence is a journey companies take to enable their companies to be truly data-driven for the best business decisions and outcomes. In order to become a modern data- driven organization, companies typically move along the data and AI maturity curve shown in Figure 2-1. At the beginning of the journey, companies use databases and data warehouses to see what happened in the past, such as his- torical sales transactions and activity logs. They obtain struc- tured data, explore it, and provide precanned reports and ad hoc queries. As  companies mature, they add data lakes to perform FIGURE 2-1: The data and AI maturity curve.

Page 16

CHAPTER 2 Explaining Data Lakehouses 11 predictive analytics on what may happen in the future. They col- lect unstructured data such as documents, social media, images, and videos to help them make more intelligent decisions on a variety of data. They want prescriptive analytics to guide them on the best courses of action. The most mature companies go beyond traditional ML by incorporating GenAI on their own proprietary data and automating the decision-making where beneficial. And good news — the lakehouse enables all of this. Sorting Out the Technical Concepts of a Lakehouse Data lakehouses take an innovative approach by combining the data warehousing attributes of reliability, performance, and quality with the openness and scale of data lakes. A lakehouse has the following key features: » Openness: The underlying technology leverages open source solutions, which offer benefits, such as lower cost, transparency, flexibility, and avoiding vendor lock-in. Lakehouses leverage open storage formats such as the popular Delta Lake and Iceberg; Unity Catalog (UC) open- source software (OSS) for governance; Spark Declarative Pipelines for building robust and reliable data pipelines that simplify the development of both batch and streaming ETL; and MLflow for streamlining the ML lifecycle with experiment tracking, model packing, and deployment. » Decoupled storage and compute: This separation enables more cost-efficient and scalable systems, unleashing massive amounts of data and concurrent users. » Unified governance for data and AI assets: UC OSS is the central source of a robust governance framework for data in lakehouses. It provides end-to-end visibility and control through audit trails, credential management for different users on different sources of data, transparency such as lineage, data discovery, and data sharing. » AI: Lakehouses support AI — both GenAI and ML — with unified data management so models can use all your corporate data assets. They facilitate ML operations (MLOps) through MLflow to develop, test, and deploy AI models. They provide compliance and governance for models and

Page 17

12 The Data Lakehouse For Dummies, 2nd Databricks Special Edition notebooks themselves — as well as the underlying data. And they enable collaboration among the data scientists, data engineers, and business analysts into one platform, for shared innovation. » Data warehousing (DW)/BI support: BI has been the most common way for business workers to get their insights. Lakehouses enable BI tools to directly access the source data in DW and beyond, reducing staleness, latency, and cost. Instead of needing to maintain multiple copies of data (in a data lake and a warehouse), it can now be stored singularly in the lakehouse. » Applications: Build, deploy, and govern secure applications on your company’s data estate, empowering teams to quickly build with the assistance of AI. With a unified lakehouse for data and AI, you can build secure apps directly on that foundation, running alongside your data and models. » Diverse data types: The best business insights often come from a variety of data types such as structured and unstruc- tured: images, video, audio, semistructured data, and text. Lakehouses support this multimodal approach. » Diverse workloads: Support includes AI, DS, ML, BI, online transaction processing (OLTP), and SQL. Multiple tools may be needed to support all these workloads, but they all rely on the same data repository. » Batch and real-time streaming: Lakehouses support batch processing, which is efficient for managing large volumes of data by processing groups of transactions collectively. Lakehouses also support streaming data — from social media to the Internet of Things (IoT) — to be ingested and analyzed as soon as it’s received. Knowing What Data Lakehouses Bring to the Table In the past, decision-making was mainly based on structured data. Today’s DM systems must be much more flexible and also support unstructured data in any format, enabling advanced AI techniques.

Page 18

CHAPTER 2 Explaining Data Lakehouses 13 With the lakehouse approach, this flexibility is achieved by deeply simplifying the data infrastructure to accelerate innovation. This is especially important because AI is revolutionizing all industries and demands an elastic infrastructure that supports speed and operational efficiency. Multimodal support of a variety of data types One significant difference among approaches is the variety of data being managed. While a data warehouse only handles structured data for most modern analysis and reporting, it’s essential to incorporate all types of data: structured, unstructured, batch, and real time. Lowers overall costs and avoids vendor lock-in Vendor lock-in happens when a customer becomes dependent on a particular vendor for its solutions and services, making the cus- tomer unable to use another vendor’s solution without substan- tial costs to switch. This issue can lead to companies paying many license fees and being forced to pay for creating multiple data copies and writing custom code to make data accessible across third-party systems. This doesn’t work for making your archi- tecture future-proof. Legacy data warehouses come with significant operational costs and vendor lock-in, which make solutions inflexible and not cost-efficient. The lakehouse approach comes with low opera- tional costs and no vendor lock-in, making the data architecture future-proof. Ability to scale and manage all types of workloads Lakehouse architecture provides nearly limitless scalability because it decouples the storage and compute, meaning you can scale one without necessarily needing to pay for the other. Scalable solutions can grow the amount of data and increase workloads, contributing to a company’s competitiveness, qual- ity, and reputation. Lakehouses can also handle all types of data workloads: big and small; long-running and quick-retrieval;

Page 19

14 The Data Lakehouse For Dummies, 2nd Databricks Special Edition batch processing, real-time analytics, and ML/AI. Organizations value this versatility, better pricing structure, and easier man- agement over traditional data warehouses. Data lakehouses also offer serverless compute. This feature allows workloads to run automatically without the need for humans to preprovision and manage the underlying infrastructure and workloads. It enables people to automate the time-consuming server management tasks and instead focus on their more impor- tant tasks. It simplifies complex cloud policies to on-demand, quicker deployment of compute and workflows, leading to effi- ciencies and more optimal resource allocation. The key benefit of the lakehouse is that it allows you to unify and govern all your data and run all your analytics and AI in a single place. Solving Problems with a Lakehouse A lakehouse enables business analytics and AI at a massive scale. A lakehouse approach can solve many challenges. It unifies your data teams on one architecture. It reduces data silos so everyone in your organization can access and process all data types: batch and streaming, structured and unstructured. And it reduces the risk of vendor lock-in by using open formats.

Page 20

CHAPTER 3 Understanding the Underlying Technology 15 Chapter 3 IN THIS CHAPTER » Recognizing data and AI benefits » Addressing data reliability » Using lakehouses for BI and DW activities » Supporting your ML and AI efforts Understanding the Underlying Technology This chapter covers the technology foundations of a well- architected lakehouse on Databricks, focusing on Delta Lake  for data management and Unity Catalog (UC) for governance. It also explores how lakehouses support data intelligence platforms for machine learning (ML) and artificial intelligence (AI). Looking into the Data and AI Benefits Without a proper data lakehouse strategy, data reliability is a big hindrance to extracting value from data across the enterprise — from raw data, batch, and real-time streaming all the way through Extract, Transform, Load (ETL) to be consumed downstream by business intelligence (BI), data warehousing (DW), online trans- action processing (OLTP), and AI.  Failed jobs can corrupt and duplicate data with partial writes. Multiple data pipelines reading and writing concurrently to your data lake can compromise data integrity. Many companies end up with their data pipeline efforts being too complex, coordinating among redundant systems with significant operational challenges to process both batch and

Tip the Site

The Data Lakehouse For Dummies®, 2nd Databricks Special Edition (Ari Kaplan Amit Kara)（Z-Library）

The Data Lakehouse For Dummies®, 2nd Databricks Special Edition (Ari Kaplan Amit Kara)（Z-Library）

Registered users can read the full content for free

AI Reading Assistant

Passage locations

Recommended for You

Tip the Site