Digital Library

Snowflake Data Engineering (Maja Ferle)(Z-Library)

Maja Ferle

Snowflake Data Engineering (Maja Ferle)(Z-Library)

Author Maja Ferle

data
Language English

A practical introduction to data engineering on the powerful Snowflake cloud data platform. Data engineers create the pipelines that ingest raw data, transform it, and funnel it to the analysts and professionals who need it. The Snowflake cloud data platform provides a suite of productivity-focused tools and features that simplify building and maintaining data pipelines. In Snowflake Data Engineering, Snowflake Data Superhero Maja Ferle shows you how to get started. In Snowflake Data Engineering you will learn how to: • Ingest data into Snowflake from both cloud and local file systems • Transform data using functions, stored procedures, and SQL • Orchestrate data pipelines with streams and tasks, and monitor their execution • Use Snowpark to run Python code in your pipelines • Deploy Snowflake objects and code using continuous integration principles • Optimize performance and costs when ingesting data into Snowflake Snowflake Data Engineering reveals how Snowflake makes it easy to work with unstructured data, set up continuous ingestion with Snowpipe, and keep your data safe and secure with best-in-class data governance features. Along the way, you’ll practice the most important data engineering tasks as you work through relevant hands-on examples. Throughout, author Maja Ferle shares design tips drawn from her years of experience to ensure your pipeline follows the best practices of software engineering, security, and data governance. Foreword by Joe Reis. Purchase of the print book includes a free eBook in PDF and ePub formats from Manning Publications. About the technology Pipelines that ingest and transform raw data are the lifeblood of business analytics, and data engineers rely on Snowflake to help them deliver those pipelines efficiently. Snowflake is a full-service cloud-based platform that handles everything from near-infinite storage, fast elastic compute services, inbuilt AI/ML capabilities like vector search, text-to-SQL, code generation, and more. This

Format PDF
Size 2.4 MB
28
Views
0
Downloads
0.00
Total Donations
(First 20 pages)

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

Page 1
M A N N I N G Maja Ferle Foreword by Joe Reis
Page 2
Data engineering building blocks with Snowflake The main building blocks of data engineering pipelines are Extracting data from source systems and ingesting it into Snowflake Performing relevant data transformations Delivering data to downstream consumers for analytics, reporting, data science, or other use cases In addition to these main building blocks, data pipelines include underlying components such as security, data modeling, data governance, software engineering, orchestration, and DataOps. Extraction layer Transformation layer Presentation layer Databases Files Streaming (IOT) Reports Dashboards Machine learning Security Data governanceData modeling Software engineeringOrchestration DataOpsSnowflake Marketplace APIs Snowflake data sharing Snowflake data sharing Querying Snowflake Marketplace Source systems ConsumersSnowflake Transform data Underlying components Extract data from source systems and ingest into Snowflake Present data to downstream consumers
Page 3
Snowflake Data Engineering
Page 4
(This page has no text content)
Page 5
Snowflake Data Engineering MAJA FERLE FOREWORD BY JOE REIS MANN I NG SHELTER ISLAND
Page 6
For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com ©2025 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. The authors and publisher have made every effort to ensure that the information in this book was correct at press time. The authors and publisher do not assume and hereby disclaim any liability to any party for any loss, damage, or disruption caused by errors or omissions, whether such errors or omissions result from negligence, accident, or any other cause, or from any usage of the information herein. Manning Publications Co. Development editor: Katie Sposato Johnson 20 Baldwin Road Technical editor: Daan Bakboord PO Box 761 Review editor: Radmila Ercegovac Shelter Island, NY 11964 Production editor: Kathy Rossland Copy editor: Kari Lucke Proofreader: Melody Dolab Technical proofreader: Rohan Pasalkar Typesetter: Dennis Dalinnik Cover designer: Marija Tudor ISBN: 9781633436855 Printed in the United States of America
Page 7
brief contents PART 1 INTRODUCING DATA ENGINEERING WITH SNOWFLAKE .........................................................1 1 ■ Data engineering with Snowflake 3 2 ■ Creating your first data pipeline 17 PART 2 INGESTING, TRANSFORMING, AND STORING DATA ............................................................35 3 ■ Best practices for data staging 37 4 ■ Transforming data 58 5 ■ Continuous data ingestion 83 6 ■ Executing code natively with Snowpark 102 7 ■ Augmenting data with outputs from large language models 125 8 ■ Optimizing query performance 144 9 ■ Controlling costs 169 10 ■ Data governance and access control 189v
Page 8
BRIEF CONTENTSviPART 3 BUILDING DATA PIPELINES .........................................205 11 ■ Designing data pipelines 207 12 ■ Ingesting data incrementally 224 13 ■ Orchestrating data pipelines 248 14 ■ Testing for data integrity and completeness 272 15 ■ Data pipeline continuous integration 297
Page 9
contents foreword xiv preface xv acknowledgments xvii about this book xix about the author xxiii about the cover illustration xxiv PART 1 INTRODUCING DATA ENGINEERING WITH SNOWFLAKE ...............................................1 1 Data engineering with Snowflake 3 1.1 Snowflake for data engineering 4 Snowflake architecture 4 ■ Snowflake features for data engineering 5 1.2 Responsibilities of a Snowflake data engineer 6 Extracting data from source systems 8 ■ Performing data transformations 9 ■ Presenting data to downstream consumers 10 ■ Applying underlying components 10 1.3 Building data pipelines 13 1.4 Data engineering with Snowflake applications 14vii
Page 10
CONTENTSviii2 Creating your first data pipeline 17 2.1 Setting up your Snowflake account 19 2.2 Staging a CSV file 20 2.3 Loading data from a staged file into a target table 22 Loading data from a staged file into a staging table 24 Merging data from the staging table into the target table 26 2.4 Transforming data with SQL commands 28 2.5 Automating the process with tasks 29 PART 2 INGESTING, TRANSFORMING, AND STORING DATA ..................................................35 3 Best practices for data staging 37 3.1 Creating external stages 39 Configuring a storage integration 40 ■ Creating an external stage using a storage integration 43 ■ Creating an external stage using credentials 44 ■ Loading data from staged files into a staging table 45 ■ Avoiding duplication when loading data from staged files 47 ■ Using a named file format 48 3.2 Viewing stage metadata with directory tables 50 3.3 Preparing data files for efficient ingestion 51 File sizing recommendations 51 ■ Organizing data by path 52 3.4 Building pipelines with external tables 53 Querying data in external stages with external tables 54 Using materialized views to improve query performance 55 4 Transforming data 58 4.1 Ingesting semistructured data from cloud storage 60 Creating a storage integration 61 ■ Creating an external stage 63 ■ Examining the JSON structure 63 ■ Ingesting JSON data into a VARIANT data type 65 4.2 Flattening semistructured data into relational tables 66 4.3 Encapsulating transformations with stored procedures 70 Creating a basic stored procedure 73 ■ Including a return value in a stored procedure 74 ■ Implementing exception handling in stored procedures 75
Page 11
CONTENTS ix4.4 Adding logging to stored procedures 75 4.5 Building robust data pipelines 78 5 Continuous data ingestion 83 5.1 Comparing bulk and continuous data ingestion 85 5.2 Preparing files in cloud storage 85 Creating a storage integration 86 ■ Creating an external stage 87 5.3 Configuring Snowpipe with cloud messaging 89 Configuring event grid messages for blob storage events 90 Creating a notification integration 92 ■ Creating a pipe object 93 ■ Ingesting data continuously 95 ■ Flattening the JSON structure to relational format 96 5.4 Transforming data with dynamic tables 98 6 Executing code natively with Snowpark 102 6.1 Introducing Snowpark 104 6.2 Creating a Snowpark procedure in a worksheet 105 6.3 Using the SQL API from a local development environment 110 Installing and configuring the local development environment 110 Creating a Snowflake session 110 ■ Providing credentials in a configuration file 111 ■ Querying data and executing SQL commands 113 6.4 Generating a date dimension in Snowpark Python 113 6.5 Working with data frames 115 6.6 Ingesting data from a CSV file into a Snowflake table 118 6.7 Transforming data with data frames 121 7 Augmenting data with outputs from large language models 125 7.1 Configuring external network access 126 7.2 Calling an API endpoint from a Snowpark function 129 Constructing the UDF that retrieves customer reviews 130 Interpreting the results from the UDF 132 ■ Storing the customer reviews in a table 134 7.3 Deriving customer review sentiments 135
Page 12
CONTENTSx7.4 Interpreting order emails using LLMs to save time 138 Creating a stored procedure that interprets customer emails 138 Constructing the prompt 139 ■ Saving the CSV result to a table 140 ■ Evaluating the output 141 8 Optimizing query performance 144 8.1 Getting data from the Snowflake Marketplace 145 8.2 Performing analysis of geographical data 148 Snowflake’s geography functions 148 ■ Copying data from the shared database 150 ■ Viewing query execution parameters using the query profile 152 8.3 Understanding Snowflake micro-partitions 154 A conceptual example of micro-partitions 154 ■ Micro-partition pruning 156 8.4 Optimizing storage with clustering 157 Viewing clustering information 157 ■ Adding clustering keys to a table 159 ■ Monitoring the clustering process 160 ■ Viewing improved query execution after clustering 161 8.5 Improving query performance with search optimization 162 Adding search optimization to a table 163 ■ Reviewing query performance after adding search optimization 164 8.6 General tips for improving query performance 165 Writing efficient SQL queries 165 ■ Identifying queries that are candidates for optimization 166 9 Controlling costs 169 9.1 Understanding Snowflake costs 170 Total Snowflake cost 171 ■ Compute resources cost 172 Virtual warehouse credits 172 9.2 Sizing virtual warehouses 173 Using persisted query results 176 ■ Comparing query statistics between differently sized warehouses 177 ■ Optimizing query performance to reduce spilling 179 9.3 Optimizing performance with data caching 181 Illustrating the metadata cache 182 ■ Utilizing the warehouse cache efficiently 183
Page 13
CONTENTS xi9.4 Reducing query queuing 183 Examining queuing 184 ■ Limiting concurrently running queries 185 9.5 Monitoring compute consumption 186 10 Data governance and access control 189 10.1 Role-based access control 190 System-defined roles 191 ■ Custom roles 191 Designing RBAC 192 10.2 Securing data with row access policies 197 10.3 Protecting sensitive data with masking policies 201 PART 3 BUILDING DATA PIPELINES ...............................205 11 Designing data pipelines 207 11.1 Designing data pipelines 208 Extracting data 208 ■ Comparing data pipeline patterns 209 Choosing data transformation layers 212 ■ Organizing data warehouse layers 213 ■ Creating schemas with access control 215 11.2 Building a sample data pipeline 216 Implementing the extraction layer 216 ■ Implementing the staging layer 218 ■ Implementing the data warehouse layer 220 Implementing the reporting layer 222 12 Ingesting data incrementally 224 12.1 Comparing data ingestion approaches 226 Full ingestion 226 ■ Incremental ingestion 227 12.2 Preserving history with slowly changing dimensions 228 SCD type 2 228 ■ Append-only strategy 229 ■ Designing idempotent data pipelines 230 12.3 Detecting changes with Snowflake streams 230 Ingesting files from cloud storage incrementally 231 ■ Preserving history when ingesting data incrementally 236 12.4 Maintaining data with dynamic tables 242 Deciding when to use dynamic tables 243 ■ Querying historical data 244
Page 14
CONTENTSxii13 Orchestrating data pipelines 248 13.1 Orchestrating with Snowflake tasks 250 Creating a schema to store the orchestration objects 251 Designing the orchestration tasks 252 ■ Creating tasks with dependencies 253 13.2 Sending email notifications 257 13.3 Orchestrating with task graphs 258 Designing the task graph 259 ■ Creating the root task 261 Creating the finalizer task 262 ■ Viewing the task graph 263 13.4 Monitoring data pipeline execution 264 Adding logging functionality to tasks 264 ■ Summarizing logging information in an email notification 266 13.5 Troubleshooting data pipeline failures 269 14 Testing for data integrity and completeness 272 14.1 Data testing methods 273 Performing data testing as steps in the pipeline 273 ■ Performing data testing independently of the pipeline 275 14.2 Incorporating data testing steps in the pipeline 275 Constructing the partner data quality task 277 ■ Constructing the product data quality task 280 ■ Executing the pipeline with the data testing tasks 280 14.3 Applying the Snowflake data metric functions 282 System-defined data metric functions 283 ■ User-defined data metric functions 284 ■ Viewing data metric function details 286 14.4 Alerting users when data metrics exceed thresholds 287 14.5 Detecting data volume anomalies 289 Generating random data 290 ■ Displaying data as a line chart in Snowsight 292 ■ Working with the anomaly detection model 292 15 Data pipeline continuous integration 297 15.1 Separating the data engineering environments 299 15.2 Database change management 300 Comparing the imperative and the declarative approach to DCM 300 ■ Organizing the code in the repository 302 15.3 Configuring Snowflake to use Git 303 Creating a Git repository stage 304 ■ Executing commands from a Git repository stage 307
Page 15
CONTENTS xiii15.4 Using the Snowflake CLI command line interface 308 Installing and configuring Snowflake CLI 308 ■ Executing scripts with Snowflake CLI 310 ■ Continuous integration with Snowflake CLI 311 15.5 Connecting to Snowflake securely 313 Configuring key-pair authentication 313 15.6 Applying what we learned in real-world scenarios 314 appendix A Configuring your Snowflake environment 317 appendix B Snowflake objects used in the examples 321 index 333
Page 16
foreword Data engineering has witnessed a massive transformation with the advent of cloud data platforms. At the forefront of this transformation (no pun intended) is Snow- flake. The ability to seamlessly scale, compute and store, share and collaborate on data, and run AI workloads all in one simple-to-use platform has made Snowflake the platform of choice for data engineers working at businesses seeking to unlock the true potential of their data. With the explosion of data and AI, data engineers have never been more critical. Yet data engineers must constantly improve their knowledge and skills to stay ahead. This book, Snowflake Data Engineering, serves as an indispensable guide for aspiring and seasoned data engineers alike, providing a comprehensive and practical explora- tion of the art and science of data engineering within the Snowflake ecosystem. Whether you are new to Snowflake or seeking to expand your data engineering expertise, this book will serve as your trusted companion on your journey to mastering the art of data engineering in the cloud. Maja’s done a fantastic job of writing this book, which is the perfect blend of technical explanations and practical examples. By the end of this book, you’ll be well-prepared to tackle the challenges of the ever-evolving data landscape and empower your organization to make informed decisions and achieve its strategic objectives. Enjoy! —JOE REIS Author, Data Engineer, “Recovering Data Scientist”xiv
Page 17
preface After years of using on-premises data analytics technologies, I was intrigued when Snow- flake emerged. The concept of a cloud-provisioned database platform without the has- sles of physical installation, sizing, purchasing, and upfront costs was captivating. My first experience with a Snowflake project was a revelation. The ease with which I could set up the infrastructure without needing heavy initial investment and commit- ment was a game-changer. The pay-as-you-go model further added to the flexibility and cost-effectiveness of the platform. Over the years, Snowflake has evolved into a comprehensive platform that can han- dle a wide range of data-related tasks. It combines storage, elastic compute, built-in AI capabilities, native applications with Python and Streamlit, data sharing, and integra- tion with third-party services and tools for data insights. With all these features hosted on a single platform, you can establish governance to manage security, compliance, privacy, and access to the data and applications. Writing a detailed book about all the features that Snowflake provides, complete with real-world examples and exercises, would be overwhelming for an author to write and for the reader to comprehend. Therefore, when I discussed the book’s content with the publisher, I asked myself: What would be the first step for someone starting to use Snowflake? Since Snowflake stores data at its core, the initial task would be to bring data into the platform to enable other functionalities. This pro- cess falls under data engineering, which involves building data pipelines that ingest data from the source, transform it as needed, and deliver it to downstream consum- ers for analytics.xv
Page 18
PREFACExvi Snowflake is hosted in the cloud, so the data ingestion process differs from tradi- tional on-premises databases. You can’t just install an ODBC driver and select the data as you would in an on-premises environment. Instead, you need to know how to retrieve data from cloud object storage, Snowflake data sharing, APIs, or third-party tools and connectors. I cover that in this book, along with detailed explanations and examples.
Page 19
acknowledgments This book would not have been possible without my acquisition editor, Jonathan Gen- nick. It is my second time working with him. I trust his judgment regarding decisions about what to include and exclude in the book, understanding the target audience, and constructing the table of contents. My development editor, Katie Sposato Johnson, was there every step of the way, providing meaningful feedback, suggestions, and encouragement. She had a keen eye for pointing out sections where I veered off course during the writing process and steered me back on track. Thank you, Katie, for always being there when I needed your input. Thanks to my technical editor, Daan Baakboord, who was meticulous in his techni- cal review, checking and rerunning my code, and providing valuable feedback and suggestions for improvement during the writing process. Daan Bakboord is a Snow- flake AI Data Cloud Consultant. He is one of the first Snowflake Data Superheroes in the world. Daan, it was wonderful having you on board! As I started writing this book, I relied on various sources such as books, blogs, training materials, and whitepapers. Some of the sources I used included Snowflake: The Definitive Guide by Joyce K. Avila and the “Snowflake for Data Engineering” train- ing course by Tomáš Sobotík. I want to express my gratitude to Joyce and Tomáš for being two of my early reviewers. Their expertise and feedback on the initial chapters of my manuscript were invaluable and provided the encouragement I needed to con- tinue. Additionally, I consulted Fundamentals of Data Engineering, which was coau- thored by Joe Reis, and I’m thankful to Joe for agreeing to write the foreword.xvii
Page 20
ACKNOWLEDGMENTSxviii I am also grateful to my technical proofreader, Rohan Pasalkar, who was very help- ful in reviewing the code for the exercises in the book. I want to thank the entire Manning Publications team for their fantastic work pre- paring the book. This includes the marketing, graphics, and production teams, some of whom I never met and whose names I don’t know. Thanks to all the reviewers who read my manuscript in various stages of completion and provided comments from the point of view of the target reader: Alain Couniot, Albert Nogués, Andriani Stylianou, Ankit Virmani, David Allen Blubaugh, David Krief, Doyle Turner, Emanuele Piccinelli, Eros Pedrini, Gabor Gollnhofer, Hilde Van Gysel, Jesús Antonino Juárez Guerrero, Jonathan Woodard, Krzysztof Kamyczek, Luke Kupka, Madiha Khalid, Nadir Doctor, Oliver Korten, Pavel Filatov, Peter G. Bishop, Rambabu Posa, Richard B. Ward, Sambasiva Andaluri, Satej Sahu, Sean Booker, Simone Sguazza, Shivani Mayekar, Sriram Macharla, Tobias Kaatz, and Ubaldo Pescatore. Their feedback also helped shape the book’s contents. I appreciate my employer, In516ht, for allowing me to work on exciting projects where I could develop my Snowflake data engineering skills. Finally, I thank Snowflake for providing us with an extraordinary cloud data platform.
The above is a preview of the first 20 pages. Register to read the complete e-book.

Support Author

0.00
Total Amount (¥)
0
Donation Count
Please enter an amount Minimum ¥1

You will be redirected to Alipay to complete payment, then return here.

Recommended for You

Loading recommended books...
Failed to load, please try again later
Back to List