Digital Library

Data Discovery in Data Lakes (Abedjan, Ziawasch, Esmailoghli, Mahdi etc.)(Z-Library)

,
,
,
,
,

Share E-Book

Data Discovery in Data Lakes (Abedjan, Ziawasch, Esmailoghli, Mahdi etc.)(Z-Library)

Author

,
,
,
,
,

Rating 3.5 AI initial rating

Log in to rate

Data
Language English

No Description

Format PDF
Size 6.2 MB
(First 20 pages)

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

Page 1
Ziawasch Abedjan Mahdi Esmailoghli Sainyam Galhotra Data Discovery in Data Lakes
Page 2
Data Discovery in Data Lakes
Page 3
Ziawasch Abedjan • Mahdi Esmailoghli • Sainyam Galhotra Data Discovery in Data Lakes
Page 4
Ziawasch Abedjan Berlin Institute for the Foundations of Learning and Data (BIFOLD) Technische Universität Berlin Berlin, Germany Sainyam Galhotra Department of Computer Science Cornell University Ithaca, NY, USA Mahdi Esmailoghli David R. Cheriton School of Computer Science University of Waterloo Waterloo, ON, Canada ISBN 978-3-032-30821-4 ISBN 978-3-032-30822-1 (eBook) https://doi.org/10.1007/978-3-032-30822-1 © The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG 2027 This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. The publisher, the authors and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. This Springer imprint is published by the registered company Springer Nature Switzerland AG The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland If disposing of this product, please recycle the paper.
Page 5
Preface At the time of writing, data discovery had emerged as a highly active research area, with entire sessions at leading database conferences, such as SIGMOD, VLDB, and ICDE, devoted to new discovery methods, indexing techniques, and benchmarks. At the same time, we observed increasing fragmentation in terminology and communication, which made it progressively harder to compare results, relate contributions to one another, and place new work into a coherent broader context. This motivated us to collect and organize the relevant body of research known to us and to develop a tutorial on data discovery for VLDB 2025, which in turn led to the invitation to write this book. This book builds on that effort by offering a deeper and more systematic treatment of the field. Our goal is not only to survey the large and rapidly growing body of research in data discovery but also to contextualize it, discuss its assumptions and limitations critically, and highlight directions that we believe are especially promising for future work. As the field is continuously growing, we acknowledge that we cannot claim to have covered all research in this area. Berlin, Germany Waterloo, ON, Canada Ithaca, NY, USA April 2026 Ziawasch Abedjan Mahdi Esmailoghli Sainyam Galhotra Competing Interests The authors have no competing interests to declare that are relevant to the content of this manuscript. v
Page 6
Contents 1 Data Discovery: Motivation and Challenges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 Introduction and Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Data Lakes: Background and History. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.3 Challenge: Ad Hoc Data Integration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.4 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1.4.1 Government Data Portals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.4.2 Data Marketplaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1.4.3 Pollution Explanation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.5 Book Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2 Discovery Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.2 Data Lake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.2.1 Centralized Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.2.2 Federated Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.2.3 Hybrid Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 2.3 Privacy and Governance Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.4 Multimodal Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.5 Temporal Dimension of Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 2.6 Summary of Discovery Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3 Discovery Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.1 Introduction and Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.2 Keyword Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.2.1 Keyword Search in Documents and Tables . . . . . . . . . . . . . . . . . . 34 3.2.2 Keyword Search Prototypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.2.3 Further Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.3 Union Search. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3.3.1 Semantics of Union Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 3.3.2 Unionability of Columns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 vii
Page 7
viii Contents 3.3.3 Union Search Prototypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 3.4 JOIN Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.4.1 Joinability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 3.4.2 Utility Augmented Joinability Scores . . . . . . . . . . . . . . . . . . . . . . . . 46 3.5 Operation Sequences and Compositions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.5.1 Operation Sequences and Grammars . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.5.2 Domain-Specific Languages for Data Discovery . . . . . . . . . . . . 52 3.6 Natural-Language Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.6.1 Natural-Language Questions Over Large Repositories . . . . . . 54 3.6.2 Multi-table QA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.7 Operations in Distributed Repositories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 3.7.1 Discovery Operations in Federated Environments . . . . . . . . . . . 56 3.8 Additional Data Discovery Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 3.8.1 Privacy-Constrained Data Discovery . . . . . . . . . . . . . . . . . . . . . . . . . 59 3.8.2 Multimodal Data Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 3.9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 4 Indexing Strategies for Data Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 4.2 Value-Based Indexes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 4.2.1 Inverted Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 4.2.2 Inverted Index for Multicolumn Retrieval . . . . . . . . . . . . . . . . . . . . 75 4.3 Column-Based Indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 4.3.1 MinHash and Locality-Sensitive Hashing for Column Similarity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.3.2 Beyond Minhash LSH. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 4.4 Indexes for Discovering Correlating Columns . . . . . . . . . . . . . . . . . . . . . . . 89 4.4.1 Sketch-Based Correlation Indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 4.4.2 Exact Correlation Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 4.4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 4.5 Row-Based Indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 4.6 Table-Based Indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 4.7 Graph-Based Indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 4.7.1 Similarity-Based Navigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 4.7.2 Topic-Driven Navigation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 4.7.3 Knowledge Graph-Enhanced Discovery . . . . . . . . . . . . . . . . . . . . . 99 4.7.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 4.8 Embedding-Based and Foundation Model Approaches . . . . . . . . . . . . . 99 4.8.1 ANN Indexes over Dense Column Embeddings . . . . . . . . . . . . . 100 4.8.2 Hybrid Retrieval: Combining Full-Text and Vector Indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 4.8.3 LLM-Augmented Discovery Pipelines with Lightweight Indexing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
Page 8
Contents ix 4.8.4 Self-Supervised Representation Learning for Table Relationships . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 4.8.5 Foundation Models as Semantic Reasoners . . . . . . . . . . . . . . . . . . 103 4.9 Coresets for Discovery Indexes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.9.1 Distribution-Aware Discovery and the Coreset Construction Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 4.10 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 5 Data Discovery Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 5.2 The Demand for End-to-End Discovery Systems . . . . . . . . . . . . . . . . . . . . 116 5.2.1 Survey by Esmailoghli et al. (2023) . . . . . . . . . . . . . . . . . . . . . . . . . . 116 5.2.2 Survey by Hulsebos et al. (2024) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 5.3 Academic Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 5.3.1 AURUM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 5.3.2 AUCTUS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 5.3.3 BLEND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 5.4 Industrial Discovery Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 5.4.1 Enterprise Discovery Platforms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 5.4.2 Marketplace Infrastructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 6 Benchmarking Data Discovery Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 6.1 Challenges in Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 6.2 Effectiveness in Data Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 6.2.1 Background: Keyword Search in Information Retrieval. . . . . 137 6.2.2 World Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 6.2.3 Adaptation of Evaluation Metrics for Tabular Data Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 6.2.4 Evaluating JOIN Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 6.2.5 Evaluating UNION Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 6.3 Available Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 6.3.1 Synthesized Lakes for CWA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 6.3.2 Limitations of CWA Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 6.3.3 Utility-Driven Data Discovery for OWA . . . . . . . . . . . . . . . . . . . . . 142 6.4 Efficiency in Data Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 6.4.1 Deployment Costs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 6.4.2 Query Runtime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 6.4.3 Ease of Use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 6.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
Page 9
x Contents 7 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 7.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 7.2 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 7.2.1 Holistic Data Discovery Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 7.2.2 Time and Lake Versioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 7.2.3 Decentralized Data Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 7.2.4 Multimodal Data Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 7.2.5 Agentic Data Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Page 10
Chapter 1 Data Discovery: Motivation and Challenges Abstract Data lakes have emerged as a dominant paradigm for retaining hetero- geneous data in its original form, but their flexibility comes with a fundamental usability challenge: They are typically underspecified, weakly documented, and difficult to exploit in an ad hoc manner. This chapter introduces data discovery as the set of methods and systems that make such repositories accessible by enabling users to search for relevant datasets without requiring detailed prior knowledge of schemas or contents. We motivate data discovery through the characteristics and historical evolution of data lakes, tracing their roots to the rise of big data infrastructures and the shift from schema-on-write to retain-first data management. Against this background, the chapter positions data discovery as a bridge between information retrieval and data integration: Like information retrieval, it addresses user-specified needs through search. Unlike traditional retrieval, it must identify datasets that are not only relevant but also usable for downstream tasks such as augmentation, joining, unioning, and exploratory analysis. We then outline the core architectural view of a vanilla data discovery system, where user queries, table artifacts, and discovery operations are supported by dedicated indexes and matching techniques. Finally, the chapter frames the scope of the book by highlighting the main technical themes that follow, including discovery settings, operations, indexing methods, end-to-end systems, and benchmarking. Overall, the chapter establishes data discovery as a central systems problem for turning raw, heterogeneous data lakes into practically usable analytical assets. Keywords Data discovery · Data lakes · Market places · Open data 1.1 Introduction and Overview Data lakes, whether intentionally created or emerging as a byproduct of organiza- tional processes or open data initiatives, share a common challenge: They are often underspecified and underdocumented (Fernandez et al. 2016; Abedjan et al. 2015). Unlike curated data warehouses, where the schema and purpose are clearly defined, data lakes typically lack predefined objectives. Despite this, there is significant © The Author(s), under exclusive license to Springer Nature Switzerland AG 2027 Z. Abedjan et al., Data Discovery in Data Lakes, https://doi.org/10.1007/978-3-032-30822-1_1 1
Page 11
2 1 Data Discovery: Motivation and Challenges interest in leveraging these repositories for augmenting in-use datasets or pure exploratory purposes. To support such applications, data discovery as a process becomes essential, enabling users to interact with the data lake not by relying on detailed schema knowledge, but by using search and matching techniques. By searching with user-provided keywords or table artifacts, users aim to uncover datasets that are relevant for their use cases. Data discovery is currently an active field of research, as building scalable solutions to effectively capture user intent, such as augmenting a given dataset fragment, presents various challenges. A key issue is the need to index thousands to millions of tables for fast value lookups and efficient alignment to evaluate joinability (Galhotra et al. 2023; Esmailoghli et al. 2022; Zhu et al. 2019) and unionability (Nargesian et al. 2018) with the user (query) table. After identifying relevant tables, assessing the usefulness of the retrieved results remains a complex problem. Often, users rely on proxy metrics, such as the downstream machine learning performance of the augmented dataset, to assess its effectiveness (Galhotra et al. 2023; Santos et al. 2021; Esmailoghli et al. 2021; Esmailoghli and Abedjan 2020). Finally, in distributed data lakes, additional constraints, such as privacy regulations (Huang et al. 2023) and pricing structures, further complicate the process. In such cases, the usefulness and relevance of datasets must often be assessed based on available metadata and summaries, rather than direct access to the full datasets. The emerging phenomena of data lakes and the necessity for data discovery naturally attracted broad attention from the data management research community. The sheer amount of novel algorithms, systems, and benchmarks lead to several efforts in systematic definition and categorization of the individual aspects of data discovery materialized as surveys and tutorials. Idreos et al. (2015) discuss efficient algorithms and systems for data exploration in the broader sense. Paton and Wu (2024) and Dong et al. (2021) cover various aspects of dataset search and navigation. Zhang and Balog (2019) discuss data discovery from the lens of information retrieval and give perspectives on Web table extraction. Nargesian et al. (2019) elevate the discussion toward discovery operations and metadata management for data lakes. The Vanilla Data Discovery System Figure 1.1 depicts a conceptual architecture of data discovery systems and its components for discovering relational artifacts. The user formulates a discovery task that typically aims at enhancing an existing table or table fragment through additional features, rows, or individual values. The formulation of the discovery task can either be the upload of a dataset whose relevant attributes for search have been marked or a more complex discovery query formulated around an input dataset fragment (Esmailoghli et al. 2025). A discovery task can range from exploratory, such as identifying all related tables, to more complex objectives, such as augmenting a dataset across multiple dimensions to improve its completeness and utility. Such tasks are then typically carried out on top of more basic operations, such as keyword search, join discovery, and union discovery. Such operations themselves
Page 12
1.2 Data Lakes: Background and History 3 Fig. 1.1 Vanilla data discovery system architecture are optimized retrieval algorithms tailored to the underlying dedicated indexes. The result of a query is a set of relevant datasets, potentially prepared for or already aligned with the input query dataset. In this book, we have a strong focus on algorithmic details and how index structures and operations have evolved and can be categorized. We further project how these approaches relate and can extend to distributed scenarios and can be combined inside holistic systems. We cover recent advances and discuss the topic from the database management perspective. Methodologically, this book is based on our previous study of data discovery use cases and our experience in building systems and algorithms that address some of its emergent challenges. Our goal is to give a structured overview of data discovery software components and dive into technical aspects that differentiate individual prototypes and approaches. A meta-goal is to bridge the language and perspective differences on what is counted as data discovery and how discovery systems in database research can be compared. While data discovery and data lakes encompass multimodal data structures, this book has a strong focus on covering recent advances in discovering relational tables. To understand the concepts that are introduced in this book, a solid background in database management and familiarity with machine learning and AI trends is recommended. 1.2 Data Lakes: Background and History Hai et al. (2023) give a systematic timeline of how the term data lakes started to surface in the early 2010s and gradually boomed midway through the same decade. The advent of data lakes is closely tied to the rise of big data technologies, such as Hadoop,1 Spark (Zaharia et al. 2012), and Flink (Alexandrov et al. 2014), in early 2010s and the resulting change in enterprise requirements. As organizations 1 http://hadoop.apache.org/
Page 13
4 1 Data Discovery: Motivation and Challenges began collecting high-volume, high-variety data such as logs, clickstreams, sensor data, and semi-structured records, the traditional data warehouse model increasingly struggled with long lead times and up-front modeling commitments, since data typically had to be transformed to fit a predefined structure before it could be stored and analyzed. In contrast, the big data stack made it practical to retain large amounts of data cheaply and process it at scale, shifting the emphasis from pre-modeling toward preserving raw data for unknown future questions. In this context, James Dixon, who was the CTO of Pentaho in 2010, popularized the term data lake to describe an alternative to “bottled” data marts: Rather than packaging data immediately for a specific consumption pattern, data flows into a lake in a more natural state, from which different users can “examine, dive in, or take samples” depending on their needs (Dixon 2010). Subsequent work has reinforced this framing by explicitly contrasting the warehouse’s schema-on-write orientation with the lake’s retain-first, interpret-later approach as a response to information loss and limited reuse under early transformation. Over the course of time, data lakes are a natural organizational outcome of three forces arriving at the same time: 1. Data collection becoming ubiquitous through applications, logs, sensors, cus- tomer interactions 2. Storage becoming cheap enough to keep “everything” and 3. A belief that retained data will eventually yield business value, whether through analytics, reporting, personalization, forecasting, or new product capabilities In practice, a data lake is often less a single deliberate project than an emergent pattern: Teams keep landing raw and semi-structured data in a central repository because it is fast, flexible, and feels future proof, even if the specific downstream use cases are not fully defined yet. That gravity of “store now, figure out later” makes data lakes both powerful and messy, especially as organizations scale and need governance, quality controls, and cost discipline. The emergence of data lakes has created strong incentives for data processing and platform companies to productize the hard parts: ingesting diverse data, managing compute vs. storage economics, enabling SQL and machine learning workloads, enforcing security and governance, and supporting reliable pipelines without locking teams into fragile bespoke systems. Vendors such as Snowflake,2 Databricks,3 and AWS,4 alongside others like Google Cloud5 and Microsoft,6 have built competing and complimentary approaches, such as cloud data warehouses and Lakehouse architectures, managed storage and query services, catalog/governance layers, and end-to-end analytics/AI platforms, because the market demand is 2 https://www.snowflake.com/ 3 https://www.databricks.com/ 4 https://aws.amazon.com/ 5 https://cloud.google.com/solutions/data-lakehouse 6 https://azure.microsoft.com/en-us/solutions/data-lake
Page 14
1.3 Challenge: Ad Hoc Data Integration 5 Table 1.1 Representative definition statements for “data lake” in the literature Source Definition excerpt (verbatim) Emphasis Terrizzano et al. (2015) “A centralized repository containing virtually inexhaustible amounts of raw (or minimally curated) data” Centralization and accumulation of raw data Nargesian et al. (2019) “Current data lakes provide reliable storage for datasets . . . along with a suite of tools for . . . data discovery, extraction, cleaning, and integration” Platform with operations for ad hoc integration Sawadogo and Darmont (2021) “A data lake is a large, raw data repository that stores and manages all company data bearing any format” Enterprise scope; any format Hai et al. (2023) “Data lakes are repositories storing raw data in its original formats and providing a common access interface” Original formats; unified access consistent: Organizations want the optionality of a lake with the usability and performance of curated systems. Table 1.1 contains several examples on how data lakes have been defined in the literature. While Terrizzano et al. (2015) put the focus on centralization, Nargesian et al. (2019) define data lakes as a combination of storage and processes. Sawadogo and Darmont (2021) highlight the relevant context of enterprise datasets, and Hai et al. (2023) emphasize the fact that data is provided in its raw and original format. In this book, we adapt to a definition that connects the commonalities of these definitions. We consider a data lake to be a repository that keeps heterogeneous data from many sources in its original form and defers imposing structure until use. In this view, the distinguishing property is not a particular processing stack but rather the commitment to retain heterogeneous data “as-is” while relying on subsequent discovery, access, and governance capabilities to make it usable for analytics (Quix and Hai 2018; Hai et al. 2023; Sawadogo and Darmont 2021; Paton et al. 2024; Dixon 2010). We will give a more formal definition of data lakes in Chap. 2, where we talk about its fundamental settings. 1.3 Challenge: Ad Hoc Data Integration So far, the aforementioned processing solutions by tech vendors serve exploratory analysis or well-curated data transformation workflows. As a result, the entirety of information is hardly accessible to a single individual without significant efforts in
Page 15
6 1 Data Discovery: Motivation and Challenges exploration. A common problem that has so far been ignored is how to use the lake in an ad hoc manner. This is where data discovery as a discipline arises and attempts to build solutions that support not only the user who is aware of the content and schema inside a data lake but also those who only have implicit knowledge about the datasets stored. What we mean with implicit knowledge is that the actual schema of tables and their constraints are not known, but the user might know that certain datasets might exist that cover an information need. At this point there is a strong initial resemblance of data discovery to information retrieval (IR). In fact, information retrieval and data discovery address related but distinct problems. The goal of information retrieval is to satisfy an information need by retrieving relevant documents or passages, whereas data discovery targets an information need or a dataset need, i.e., helping users find data assets, such as tables, files, streams, and features, that can be analyzed or reused, not just read. Accordingly, information retrieval queries are typically expressed as unstructured keywords, while data discovery queries often combine keywords with structured elements such as schema terms, columns, tables, or quality constraints. Finally, information retrieval systems usually present a ranked list of results optimized for topical relevance, whereas data discovery systems commonly go beyond ranking to support ad hoc integration, through previewing, joining, sampling, imputing, and assembling candidate datasets to make them immediately usable in downstream analytics and decision-making. With the goal for ad hoc data integration, data discovery faces several data integration related challenges that go beyond the performance challenges that are addressed by current vendors. The questions of relevance of discovered objects and their usefulness require heuristics-based solutions that rely on metrics, such as coverage and similarity. For each individual data discovery operation, such as Union or Join search, which we discuss in detail in Chap. 3 of this book, both exact matches of values and fuzzy matches are relevant. Further questions include how to holistically integrate such operations on top of unified data models, how to optimize data discovery tasks consisting of multiple operations holistically, and how to execute such operations in scenarios where data is distributed and access is subject to constraints. 1.4 Case Studies To motivate the case for researching data discovery systems and operations, we provide three different examples of real-world scenarios where data discovery capabilities support end users and analysis. First we discuss the emergence of governmental open data frameworks and then the data market place landscape, and finally we conclude with a concrete example of ad hoc data integration for inference.
Page 16
1.4 Case Studies 7 1.4.1 Government Data Portals Open Government Data Portals (OGDPs) exist on multinational (EU) national (USA, UK, Canada, and Germany) and municipal (NYC) levels. They serve as open-data catalogs that are increasingly modeled as large, heterogeneous “open data lakes” of tables rather than as well-curated databases. For example, the US federal portal grew its indexed files by an order of magnitude since 2016. Further, a crawl of US/UK/Canada portals already contained 215,393 CSV7 tables with millions of attributes, yielding a repository whose scale and publisher heterogeneity make a priori knowledge of available schemata unlikely for end users (Miller 2018). • Data.gov: The US federal open-data portal8 is an archetypal OGDP that foregrounds discovery and reuse at catalog scale: It presents itself as “the home of the U.S. Government’s Open Data,” explicitly emphasizing downstream reuse for research, application development, and visualization to inform decisions, drive innovation, and strengthen transparency. Operationally, its public-facing catalog is powered by CKAN9 and exposes a Comprehensive Knowledge Archive Network-Application Programming Interface (CKAN-API). The plat- form documentation stresses that the CKAN end point is a metadata service— containing dataset descriptions and URLs rather than necessarily hosting the underlying data. So discovery often entails navigating from portal metadata to distributed agency systems. This architecture makes data.gov a representative case where data discovery quality depends on metadata completeness and consistency. • Data.europe.eu: The European portal10 aims to serve as a “single point of access to open data from European countries” and EU institutions, and it is explicitly engineered around semantic interoperability and cross-portal harvesting. A key technical feature is that portal metadata is stored as Resource Description Framework (RDF) and are queryable via a public SPARQL end point (Angles and Gutierrez 2008), aligning with the Data Catalogue vocabulary - Application Profile (DCAT-AP) profile that standardizes public-sector dataset descriptions across Europe. The design rationale and implementation details are documented in the EGOV 2019 paper by Kirstein et al., which describes the then European Data Portal as a central access point harvesting metadata from dozens of national providers and bridging conventional portal expectations regarding UI and API compatibility with native RDF storage and quality assessment (Kirstein et al. 2019). Conceptually, the European portal thereby illustrates a portal model where data discovery is supported not only through conventional catalog search but also 7 comma-separated values 8 data.gov 9 https://ckan.org/ 10 data.europe.eu
Page 17
8 1 Data Discovery: Motivation and Challenges through machine-actionable graph representations intended to enable federation, linkage, and quality monitoring at continental scale. The dominant failure mode of search on the aforementioned portals is that meaningful relatedness is often expressed at the instance level, i.e., values and keywords, and via latent inter-table relationships rather than in reliable metadata. Accordingly, it is acknowledged that there is a “lack of consistent and reliable metadata” in open data and proposes interactive discovery of dataset linkages that can be navigated as joins, making “unrelated and unlinked dynamic collections” appear as a connected cloud that users can combine in real time (Zhu et al. 2017). 1.4.2 Data Marketplaces Data marketplaces represent a distinct and commercially motivated instantiation of federated data discovery. Unlike government portals, where datasets are published for transparency and public reuse, data marketplaces are platforms where data is treated as a tradable commodity: Providers monetize datasets they have collected or curated, and buyers acquire them to enrich their own analytical pipelines. Figure 1.2 presents the key components of a data marketplace. At its core, every marketplace must solve a discovery problem: Given a buyer’s need and a corpus of available datasets, how does the right data find the right buyer? The answer turns out to depend heavily on what the buyer knows about their need, what the seller is willing to expose before a transaction is committed, and what constraints (commercial, legal, or technical) govern the transfer. These factors vary dramatically across marketplace types. Below, we discuss examples of popular types of marketplaces. 1.4.2.1 Clean Rooms: Discovery Without Visibility Enterprise data clean rooms are privacy-preserving environments in which two organizations collaborate over their combined data without either party seeing the Fig. 1.2 Key components of a data marketplace
Page 18
1.4 Case Studies 9 other’s raw records. Popular clean rooms include Snowflake clean room,11 AWS clean room,12 Google BigQuery Analytics Hub,13 and InfoSum.14 As an example, Walmart may have a loyalty program and they collect in-store purchase data of their customers. Discover bank holds transaction and demographic data, and both want to measure whether advertising exposure drove purchases but PII regulations prohibit sharing raw customer records. A clean room mediates a join over anonymized keys, allowing both parties to compute sales lift and cross-channel attribution on the combined dataset. From a discovery standpoint, clean rooms represent the most constrained setting: The buyer cannot inspect any of the seller’s data before committing to collaboration. Discovery is reduced to catalog browsing and bilateral invitation, as the buyer must decide whether a potential partner holds relevant data based solely on the partner’s identity, reputation, and a high-level schema description. The central question shifts from "does this dataset contain the values I need?" to "does this organization hold data about the same entities I care about?" This is a setting where traditional content- based discovery operations like join search, union search, and keyword search over values are unavailable and where the design of useful discovery mechanisms under zero-visibility constraints is an open problem. We return to privacy-constrained discovery in Chap. 2. 1.4.2.2 Labeling Markets: Discovery as Task Specification Data labeling markets such as Scale AI15 and Surge AI16 invert the conventional discovery model. Rather than searching for data that already exists, the buyer specifies what data should exist, and the platform produces it. The company Waymo, for instance, generates millions of raw LiDAR frames from its autonomous vehicle fleet but needs structured 3D bounding boxes and semantic segmentation labels to train perception models. It submits task definitions and raw frames to Scale AI, whose labeler network and AI-assisted workflows produce the annotations to order. This inversion is instructive for data discovery because it highlights a class of information need that retrieval-based discovery cannot satisfy: When the required data does not yet exist in any repository, no amount of indexing or search will find it. The discovery problem here is one of gap identification: recognizing that existing datasets do not cover the buyer’s need and that new data must be created. Understanding when retrieval-based discovery is sufficient and when it must be 11 https://docs.snowflake.com/en/user-guide/cleanrooms 12 https://aws.amazon.com/clean-rooms/ 13 https://cloud.google.com/analytics-hub 14 https://www.infosum.com/ 15 https://scale.com/ 16 https://surgehq.ai/
Page 19
10 1 Data Discovery: Motivation and Challenges supplemented by data production is a recurring tension in practical data lake management. 1.4.2.3 Alternative Data Brokers Alternative data brokers such as Thinknum,17 Bloomberg,18 and YipitData19 curate datasets about companies, products, and markets, sourced from Web crawls, sensors, and proprietary feeds, and sell them to investors and analysts seeking informational advantages. Thinknum, for instance, continuously crawls hiring pages, app store rankings, and retail inventory listings, differences the data day-to-day, and sells cleaned change-signal feeds. A subscriber might monitor a competitor’s engineering hiring rate as a proxy for product development pace or track retail pricing changes ahead of earnings announcements. The discovery challenge here is particularly instructive: Buyers often do not know which datasets are relevant to a given hypothesis until they have examined the data, but the data is too commercially sensitive to expose fully before purchase. Platforms respond with curated thematic catalogs and team-based access workflows that allow analysts to explore related products before committing. This is a practical instance of the utility gap introduced in Chap. 2, the disconnect between what a buyer can learn from prepurchase metadata and what the data actually contains. Designing discovery mechanisms that close this gap without compromising seller interests is one of the central open problems in data marketplace research. Dataset Search Engines: Metadata-Only Discovery at Web Scale Dataset search engines such as Google Dataset Search index metadata exposed by publishers using the Schema.org vocabulary and return links to dataset landing pages rather than the datasets themselves. Because no raw data is ever accessed, discovery operates entirely over declared descriptions and fields, making this the purest possible instance of metadata-only discovery. The resulting discovery quality is often not ideal. Two datasets that share geographic identifiers and are therefore highly joinable will not be surfaced as related unless their metadata descriptions use compatible terminology. A dataset of European traffic counts and a dataset of European air quality measurements may share NUTS region codes as a natural join key, but no metadata-based search engine will surface the relationship if neither publisher mentioned it. Google Dataset Search’s own designers identify metadata sparsity and inconsistency as the primary limitation of the system (Brickley et al. 2019). The system therefore defines the performance floor of data discovery, what is achievable with metadata alone, against which richer, content-aware discovery operations should be measured. 17 https://www.thinknum.com/ 18 https://data.bloomberg.com/ 19 https://www.yipitdata.com/
Page 20
1.4 Case Studies 11 Understanding where metadata-only discovery fails is as important for the field as understanding where it succeeds. 1.4.2.4 Model-as-Data: Discovery Through Task and Community Signals Hugging Face20 Datasets have redefined what it means to discover a dataset in the machine learning context. Rather than searching by content or schema, practitioners discover datasets through task tags, benchmark affiliations, and citation counts. A practitioner building a question-answering model searches for datasets that are tagged as question answering, filtered by language and license, and ranked by how frequently the dataset appears in published work. This modality exposes a discovery mechanism that content-based systems largely ignore: social and task-structural signals. A dataset that is well-documented, benchmarked, and cited in influential papers is discoverable not because of its content but because of its position in a community of practice. This is particularly relevant for data discovery in scientific and ML contexts, where the downstream task is well-defined and community consensus about which datasets are appropriate for a given task carries significant information. It also highlights a limitation of purely content-based approaches: Two datasets may be semantically equivalent by any value-based measure, but one is far more discoverable than the other because of the social infrastructure surrounding it. 1.4.2.5 Academic Repositories: Schema-Driven Discovery Platforms such as ICPSR21 catalog thousands of research datasets with stan- dardized variable-level metadata variable names, value labels, question wording, and collection methodology: enabling discovery queries far more precise than keyword search. A researcher studying voter turnout can search not just for datasets mentioning “voting” but also for datasets containing a variable recording turnout as a percentage, collected via probability sampling, in US congressional districts, between 1990 and 2020. This schema-aware discovery is only possible because Inter-university Consor- tium for Political and Social Research (ICPSR) enforces a metadata standard that captures the semantics of individual variables rather than just the dataset as a whole. It represents one of the few settings where the metadata is rich enough to support structured, predicate-based discovery: the kind of query that resembles a database WHERE clause more than a keyword search. The gap between what ICPSR achieves through careful human curation and what automated discovery systems achieve over uncurated data lakes is one of the most important motivating observations for the 20 https://huggingface.co/ 21 https://www.icpsr.umich.edu/sites/icpsr/home
The above is a preview of the first 20 pages. Register to read the complete e-book.

Recommended for You

Loading recommended books...
Failed to load, please try again later

Tip the Site

Scan the WeChat Pay or Alipay code to tip. No login required.

WeChat Pay
Alipay
← Back to List