Page
1
(This page has no text content)
Page
3
Praise for CockroachDB: The Definitive Guide, Second Edition CockroachDB isn’t just another database. It offers the benefits of relational databases, but goes further by solving the incredibly difficult challenge of handling distributed transactions with complete data integrity. This book provides everything you need to effectively use CockroachDB, making this complex challenge workable. —Rick Greenwald, retired senior director for DBMS analysis, Gartner, Inc. CockroachDB gives you another option that’s not strictly SQL and not strictly NoSQL. It’s kind of this next-generation database that helps us solve a lot of challenges as we become more dependent on data and need to scale out further and further. —Bryan Call, senior principal engineer, Route
Page
4
Today’s customers are focused on building resilient, highly durable systems—such as mission-critical trading applications—that need to operate seamlessly across multiple regions. CockroachDB ensures their resilience, at scale, effortlessly. —Peter Williams, global head of financial services partner technology, AWS Modern distributed SQL systems bring together the best ideas of databases from the last 50 years: the relational model, strong consistency guarantees, and elastic scalability. This trinity is what modern applications need to stay in the game. —Andy Pavlo, associate professor of databases, Carnegie Mellon University; cofounder, OtterTune
Page
5
CockroachDB: The Definitive Guide SECOND EDITION Distributed Data at Scale Guy Harrison, Jesse Seldess, Ben Darnell & Rob Reid
Page
6
CockroachDB: The Definitive Guide by Guy Harrison, Jesse Seldess, Ben Darnell, and Rob Reid Copyright © 2025 O’Reilly Media, Inc. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Aaron Black Development Editor: Angela Rufino Production Editor: Gregory Hyman Copyeditor: Dwight Ramsey Proofreader: Stephanie English Indexer: nSight, Inc.
Page
7
Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea March 2025: Second Edition Revision History for the Second Edition 2025-03-17: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781098179847 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. CockroachDB: The Definitive Guide, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without
Page
8
limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. This work is part of a collaboration between O’Reilly and Cockroach Labs. See our statement of editorial independence. 978-1-098-17985-4 [LSI]
Page
9
Preface Welcome to CockroachDB: The Definitive Guide, and thank you for being here! With this book, we want to help you learn to build and deploy applications with CockroachDB, the distributed SQL database that meets you where you’re at on your journey—in the cloud, on-premises, or hybrid. First, the question everyone asks: Why the name CockroachDB? One immutable fact of engineering is that things break. At large enough scale, things are breaking all the time. The kinds of failures that might happen once a year on a single machine become daily occurrences when you’re running thousands of them. A system that aspires to handle large scale must treat fault tolerance as a core responsibility. This was one of the key insights of Google’s MapReduce framework. By requiring all computation to fit within a relatively restrictive framework, it became straightforward for the system to automatically rerun the necessary pieces of work after a failure. We believe that this is how it should be: highly available replication should be the default state of a database from Day One, not the result of painstaking configuration work. And looking ahead to Day Two (or Day Two Hundred), the database
Page
10
must be able to grow along with the application so that runaway success is a cause for celebration, not panic. When Cockroach Labs set out to build a relational database from scratch, we wanted to bring consistency, native resilience, data locality, and massive scale to modern applications. Our vision was of a system able to colonize any resource that you gave it and then relentlessly optimize itself. A database that would use available space and reach equilibrium across a coordinated set of distributed nodes so that it would not only incorporate new resources, but—if a machine or data center or even an entire region went down—the database would simply equalize the remaining available resources. We aimed to build a database that would be impossible to eradicate…just like periplaneta brunnea, the pesky common cockroach. And so CockroachDB was born. Building CockroachDB CockroachDB’s origin is a tale of necessity. Quite literally, Spencer Kimball, Peter Mattis, and I, Ben Darnell, set out to build the relational database that we ourselves needed.
Page
11
After working together at Google, the three of us went in different directions for a while. Eventually we ended up back together (along with Spencer’s brother Andy, a Microsoft SQL Server veteran) at a startup called Viewfinder, building an application for photo organization and social sharing. We, of course, hoped it would be successful, so we wanted to build for scale. After all, as Spencer says, “Sometimes shit happens. And, at scale, shit’s always happening.” It turns out that, given the history of that project, we didn’t do well enough to ever need massive scalability, but we wanted to plan for it from the beginning. We also didn’t want to lock ourselves into a monolithic database, sharding MySQL or anything like that. We had been down that road before and knew it didn’t lead where we wanted to go. This left us looking at NoSQL options as the best alternative and eventually we settled on DynamoDB. DynamoDB ticked a lot of boxes: it’s scalable, it’s fast, it’s predictable. It’s very similar to Bigtable, which I had used as the backend for Google Reader while I was at Google. And so, on paper at least, we really liked that model. But as we got experience with it, we realized we had a couple of fundamental problems with NoSQL.
Page
12
The biggest one was we found that we really needed secondary indexes. The Bigtable/DynamoDB model is really primary keys only; there’s no concept of secondary indexes. To get them, we had to build our own partial implementation of transactions on top of DynamoDB, which was workable. Once we were using this system, however, we found a subsequent problem. DynamoDB on its own is very fast, predictable, and scalable— but once you start doing things that combine multiple records, as we were doing with our secondary indexes, then things got difficult. Ultimately, we came up with a system that worked, but the layering of transactions and indexes on top of DynamoDB was inefficient, both in terms of performance and our engineering efforts. During this time we did start talking about the idea of building a database ourselves, but Viewfinder was a startup with a small team. We decided, You know, DynamoDB is not ideal, but it exists and we can use it today, so that’s what we’ll do. But then a year or two later Square bought Viewfinder, and we found out they were struggling with sharded MySQL. We realized then, Okay, it’s not just us, this is a real database need that’s not getting met. At Square we were now in a good position to try building a system that was both inherently scalable and supported
Page
13
secondary indexes. So that’s where the effort really started in earnest. Spencer started on it as a side project on nights and weekends at first. And then he got some more contributors, and eventually it even became an official project at Square. But we knew that the real destiny of this project was to serve the entire database market, so that’s when we decided to leave Square and start a company to do just that: Cockroach Labs was founded in 2015. Next Steps Having set out to build a database that is as indestructible as an intrusion of cockroaches, the next question was, of course, what other capabilities it would need. We believed that the right approach was to embrace distributed transactions from the start and make them an integral part of the architecture. And while Structured Query Language (SQL) is hardly anyone’s idea of a perfect query language, it is the one that everyone knows, with a time-tested feature set. SQL’s declarative schema management and statistics-based query
Page
14
optimization give operators powerful tools to manage their application’s performance at runtime. So the story of getting to CockroachDB version 1.0 was solving the new problem of scale plus transactions using SQL. The result is a globally distributed SQL database built on top of a transactional and consistent key-value (KV) store. The primary design goals are support for atomic, consistent, isolated, and durable (ACID) transactions; horizontal scalability; low latency/high availability; and survivability. CockroachDB implements a Raft consensus algorithm for consistency and aims to tolerate disk, machine, rack, and even data center failures with minimal disruption and no manual intervention. Once we started building and deploying this, we found that the greatest interest in what we were doing came from people working with global deployments. It turns out this is something Cockroach is uniquely capable of handling, and so with version 2.0 we started building more and more features for global and geographic distribution. It’s been a long time since we launched CockroachDB 2.0 and multiregion. Since then, the biggest developments have been CockroachDB Cloud followed by CockroachDB Cloud Basic. These fulfilled our next mission: to make data easy.
Page
15
Our Viewfinder startup days showed us the need for a database that can be used for projects from Day One and would also grow along with it, without breaking the bank. You can self-host CockroachDB locally on a laptop, on a corporate dev cluster or private cloud, as well as on any public cloud infrastructure, but the price of entry is still fairly substantial. The first step toward making CockroachDB accessible was to take away the operational burden from the user, and we achieved that with our cloud-hosted, fully managed service now called CockroachDB Cloud Advanced. Now we have added CockroachDB Cloud Basic to give users the low-cost or even no-cost foot in the door that you can use from the beginning, and then it scales up affordably with your usage. It’s now inexpensive enough for everyone to simply build multiregion by default. What comes after that? Well, we are continuing to explore ways to use cloud native features to gain even greater efficiencies and drive down operating costs. Things like looking at the menu of various storage solutions that the clouds offer and choosing the appropriate ones for our different kinds of storage needs instead of using block storage devices for everything as we currently do.
Page
16
Along the way, we’ve added user-defined functions and stored procedures. This is a lightning rod for discussion because people have strong opinions, but I think that when you combine user-defined functions and stored procedures with a scalable database, it gets really interesting. It can really transform the way you think about your application architecture because it lets you shift more of your logic into the database. More speculatively? As CockroachDB’s cofounder, I’d like to explore ways that you could host your entire application, just straight out of the database, using stored procedures. Even saying that out loud is going to be a bit controversial because that’s the sort of thing that people tried to do back in, like, 1998 and, we admit, it didn’t work well then. However, I think things have evolved to the point that now may be the right time. That’s still very speculative, just something we’re thinking about and not something that is officially on the Cockroach Labs roadmap at this time. But we’d like to try to hack out some sort of prototype instead of just talking about it. Maybe now that this book is published, that will free up some time to experiment.
Page
17
Why We Wrote This Book This book aims to help you, the reader, understand the architecture and capabilities of CockroachDB as well as suitable use cases for CockroachDB. By the end of this book, you will be able to get started with CockroachDB, build effective applications on it, and, ultimately, run a cluster in production. The chapters teach developers, architects, and DevOps teams how to build, optimize, and manage applications that run on CockroachDB. We wrote this book to give any developer, no matter how they want to work with CockroachDB, the knowledge and tools they need to do it effectively. Ben Darnell March 2025 New York City
Page
18
Who This Book Is For Whether you’re an architect designing systems, a developer crafting code, or a DevOps engineer streamlining deployments, this book will guide you through everything you need to know about CockroachDB—from fundamental concepts and basic operations to advanced features and production deployment strategies: Innovative developers CockroachDB: The Definitive Guide is for developers building modern applications, whether in a small startup or a large enterprise. This guide will teach developers how to build and ship apps with fewer obstacles using a powerful distributed SQL database that just works. Architects The Definitive Guide is for modern architects delivering scalable, resilient apps across their IT ecosystem. Architects will learn how to design distributed applications to provide low latency, high availability, and faster performance. DevOps teams
Page
19
The Definitive Guide is for strategic IT operators managing applications with data-intensive workloads. Operators will learn how to optimize CockroachDB’s inherent abilities of scale and resilience, efficiently distributing data to meet any workload demand—wherever it’s deployed. Those already familiar with distributed systems will discover the benefits of strong data correctness and consistency guarantees as well as optimizations for delivering low-latency transactions to users anywhere in the world. It includes specific guidance for anyone transitioning from a monolithic database (e.g., MySQL or PostgreSQL) to a distributed architecture, as well as practical examples for anyone more familiar with NoSQL systems.
Page
20
How This Book Is Organized This book contains 15 chapters divided into three sections: Part I, “Introduction to CockroachDB”, establishes the historical context of CockroachDB, covering the evolution of databases and the emergence of distributed cloud databases. Readers cover core concepts of distributed SQL, then dive into the architecture and capabilities of CockroachDB, followed by examinations of suitable use cases for CockroachDB. Part I ends with a hands-on guide to getting started, both with a local installation and Cockroach Labs’ free cloud service, and a thorough guide to CockroachDB’s SQL dialect. Part II, “Developing Applications with CockroachDB”, covers the fundamentals of CockroachDB schema design. Next, readers work through application design and implementation as well as integration with and/or migration from other databases. You will practice working with data in CockroachDB, including managing transactions, exploring change data capture, and building SQL tuning skills. Part III, “Deploying and Administering CockroachDB”, covers the planning and execution of single and multiregion deployments. Essential topics include backup and disaster