Distributed Services with Go Your Guide to Reliable, Scalable, and Maintainable Systems (Travis Jeffery) (z-library.sk, 1lib.sk, z-lib.sk)

📄 File Format: PDF

💾 File Size: 5.2 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

(This page has no text content)

📄 Page 2

(This page has no text content)

📄 Page 3

Early Praise for Distributed Services with Go Having built most of the technologies in this book without the benefit of this book, I can wholeheartedly recommend Distributed Services with Go. Travis delivers years of practical experience distilled into a clear and concise guide that takes the reader step by step from foundational knowledge to production deployment. This book earns my most hearty endorsement. ➤ Brian Ketelsen Principal Developer Advocate, Microsoft; and Organizer, GopherCon In this practical, engaging book, Travis Jeffery shines a light on the path to building distributed systems. Read it, learn from it, and get coding! ➤ Jay Kreps CEO, Confluent, Inc., and Co-Creator of Apache Kafka Travis Jeffery distills the traditionally academic topic of distributed systems down to a series of practical steps to get you up and running. The book focuses on the real-world concepts used every day by practicing software engineers. It’s a great read for intermediate developers getting into distributed systems or for senior engineers looking to expand their understanding. ➤ Ben Johnson Author of BoltDB For any aspiring Gopher, Travis provides a gentle introduction to complex topics in distributed systems and provides a hands-on approach to applying the concepts. ➤ Armon Dadgar HashiCorp Co-Founder

📄 Page 4

A must-have for Gophers building systems at scale. ➤ William Rudenmalm Lead Developer, CREANDUM This book is a great resource for Go developers looking to build and maintain distributed systems. It pairs an incremental development process with extensive code examples to teach you how to write your own distributed service, understand how it works under the hood, and how to deploy your service so others may start using it. ➤ Nishant Roy Tech Lead

📄 Page 5

Distributed Services with Go Your Guide to Reliable, Scalable, and Maintainable Systems Travis Jeffery The Pragmatic Bookshelf Raleigh, North Carolina

📄 Page 6

Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and The Pragmatic Programmers, LLC was aware of a trademark claim, the designations have been printed in initial capital letters or in all capitals. The Pragmatic Starter Kit, The Pragmatic Programmer, Pragmatic Programming, Pragmatic Bookshelf, PragProg and the linking g device are trade- marks of The Pragmatic Programmers, LLC. Every precaution was taken in the preparation of this book. However, the publisher assumes no responsibility for errors or omissions, or for damages that may result from the use of information (including program listings) contained herein. For our complete catalog of hands-on, practical, and Pragmatic content for software devel- opers, please visit https://pragprog.com. The team that produced this book includes: CEO: Dave Rankin COO: Janet Furlow Managing Editor: Tammy Coron Development Editor: Dawn Schanafelt and Katharine Dvorak Copy Editor: L. Sakhi MacMillan Indexing: Potomac Indexing, LLC Layout: Gilson Graphics Founders: Andy Hunt and Dave Thomas For sales, volume licensing, and support, please contact support@pragprog.com. For international rights, please contact rights@pragprog.com. Copyright © 2021 The Pragmatic Programmers, LLC. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form, or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher. ISBN-13: 978-1-68050-760-7 Encoded using the finest acid-free high-entropy binary digits. Book version: P1.0—March 2021

📄 Page 7

Contents Acknowledgments . . . . . . . . . . . ix Introduction . . . . . . . . . . . . . xi Part I — Get Started 1. Let’s Go . . . . . . . . . . . . . . 3 How JSON over HTTP Services Fits into Distributed Systems 4 Set Up the Project 5 Build a Commit Log Prototype 6 Build a JSON over HTTP Server 7 Run Your Server 10 Test Your API 11 What You Learned 11 2. Structure Data with Protocol Buffers . . . . . . . 13 Why Use Protocol Buffers? 14 Install the Protocol Buffer Compiler 16 Define Your Domain Types as Protocol Buffers 17 Compile Protocol Buffers 19 Work with the Generated Code 20 What You Learned 21 3. Write a Log Package . . . . . . . . . . . 23 The Log Is a Powerful Tool 23 How Logs Work 25 Build a Log 26 What You Learned 51

📄 Page 8

Part II — Network 4. Serve Requests with gRPC . . . . . . . . . 55 What Is gRPC? 55 Goals When Building a Service 56 Define a gRPC Service 58 Compile with the gRPC Plugin 60 Implement a gRPC Server 60 Register Your Server 68 Test a gRPC Server and Client 68 What You Learned 73 5. Secure Your Services . . . . . . . . . . 75 Secure Services in Three Steps 76 Authenticate the Server with TLS 78 Authenticate the Client with Mutual TLS Authentication 86 Authorize with Access Control Lists 88 What You Learned 98 6. Observe Your Systems . . . . . . . . . . 99 Three Types of Telemetry Data 99 Make Your Service Observable 103 What You Learned 109 Part III — Distribute 7. Server-to-Server Service Discovery . . . . . . . 113 Why Use Service Discovery? 113 Embed Service Discovery 114 Discover Services with Serf 116 Request Discovered Services and Replicate Logs 123 Test Discovery and the Service End-to-End 134 What You Learned 138 8. Coordinate Your Services with Consensus . . . . . 141 What Is Raft and How Does It Work? 141 Implement Raft in Our Service 144 Multiplex to Run Multiple Services on One Port 163 What You Learned 169 Contents • vi

📄 Page 9

9. Discover Servers and Load Balance from the Client . . . 171 Three Load-Balancing Strategies 171 Load Balance on the Client in gRPC 172 Make Servers Discoverable 174 Resolve the Servers 177 Route and Balance Requests with Pickers 183 Test Discovery and Balancing End-to-End 188 What You Learned 189 Part IV — Deploy 10. Deploy Applications with Kubernetes Locally . . . . 193 What Is Kubernetes? 193 Install kubectl 194 Use Kind for Local Development and Continuous Integration 195 Write an Agent Command-Line Interface 196 Build Your Docker Image 201 Configure and Deploy Your Service with Helm 202 Advertise Raft on the Fully Qualified Domain Name 213 What You Learned 217 11. Deploy Applications with Kubernetes to the Cloud . . . 219 Create a Google Kubernetes Engine Cluster 220 Create Custom Controllers with Metacontroller 222 Deploy to the Internet 228 What You Learned 228 Index . . . . . . . . . . . . . . 229 Contents • vii

📄 Page 10

Acknowledgments I write this, having finished the book, two and a half years after I began. Writing this book was the hardest thing I’ve done. I’ve built a few startups and several open source projects—this was much harder. I set out to write a good book people would enjoy and find useful. I’m critical of myself and my work and wouldn’t put out anything I didn’t deem worthy. It took me a long time to write because I didn’t want to compromise. I’m happy with this book and proud of myself. I thank my editors, Dawn Schanafelt and Katharine Dvorak, for their patience and for helping me to improve my writing and motivating me in hard times. Thank you to my publisher, The Pragmatic Bookshelf, for the guidance I received in writing my first book and for all of the work out of view. I thank my book’s reviewers and beta readers for giving me their impressions of the book and contributing suggestions and errata to help me improve the book. Thank you to Clinton Begin, Armon Dadgar, Ben Johnson, Brian Ketelsen, Jay Kreps, Nishant Roy, William Rudenmalm, and Tyler Treat. Thank you to the free and open source software communities for putting out code to study, change, and run. Special thanks to the people at Hashicorp for open-sourcing their Raft and Serf packages I use in this book and their services like Consul, whose source I studied and learned from a lot. Thank you to the Emacs and Linux contributors—the text editor and operating system I wrote this book with. Thank you to the Go team for creating a simple, stable, useful language. Thank you to my parents, Dave and Tricia Jeffery, for buying my first com- puter and programming books and encouraging me with a strong work ethic. Thank you to my high school English teacher, Graziano Galati, for giving me the right reading at the right time in my life. report erratum • discuss

📄 Page 11

Thank you to J. R. R. Tolkien for authoring The Lord of the Rings. I read it while writing this book, and the rapport with Frodo and Samwise aided me on the journey. I thank my cat, Callie Jeffery. I adopted her a quarter of the way through writing the book, and her useful contributions to the discussion helped speed up my writing pace. Thank you to Emily Davidson for her love and support and for fueling me with broccoli soup, ginger kombucha, and matcha tea. Thank you, dear reader, for independently furthering your skills and knowl- edge and having the ambition to put your dent in the universe. —Travis Jeffery Acknowledgments • x report erratum • discuss

📄 Page 12

Introduction Go has become the most popular language for building distributed services, as shown by projects like Docker, Etcd, Vault, CockroachDB, Prometheus, and Kubernetes. Despite the number of prominent projects such as these, however, there’s no resource that teaches you why or how you can extend these projects or build your own. Where do you begin if you want to build a distributed service? When I began learning how to build distributed services, I found the existing resources to be of two extremes: • Concrete code—distributed services are large, complex projects, and the prominent ones have had teams working on them for years. The layout of these projects, their technical debt, and their spaghetti code bury the ideas you’re interested in, which means you have to dig them out. At best, learning from code is inefficient. Plus there’s the risk that you may uncover outdated and irrelevant techniques that you’re better off avoiding in your own projects. • Abstract papers and books—papers and books like Designing Data- Intensive Applications by Martin Kleppmann1 describe how the data structures and algorithms behind distributed services work but cover them as discrete ideas, which means you’re left on your own to connect them before you can apply them in a project. These two extremes leave a chasm for you to cross. I wanted a resource that held my hand and taught me how to build a distributed service—a resource that explained the big ideas behind distributed services and then showed me how to make something of them. I wrote this book to be that resource. Read this book, and you’ll be able to build your own distributed services and contribute to existing ones. 1. https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063 report erratum • discuss

📄 Page 13

Who This Book Is For This book is for intermediate to advanced developers who want to learn how to build distributed services. I’ve geared the book toward Go programmers, and prior Go experience will help, but you don’t have to be an expert. This book shows you how to build distributed services, and the concepts are the same regardless of what language you use. So if you’re writing distributed services in Go, you can take full advantage of this book; if not, you can apply the ideas I present here in any language. This book’s code is compatible with Go 1.13+. What’s in This Book We will design, develop, and deploy a distributed service to explore what Go can do. We’ll develop and deploy the service in layers: from the bare essentials of storage handling, to the networking of a client and server, to distributing server instances, deployment, and testing. I divided this book into four parts that parallel those layers. (Don’t worry if you aren’t familiar with the technolo- gies I mention next—I explain them in the relevant chapters.) Part I — Get Started We’ll begin with the basic elements: building our project’s storage layer and defining its data structures. In Chapter 1, Let’s Go, on page 3, we’ll kick off our project by building a simple JSON over HTTP commit log service. In Chapter 2, Structure Data with Protocol Buffers, on page 13, we’ll set up our protobufs, generate our data structures, and set up automation to quickly generate our code as we make changes. In Chapter 3, Write a Log Package, on page 23, we’ll build a commit log library that’ll serve as the heart of our service, storing and looking up data. Part II — Network This part is where we’ll make our service work over a network. In Chapter 4, Serve Requests with gRPC, on page 55, we’ll set up gRPC, define our server and client APIs in protobuf, and build our client and server. Introduction • xii report erratum • discuss

📄 Page 14

In Chapter 5, Secure Your Services, on page 75, we’ll make our connections secure by authenticating our server with SSL/TLS to encrypt data exchanged between client and server and by authenticating requests with access tokens. In Chapter 6, Observe Your Systems, on page 99, we’ll make our service observable by adding logs, metrics, and tracing. Part III — Distribute In this part we’ll make our service distributed—highly available, resilient, and scalable. In Chapter 7, Server-to-Server Service Discovery, on page 113, we’ll build dis- covery into our service to make server instances aware of each other. In Chapter 8, Coordinate Your Services with Consensus, on page 141, we’ll add consensus to coordinate the efforts of our servers and turn them into a cluster. In Chapter 9, Discover Servers and Load Balance from the Client, on page 171, we’ll code discovery in our gRPC clients so they discover and connect to the servers with client-side load balancing. Part IV — Deploy Here’s where we’ll deploy our service and make it live. In Chapter 10, Deploy Applications with Kubernetes Locally, on page 193, we’ll set up Kubernetes locally and run a cluster on your local machine. And we’ll prepare to deploy to the cloud. In Chapter 11, Deploy Applications with Kubernetes to the Cloud, on page 219, we’ll create a Kubernetes cluster on Google Cloud’s Kubernetes Engine and deploy our service to the cloud so that people on the Internet can use it. If you plan on building the project as you read (which is a great idea), read the parts in order so that your code works. It’s also fine to skip around in the book as well; the ideas we’ll explore in each chapter have value on their own. Online Resources The code we’ll develop is available on the Pragmatic Bookshelf website: https://pragprog.com/book/tjgo. You’ll also find an errata-submission form there for you to ask questions, report any problems with the text, or make suggestions for future versions of this book. Let’s get Going! report erratum • discuss Online Resources • xiii

📄 Page 15

Part I Get Started

📄 Page 16

CHAPTER 1 Let’s Go Throughout my career I’ve written programs in C, Ruby, Python, JavaScript, Java, Elixir, Erlang, Bash, and more. Each of these languages had a lot of great things going for it but always at least a few things that bugged me a lot. C didn’t have modules, Ruby wasn’t fast enough, JavaScript and its type system made you question your sanity, and so on. This meant that each language had a specific use case, like all the different knives a chef uses. For example, a chef uses a cleaver to cut through big bones. Similarly, I’d use Java when writing big, objective-oriented programs and wanted to make a cup of tea between the time I started the program and it was ready to run. A chef uses a paring knife when making small, delicate cuts, and I’d use Bash when writing small, portable scripts. But I always wished I could find a lan- guage that was useful in almost all situations and didn’t irritate me. Finally, I came upon Go, a language that can: • Compile and run your programs faster than an interpreted language like Ruby; • Write highly concurrent programs; • Run directly on the underlying hardware; and • Use modern features like packages (while excluding a lot of features I didn’t need, like classes). Go had more things going for it. So there had to be something that bugged me, right? But no, it was as if the designers of Go had taken all the stuff that bothered me about other languages and stripped them out, leaving the lean, mean programming language that is Go. Go gave me the same feeling that made me first fall in love with programming: that if something was wrong it was my fault, me getting in my way instead of the language burying me under the weight of all its features. If Java is the cleaver and Bash the paring knife, then Go is the katana. Samurai felt that katanas were extensions of report erratum • discuss

📄 Page 17

themselves, things they could spend a lifetime with while pursuing mastery of their craft. That’s the way I feel about Go. If you were to pick the software field where Go has had the biggest impact, it would have to be distributed systems. The developers of projects like Docker, Kubernetes, Etcd, and Prometheus all decided to use Go for good reason. Google developed Go and its standard library as an answer to software prob- lems at Google: multicore processors, networked systems, massive computa- tion clusters—in other words, distributed systems, and at large scale in terms of lines of code, programmers, and machines. Because you’re a Go program- mer, you likely use systems like these and want to know how they work, how to debug them, and how to contribute to them, or you want to build similar projects of your own. That’s the case for me: the companies I’ve worked for used Docker and Kubernetes, and I’ve built my own projects like Jocko, an implementation of Kafka (the distributed commit log) in Go. So how do you start down the path of knowing how to do all that in Go? Building a distributed service isn’t the easiest or smallest project in the world. If you try to build all the pieces at once, all you’ll end up with is a big, stinking mess of a code base and a fried brain. You build the project piece by piece. A good place to start is a commit log JSON over HTTP service. Even if you’ve never written an HTTP server in Go before, I’ll teach you how to make an accessible application programming interface (API) that clients can call over the network. You’ll learn about commit log APIs and, because we’re working on one project throughout this book, you’ll be set up to write the code we’ll work on in the following chapters. How JSON over HTTP Services Fits into Distributed Systems JSON over HTTP APIs are the most common APIs on the web, and for good reason. They’re simple to build since most languages have JSON support built in. And they’re simple and accessible to use since JSON is human readable and you can call HTTP APIs via the terminal with curl, by visiting the site with your browser, or using any of the plethora of good HTTP clients. If you have an idea for a web service that you want to hack up and have people try as soon as pos- sible, then implementing it with JSON/HTTP is the way to go. JSON/HTTP isn’t limited to small web services. Most tech companies that provide a web service have at least one JSON/HTTP API acting as the public API of their service either for front-end engineers at their company to use or for engineers outside the company to build their own third-party applications Chapter 1. Let’s Go • 4 report erratum • discuss

📄 Page 18

on. For their internal web APIs, the company may take advantage of technolo- gies like protobuf for features that JSON/HTTP doesn’t provide—like type checking and versioning—but their public one will still be JSON/HTTP for accessibility. This is the same architecture I’ve used at my current and previ- ous companies. At Segment we had a JSON/HTTP-based architecture that for years handled billions of API calls a month before we changed our internal services to use protobuf/gRPC to improve efficiency. At Basecamp, all services were JSON/HTTP-based and (as far as I know) still are to this day. JSON/HTTP is a great choice for the APIs of infrastructure projects. Projects like Elasticsearch (a popular open source, distributed search engine) and Etcd (a popular distributed key-value store used by many projects, including Kubernetes) also use JSON/HTTP for their client-facing APIs, while employing their own binary protocols for communication between nodes to improve perfor- mance. JSON/HTTP is no toy—you can build all kinds of services with it. Go has great APIs in its standard library for building HTTP servers and working with JSON, making it perfect for building JSON/HTTP web services. I’ve worked on JSON/HTTP services written in Ruby, Node.js, Java, Python, and I’ve found Go to be the most pleasant by far. This is because of the interaction between Go’s declarative tags and the great APIs in the JSON encoding package (encoding/json) in the standard library that save you from the fiddling marshaling code you have to write in other languages. So let’s dive right in. Set Up the Project The first thing we need to do is create a directory for our project’s code. Since we’re using Go 1.13+, we’ll take advantage of modules1 so you don’t have to put your code under your GOPATH. We’ll call our project proglog, so open your terminal to wherever you like to put your code and run the following commands to set up your module: $ mkdir proglog $ cd proglog $ go mod init github.com/travisjeffery/proglog Replace travisjeffery with your own GitHub username or with github.com if you use something like Bitbucket, but keep in mind as you’re working through this book that the code examples all have github.com/travisjeffery/proglog as the import path, so if you’re using your own import path, you must change the code examples to use that import path. 1. https://github.com/golang/go/wiki/Modules report erratum • discuss Set Up the Project • 5

📄 Page 19

Build a Commit Log Prototype We’ll explore commit logs in depth in Chapter 3, Write a Log Package, on page 23, when we build a persisted commit log library. For now, all you need to know about commit logs is that they’re a data structure for an append-only sequence of records, ordered by time, and you can build a simple commit log with a slice. Create an internal/server directory tree in the root of your project and put the following code under the server directory in a file called log.go: LetsGo/internal/server/log.go package server import ( "fmt" "sync" ) type Log struct { mu sync.Mutex records []Record } func NewLog() *Log { return &Log{} } func (c *Log) Append(record Record) (uint64, error) { c.mu.Lock() defer c.mu.Unlock() record.Offset = uint64(len(c.records)) c.records = append(c.records, record) return record.Offset, nil } func (c *Log) Read(offset uint64) (Record, error) { c.mu.Lock() defer c.mu.Unlock() if offset >= uint64(len(c.records)) { return Record{}, ErrOffsetNotFound } return c.records[offset], nil } type Record struct { Value []byte `json:"value"` Offset uint64 `json:"offset"` } var ErrOffsetNotFound = fmt.Errorf("offset not found") Chapter 1. Let’s Go • 6 report erratum • discuss

📄 Page 20

To append a record to the log, you just append to the slice. Each time we read a record given an index, we use that index to look up the record in the slice. If the offset given by the client doesn’t exist, we return an error saying that the offset doesn’t exist. All really simple stuff, as it should be since we’re using this log as a prototype and want to keep moving. Ignore Chapter Namespaces in the File Paths You may have noticed that code snippet’s file path said LetsGo/inter- nal/server/log.go instead of internal/server/log.go and that subsequent code snippets have similar per-chapter directory namespaces. These namespaces were needed to structure the code for the book build. When writing your code, pretend that these namespaces don’t exist. So for the previous example, the internal directory would go at the root of your project. Build a JSON over HTTP Server Now we’ll write our JSON/HTTP web server. A Go web server comprises one function—a net/http HandlerFunc(ResponseWriter, *Request)—for each of your API’s endpoints. Our API has two endpoints: Produce for writing to the log and Consume for reading from the log. When building a JSON/HTTP Go server, each handler consists of three steps: 1. Unmarshal the request’s JSON body into a struct. 2. Run that endpoint’s logic with the request to obtain a result. 3. Marshal and write that result to the response. If your handlers become much more complicated than this, then you should move the code out, move request and response handling into HTTP middle- ware, and move business logic further down the stack. Let’s start by adding a function for users to create our HTTP server. Inside your server directory, create a file called http.go that contains the following code: LetsGo/internal/server/http.go package server import ( "encoding/json" "net/http" "github.com/gorilla/mux" ) func NewHTTPServer(addr string) *http.Server { httpsrv := newHTTPServer() r := mux.NewRouter() report erratum • discuss Build a JSON over HTTP Server • 7