Build an Orchestrator in Go (From Scratch) (Tim Boring) (Z-Library)

📄 File Format: PDF

💾 File Size: 5.3 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

(This page has no text content)

📄 Page 2

inside front cover

📄 Page 3

(This page has no text content)

📄 Page 4

Build an Orchestrator in Go (From Scratch) Tim Boring To comment go to liveBook

📄 Page 5

Manning Shelter Island For more information on this and other Manning titles go to www.manning.com

📄 Page 6

Copyright For online information and ordering of these and other Manning books, please visit www.manning.com. The publisher offers discounts on these books when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com ©2024 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning

📄 Page 7

Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. ♾ Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. Manning Publications Co. 20 Baldwin Road Technical PO Box 761 Shelter Island, NY 11964 Development editor: Katie Sposato Johnson Technical development editor: Mike Shepard Review editor: Adriana Sabo and Dunja Nikitović Production editor: Keri Hales Copy editor: Alisa Larson Proofreader: Melody Dolab Technical proofreader: Mike Haller Typesetter: Dennis Dalinnik Cover designer: Marija Tudor

📄 Page 8

ISBN: 9781617299759

📄 Page 9

dedication For Jennifer, who has always believed in me, even when I didn’t believe in myself.

📄 Page 10

contents Front matter preface acknowledgments about this book about the author about the cover illustration Part 1 Introduction 1 What is an orchestrator? 1.1 Why implement an orchestrator from scratch? 1.2 The (not so) good ol’ days 1.3 What is a container, and how is it different from a virtual machine? 1.4 What is an orchestrator? 1.5 The components of an orchestration system The task The job The scheduler The manager The worker The cluster Command-line interface 1.6 Meet Cube 1.7 What tools will we use?

📄 Page 11

1.8 A word about hardware 1.9 What we won’t be implementing or discussing Distributed computing Service discovery High availability Load balancing Security 2 From mental model to skeleton code 2.1 The task skeleton 2.2 The worker skeleton 2.3 The manager skeleton 2.4 The scheduler skeleton 2.5 Other skeletons 2.6 Taking our skeletons for a spin 3 Hanging some flesh on the task skeleton 3.1 Docker: Starting, stopping, and inspecting containers from the command line 3.2 Docker: Starting, stopping, and inspecting containers from the API 3.3 Task configuration 3.4 Starting and stopping tasks Part 2 Worker 4 Workers of the Cube, unite! 4.1 The Cube worker 4.2 Tasks and Docker

📄 Page 12

4.3 The role of the queue 4.4 The role of the DB 4.5 Counting tasks 4.6 Implementing the worker’s methods Implementing the StopTask method Implementing the StartTask method An interlude on task state Implementing the RunTask method 4.7 Putting it all together 5 An API for the worker 5.1 Overview of the worker API 5.2 Data format, requests, and responses 5.3 The API struct 5.4 Handling requests 5.5 Serving the API 5.6 Putting it all together 6 Metrics 6.1 What metrics should we collect? 6.2 Metrics available from the /proc filesystem 6.3 Collecting metrics with goprocinfo 6.4 Exposing the metrics on the API 6.5 Putting it all together Part 3 Manager 7 The manager enters the room

📄 Page 13

7.1 The Cube manager The components that make up the manager 7.2 The Manager struct 7.3 Implementing the manager’s methods Implementing the SelectWorker method Implementing the SendWork method Implementing the UpdateTasks method Adding a task to the manager Creating a manager 7.4 An interlude on failures and resiliency 7.5 Putting it all together 8 An API for the manager 8.1 Overview of the manager API 8.2 Routes 8.3 Data format, requests, and responses 8.4 The API struct 8.5 Handling requests 8.6 Serving the API 8.7 A few refactorings to make our lives easier 8.8 Putting it all together 9 What could possibly go wrong? 9.1 Overview of our new scenario 9.2 Failure scenarios Application startup failure Application bugs Task startup failures due to resource problems Task failures due to Docker daemon crashes and restarts Task failures due to machine crashes and restarts

📄 Page 14

Worker failures Manager failures 9.3 Recovery options Recovery from application failures Recovering from environmental failures Recovering from task-level failures Recovering from worker failures Recovering from manager failures 9.4 Implementing health checks Inspecting a task on the worker Implementing task updates on the worker Healthchecks and restarts 9.5 Putting it all together Part 4 Refactorings 10 Implementing a more sophisticated scheduler 10.1 The scheduling problem 10.2 Scheduling considerations 10.3 Scheduler interface 10.4 Adapting the round-robin scheduler to the scheduler interface 10.5 Using the new scheduler interface Adding new fields to the Manager struct Modifying the New helper function 10.6 Did you notice the bug? 10.7 Putting it all together 10.8 The E-PVM scheduler The theory

📄 Page 15

In practice 10.9 Completing the Node implementation 10.10 Using the E-PVM scheduler 11 Implementing persistent storage for tasks 11.1 The storage problem 11.2 The Store interface 11.3 Implementing an in-memory store for tasks 11.4 Implementing an in-memory store for task events 11.5 Refactoring the manager to use the new in-memory stores 11.6 Refactoring the worker 11.7 Putting it all together 11.8 Introducing BoltDB 11.9 Implementing a persistent task store 11.10 Implementing a persistent task event store 11.11 Switching out the in-memory stores for permanent ones Part 5 CLI 12 Building a command-line interface 12.1 The core components of CLIs 12.2 Introducing the Cobra framework 12.3 Setting up our Cobra application 12.4 Understanding the new main.go 12.5 Understanding root.go 12.6 Implementing the worker command

📄 Page 16

12.7 Implementing the manager command 12.8 Implementing the run command 12.9 Implementing the stop command 12.10 Implementing the status command 12.11 Implementing the node command 13 Now what? 13.1 Working on Kubernetes and related tooling 13.2 Manager-worker pattern and workflow systems 13.3 Manager-worker pattern and integration systems 13.4 In closing appendix Environment setup index

📄 Page 17

Front matter preface I was introduced to orchestrators when I started at Google in 2007. And my introduction was not to Borg but rather Ganeti. Ganeti was an internally developed cluster management system that operated on virtual machines. At the time, it was a basic wrapper around the open source version of Xen, and it provided a clustered solution that allowed us to provide virtual (instead of physical) machines to engineers. We didn’t refer to Ganeti as an orchestrator, nor did we talk about it in the same vein as Borg. In hindsight, I don’t think it’s too much of a stretch to consider Ganeti a kind of orchestrator. Instead of operating on tasks (in the form of containers), it operated on virtual machines. Internally at Google, Ganeti served as a bridge from a world where some engineers could run their applications on physical machines to a world where every engineer ran their applications on Borg. Several years later, I got a proper introduction to Borg when we rewrote the life cycle management system we built to manage Ganeti clusters and virtual machines. We ran it on Borg. Fast-forward to 2020. The COVID pandemic hit, and like everyone else, I found myself working from home. Suddenly, I had three-plus extra hours per day as a result

📄 Page 18

of not having to commute to a Manhattan office. What to do? Of course, the obvious thing to do was to start a personal project of some kind. But what? After 13 years of working with orchestrators, I thought it might be fun to try to write one from scratch. How hard could it be? I spent most of the summer of 2020 working on my orchestrator. I called it Cube in an effort to continue the Star Trek theme. Surprisingly, I got it working in less than 3,000 lines of code. Around the same time, I read Thorsten Ball’s Writing an Interpreter in Go. While I wasn’t necessarily interested in interpreters or writing programming languages, I was interested in learning how they work. And then it hit me! I could do a book about writing an orchestrator in Go. It would be the book that I wish I’d had back in 2007! Thus was born the book you have in front of you now. I realized early on in the writing process that orchestration is a big topic. It’s easy to get distracted by secondary concerns when talking about orchestration systems. How do you handle service discovery? How do you handle DNS? What about consensus? I wanted to strip away all the stuff that gets piled on top of orchestration systems and present just the core, the foundation on which all that other stuff sits. Not that things like service discovery, DNS, and load balancing are unimportant. But in the context of an orchestration system, we talk about those things because they are tools in service to the core function of an

📄 Page 19

orchestrator: scheduling applications to run on a pool of nodes and managing their life cycle. So in a nutshell, that’s what this book is about: taking a request from a user to run an application, identifying a machine that can run the application, and then sending a request to the chosen machine to start the application. It seems simple when you put it that way, doesn’t it? In addition to presenting the foundational concepts of an orchestration system, another goal in writing this book is to make the content approachable to a broad audience. So while I’ve chosen to write the Cube orchestrator in the Go programming language, my hope is that anyone can work through the book and get the code working, even if you’ve never written a line of Go in your life. All of the code uses basic Go features. While we do use goroutines to do some basic concurrency, we don’t use channels (there are many great resources to learn about concurrency if you’re interested). And we don’t use generics. (Shortly after Go 1.18 was released, I did attempt to refactor the code and manuscript to use generics. While I got the code working, I realized it introduced unnecessary complexity to the book. It became one more thing to explain in an already long list of things to explain.) I hope you have fun while you read this book. And in the process of having fun, I hope you learn as much from this book as I did writing it.

📄 Page 20

acknowledgments With most things in life, we get a lot of help from others. And this book is no different. I’d like to start by acknowledging all the folks at Manning who helped make this book a reality. I’d like to thank Andy Waldron for taking on my book and believing in it throughout its many shapes. Katie Sposato Johnson was instrumental in helping me navigate the Manning process. Without her help, this book would not exist. I’d also like to thank the many other folks at Manning who have worked on the production and marketing of the book. I want to thank the reviewers who read the manuscript at various stages and provided thoughtful feedback: Alain Lompo, Alessandro Campeis, Andres Sacco, Becky Huett, Bobby Lin, Christopher Villanueva, Clifford Thurber, David Paccoud, Emanuele Piccinelli, Ernesto Bossi, Fernando Bernardino, Fernando Rodrigues, Geert Van Laethem, Gregory Reshetniak, Katia Patkin, Kosmas Chatzimichalis, Larry Cai, Lucian Enache, Madiha Khalid, Matthias Busch, Michael Bright, Muneeb Shaikh, Nathan B. Crocker, Nghia To, Richard Vaughan, Sanket Naik, Simone Sguazza, Thomas Dybdahl, Timothy R. J. Langford, Tim van Deurzen, and Vamsi Krishna. Special thanks also go out to Mike Haller, technical proofreader, for his thorough review of the code shortly before the book went into production. It’s quite challenging to keep the code presented in the book in sync with the