DevOps for the Desperate A Hands-On Survival Guide [EARLY ACCESS] (Bradley Smith) (Z-Library)

📄 File Format: PDF

💾 File Size: 3.5 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

EARLY ACCESS

📄 Page 2

N O S T A R C H P R E S S E A R LY A C C E S S P R O G R A M : F E E D B A C K W E L C O M E ! Welcome to the Early Access edition of the as yet unpublished DevOps for the Desperate by Bradley Smith! As a prepublication title, this book may be incomplete and some chapters may not have been proofread. Our goal is always to make the best books possible, and we look forward to hearing your thoughts. If you have any comments or questions, email us at earlyaccess@nostarch.com. If you have specific feedback for us, please include the page number, book title, and edition date in your note, and we’ll be sure to review it. We appreciate your help and support! We’ll email you as new chapters become available. In the meantime, enjoy!

📄 Page 3

DEVOPS FOR THE DESPERATE DevOps for the Desperate (Early Access) © 2022 by Bradley Smith

📄 Page 4

DevOps for the Desperate (Early Access) © 2022 by Bradley Smith

📄 Page 5

San Francisco D E V O P S F O R T H E D E S P E R AT E A H a n d s - o n S u r v i v a l G u i d e Bradley Smith DevOps for the Desperate (Early Access) © 2022 by Bradley Smith

📄 Page 6

[S] DEVOPS FOR THE DESPERATE. Copyright © 2022 by Bradley Smith. All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. Printed in the United States of America First printing 26 25 24 23 22     1 2 3 4 5 6 7 8 9       ISBN-13: 978-1-7185-0248-2 (print) ISBN-13: 978-1-7185-0249-9 (ebook) Publisher: William Pollock Managing Editor: Jill Franklin Production Editor: Paula Williamson Developmental Editor: Jill Franklin Cover Design: Interior Design: Octopod Studios Technical Reviewers: Quentin Hartman and Kyle Terrien Copyeditor: Doug McNair Compositor: Happenstance Type-O-Rama Proofreader: ??? For information on distribution, bulk sales, corporate sales, or translations, please contact No Starch Press, Inc. directly at info@nostarch.com or: No Starch Press, Inc. 245 8th Street, San Francisco, CA 94103 phone: 1-415-863-9900 www.nostarch.com Library of Congress Cataloging-in-Publication Data Names: Smith, Bradley (Software engineer), author. Title: DevOps for the desperate : a hands-on survival guide / Bradley Smith. Description: San Francisco, CA : No Starch Press, Inc., [2022] | Includes index. Identifiers: LCCN 2021060922 (print) | LCCN 2021060923 (ebook) | ISBN 9781718502482 (paperback) | ISBN 9781718502499 (ebook) Subjects: LCSH: Computer software--Development--Management. | Software engineering--Management. Classification: LCC QA76.76.D47 S567 2022 (print) | LCC QA76.76.D47 (ebook) | DDC 005.1068--dc23/eng/20220111 LC record available at https://lccn.loc.gov/2021060922 LC ebook record available at https://lccn.loc.gov/2021060923 No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc. Other product and company names mentioned herein may be the trademarks of their respective owners. Rather than use a trademark symbol with every occurrence of a trademarked name, we are using the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The information in this book is distributed on an “As Is” basis, without warranty. While every precaution has been taken in the preparation of this work, neither the author nor No Starch Press, Inc. shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in it. DevOps for the Desperate (Early Access) © 2022 by Bradley Smith

📄 Page 7

This book is for all the engineers slogging through on call. DevOps for the Desperate (Early Access) © 2022 by Bradley Smith

📄 Page 8

DevOps for the Desperate (Early Access) © 2022 by Bradley Smith

📄 Page 9

About the Author Bradley Smith is a director of infrastructure and resides in Denver, Colorado. He has been an engineer for more than 20 years at many startups and businesses, large and small. He has built, trained, and been a member of numerous DevOps, SRE, and software-engineering teams. A Boston native, Bradley graduated from the University of Massachusetts Lowell. About the Technical Reviewer Quentin Hartman has been living and breathing DevOps since before it had a name. He loves the tech, but more than that, he loves seeing how DevOps practices make software and the lives of people who build it better. Over the course of his nearly 25-year career in technology, Quentin has worked in public education, higher education, nonprofits, and private businesses with anywhere from 3 to 300,000 employees. He has managed telecom systems, datacenters, and public and private clouds. He has acted as a sysadmin, a DBA, a network engineer, an incident responder, and a leader. This broad experience has given him an especially strong foundation in DevOps, which has been his primary focus since 2012. Wherever Quentin is, he puts people before tech and is only really happy when he’s working on a social-impact mission using open-source tools. Quentin lives near Denver, Colorado, with his family. He can often be found building things, cooking, and wandering in the woods. He can be reached as qhartman on many platforms, includ- ing Mastodon.social, Twitter, and LinkedIn. DevOps for the Desperate (Early Access) © 2022 by Bradley Smith

📄 Page 10

DevOps for the Desperate (Early Access) © 2022 by Bradley Smith

📄 Page 11

B R I E F C O N T E N T S Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xvii Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix PART I: INFRASTRUCTURE AS CODE, CONFIGURATION MANAGEMENT, SECURITY, AND ADMINISTRATION . . . . . . . . . . . . 1 Chapter 1: Setting Up a Virtual Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Chapter 2: Using Ansible to Manage Passwords, Users, and Groups . . . . . . . . . . . . . . . . . 13 Chapter 3: Using Ansible to Configure SSH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Chapter 4: Controlling User Commands with sudo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Chapter 5: Automating and Testing a Host-Based Firewall . . . . . . . . . . . . . . . . . . . . . . . . . 49 PART II: CONTAINERIZATION AND DEPLOYING MODERN APPLICATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .59 Chapter 6: Containerizing an Application with Docker . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 Chapter 7: Orchestrating with Kubernetes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 Chapter 8: Deploying Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 PART III: OBSERVABILITY AND TROUBLESHOOTING . . . . . . . . . . . . . . . .107 Chapter 9: Observability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Chapter 10: Troubleshooting Hosts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 DevOps for the Desperate (Early Access) © 2022 by Bradley Smith

📄 Page 12

DevOps for the Desperate (Early Access) © 2022 by Bradley Smith

📄 Page 13

C O N T E N T S I N D E T A I L ACKNOWLEDGMENTS xvii INTRODUCTION xix What Is the Current State of DevOps? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx Who Should Read This Book? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi How This Book Is Organized . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Part I: Infrastructure as Code, Configuration Management, Security, and Administration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii Part II: Containerization and Deploying Modern Applications . . . . . . . . . . . . . xxii Part III: Observability and Troubleshooting . . . . . . . . . . . . . . . . . . . . . . . . . . xxii What You’ll Need . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii Downloading and Installing VirtualBox . . . . . . . . . . . . . . . . . . . . . . . . . . . .xxiii Companion Repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xxiv Editor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xxiv PART I: INFRASTRUCTURE AS CODE, CONFIGURATION MANAGEMENT, SECURITY, AND ADMINISTRATION 1 1 SETTING UP A VIRTUAL MACHINE 3 Why Use Code to Build Infrastructure? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Getting Started with Vagrant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Anatomy of a Vagrantfile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Basic Vagrant Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Getting Started with Ansible . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Key Ansible Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Ansible Playbook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Basic Ansible Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Creating an Ubuntu VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 2 USING ANSIBLE TO MANAGE PASSWORDS, USERS, AND GROUPS 13 Enforcing Complex Passwords . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Installing libpam-pwquality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Configuring pam_pwquality to Enforce a Stricter Password Policy . . . . . . . . . . 15 Linux User Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Getting Started with the Ansible User Module . . . . . . . . . . . . . . . . . . . . . . . . 16 Generating a Complex Password . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 DevOps for the Desperate (Early Access) © 2022 by Bradley Smith

📄 Page 14

xii   Contents in Detail Linux Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Getting Started with the Ansible Group Module . . . . . . . . . . . . . . . . . . . . . . 18 Assigning a User to the Group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Creating Protected Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Updating the VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Testing User and Group Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3 USING ANSIBLE TO CONFIGURE SSH 25 Understanding and Activating Public Key Authentication . . . . . . . . . . . . . . . . . . . . . . . 26 Generating a Public Key Pair . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Using Ansible to Get Your Public Key on the VM . . . . . . . . . . . . . . . . . . . . . . 27 Adding Two-Factor Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Installing Google Authenticator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Configuring Google Authenticator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Configuring PAM for Google Authenticator . . . . . . . . . . . . . . . . . . . . . . . . . 30 Configuring the SSH Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Restarting the SSH Server with a Handler . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Provisioning the VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Testing SSH Access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4 CONTROLLING USER COMMANDS WITH SUDO 37 What Is sudo? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Planning a sudoers Security Policy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Installing the Greeting Web Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Anatomy of a sudoers File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Creating the sudoers File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 The sudoers Template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Provisioning the VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Testing Permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Accessing the Web Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Editing greeting .py to Test the sudoers Policy . . . . . . . . . . . . . . . . . . . . . . . . 46 Stopping and Starting with systemctl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Audit Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 5 AUTOMATING AND TESTING A HOST-BASED FIREWALL 49 Planning the Firewall Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Automating UFW Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Provisioning the VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Testing the Firewall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Scanning Ports with Nmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 Firewall Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 Rate Limiting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 DevOps for the Desperate (Early Access) © 2022 by Bradley Smith

📄 Page 15

Contents in Detail   xiii PART II: CONTAINERIZATION AND DEPLOYING MODERN APPLICATIONS 59 6 CONTAINERIZING AN APPLICATION WITH DOCKER 61 Docker from 30,000 Feet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Getting Started with Docker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Dockerfile Instructions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Container Images and Layers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 Containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 Namespaces and Cgroups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 Installing and Testing Docker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 Installing the Docker Engine with Minikube . . . . . . . . . . . . . . . . . . . . . . . . . . 65 Installing the Docker Client and Setting Up Docker Environment Variables . . . . 66 Testing the Docker Client Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 Containerizing a Sample Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 Dissecting the Example telnet-server Dockerfile . . . . . . . . . . . . . . . . . . . . . . . 67 Building the Container Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 Verifying the Docker Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Running the Container . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 Other Docker Client Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 exec . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 rm 72 inspect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 history . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 stats 74 Testing the Container . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Connecting to the Telnet-Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Getting Logs from the Container . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 7 ORCHESTRATING WITH KUBERNETES 77 Kubernetes from 30,000 Feet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 Kubernetes Workload Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Pods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 ReplicaSet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Deployments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 StatefulSets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 Volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 Secrets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 ConfigMaps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Namespaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Deploying the Sample telnet-server Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 Interacting with Kubernetes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 Reviewing the Manifests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 Creating a Deployment and Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 Viewing the Deployment and Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 DevOps for the Desperate (Early Access) © 2022 by Bradley Smith

📄 Page 16

xiv   Contents in Detail Testing the Deployment and Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Accessing the Telnet Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Troubleshooting Tips . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 Killing a Pod . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Logs 93 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 8 DEPLOYING CODE 95 CI/CD in Modern Application Stacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 Setting Up Your Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Reviewing the skaffold .yaml File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 Reviewing the Container Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Simulating a Development Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Making a Code Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 Testing the Code Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Testing a Rollback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 Other CI/CD Tooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 PART III: OBSERVABILITY AND TROUBLESHOOTING 107 9 OBSERVABILITY 109 Monitoring Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 Monitoring the Sample Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Installing the Monitoring Stack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 Verifying the Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 Golden Signals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 Adjusting the Monitoring Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 The telnet-server Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 PromQL: A Primer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118 Alerts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Reviewing Golden Signal Alerts in Prometheus . . . . . . . . . . . . . . . . . . . . . . 119 Routing and Notifications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 10 TROUBLESHOOTING HOSTS 125 Troubleshooting and Debugging: A Primer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 Scenario: High Load Average . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 uptime . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 top 128 Next Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 DevOps for the Desperate (Early Access) © 2022 by Bradley Smith

📄 Page 17

Contents in Detail   xv Scenario: High Memory Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 free 129 vmstat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 ps 131 Next Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Scenario: High iowait . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 iostat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 iotop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Next Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Scenario: Hostname Resolution Failure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 resolv .conf . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 resolvectl . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 dig 136 Next Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 Scenario: Out of Disk Space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 df 138 find 138 lsof 139 Next Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Scenario: Connection Refused . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140 curl 140 ss 140 tcpdump . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 Next Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 Searching Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 Common Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Common journalctl Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 Parsing Logs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 Probing Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 strace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 INDEX 153 DevOps for the Desperate (Early Access) © 2022 by Bradley Smith

📄 Page 18

DevOps for the Desperate (Early Access) © 2022 by Bradley Smith

📄 Page 19

A C K N O W L E D G M E N T S When writing acknowledgments, you quickly realize how many people make publishing a book possible. This would be a very long section if I thanked everyone who contributed in some way, and since this is not a Nobel Prize acceptance speech, I will try to keep it short and sweet. If I do not mention you below, please know I appreciate your help tremendously. First, I want to thank everyone at No Starch Press. Without you, this book would not have been possible. The guidance from my editor, Jill Franklin, and technical editors, Kyle Terrien and Quentin Hartman, have been invaluable. Thank you so much for wrangling this idea into a book. I appreciate you all. We all need help from our friends, and this book has my friends’ finger- prints all over it. Many of you provided feedback, and I thank you all so much. In particular, I want to thank Rishi Malik, Jaden Grossman, and Jeffrey Matthias. You provided support and (more importantly) lent me your precious time. I owe you! Finally, I want to thank my family. Countless times, I asked you to read a sentence or a paragraph and tell me what you thought of it—even though you had no idea what I was talking about. To my wife, Leilani, you have always encouraged me and made me believe I could do this. Thank you for making time in our lives so I could work on this book. To my daughters, Aiden and Akira, you are my inspiration, and you make me want to be the best person I can be. I love the three of you, always. DevOps for the Desperate (Early Access) © 2022 by Bradley Smith

📄 Page 20

DevOps for the Desperate (Early Access) © 2022 by Bradley Smith