Argo CD Up Running A Hands-On Guide to GitOps and Kubernetes (Andrew Block, Christian Hernandez) (z-library.sk, 1lib.sk, z-lib.sk)

Argo CD Up & Running A Hands-On Guide to GitOps and Kubernetes Andrew Block & Christian Hernandez

The GitOps Platform for K8s From the creators of Argo and Kargo, the Akuity Platform is the only end-to-end GitOps platform for enterprises. 
 Book a demo today! 
 Engineering Efficiency:  Reduced MTTR, fewer manual approvals, fewer deployment failures Deployment Speed:  Faster lead time, increased deployment frequency Risk Reduction:  Fewer security vulnerabilities, better compliance Developer Experience:  More self-service deployments, higher K8s adoption. PlatformPortal Akuity Platform Repo Server Managed ClusterControl Plane Repo Server Managed Cluster Repo Server Managed Cluster Agent Based Application Management Controller Redis get Akuity Agent Controller Akuity Agent Controller Akuity Agent Controller Deploy – Hybrid agent-based architecture for enterprise scale. Promote – Leverage stage-to-stage promotion using GitOps principles without bespoke automation or CI pipeline. 
 Monitor – Real-time monitoring & comprehensive K8s dashboards across all cloud providers, regions, on-prem and edge locations. Subscription Guestbook Security: The agent-based 
 architecture isolates secrets within 
 the cluster, improving security Performance: Resource updates 
 are processed locally and only 
 relevant changes are sent back to 
 the control plane. Scalability: built-in and configured autoscaler automatically adjusts Argo 
 CD resources. ghcrJa/jessesuen/gue... Refresh dev Created 5 months ago LAST PROMO: 5 days ago looping-elk staging Created 5 months ago LAST PROMO: 5 days ago looping-elk prod Created 5 months ago LAST PROMO: 27 days ago illmannered-arachnid > > > https://akuity.io/get-in-touch 


Praise for Argo CD: Up and Running This book from two leading Argo experts is clear and to the point. You’ll be up to speed quickly and well on your way to being an advanced Argo practitioner. —Michael Crenshaw, staff SWE and lead Argo CD maintainer, Intuit If you want a guide that masterfully demystifies the Argo CD and GitOps world, look no further. Whether you’re just beginning or fine-tuning a production setup, the authors distill years of practical experience condensed into this book that will serve as a trusted reference long after the first read. —Lipi Deepaakshi Patnaik, senior software developer, Zeta Suite I wish I had this book when I first started learning Argo CD—it would have made implementation so much easier. —Werner Dijkerman, Kubernetes and DevOps engineer, Awesome Cloud Andrew and Christian discuss several applicable examples in-depth at an enjoyable reading pace—a practical reference! —Nadir Doctor, architect This book is a must-read for anyone adopting GitOps with Kubernetes and Argo CD. It provides the practical guidance needed to effectively get started with Argo CD and scale it for use in multi-cluster environments. —Manuel Dewald, lead software architect at Codesphere and coauthor of Operating OpenShift

Working with Andy and Christian, you naturally learn by osmosis. I’m thrilled they’ve captured their deep knowledge of real-world GitOps patterns and advanced Argo CD in this book, allowing anyone to benefit from their proven experience and be inspired by their passion. —Natale Vinto, director of developer advocacy, Red Hat In my experience, GitOps and Argo CD are widely deployed but commonly misunderstood. Andrew and Christian are working hard to change this, covering both the theory and the execution. This book is my go-to reference for everything from deploying applications to operationalizing Argo CD. —Daniel Bryant, platform engineer and PMM, Syntasso The authors have done an outstanding job curating a thoughtful and thorough journey through Argo CD. Whether you’re deploying your first application or scaling GitOps in an enterprise setting, this book equips you with the tools and mindset you need to succeed. A standout resource in this ecosystem. —Samyak Ahuja, software engineer, Uber

Andrew Block and Christian Hernandez Argo CD: Up and Running A Hands-On Guide to GitOps and Kubernetes

979-8-341-63488-6 [LSI] Argo CD: Up and Running by Andrew Block and Christian Hernandez Copyright © 2025 Andrew Block and Christian Hernandez. All rights reserved. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Megan Laddusaw Development Editor: Jill Leonard Production Editor: Kristen Brown Copyeditor: nSight, Inc. Proofreader: Emily Wydeven Indexer: Sue Klefstad Cover Designer: Susan Thompson Cover Illustrator: Karen Montgomery Interior Designer: David Futato Interior Illustrator: Kate Dullea June 2025: First Edition Revision History for the First Edition 2025-06-16: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781098142001 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Argo CD: Up and Running, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. This work is part of a collaboration between O’Reilly and Akuity. See our statement of editorial independ‐ ence.

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii 1. Introduction to Argo CD. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 What Is Argo CD? 1 Why Argo CD? 3 Unifying Application Definitions 3 Configuration Drift 3 Rollback and Disaster Recovery 3 The GitOps Movement 4 Origins of GitOps 4 OpenGitOps Principles 4 Comparison of GitOps Tools in the Ecosystem 6 Flux 6 Argo CD 6 Comparison of Flux and Argo CD 7 The Argo Ecosystem 7 Summary 8 2. Installing Argo CD. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Argo CD Architecture 9 Kubernetes Controller Pattern 9 Argo CD Architecture Overview 11 Argo CD Key Patterns 14 Installing Argo CD 14 Installation Types 15 Deploying Argo CD 17 Summary 22 vii

3. Interacting with Argo CD. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 The User Interface in Depth 23 The Argo CD Command-Line Interface (CLI) 29 Additional Methods for Managing Argo CD 31 Summary 35 4. Managing Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Application Overview 37 Application Sources 39 Git 40 Helm 40 Destinations 40 Tools 41 Helm 42 Kustomize 43 Beyond Helm and Kustomize 44 Deploying Your First Application 44 Deleting Applications 47 Finalizers 48 Summary 49 5. Synchronizing Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Managing How Applications Are Synchronized 51 Sync Options 52 Application-Level Options 53 Resource-Level Options 54 Sync Order and Hooks 56 Hooks 56 Sync Waves 58 Comparing Options 59 Managing Resource Differences 60 Application-Level Diffing 60 System-Level Diffing 61 Use Case: Database Schema Setup 61 Argo CD Application Overview 62 Manifest Sync Wave Overview 63 Importance of Probes 65 Seeing It in Action 66 Summary 73 viii | Table of Contents

6. Authentication and Authorization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Managing Users 75 The Admin User 76 Local Users 77 SSO 83 Role-Based Access Control 109 Argo CD RBAC Basics 109 Custom Role Creation 111 RBAC Defaults 113 Anonymous Access 113 Summary 114 7. Cluster Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 Cluster Architecture 115 Local Versus Remote Clusters 116 Hub-and-Spoke Design 117 How Clusters Are Defined 117 Adding Remote Clusters 121 Creating a Cluster 121 Adding a Cluster with the CLI 122 Adding a Cluster Declaratively 124 Deploying Applications to Multiple Clusters 127 App-of-Apps Pattern 127 Using Helm 129 ApplicationSets 129 Summary 131 8. Multi-Tenancy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Argo CD Installation Modes 133 Cluster Scoped 134 Namespace Scoped 134 Projects 135 Resource Management 136 Use Case: GitOps Dashboard 138 Create Project 138 Deploy Applications 139 Configure Project 139 Test Setup 140 Summary 142 Table of Contents | ix

9. Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Securing Argo CD 143 Configuring TLS Certificates 146 Generating Argo CD TLS Certificates 147 Repository Access 149 Configuring TLS Repository Certificates 154 Protected Repositories 155 HTTPS Credentials 155 SSH-Based Authentication 158 Enabling Reuse Through Credential Templates 161 Enforcing Signature Verification 162 Enable Signature Verification 162 Signature Verification in Action 163 Application Sync Impersonation 165 Enable Sync with Impersonation 166 Define the Service Account to Use for Impersonation 167 Deploying an Application with Impersonation 168 Summary 169 10. Applications at Scale. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 Argo CD Application Drawbacks 172 Consideration and Best Practices 173 Set Up Probes 173 Argo CD Health Checks 174 Application Health 175 Eventual Consistency 176 Use Case Setup 177 Inspecting Probes 177 Adding Argo CD Health Checks 178 Use Case: App-of-Apps with Sync Waves 178 ApplicationSets 183 Progressive Sync 183 Use Case: Using Progressive Sync 184 Summary 188 11. Extending Argo CD. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 Config Management Plugins 189 The ConfigManagementPlugin Manifest 190 Registering the Plugin 194 Customizing Plugin Execution 199 x | Table of Contents

Environment Variables 199 Parameters 200 User Interface Customization 202 Banner Notifications 202 Custom Styles 203 UI Extensions 206 Summary 208 12. Integrating CI with Argo CD. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 Reconciliation Response Time 209 Modifying Reconciliation 210 Setting Up Webhooks 211 CI/CD Integration via Tekton 215 Building a Tekton Pipeline 216 Triggering Tekton Pipelines 222 Summary 227 13. Operationalizing Argo CD. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 Monitoring 229 Installing Prometheus Stack 230 Configuring Argo CD for Prometheus 231 Accessing Grafana 233 Notifications 235 Installing Mattermost 236 Configuring Mattermost 237 Setting Up Argo CD Notifications 247 High Availability 249 Scalability 251 Scaling Up 251 Sharding 253 Summary 256 14. Future Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257 GitOps Is Still Evolving 257 GitOps Directory Structure Considerations 258 Rendered Manifests Pattern 261 GitOps Workflow Best Practices 263 Interacting with the Community 265 Slack 265 GitHub 266 Table of Contents | xi

Next Steps 266 Progressive Delivery 266 GitOps Promotions 267 Summary 268 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 xii | Table of Contents

Preface Cloud native technologies, regardless of where they reside (on the public cloud or in a private datacenter) continue to proliferate. For those running containerized applications, Kubernetes has become the de facto solution for running and managing these applications at scale and, as a result, several different architectural patterns have emerged over time. GitOps is one such pattern that describes a set of processes for managing infrastructure and applications within source code stored within a Git repository. While GitOps is not exclusive to Kubernetes, it has strong ties to Kuber‐ netes, as the practices and principles have become the cornerstone for managing the platform. While GitOps provides a framework that defines how to align infrastructure as code (IaC) concepts for managing resources using content stored within source code management tools, there is still a need for a tool that can realize these goals and the declarative nature of the content. In the world of Kubernetes, Argo CD has become one of the most popular tools for implementing GitOps paradigms. Given its broad adoption within the Kubernetes community for use by both infrastructure and application teams, having an understanding of how it can be used effectively is essential. Who Should Read This Book This book is primarily written for Kubernetes administrators and developers who want to utilize GitOps practices to improve the user experience around cloud native technologies, along with those looking to operationalize Argo CD using the full set of features provided by the tool. However, since many development teams are also lever‐ aging Argo CD to deploy and manage their own applications, these teams will also find most of the content applicable for their use as well. Upon the completion of this book, you will be better equipped to implement Argo CD within your organization in a manner that supports production use. xiii

Whether you just started your Argo CD journey or are a seasoned power user, we wrote this book to be applicable for all levels of experience. By including key topics and a set of relatable examples, this book will become a reference that you can use from day one and beyond. Why We Wrote This Book Argo CD is one of the most popular toolsets in the Cloud Native Computing Foun‐ dation (CNCF) and is quickly becoming the de facto standard in GitOps implementa‐ tion. Even with its popularity, best practices and getting-started guides are sparse and scattered throughout the ecosystem. We wrote this book as a central place for those looking into operationalizing Argo CD without having to scour the internet for the information. Both of us have spent a large amount of time in the open source com‐ munity, as well as various enterprise organizations, assisting in the implementation of Argo CD in their own environment. We’ve collected our shared experiences and seek to be able to share them broadly so that others, like yourself, can become successful in your Argo CD journey. Navigating This Book The adoption of cloud native concepts is a journey. The following is a glimpse of what you can expect as you make your way through this book: • Chapters 1–3 cover everything that you need for beginning to be productive working with Argo CD, including the goals the project seeks to achieve, the installation methods, and common methods for interacting with the platform. • Chapters 4–5 place an emphasis on one of the most important topics within Argo CD: Applications. As the primary vehicle for managing resources in Kubernetes using GitOps, an in-depth overview of Applications will be provided, including the tools that can be used to define Kubernetes manifests, the content source for these manifests, and how and when they are applied to Kubernetes clusters. • Chapters 6–9 cover a number of topics that focus on the management of Argo CD, including authentication and authorization, cluster management, multi-tenancy and security. • Chapters 10–11 go beyond the basics, including advanced Application design and deployment patterns and extending the base functionality of Argo CD to take GitOps to new heights. • Chapters 12–13 discuss some of the key areas that are applicable for using Argo CD within large organizations, including how both the tool as well as GitOps in general can be incorporated into continuous integration/continuous delivery (CI/CD) workflows, as well as how to operationalize the platform at scale. xiv | Preface

• Chapter 14 might appear to be the end of our journey with Argo CD. However, it is just beginning, as this concluding chapter provides a number of resources for how to keep the conversation going with other members of the Argo CD community, as well as areas for further exploration. What This Book Will Not Cover This book will focus on how to get up and running with Argo CD in a Kubernetes environment. This book will not go over how to install Kubernetes nor how to manage the lifecycle of a Kubernetes cluster. Furthermore, there are many ways to do the same thing. We will be focusing a lot on Helm in this book; however, that is not to say that using other methods aren’t valid. It is impossible to go over every valid option. There are also many tools/projects that do similar things. Beyond Argo CD, usage of a particular tool over another doesn’t mean we are endorsing that tool or that we would use that particular tool all the time in every scenario. A lot of the time, we chose the tool for the sake of brevity. We will try and call out all these exceptions as we go over them. Prerequisites Before getting started, we will go through some of the prerequisites you might need in order to follow along in this book. We assume that you have access to an operational Kubernetes cluster; we will describe how to run an environment on your local machine using kind. However, we recommend that you test these out on a test system (and for that, we recommend kind). kind Although the steps outlined in this book should “just work” with most Kubernetes implementations, the exercises will make use of kind, a tool for running local Kuber‐ netes clusters within container “nodes.” You can get started with kind by visiting https://kind.sigs.k8s.io. The kind website includes instructions on how to install the kind binary and any of the other prerequisites. Several providers are available, which map to popular con‐ tainer runtimes, including Docker, Podman, or nerdctl (containerd), which enables its use among a greater set of end users. Helm We use Helm routinely throughout the course of this book, so it will be necessary to have the Helm binary available in your $PATH. You can visit Helm for installation guidelines. Preface | xv

Kubernetes Client Since we will be interacting with Kubernetes clusters, it will be important to have the kubectl client available. You can follow the instructions on the official Kubernetes documentation site. Argo CD CLI Client Argo CD comes with the argocd CLI client that interacts with the Argo CD API server. You can follow the instructions found on the Argo CD website for installation of this client. YAML/JSON Processing To make things easier, we use a lot of jq and yq to modify/update JSON/YAML in place. You can find information about these tools by visiting their respective websites: jq and yq. If you’re using Linux or a Mac, you might be able to find these utilities using their respective package manager (for example; you can run brew install jq on a Mac). Companion Git Repository Throughout this book, you will work through a series of exercises and examples as you expand your knowledge of Argo CD. These resources are available within a Git repository. Since Git is the source code management (SCM) tool for not only interacting with the companion repository but also GitOps as a whole as well as Argo CD, it is important that you also have Git installed locally on your machine. Information related to Git, including the supported installation options and platforms can be found on the Git website. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. xvi | Preface

Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://oreil.ly/argoCD_UR_repo. If you have a technical question or a problem using the code examples, please send email to support@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. Preface | xvii

We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Argo CD: Up and Run‐ ning by Andrew Block and Christian Hernandez (O’Reilly). Copyright 2025 Andrew Block and Christian Hernandez, 978-1-098-14200-1.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-889-8969 (in the United States or Canada) 707-827-7019 (international or local) 707-829-0104 (fax) support@oreilly.com https://oreilly.com/about/contact.html We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/argoCD_UR. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly-media. Watch us on YouTube: https://youtube.com/oreillymedia. xviii | Preface

Acknowledgments Andy Block: They say that it takes a village to raise a child, and this sentiment is certainly true for both the GitOps and Argo CD communities. It would not be possible to pro‐ duce a publication, such as this book on Argo CD, without the continued support of the open source community. In particular, I would like to thank Dan Garfield, who has helped shed light into what it takes to build a business that is focused primarily on GitOps. In addition, I wanted to also thank Michael Crenshaw. whose unbelievably deep knowledge of Argo CD has helped me time after time better understand all of the minute details of the project. These insights directly translated into the ongoing support that I am able to provide to community members along with material within this book. Of course, I could not forget my colleagues at Red Hat who have helped and supported my endeavors within the GitOps space. From Raffaele Spazzoli and our endless conversations on Helm, Kustomize, and various GitOps patterns to Gerald Nunn and our thoughts and designs for what it takes to properly architect and operate GitOps as a platform service within some of the most regulated organizations in the world. And, to the entire OpenShift GitOps team. Thank you for making me feel like an extended member of your team, where our ongoing collaboration has enabled our customers to apply GitOps principles at scale, using some of the most secure and trusted software available. Finally, Argo CD is just one of many GitOps tools in the industry. There will never be a single GitOps tool, and we are all better because of that fact. A big thank you goes out to those in the GitOps community, including Scott Rigby, Alexis Richardson, and Stacey Potter. Your continued partnership and collabora‐ tion is truly appreciated! Christian Hernandez: The path to being a subject matter expert in a particular technology—to the point where you write a book—isn’t a path you take alone. There have been many people in my career who have helped me get where I am. I would like to take this opportunity to give many thanks to those people. My time at Red Hat was paramount to my development, and I couldn’t have done it without the mentorship and leadership I received from Scott Cranton, Chris Morgan, and Erik Jacobs. Your willingness to let me grow was pivotal in my success. I cannot express my gratitude enough for everything. Also, to my “OG OpenShift TigerTeam” coworkers. We were lucky enough to work together during the best time of my career. Being able to work with experts in the field propelled me to be the best I can be. Also, a very special thanks to Chris Short, who always pushed me to be the “Kelsey Hightower of GitOps.” Preface | xix

Lastly, I would like to thank Hong Wang, Jesse Suen, and Alexander Matyushent‐ sev. Creating the Argo Project was a bold and brave thing to do (even if you all didn’t know it at the time). Growing with the Argo Project has been a privilege; and now working with you all directly has elevated me to a level of expertise that I wouldn’t have imagined. I am proud to be a part of your journey, and I wouldn’t be here without what you three have created. Both: We are deeply grateful to the tech reviewers for their meticulous attention to detail and technical expertise, which greatly enhanced the accuracy and quality of this book. We would like to thank the following: • Vladislav Bilay • Manuel Dewald • Werner Dijkerman • Nadir Doctor • Predrag Knežević • Jess Males • Benjamin Muschko • Gerald Nunn • Lipi Deepaakshi Patnaik • Rick Rackow Your invaluable feedback helped us refine complex concepts, ensuring clarity and precision for readers. The insights and suggestions you provided were instru‐ mental in strengthening the technical depth and real-world applicability of the content. We sincerely appreciate the time and effort you dedicated to reviewing, catching errors, and offering thoughtful recommendations. This book is stronger because of your contributions, and we are truly thankful for your commitment to making it the best resource possible. xx | Preface