CICD DESIGN PATTERNS Design and Implement Cicd Through Tried and Tested (Garima Bajpai, Pawel Piwosz etc.) (z-library.sk, 1lib.sk, z-lib.sk)

CI/CD Design Patterns Design and implement CI/CD using proven design patterns Garima Bajpai Michel Schildmeijer Muktesh Mishra Pawel Piwosz

CI/CD Design Patterns Copyright © 2024 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. The authors acknowledge the use of cutting-edge AI, specifically Grammarly writing assistant, with the sole aim of enhancing text flow, performing plagiarism checks, and correcting certain sentences within the book, thereby ensuring a smooth reading experience for readers. It is important to note that the content itself has been crafted by the authors and edited by a professional publishing team. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Group Product Manager: Preet Ahuja Publishing Product Manager: Suwarna Patil Book Project Manager: Ashwini Gowda Senior Editor: Mohd Hammad Technical Editor: Arjun Varma Copy Editor: Safis Editing Proofreader: Mohd Hammad Indexer: Rekha Nair Production Designer: Gokul Raj S.T DevRel Marketing Coordinator: Rohan Dobhal First published: December 2024 Production reference: 1151124 Published by Packt Publishing Ltd. Grosvenor House 11 St Paul’s Square Birmingham B3 1RB, UK ISBN 978-1-83588-964-0 www.packtpub.com

I would like to thank the Canada DevOps Community of Practice, where I have been able to brainstorm a lot of ideas with chapter leads and senior community members. I would also like to thank the open-source community, which provides us with a platform to steer thought leadership in the DevOps and continuous delivery space, especially the Continuous Delivery Foundation ambassadors. My gratitude toward my co-authors and the entire team at Packt; without them, I would not have been able to bring this book to life. Lastly, I would like to express my gratitude toward my husband, Amit Mishra, and my daughter, Akira, for their relentless support and encouragement. – Garima Bajpai Writing this book and sharing my learnings with others has been one of the most rewarding experiences of my life. I would like to extend my heartfelt thanks to my co-authors, Garima, Michel, and Pawel—you all are awesome! I am also deeply grateful to the Packt team, the Continuous Delivery Foundation, and the incredible open source community for their invaluable support. A special thanks to my wife, Anita, and my family for their love, patience, and unwavering support, without which this book would not have been possible. You are my constant source of strength and inspiration. – Muktesh Mishra

Foreword When DevOps emerged in 2009, and for several years after, teams in organizations adopted continuous integration/continuous delivery (CI/CD) in a haphazard manner. It was mostly tactical, often an experiment in version control or build automation. It was developers trying to avoid merge hell, trying to deliver something better to their friends in IT operations. It was the dawn of DevOps. More teams in more organizations tried out more tools, hung together DevOps toolchains by accident, bolting on bit after bit, striving for test coverage, test categories, and all the tests needed. Security and performance got involved. People started to wonder what continuous deployment was and what they could do to limit blast radius using canary testing and other progressive deployment techniques such as blue-green. Infrastructure as Code was part of it. It was all a bit chaotic. Heterogeneous pipelines spawned across departments. Arguments broke out over tool standardization. Platform engineering emerged. And now times have changed. This book is a fantastic example of what we have learned along these paths we have traveled, the paths all the authors have taken with their twists and turns and dead-ends and lights at the ends of tunnels. Together, they bring their wide and varied expertise of what’s possible when designing CI/CD solutions for your teams in your organization and how to pick the right design pattern for the outcome you seek to achieve, for the constraints that exist for you. You will benefit from all the learnings that have come before, distilled into these patterns, abstracted to make it easy for you to collaborate with your colleagues on what’s going to work best for you. They address the most common design problems and are adaptable and flexible, so you can make them fit your unique requirements and environments. You truly are standing on the shoulders of giants. Helen Beal Head of Ambassador Program, DevOps Institute (PeopleCert)

Contributors About the authors Garima Bajpai is an industry leader in DevOps and cloud technologies. She is the founder of the Canada DevOps Community of Practice. She leads the ambassador program for the Continuous Delivery Foundation. Some might know her as a course contributor and instructor for various DevOps courses from leading certification bodies. She has over two decades of experience leading large-scale R&D with a variety of different teams and has helped them adapt DevOps to be able to increase team productivity when it comes to cloud resource deployment. Furthermore, she has collaborated on and contributed to many international conference talks, written several technical blog posts, and published white papers. Michel Schildmeijer started his career in pharma and then switched to IT, where he increased his multiple-industry knowledge in his role as solutions or IT architect in several industries. At the time of writing this book, he fulfills the role of enterprise architect at the Dutch government. He has received the IT industry-recognized title of Oracle ACE for being an ambassador and community leader in his area of expertise. Michel speaks regularly about technology and the impact of innovation at national and international conferences. He contributes to the open source community and solutions regarding containerization, CI/CD, and DevOps. Muktesh Mishra is a passionate engineer and open source enthusiast with expertise in cloud computing, DevOps, software architecture, distributed systems, and artificial intelligence. He loves programming across multiple languages and solving complex software problems. Beyond his professional pursuits, Muktesh is a keen photographer and cook. He has presented at numerous global conferences on topics such as CI/CD, developer productivity, platform engineering, and open source technologies. Committed to giving back to the community, Muktesh actively contributes to Apache Software Foundation, Microsoft TEALS, and open source forums. He is a proud ambassador for the Continuous Delivery Foundation and the founder of the Sunnyvale Java User Group.

Pawel Piwosz has a long career in SysOps and DevOps areas. With experience in small start-ups and huge, global organizations, he has unique knowledge of processes and executions of DevOps workloads. He is deeply interested in Infrastructure as Code and CI/CD. He authored a CI/CD design framework, which helps organizations create and assess CI/CD processes. He is a DevOps Institute Ambassador, Continuous Delivery Foundation ambassador, and AWS Community Builder. He speaks about DevOps and DevSecOps at multiple conferences and meetups, and he led the DevOps Academy where he was responsible for introducing new engineers to the DevOps world. He is a mentor in the Tech Leaders mentoring program.

About the reviewers Roshan Mahant is a seasoned senior J2EE/AMANDA solution architect with over 20 years of experience in developing enterprise applications for various industries, including finance, gaming, and e-governance. He has expertise in Java, J2EE, Spring Boot, microservices, and cloud computing (AWS). Roshan has worked on digital transformation projects for state agencies, customizing platforms such as AMANDA for efficient governance. His background also encompasses academic roles in programming and engineering education. I would like to express my gratitude to my family and friends for their unwavering support throughout this journey. Special thanks to my colleagues for their collaboration and insights, which have greatly contributed to my professional growth. Mayank Jadhav is a seasoned DevOps engineer with over five years of experience in crafting and optimizing mission-critical deployments. His expertise lies in automating processes, streamlining CI/CD pipelines, and ensuring robust application and infrastructure monitoring. Mayank’s DevOps expertise enables him to deliver business-driven solutions. His focus on automation and monitoring consistently drives positive results in his projects. His dedication to staying updated with industry trends is reflected in his ongoing pursuit of knowledge through technical book reviews and contributions to the DevOps community through his LinkedIn presence, where he is recognized as a Top Cloud Computing Voice.

Preface xvii Part 1: Introduction to CI/CD Design Patterns 1 Foundations of CI/CD Design Patterns 3 An overview of design patterns 3 Types of design patterns 4 Where design patterns meet CI/CD 12 Understanding CI/CD and design patterns 13 The build cycle 14 Manual deployment 14 The evolution of CI/CD 14 The maturity journey of CI/CD 15 The evolution of CI/CD deployments – push and pull-based implementation 16 Implementing testing in CI/CD 17 The evolution of CI/CD deployments – infrastructure 19 The key components of design patterns – pipelines and infrastructures 21 Pipelines – definition and characteristics 21 Pipelines – techniques and tools 23 Orchestration – coordinating pipeline execution 24 Testing – ensuring code quality and reliability 25 Deployment strategies – efficiently releasing software 25 Logging and monitoring 26 Pipelines and Infrastructures 26 The deployment of CI/CD 27 Rolling deployments 27 Blue-green (red-black) deployments 29 Canary deployments 30 Summary 32 Further reading 32 Table of Contents

Table of Contentsx 2 Understanding Types of CI/CD Design Patterns and Their Components 33 Common CI/CD design patterns 34 The build and deploy model 34 Key component – Pipeline as Code 36 Faster team feedback/feedback loops 36 Design pattern principles – sequential and parallel execution 37 How to choose between sequential and parallel execution 40 Design pattern components – Pipeline as Code 40 How do you implement PaC? 41 Design pattern components – Infrastructure as Code 46 Example Argo CD applications (the Observer pattern) 53 Summary 55 3 Advancing on CI/CD Design Patterns – from Testing to Deployment 57 Design pattern components – test automation 57 Design pattern components – artifacts 62 Using artifacts 65 Artifact components related to CI/CD 66 Design pattern components – deployment types 68 Test automation patterns 70 Branching strategies 72 Other CI/CD patterns 73 Industry best practices and challenges 73 What are the components of a CI/CD pipeline? 74 What steps should be taken to implement CI/CD? 75 Why are these industry best practices? 76 Challenges around implementing CI/CD 76 Summary 76 4 Business Outcome Alignment with CI/CD Design Patterns 79 CI/CD design patterns – strategic intent 80 State of play – CI/CD design patterns 82 A decade of action using CI/CD design patterns 83 Introduction to CI/CD measurements 86

Table of Contents xi North Star for CI/CD design patterns 89 Scaling adoption of CI/CD design patterns 90 Removing the barrier for CI/CD design pattern adoption 91 Resistance to change – cultural aspects 91 Summary 92 Further reading 92 Part 2: Structural Design Patterns for CI/CD 5 Exploring Structural CI/CD Design Patterns 95 Introducing structural design patterns 95 CI/CD design pattern principles – monorepo and polyrepo scalability 97 Design examples for the monorepo and polyrepo approaches 100 CI/CD design patterns – dependencies between pipelines 103 Key components – tools, code, and modularity 105 Tools 106 Code 108 Modularity 110 Key components – level of control 116 Summary 117 6 Deployment Strategies for Structural Design Patterns for CI/CD 119 Structural CI/CD Patterns – deployment strategies 120 Types of deployment strategies 121 Blue-green deployments 122 Canary deployments 123 Feature toggle deployments 126 A/B testing deployments 127 Dark launch deployments 128 Structural CI/CD design patterns and integration of data 129 Code phase 129 Build phase 130 Test phase 131 Deploy phase 131 Post-deploy phase 132 CDEvents 133 Supply-chain Levels for Software Artifacts 133 Setting boundaries through access management and policy enforcement 134 SoD 134 Summary 138

Table of Contentsxii Part 3: Behavioral and Domain-Driven Design Patterns for CI/CD 7 Understanding Behavioral Design Patterns for CI/CD 141 An overview of behavioral CI/CD design patterns 141 Behavioral CI/CD design pattern principles 146 Key components – tools and processes for AI-based observability 149 Tools and processes 149 AI-based observability 153 Challenges in implementing behavioral design patterns 154 Implementing design patterns in CI/CD 155 Implementation steps in CI/CD – a practical example 156 Summary 158 8 Domain-Driven Design Patterns for Regulated Sectors 159 An overview of a domain-driven CI/CD design pattern 160 Domain-driven CI/CD design pattern principles – implementing regulation actions 162 Key components – managing access control 164 Key components – building artifacts according to regulations 169 The importance of DDD 171 Performance considerations of DDD in CI/CD 172 An example of DDD in CI/CD 173 A real-life case study 176 Practical implementation of DDD in CI/CD 178 Summary 179 Further reading 180

Table of Contents xiii Part 4: Creational CI/CD Design Patterns 9 Applying Creational CI/CD Design Patterns 183 Creational design patterns – the concept 184 Creational CI/CD design pattern principles – scalability and resilience 185 Singleton pattern 186 Factory Method pattern 187 Abstract Factory pattern 189 Builder pattern 190 Prototype pattern 192 CI/CD cloud-native creational design patterns 193 Key components – containerization, microservices, and the serverless ecosystem 197 Containerization 197 Microservices 201 Serverless 202 A real-life example of microservices in a CI/CD ecosystem 203 Key components – ensuring security and compliance 205 Implementing CI/CD pipelines based on creational design patterns 207 Factory Method implementation 207 Abstract Factory pattern implementation 209 Builder pattern implementation 212 Prototype pattern implementation 213 Singleton pattern implementation 214 Summarizing the implementation of creational design patterns 215 Prerequisites of scaling and optimization 216 Summary 217 10 Understanding Deployment Strategies – Creational CI/CD with Cloud Providers 219 Creational design pattern used in CI/CD 220 Landing zones 220 Code repositories as a singleton 221 Singleton creational pattern in a microservices world 222 Singleton pattern in pipelines 223 Singletons in clouds 224 Cloud-native development and its influence on CI/CD patterns 225 Immutable infrastructure 225 Microservices 226 API 226 IaC 226 Cloud provider offerings 227 API-driven communication for microservices 227 API-driven communication for serverless 228 Cloud offerings for CI/CD toolset 231 Microsoft Azure – Azure DevOps 231

Table of Contentsxiv AWS CodePipeline 231 Deployment strategies 232 Blue-green deployment strategy 232 Blue-green modification – blue-violet 235 Blue-green modification – red-black 237 Canary deployment 238 Rolling deployment 240 Team topologies 241 Example – Landing zones, CI/CD, and team topologies 242 Practical examples 244 CodePipeline Build block – the CI part of the process 244 CodePipeline Deploy – the CD part 246 Summary 247 Further reading 248 11 Auditing and Assessment of Design Patterns 249 Overview – taxonomy of assessment and audits 250 Conducting a comprehensive assessment of CI/CD design patterns 251 Tools and techniques for comprehensive assessments 251 Conducting comprehensive audits on CI/CD design patterns 252 Shifting the auditing process left 253 Common audit checklist for the CI/ CD design patterns 254 General audit points 254 Specific audit points 255 Implementing audit points with quality gates 256 Impact of audits and assessments 258 Business value impact 259 Maturity model of design patterns 259 Common challenges for comprehensive audits and assessments 261 Next steps for comprehensive audits and assessments 261 Summary 261 Further reading 262 Part 5: Advanced Design Patterns and Anti- Patterns for CI/CD 12 Advanced CI/CD Design Patterns and Use Cases 265 Self-learning CI/CD design practices 266 Understanding real-time utility-based CI/CD design patterns 269

Table of Contents xv The challenges of implementing real-time utility-based CI/CD systems 273 A real-life scenario of implementation challenges 274 Further evolution and roadmap considerations 277 Adoption of new architectures and technologies 278 Scalability 278 Practical implementation 278 CI/CD in the near future – generative AI 279 Summary 281 13 Exploring Anti-Patterns for CI/CD Design Pattern Deployments 283 Anti-patterns for CI/CD design patterns 283 What is an anti-pattern? 284 Best fit design patterns – considerations 296 CI/CD design pattern and platform engineering 300 Exploring a case study of CI/CD integration in platform engineering 301 The integration of CI/CD in platform engineering 301 Summary 304 Part 6: Case Studies 14 Appendix – Knowledge Test and Case Studies 307 Knowledge test 307 The quiz 308 Answers 310 Case studies 311 Case 1 – Improving unnecessary “wiring” 311 Case 2 – Fixing bad decision 314 Case 3 – Embracing the platform team 314 Summary 315 Index 317 Other Books You May Enjoy 330

Preface Hello there! In the evolving landscape of software development, continuous integration and continuous delivery (CI/CD) have become the lifeblood of innovation and efficiency. CI/CD practices empower teams to bring new features or update existing software in order to market in a faster and more reliable way with improved quality. For organizations of all sizes, adopting CI/CD practices can mean the difference between thriving in a competitive market or lagging behind. CI/CD has become a transformative approach that every organization can benefit from, whether they’re a start-up building their first product or an enterprise with established systems. This book aims to provide a roadmap through the complex landscape of CI/CD, bridging foundational principles with practical design patterns. We’ll explore how CI/CD can be optimally structured to meet various organizational needs while being scalable and resilient to change. The need for CI/CD is evident, but implementing it effectively is often challenging, especially on an organization’s scale. This book is designed to fill that gap, offering a pattern-driven approach that takes into account the nuanced requirements of various organizational sizes and development methodologies. You’ll find practical advice, industry insights, and pattern-based solutions that can be readily applied to real-world CI/CD challenges. Whether you’re a developer, DevOps engineer, architect, or technology leader, this book will serve as both a reference and a guide. Together, we’ll uncover how CI/CD, underpinned by these structural, behavioral, and creational patterns, can elevate your development processes, reduce risk, and increase delivery speed—ultimately, helping your organization move forward with confidence in a fast-paced world. Who this book is for The following persona types will specifically benefit from the book: • Senior/principal developers and software architects: Learn about reusable patterns and practices while delivering the software • Site reliability engineering architects, DevOps architects, and cloud architects: Learn about reliable and quality-driven software delivery and how to design complex pipelines with reusable components utilizing different design patterns • Engineering managers: Learn how to gauge the overall quality of the pipeline and delivery of the software along with best practices

Prefacexviii This book highlights the relationships and interactions between different tools and practices within CI/CD. The objective is to speed up the development process by providing well-tested, proven development/design paradigms when it comes to CD and its adoption. Overall, It can be considered a model that you can use to implement CD in specific environments with similar needs. What this book covers Chapter 1, Foundations of CI/CD Design Patterns, explains the significance of efficient CI/CD in modern organizations. It also provides an overview of the delivery pipeline and its components. Chapter 2, Understanding Types of CI/CD Design Patterns and Their Components, talks about key considerations while designing pipelines and also explains the significance of Infrastructure as Code and pipeline as code in the deployment pipeline. Chapter 3, Advancing on CI/CD Design Patterns – from Testing to Deployment, focuses on pipeline components and stages around artifact management and test automation. It also talks in depth about testing strategies and their significance while designing the delivery pipeline. Chapter 4, Business Outcome Alignment with CI/CD Design Patterns, talks about how efficient CI/CD practices and architecture can be aligned with business goals such as improved user experiences. It also provides insights to users on some of the best practices to align implementation and adoption of CI/CD design patterns toward bigger business outcomes. Chapter 5, Exploring Structural CI/CD Design Patterns, provides an overview of structural components in the pipeline and their interaction, especially in terms of monorepo or polyrepo source code repositories. It also talks about how to consider constructs such as the modularity of the pipelines and components while designing a pipeline. Chapter 6, Deployment Strategies for Structural Design Patterns for CI/CD, talks about different deployment strategies and their trade-offs. It also introduces concepts such as data-driven DevOps where data/feedback loops are integrated into different components by default. Chapter 7, Understanding Behavioral Design Patterns for CI/CD, discusses the pipeline components and their behavior. It also explains the importance of designing observability into the pipeline and its benefits, such as enhanced debuggability and reduced downtimes. More importantly, it also highlights using observability as a mechanism to improve the pipeline behavior and performance. Chapter 8, Domain-Driven Design Patterns for Regulated Sectors, discusses the importance and challenges of applying the CI/CD practices in a regulated sector such as banking/finance or healthcare. This chapter also talks about security and access paradigms such as access controls and approval workflows and their importance and implementations.

Preface xix Chapter 9, Applying Creational CI/CD Design Patterns, focuses on applying creational design patterns such as built-in resilience and scalability into the pipeline design considerations. It provides their implementation in terms of pipeline design for complex and modern software and different workloads (serverless to containers). Chapter 10, Understanding Deployment Strategies – Creational CI/CD with Cloud Providers, covers the various implementations from different cloud providers and their examples. This chapter also talks about implementing CI/CD from a team topologies perspective and considerations. Chapter 11, Auditing and Assessment of Design Patterns, discusses a maturity model for design patterns and also running audits while implementing one or multiple patterns for defining the pipeline. Chapter 12, Advanced CI/CD Design Patterns and Use Cases, covers pipeline design for advanced use cases such as machine learning-based workflows or real-time utility-based workflows. It also talks about various challenges such as performance or implementation and how to solve them with different strategies. Chapter 13, Exploring Anti-Patterns for CI/CD Design Pattern Deployments, talks about common mistakes to avoid while designing the pipelines. Chapter 14, Appendix – Knowledge Test and Case Studies, talks about some real-world case studies and examples for designing pipelines. To get the most out of this book Here, we will give you some specifications for trying the examples out yourself. Software/hardware covered in the book Operating system requirements Jenkins Windows, macOS, or Linux (Any) OpenTofu Windows, macOS, or Linux (Any) Kubernetes Windows, macOS, or Linux (Any) Download the example code files There are a few code examples that are also supplemented at multiple places in the book. You can refer to or download the example code files for this book from GitHub at https://github. com/PacktPublishing/CI-CD-Design-Patterns. If there’s an update to the code, it will be updated in the GitHub repository. We also have other code bundles from our rich catalog of books and videos available at https:// github.com/PacktPublishing/. Check them out!