Kubernetes Security and Observability A Holistic Approach to Securing Containers and Cloud Native Applications (Brendan Creane, Amit Gupta) (Z-Library)

C rea ne & G up ta Kub ernetes Security a nd O b serva b ility Brendan Creane & Amit Gupta Kubernetes Security and Observability A Holistic Approach to Securing Containers and Cloud Native Applications Compliments of

SECURIT Y AND OBSERVABILIT Y “This book is a must-read for anyone who cares about securing their Kubernetes clusters or workloads! These concepts are largely applicable regardless of industry, vertical, or company size. Tin foil hats are not included.” —Seth Vargo Senior Staff Engineer, Google Kubernetes Security and Observability Twitter: @oreillymedia facebook.com/oreilly Securing, observing, and troubleshooting containerized workloads on Kubernetes can be daunting. It requires a range of considerations, from infrastructure choices and cluster configuration to deployment controls and runtime and network security. With this practical book, you’ll learn how to adopt a holistic security and observability strategy for building and securing cloud native applications running on Kubernetes. Whether you’re already working on cloud native applications or are in the process of migrating to its architecture, this guide introduces key security and observability concepts and best practices to help you unleash the power of cloud native applications. Authors Brendan Creane and Amit Gupta from Tigera take you through the full breadth of new cloud native approaches for establishing security and observability for applications running on Kubernetes. • Learn why you need a security and observability strategy for cloud native applications and determine your scope of coverage • Understand key concepts behind the book’s security and observability approach • Explore the technology choices available to support this strategy • Discover how to share security responsibilities across multiple teams or roles • Learn how to architect Kubernetes security and observability for multicloud and hybrid environments Brendan Creane is head of engineering at Tigera. He has several decades of experience building enterprise security, observability, and networking products. Amit Gupta is vice president of product management and business development at Tigera. He is a hands- on product executive with expertise in building software products and services across various domains, including cloud security, cloud native applications, and public and private cloud infrastructure. C rea ne & G up ta Kub ernetes Security a nd O b serva b ility ISBN: 978-1-098-10711-6

Get started with a free trial of Calico Cloud Trusted by Innovators tigera.io/tigera-products/cloud-trial Modern security and observability for distributed cloud-native applications. COMPLIANCE CLOUD -NATIVE NETWORK SECURITY CONTAINER SECURITY OBSERVABILITY Comprehensive protection for containers and Kubernetes Container firewall, workload access controls, and microsegmentation Continuously monitor and enforce compliance controls, easily create custom reports for audit Visibility and troubleshooting with service graph, packet capture, and ML based anomaly detection

Brendan Creane and Amit Gupta Kubernetes Security and Observability A Holistic Approach to Securing Containers and Cloud Native Applications Boston Farnham Sebastopol TokyoBeijing

978-1-098-10711-6 [LSI] Kubernetes Security and Observability by Brendan Creane and Amit Gupta Copyright © 2022 O’Reilly Media. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: John Devins Development Editor: Virginia Wilson Production Editor: Beth Kelly Copyeditor: J. M. Olejarz Proofreader: Kim Wimpsett Indexer: Sue Klefstad Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea November 2021: First Edition Revision History for the First Edition 2021-11-26: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781098107109 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Kubernetes Security and Observability, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. This work is part of a collaboration between O’Reilly and Tigera. See our statement of editorial independence.

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi 1. Security and Observability Strategy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Security for Kubernetes: A New and Different World 1 Deploying a Workload in Kubernetes: Security at Each Stage 3 Build-Time Security: Shift Left 5 Deploy-Time Security 6 Runtime Security 7 Observability 13 Security Frameworks 15 Security and Observability 17 Conclusion 18 2. Infrastructure Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Host Hardening 20 Choice of Operating System 20 Nonessential Processes 21 Host-Based Firewalling 21 Always Research the Latest Best Practices 21 Cluster Hardening 22 Secure the Kubernetes Datastore 22 Secure the Kubernetes API Server 23 Encrypt Kubernetes Secrets at Rest 23 Rotate Credentials Frequently 25 Authentication and RBAC 25 Restricting Cloud Metadata API Access 26 Enable Auditing 26 Restrict Access to Alpha or Beta Features 28 v

Upgrade Kubernetes Frequently 29 Use a Managed Kubernetes Service 29 CIS Benchmarks 29 Network Security 30 Conclusion 32 3. Workload Deployment Controls. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Image Building and Scanning 33 Choice of a Base Image 33 Container Image Hardening 35 Container Image Scanning Solution 36 Privacy Concerns 37 Container Threat Analysis 37 CI/CD 38 Scan Images by Registry Scanning Services 39 Scan Images After Builds 40 Inline Image Scanning 41 Kubernetes Admission Controller 42 Securing the CI/CD Pipeline 42 Organization Policy 43 Secrets Management 43 etcd to Store Secrets 43 Secrets Management Service 44 Kubernetes Secrets Store CSI Driver 44 Secrets Management Best Practices 44 Authentication 46 X509 Client Certificates 46 Bearer Token 47 OIDC Tokens 47 Authentication Proxy 47 Anonymous Requests 47 User Impersonation 48 Authorization 48 Node 48 ABAC 48 AlwaysDeny/AlwaysAllow 48 RBAC 49 Namespaced RBAC 50 Privilege Escalation Mitigation 50 Conclusion 51 vi | Table of Contents

4. Workload Runtime Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Pod Security Policies 53 Using Pod Security Policies 54 Pod Security Policy Capabilities 56 Pod Security Context 58 Limitations of PSPs 59 Process Monitoring 59 Kubernetes Native Monitoring 60 Seccomp 62 SELinux 64 AppArmor 66 Sysctl 67 Conclusion 68 5. Observability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Monitoring 69 Observability 72 How Observability Works for Kubernetes 72 Implementing Observability for Kubernetes 76 Linux Kernel Tools 79 Observability Components 80 Aggregation and Correlation 81 Visualization 84 Service Graph 84 Visualization of Network Flows 85 Analytics and Troubleshooting 86 Distributed Tracing 86 Packet Capture 87 Conclusion 87 6. Observability and Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Alerting 89 Machine Learning 92 Examples of Machine Learning Jobs 93 Security Operations Center 94 User and Entity Behavior Analytics 96 Conclusion 98 7. Network Policy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 What Is Network Policy? 99 Why Is Network Policy Important? 100 Network Policy Implementations 101 Table of Contents | vii

Network Policy Best Practices 102 Ingress and Egress 103 Not Just Mission-Critical Workloads 103 Policy and Label Schemas 103 Default Deny and Default App Policy 105 Policy Tooling 107 Development Processes and Microservices Benefits 107 Policy Recommendations 108 Policy Impact Previews 108 Policy Staging and Audit Modes 109 Conclusion 109 8. Managing Trust Across Teams. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Role-Based Access Control 112 Limitations with Kubernetes Network Policies 112 Richer Network Policy Implementations 113 Admission Controllers 117 Conclusion 119 9. Exposing Services to External Clients. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Understanding Direct Pod Connections 122 Understanding Kubernetes Services 123 Cluster IP Services 123 Node Port Services 124 Load Balancer Services 125 externalTrafficPolicy:local 126 Network Policy Extensions 127 Alternatives to kube-proxy 128 Direct Server Return 129 Limiting Service External IPs 130 Advertising Service IPs 132 Understanding Kubernetes Ingress 133 Conclusion 136 10. Encryption of Data in Transit. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 Building Encryption into Your Code 138 Sidecar or Service Mesh Encryption 139 Network-Layer Encryption 140 Conclusion 142 11. Threat Defense and Intrusion Detection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Threat Defense for Kubernetes (Stages of an Attack) 143 viii | Table of Contents

Intrusion Detection 147 Intrusion Detection Systems 147 IP Address and Domain Name Threat Feeds 147 Special Considerations for Domain Name Feeds 150 Advanced Threat Defense Techniques 154 Canary Pods/Resources 154 DNS-Based Attacks and Defense 155 Conclusion 156 Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Table of Contents | ix

Preface Kubernetes is not secure by default. Existing approaches to enterprise and cloud secu‐ rity are challenged by the dynamic nature of Kubernetes and the goal of increased organizational agility that often comes with using it. Successfully securing, observing, and troubleshooting mission-critical microservices in this new environment requires a holistic understanding of a breadth of considerations. These include organizational challenges, how new cloud native approaches can help meet the challenges, and the new best practices and how to operationalize them. While there is no shortage of resources on Kubernetes, navigating through them and formulating a comprehensive security and observability strategy can be a daunting task and in many cases leads to gaps that significantly undermine the desired security posture. That’s why we wrote this book—to guide you toward a holistic security and observa‐ bility strategy across the breadth of these considerations and to give you best practi‐ ces and tools to help you as you move applications to Kubernetes. Over our years of working at Tigera and building Calico, a networking and security tool for Kubernetes, we have gotten to see the user journey up close. We have seen many users focus on getting their workloads deployed in Kubernetes without think‐ ing through their security or observability strategy, and then struggle as they try to understand how to secure and observe such a complex distributed system. Our goal with this book is to help minimize this pain as much as possible by sharing with you what we’ve learned. We mention a number of tool examples throughout, and Calico is among them. We believe that Calico is an excellent and popular option, but there are many good tools, like Weave Net, VMware Tanzu, Aqua Security, and Datadog, to choose from. Ultimately, only you can decide which is best for your needs. xi

The Stages of Kubernetes Adoption Any successful Kubernetes adoption journey follows three distinct stages: The learning stage As a new user, you begin by learning how Kubernetes works, setting up a sand‐ box environment, and starting to think about how you can use Kubernetes in your environment. In this stage you want to leverage the online Kubernetes resources available and use open source technologies. The pilot/pre-production stage Once you familiarize yourself with Kubernetes and understand how it works, you start thinking about a high-level strategy to adopt Kubernetes. In this stage you typically do a pilot project to set up your cluster and onboard a couple of applica‐ tions. As you progress in this stage, you will have an idea about which platforms you’re going to use and whether they will be on-premise or in the cloud. If you choose cloud, you will decide whether to host the cluster yourself or leverage a managed Kubernetes service from a cloud provider. You also need to think about strategies to secure your applications. By this time, you would have realized that Kubernetes is different due to its declarative nature. This means that the platform abstracts a lot of details about the network, infrastructure, host, etc., and there‐ fore makes it very easy for you to use the platform for your applications. Because of this, the current methods you use to secure your applications, infrastructure, and networks simply do not work, so you now need to think about security that is native to Kubernetes. The production stage By this point, you have completed your pilot project and successfully onboarded a few applications. Your focus is on running mission-critical applications in pro‐ duction and on considering whether to migrate most of your applications to Kubernetes. In this stage you need to have detailed plans for security, compliance, troubleshooting, and observability in order to safely and efficiently move your applications to production and realize all the benefits of the Kubernetes platform. The popularity and success of Kubernetes as a platform for container-based applications has many people eager to adopt it. In the past couple of years, there has been an effort by managed Kubernetes service providers to innovate and make adoption eas‐ ier. New users may be tempted to go past the learning and pilot stages in order to get to the production stage quickly. We caution against skipping due diligence. You must consider security and observability as critical first steps before you onboard mission- critical applications to Kubernetes; your Kubernetes adoption is incomplete and potentially insecure without them. xii | Preface

Who This Book Is For This book is for a broad range of Kubernetes practitioners who are in the pilot/pre- production stage of adoption. You may be a platform engineer or part of the security or DevOps team. Some of you are the first in your organization to adopt Kubernetes and want to do security and observability right from the start. Others are helping to establish best practices within an organization that has already adopted Kubernetes but has not yet solved the security and observability challenges Kubernetes presents. We assume you have basic knowledge of Kubernetes—what it is and how to use it as an orchestration tool for hosting applications. We also assume you understand how applications are deployed and their distributed nature in a Kubernetes cluster. Within this broad audience, there are many different roles. Here is a nonexhaustive list of teams that help design and implement Kubernetes-based architectures that will find value in this book. Please note that the role names may be different in your orga‐ nization, so please look at the responsibilities for each to identify the corresponding role in your organization. We will use these names throughout the book to help you understand how a concept impacts each role. The Platform Team The platform engineering team is responsible for the design and implementation of the Kubernetes platform. Many enterprises choose to implement a container as a ser‐ vice platform (CaaS) strategy. This is a platform that is used across the enterprise to implement container-based workloads. The platform engineering team is responsible for the platform components and provides them as a service to application teams. This book helps you understand the importance of securing the platform and best practices to help secure the platform layer—that way you can provide application teams a way to onboard applications on a secure Kubernetes platform. It will also help you learn how to manage the security risk of new applications to the platform. The Networking Team The networking team is responsible for integrating Kubernetes clusters in an enter‐ prise network. We see these teams play different roles in an on-premise deployment of Kubernetes and in a cloud environment where Kubernetes clusters are self-hosted or leverage a managed Kubernetes service. You will understand the importance of network security and how to build networks with a strong security posture. Best practices for exposing applications outside the Kubernetes platform as well as net‐ work access for applications to external networks are examples of topics covered in this book. You will also learn how to collaborate with other teams to implement net‐ work security to protect elements external to Kubernetes from workloads inside Kubernetes. Preface | xiii

The Security Team The security team in enterprises is the most impacted by the movement toward cloud native applications. Cloud native applications are those designed for cloud environ‐ ments and are different from traditional applications. As an example, these applica‐ tions are distributed across the infrastructure in your network. This book will help you understand details about how to secure a Kubernetes platform that is used to host applications. It will provide you a complete view of how to secure mission-critical workloads. You will learn how to collaborate with various teams to effectively imple‐ ment security in the new and different world of Kubernetes. The Compliance Team The compliance team in an enterprise is responsible for ensuring operations and pro‐ cesses in an organization to meet the requirements of compliance standards adopted by an organization. You will understand how to implement various compliance requirements and how to monitor ongoing compliance in a Kubernetes-based plat‐ form. Note that we will not cover detailed compliance requirements and various standards, but we will provide you with strategies, examples, and tools to help you meet compliance requirements. The Operations Team The operations team is the team of developers/tools/operations engineers responsible for building and maintaining applications. They are also known as DevOps or site reliability engineers (SREs). They ensure that applications are onboarded and meet the required service level agreements (SLAs). In this book you will learn about your role in securing the Kubernetes cluster and collaboration with the security team. We will cover the concept of shift-left security, which says security needs to happen very early in the application development life cycle. Observability in a Kubernetes plat‐ form means the ability to infer details about the operation of your cluster by viewing data from the platform. This is the modern way of monitoring a distributed applica‐ tion, and you will learn how to implement observability and what its importance to security is. What You Will Learn In this book you will learn how to think about security as you implement your Kuber‐ netes strategy, from building applications to building infrastructure to hosting appli‐ cations to deploying applications to running applications. We will present security best practices for each of these with examples and tools to help you secure your Kubernetes platform. We will cover how to implement auditing, compliance, and other enterprise security controls like encryption. xiv | Preface

You will also learn best practices with tools and examples that show you how to implement observability and demonstrate its relevance to security and troubleshoot‐ ing. This enhanced visibility into your Kubernetes platform will drive actionable insights relevant to your unique situation. By the end of the book, you will be able to implement these best practices for security and observability for your Kubernetes clusters. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This element signifies a general note. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/tigera/k8s-security-observability-book. If you have a technical question or a problem using the code examples, please send email to bookquestions@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly Preface | xv

books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Kubernetes Security and Observability by Brendan Creane and Amit Gupta (O’Reilly). Copyright 2022 O’Reilly Media, 978-1-098-10711-6.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/KSO. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For news and information about our books and courses, visit http://oreilly.com. xvi | Preface

Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://youtube.com/oreillymedia Acknowledgments It was a great experience writing this book and it would not have been possible without the help, support and guidance of several people. Firstly, we want to thank the community, developers and maintainers of Project Calico, your innovation and contributions to Kubernetes and Kubernetes security and observability, have enabled us to write this book. The amazing engineering and the security research teams at Tigera have built products to address the complex challenges for security and observ‐ ability, and this enabled us to get a clear understanding of the challenges facing the users. This was very helpful as we wrote this book to guide users to a holistic security and observability solution. We also wanted to thank the reviewers who provided their opinions and subject mat‐ ter expertise. Their comments and guidance have greatly enriched the content of this book. Special mention to Manish Sampat, Alex Pollitt, Virginia Wilson, Seth Vargo, Tim Mackey, Ian Lewis, Puja Absassi, and Jose Ruiz—you are awesome! Finally, we want to thank everyone in the community that is contributing to Kuber‐ netes security and observability. It is amazing to see the innovation in this area, and we are thrilled to be involved with Kubernetes security and observability. Preface | xvii

CHAPTER 1 Security and Observability Strategy In this chapter, we will cover a high-level overview of how you can build a security and observability strategy for your Kubernetes implementation. Subsequent chapters will cover each of these concepts in more detail. You need to think about a security strategy when you are in the pilot/pre-production phase of your Kubernetes journey, so if you are part of the security team, this chapter is very important. If you are part of the network, platform, or application team, this chapter shows how you can be a part of the security strategy and discuss the importance of collaboration between the security, platform, and application teams. We will cover the following concepts that will guide you with your security and observability strategy: • How securing Kubernetes is different from traditional security methods • The life cycle of deploying applications (workloads) in a Kubernetes cluster and best practices for each stage • How you should implement observability to help with security • Well-known security frameworks and how to use them in your security strategy Security for Kubernetes: A New and Different World In this section we’ll highlight how Kubernetes is different and why traditional security methods do not work in a Kubernetes implementation. As workloads move to the cloud, Kubernetes is the most common orchestrator for managing them. The reason Kubernetes is popular is its declarative nature: It abstracts infrastructure details and allows users to specify the workloads they want to run and the desired outcomes. The application team does not need to worry about 1