Kafka for Architects Event-driven architecture, logs, microservices, real-time event processing (Katya Gorshkova) (z-library.sk, 1lib.sk, z-lib.sk)

M A N N I N G Katya Gorshkova Foreword by Viktor Gamov Event-driven architecture, logs, microservices, real-time event processing

Storing and Processing Data in Kafka Data is organized into topics, which are further divided into partitions. The basic units of information are messages, which contain keys, values, and headers. Transactions Topics, where messages are sent A consumer, which reads messages Producers, sending messages to topic Structure of a topic: a topic is broken down into partitions for parallel processing. Structure of a message: a message consists of a key, value, headers, and a timestamp. Broker 1 Broker 2 Broker 3 A Kafka cluster, which consists of several brokers key value headers timestamp Profiles Partition 0 Partition1 The “C” arrow shows data flowing from Kafka to the consumer (pull model). P P C C The “P” arrow shows data being sent from the producer to Kafka. TransactionService ProfileService Customer360Service

MANN I NG Shelter ISland Katya Gorshkova Kafka for Architects Event-driven architecture, logs, microservices, real-time event processing Foreword by Viktor Gamov

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com © 2026 Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid- free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. ∞ Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 ISBN 9781633436411 Printed in the United States of America The author and publisher have made every effort to ensure that the information in this book was correct at press time. The author and publisher do not assume and hereby disclaim any liability to any party for any loss, damage, or disruption caused by errors or omissions, whether such errors or omissions result from negligence, accident, or any other cause, or from any usage of the information herein. Development editor: Frances Lefkowitz Technical editor: Simeon Leyzerzon Review editor: Radmila Ercegovac Production editor: Kathy Rossland Copy editor: Andy Carroll Proofreader: Olga Milanko Typesetter: Tamara ŠveliÊ SabljiÊ Cover designer: Marija Tudor

iii contents foreword x preface xi acknowledgments xiii about this book xv about the author xvii about the cover illustration xviii Part 1 Exploring Kafka building blocks ..............1 1 Getting to know Kafka as an architect 3 1.1 How an architect sees Kafka 4 Event-driven architecture 5  ■  Handling myriads of data 6 1.2 Field notes: Journey of an event-driven project 7 1.3 Key players in the Kafka ecosystem 8 Brokers and clients 8  ■  Controllers: Managing cluster metadata 9 1.4 Applying Kafka’s architectural principles 10 The publish-subscribe pattern 11  ■  Reliable delivery 12 The commit log 13 1.5 Designing and managing data flows 13 Schema Registry: Handling data contracts 14  ■  Kafka Connect: Data replication without code 15  ■  Streaming frameworks: Processing data in real time 16

iv contents 1.6 Addressing operations and infrastructure 18 Kafka tuning and maintenance 18  ■  On-premises and cloud options 19  ■  Solutions from other cloud providers 19 1.7 Applying Kafka in enterprise 19 Using Kafka for sending messages 20  ■  Using Kafka for storing data 20  ■  How Kafka is different 20 1.8 Online resources 22 2 Kafka cluster data architecture 24 2.1 Inside the Kafka cluster 25 2.2 Core concepts of data processing 28 Partitioning the topic 29  ■  Processing data concurrently 30 Ordering within a topic 31  ■  AsyncAPI: Capturing the architecture of topics, partitions, and more 32 2.3 Replicating partitions 34 Replica leaders and followers 34  ■  Choosing replication factor and minimal number of in-sync replicas 36  ■  Extending topic configuration with replication information 37 2.4 Inside the topic 38 Messages: Keys, values and headers 39  ■  First draft for documenting messages in AsyncAPI 41  ■  Message batches and offsets 42  ■  Physical representation of a topic 43  ■  Data retention 44  ■  Selecting the number of partitions 45 Configuring topic metadata 46 2.5 Compacted topics 47 The idea of compaction 48  ■  How compaction works 49 When compaction happens 50  ■  Making decisions about the compaction policy 51 2.6 Online resources 52 3 Kafka clients and message production 54 3.1 Communicating with Kafka 55 How producers send messages to brokers 56  ■  Configuring clients 58  ■  Connecting to Kafka 59  ■  Serializing and deserializing data 62  ■  Setting quotas 63  ■  Field notes: Setting up the Customer 360 operational data store 63

vcontents 3.2 Sending a message 65 Partitioning strategy 67  ■  Field notes: Partitioning strategy for the Customer 360 ODS 69  ■  Acknowledgment strategies 69 Field notes: Implementing an acknowledgment strategy 71 Batches and timeouts 71  ■  Common producer challenges 73 3.3 Online resources 75 4 Creating consumer applications 77 4.1 Organizing consumer applications 78 4.2 Receiving a message 80 Reading data in parallel 81  ■  Setting initial consumer configuration for the Customer 360 project 83  ■  Group leader and group coordinator 87  ■  Committing the offsets 89 Specifying the strategy for committing offsets for an ODS 90 Creating batches 90  ■  Timeouts and partition rebalance 91 Static group membership 94  ■  Partition assignment strategies 94 The next-gen consumer rebalance protocol 97  ■  Subscriptions and assignment 100  ■  Reading data from compacted topics 100 Consumer considerations for the ODS project 101 4.3 Common consumer problems 103 Consumer scalability challenges 103  ■  Optimizing batch size configuration 106  ■  Timeout management strategies 106 Error-proof deserialization processes 106  ■  Offset initialization strategies for new consumers 107  ■  Accurate offset commitment practices 108  ■  Coordinating transactions across external systems 108 4.4 Data compression 108 4.5 Accessing Kafka through the Confluent REST Proxy 110 4.6 Online resources 111 Part 2 Solving problems with Kafka ................. 113 5 Kafka in real-world use cases 115 5.1 Navigating real-world implementation 116 Event-driven microservices 117  ■  Data integration 121 Collecting logs 126  ■  Real-time data processing 130 5.2 Differences from other messaging platforms 133 Publish-subscribe model 133  ■  Partitioned data 133 Lack of broker-side logic 134  ■  Sequential data access 134

vi contents Message persistence 134  ■  Limitations in handling large messages 135  ■  Scalability and high throughput 136 Fault tolerance 136  ■  Batch processing 137  ■  Lack of global ordering 138 5.3 Kafka alternatives 139 RabbitMQ 139  ■  Apache Pulsar 141  ■  Solutions from cloud providers 143 5.4 Online resources 144 6 Defining data contracts 146 6.1 How Kafka handles event structure 148 6.2 Designing events 149 Challenges in event design 149  ■  Fact and delta events: Representing state changes 152  ■  Composite, atomic, and aggregate events: Representing event structure 155 Pulling state on notification 156  ■  Evolution of types 157 Mapping events to Kafka messages 161  ■  Data strategies for the Customer 360 ODS 162 6.3 Event governance 162 Data formats 163  ■  Selecting a data format for the Customer 360 ODS 167  ■  Data ownership 167  ■  Organizing data and communicating changes 167  ■  Designing events for the Customer 360 ODS 168 6.4 Schema Registry 169 Schema Registry in Kafka ecosystem 169  ■  Registering schemas 172  ■  A concept of subject 172  ■  Compatibility rules 175  ■  Alternatives to Schema Registry 178 Handling data contracts without the centralized server 178 Commercial extensions for data contracts 179 6.5 Common problems in handling data contracts 180 Absence of server-side validation 180  ■  Handling incompatible changes for non-compacted topics 181  ■  Migrating state 181 Automatic registration of schemas 182 6.6 Online resources 183 7 Kafka interaction patterns 185 7.1 Using Kafka in microservices 187 Smart endpoints and dumb pipes 187  ■  Request-response pattern 190  ■  CQRS pattern 195  ■  Event sourcing with snapshotting 197  ■  Having “hot” and “cold” data 198

viicontents 7.2 Decentralizing analytical data with a data mesh 200 Domain ownership 201  ■  Data as a product 201  ■  Federated governance 202  ■  Self-serve platform 202 7.3 Using Kafka Connect 203 Kafka Connect at a glance 204  ■  Internal Kafka Connect architecture 206  ■  Converters 207  ■  Single message transformations 208  ■  Source connectors 209 Sink connectors 211  ■  Changes in the incoming data structure 212  ■  Integrating Kafka and databases 213 Creating a connector for the Customer 360 ODS 214 Common Kafka Connect problems 215 7.4 Ensuring delivery guarantees 217 Producer idempotence 217  ■  Understanding Kafka transactions 218  ■  Transactional outbox pattern 222 7.5 Online resources 223 8 Designing streaming applications 226 8.1 Introducing Kafka Streams 227 ETL, ELT, and stream processing 229  ■  The Kafka Streams framework 232  ■  Benefits of using Kafka Streams 233 8.2 Sketching out the ODS with Kafka Streams 234 8.3 Processing data 235 Stateless operations 236  ■  Stateful operations 236 The Processor API 239  ■  Kafka Streams internal architecture 240  ■  Windowing operations 242 Joining streams 243  ■  Implementing CustomerJoinService in the example ODS 246  ■  Interactive queries 246 8.4 Alternative solutions 248 Confluent ksqlDB 248  ■  Apache Flink 253  ■  Solutions from cloud providers 260 8.5 Common streaming application challenges 260 Memory and disk capacity planning 261  ■  Incorrect topic partitioning 261  ■  Out-of-order data 262  ■  Late-arriving data 262  ■  State store initialization 262  ■  Monitoring and debugging challenges 263 8.6 Online resources 263

viii contents Part 3 Delivering projects with Kafka ............ 265 9 Managing Kafka within the enterprise 267 9.1 Managing metadata 268 Introducing KRaft controllers 270  ■  Example of cluster configuration 276  ■  Failover scenarios 277  ■  Using ZooKeeper 279 9.2 Choosing a deployment solution 282 Choosing between on-premises and cloud Kafka deployment 282 Hybrid approach 285  ■  Choosing the right deployment for the Customer 360 ODS 288 9.3 Creating a security solution 289 Kafka security overview 289  ■  Encrypting using TLS 289 Authentication 292  ■  Authorization 295  ■  Protecting data at rest 296  ■  Enabling security in the Customer 360 ODS 298 9.4 Online resources 299 10 Organizing a Kafka project 301 10.1 Defining Kafka project requirements 302 Identifying event-driven workflows 302  ■  Turning business workflows into events 303  ■  Gathering functional requirements for Kafka topics 305  ■  Identifying nonfunctional requirements 308 10.2 Maintaining cluster structure 311 Using CLI and UI tools 311  ■  Using GitOps for Kafka configurations 312  ■  Using the Kafka Admin API 316 Setting up environments 317  ■  Choosing a solution for the Customer 360 ODS 318 10.3 Testing Kafka applications 319 Unit testing 319  ■  Integration testing 322 Performance tests 327 10.4 Online resources 328 11 Operating Kafka 330 11.1 Cluster evolution and upgrades 331 Adding brokers and distributing the load 332  ■  Removing a broker from the cluster 333  ■  Upgrading clients 334 Data mobility 334

ixcontents 11.2 Monitoring a Kafka cluster 335 Types of metrics in monitoring 335  ■  Kafka monitoring objects 336  ■  Ownership of monitoring responsibilities 338 Monitoring stacks and tools 339 11.3 Performance tuning clinic 340 Balancing throughput and latency 340  ■  Balancing data safety and uptime 343 11.4 Disaster recovery and failover 345 RTO/RPO engineering 345 11.5 Online resources 348 12 What’s next for Kafka 350 12.1 Kafka as an orchestration platform 351 12.2 Integration with new runtimes 353 Kafka with WebAssembly 353  ■  Serverless Kafka 354 Kafka at the edge 356 12.3 Diskless Kafka: Decoupling storage from brokers 357 12.4 Kafka in AI/ML world 358 Incremental learning 359  ■  Feature engineering in motion 361 Kafka and AI agents 362 index 365

x foreword Even skilled software architects and development teams face challenges, such as scal- ing, updating data, and maintaining system stability. Kafka can help solve these com- plex problems, but it can feel like a powerful tool without enough instructions, so empathy and practical advice are essential. This is where Katya Gorshkova’s book Kafka for Architects stands out. With twenty years of experience as a software engineer and architect, and a gift for teaching even during tough times, Katya makes Kafka easy to understand and learn. She focuses on what architects care about most: big ideas, hard choices, and useful advice. The book takes you from the basics of EDA in chapter 1 through to real-world enterprise challenges, helping you feel confident using Kafka. Katya is open about what Kafka is really like, talking about both its strengths and its delays and challenges, so readers feel understood. Chapters on data and practical patterns help teams avoid expensive mistakes. Other chapters give clear advice on gov- ernance and scaling, and her look at AI-powered streams helps readers feel confident about Kafka’s future. Many Kafka books focus on the step-by-step technical details. This book is different because it helps you see the big picture and make smart architectural choices. It stands out by guiding leaders to support EDA with a realistic approach, using real stories and showing both the pros and cons. After reading this book, you’ll feel ready to build not just systems, but ecosystems that can grow and last. Event-driven design shapes modern architecture. With Katya’s guidance, Kafka becomes more than just a tool. It’s a strategic foundation for the future! —Viktor Gamov, Principal Developer Advocate, Confluent, coauthor of Kafka in Action

xi preface One day, my team leader came to me with a simple question: “Hey, there’s a new mes- saging system out there. Can you check if it could be useful for us?” That was how my journey with Apache Kafka began. At first glance, Kafka’s architecture felt clean and elegant. But implementing that first project was anything but seamless. It was too distributed, too low-level, and it lacked the tooling we take for granted today. I quickly learned that while Kafka had immense potential, it also demanded a deeper understanding than most systems I had worked with before. The next chapter in my relationship with Kafka began when I was asked to create a course explaining its concepts. Teaching forced me to find clear ways to communicate the ideas behind event-driven architecture—not just how Kafka works, but how to think about it. And I discovered that most people weren’t as interested in the implementation details as they were in the bigger question: How can we incorporate Kafka into our project? The Kafka community has done a great job of building documentation, but most tutorials stop at “how to get Kafka running.” Very few address the harder questions: Should we even use Kafka for this project? How can we fit it into our existing architecture? What patterns will help us design Kafka systems that stand the test of time? When I started, answers to these questions were scattered, incomplete, or hard-won through trial and error. I kept encountering the same challenges over and over again—how to model events effectively, how to evolve schemas safely, how to balance throughput, ordering guaran- tees, and fault tolerance without overengineering, and, when something went wrong, how to trace responsibility and find the root cause. Kafka for Architects grew out of those experiences. Its goal is to explore the questions that are often left untouched—the ones that sit between “Kafka is installed” and “Kafka

xii preface is delivering real business value.” It’s about design decisions, trade-offs, and the mental models that guide successful implementations. If you’re holding this book, you’ve probably already decided that Kafka could play a role in your systems. My hope is that the following pages will help you not only make it work, but make it work well—for your architecture, your team, and the years ahead.

xiii acknowledgments I never intended to write a book—it happened all of a sudden. I set out on this journey without any idea of what lay ahead. It turned out to be a much longer road than it first appeared, and I am deeply grateful to my family and colleagues who supported me throughout. I would like to thank all the contributors to the Apache Kafka project for creating such powerful tools and making them open source. Kafka for Architects wouldn’t exist without your work. Many thanks to the participants of my training sessions. You often stopped me from explaining what could be easily found in the documentation and instead asked how things really work. Your questions shaped this book—it is, in many ways, a collection of answers to your curiosity. You all deserve co-author credit. I am deeply grateful to my family for their patience and love during this journey. I especially thank Stanislav for his continuous encouragement, Anna for helping me with the graphics, and my dog Artie for calmly waiting until I finished writing before taking him for a walk. I am also thankful to my dear friends and colleagues for our endless discussions. I especially want to thank Peter Vašek, my earliest reviewer, for tirelessly reading and commenting on the manuscript again and again (and again). My special thanks go to Daniel Buchta, whose help in transforming my documentation-style drafts into read- able content made this book accessible to a much wider audience. I want to express my deep gratitude to Professor Boris Novikov from the University of Saint Petersburg, who believed in me from the very beginning. His early support and guidance played a significant role in shaping me into the professional I am today.

xiv acknowledgments Thank you to my editors: development editor Frances Lefkowitz, for persistently challenging me to explain complex ideas clearly, and technical editor Simeon Leyz- erzon, whose contributions to the content were invaluable. Simeon is a seasoned IT professional with over three decades of experience in software development and archi- tecture. He excels in seamlessly integrating Kafka and other middleware solutions to develop high-performing, scalable enterprise applications, addressing complex techni- cal challenges with creativity and precision. Finally, to all the reviewers—Amarjit Bhandal, Afzal Mazhar, Alessandro Campeis, Bassam Ismail, Christoph Kappel, Danica Fine, Dinesh Reddy Chittibala, Erin Colvin, Francisco Lopez-Sancho Abraham, George Haines, Giuseppe Catalano, Israel Ekpo, James Black, Jeff Patterson, Karol Skorek, Krishna Kumaar, Mandar Kulkarni, Michael Williams, Mohammad Shahnawaz Akhter, Monika Rathor, Nathan B. Crocker, Nuwan Dias, Oscar Caraballo, Paul Grebenc, Pradeep Kumar Goudagunta, Rajiv Moghe, Ralph M. Debusmann, Richard Jepps, Sachin Handiekar, Sandhya Vinjam, Shay Elkin, Srinath Chandramohan, Sunil Murali, Swapneelkumar Deshpande, and Vlad Bezden—your suggestions helped make this a better book.

xv about this book An architect’s role involves choosing how systems communicate, and one proven method for coordinating distributed work is through events. Several technologies exist, each with trade-offs in reliability, flexibility, and integration complexity. Apache Kafka is a widely adopted, open source event-streaming platform that started as a mes- saging system and has grown into an ecosystem for real-time processing, backed by durable event storage. But while Kafka is powerful, its adoption isn’t straightforward. Most tutorials and books focus on code and configuration, neglecting architectural and design questions crucial for success. This book fills that gap by focusing on archi- tectural choices—fit, event design, patterns, and governance—ensuring deliberate Kafka adoption. Who should read this book Kafka for Architects is for software architects, technical leads, and developers who need to understand Kafka at a system level—how it operates and how to fit it into broader architectures. You do not need to know any specific programming language or technol- ogy stack to benefit from this book, but a basic grasp of distributed systems concepts, general architecture principles, and some experience delivering enterprise projects will be helpful. My goal is to equip you with the insight to evaluate Kafka’s role in your architecture, make informed trade-offs, and design solutions that last, regardless of the tools or lan- guages you use.

xvi about this book How this book is organized: A road map This book begins by introducing the main concepts and ideas that underpin Apache Kafka, and it’s divided into three parts. Part 1 (chapters 1–4) provides a foundational understanding: the key architectural principles, the organization of a Kafka cluster, and a detailed exploration of how pro- ducing and consuming messages work. These chapters also cover how Kafka fits into the broader ecosystem, giving you the context needed to reason about it as part of a larger system. With this groundwork in place, part 2 (chapters 5–8) moves into applied architec- ture. Here, we’ll explore real-world use cases, strategies for defining and managing data contracts, and architectural patterns (and anti-patterns) for integrating Kafka into diverse environments. After exploring how Kafka is applied across various architectures, chapter 8 shifts focus to one of its most impactful uses—processing data in real time. This chapter dis- cusses data transformation with Kafka Streams and other technologies designed to han- dle information in motion. Part 3 (chapters 9–12) starts with the organizational and operational aspects of run- ning Kafka and concludes with a look at emerging trends and the future of the plat- form. Chapter 9 discusses how to integrate Kafka into the enterprise infrastructure, chapter 10 covers organizing and managing Kafka projects, and chapter 11 dives into day-to-day operations, maintenance, and monitoring. Chapter 12 concludes the book by exploring emerging trends, upcoming features, and possible directions for Kafka’s future. liveBook discussion forum Purchase of Kafka for Architects includes free access to liveBook, Manning’s online read- ing platform. Using liveBook’s exclusive discussion features, you can attach comments to the book globally or to specific sections or paragraphs. It’s a snap to make notes for yourself, ask and answer technical questions, and receive help from the author and other users. To access the forum, go to https://livebook.manning.com/book/ kafka-for-architects/discussion. Manning’s commitment to our readers is to provide a venue where a meaningful dia- logue between individual readers and between readers and the author can take place. It is not a commitment to any specific amount of participation on the part of the author, whose contribution to the forum remains voluntary (and unpaid). We suggest you try asking the author some challenging questions lest her interest stray! The forum and the archives of previous discussions will be accessible from the publisher’s website as long as the book is in print.

xvii about the author Ekaterina Gorshkova graduated from the University of Saint Petersburg with a degree in Applied Mathemat- ics but shifted her focus to software engineering early in her career. Starting as a Junior Software Engineer, she specialized in the Java stack and data-intensive applica- tions. She later pursued a PhD in Computer Science and moved to the Czech Republic. Katya has extensive experience in the FinTech sec- tor, serving as both a Software Engineer and a Solution Architect. About seven years ago, she developed a proof- of-concept project using Apache Kafka, sparking a deep passion for the technology. This led her to become a Kafka engineer, providing consultancy services for messaging systems architecture and developing applications for real-time data processing.

xviii about the cover illustration The figure on the cover of Kafka for Architects is “Il Galantariaro,” or “a petty trader,” taken from Usi e Costumi di Napoli e Contorni Descritti e Dipinti by Francesco de Bourcard, published in 1853. Each illustration is finely drawn and colored by hand. In those days, it was easy to identify where people lived and what their trade or station in life was just by their dress. Manning celebrates the inventiveness and initiative of the computer business with book covers based on the rich diversity of regional culture cen- turies ago, brought back to life by pictures from collections such as this one.