Observability For Legacy Systems Methods and Solutions with OpenTelemetry and AIOps (Hyen Seuk Jeong) (z-library.sk, 1lib.sk, z-lib.sk)

Page 1

Observability For Legacy Systems Methods and Solutions with OpenTelemetry and AIOps — Hyen Seuk Jeong

Page 2

Observability For Legacy Systems Methods and Solutions with OpenTelemetry and AIOps Hyen Seuk Jeong

Page 3

Observability For Legacy Systems: Methods and Solutions with OpenTelemetry and AIOps ISBN-13 (pbk): 979-8-8688-1687-1 ISBN-13 (electronic): 979-8-8688-1688-8 https://doi.org/10.1007/979-8-8688-1688-8 Copyright © 2025 by Hyen Seuk Jeong This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Managing Director, Apress Media LLC: Welmoed Spahr Acquisitions Editor: Anandadeep Roy Editorial Project Manager: Jessica Vakili Copy Editor: Kezia Endsley Distributed to the book trade worldwide by Springer Science+Business Media New York, 1 New York Plaza, New York, NY 10004. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com, or visit www.springeronline.com. Apress Media, LLC is a Delaware LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please e-mail booktranslations@springernature.com; for reprint, paperback, or audio rights, please e-mail bookpermissions@springernature.com. Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales web page at http://www.apress.com/bulk-sales. Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub. For more detailed information, please visit https://www.apress.com/gp/services/ source-code. If disposing of this product, please recycle the paper Hyen Seuk Jeong Goyang-si, Gyeonggi-do, Republic of Korea

Page 4

To my most cherished and respected family

Page 5

v About the Author ������������������������������������������������������������������������������������������������������������������������������xiii Introduction ��������������������������������������������������������������������������������������������������������������������������������������� xv Chapter 1: RCA Essentials �������������������������������������������������������������������������������������������������������������������1 1.1. Introduction .............................................................................................................................................................. 1 1.2. Root Cause Analysis (RCA) Process ........................................................................................................................ 19 1.2.1. Identify the Problem Area ............................................................................................................................. 20 1.2.2. Analyze Individual Requests ......................................................................................................................... 29 1.2.3. Understand Low-Level Methods ................................................................................................................... 36 1.3. Observability Signals .............................................................................................................................................. 46 1.3.1. Logs .............................................................................................................................................................. 47 1.3.2. RUM (Real User Monitoring) ......................................................................................................................... 55 1.3.3. Profiles ......................................................................................................................................................... 69 1.3.4. Debugging .................................................................................................................................................... 77 1.3.5. Events ........................................................................................................................................................... 78 1.3.6. Root Cause Analysis Data Model .................................................................................................................. 87 1.4. Summary ................................................................................................................................................................ 92 Chapter 2: The RCA Approach ������������������������������������������������������������������������������������������������������������95 2.1. Introduction ............................................................................................................................................................ 95 2.2. Using Correlations .................................................................................................................................................. 96 2.2.1. From Distributed to System Traces .............................................................................................................. 96 2.2.2. From Events to Traces .................................................................................................................................. 98 2.2.3. From Traces to Logs ..................................................................................................................................... 99 2.2.4. From Traces to Profiles ............................................................................................................................... 100 2.2.5. From Logs to Profiles ................................................................................................................................. 101 2.2.6. From Metrics to Traces ............................................................................................................................... 102 2.2.7. From RUMs to Traces .................................................................................................................................. 102 2.2.8. From Events to Anomalies .......................................................................................................................... 103 2.2.9. From Logs to Traces ................................................................................................................................... 104 Table of Contents

Page 6

vi 2.2.10. From Networks to Traces .......................................................................................................................... 105 2.2.11. From Traces to Metrics ............................................................................................................................. 106 2.2.12. From Traces to Service Maps ................................................................................................................... 106 2.2.13. From Service Maps to Metrics ................................................................................................................. 107 2.2.14. From Metrics to Logs ................................................................................................................................ 107 2.3. Grafana Correlations ............................................................................................................................................ 108 2.3.1. From Metrics to Traces ............................................................................................................................... 108 2.3.2. From Traces to Metrics ............................................................................................................................... 110 2.3.3. From Logs to Traces ................................................................................................................................... 114 2.3.4. From Traces to Logs ................................................................................................................................... 115 2.4. Using a New Stack ............................................................................................................................................... 116 2.4.1. Introducing The New Stack (TNS) ............................................................................................................... 117 2.4.2. Configure TNS ............................................................................................................................................. 120 2.4.3. TNS Correlation ........................................................................................................................................... 122 2.5. Grafana Observability ........................................................................................................................................... 131 2.5.1. Code Description ........................................................................................................................................ 134 2.5.2. System Configuration ................................................................................................................................. 136 2.5.3. Setting Up Grafana ..................................................................................................................................... 139 2.6. Running the o11y Shop Demo .............................................................................................................................. 141 2.6.1. Demo Architecture ...................................................................................................................................... 141 2.6.2. Demo Process............................................................................................................................................. 142 2.6.3. Code Description ........................................................................................................................................ 143 2.6.4. Install the Demo ......................................................................................................................................... 146 2.6.5. Checking Results in OpenSearch................................................................................................................ 148 2.6.6. Summary .................................................................................................................................................... 152 2.7. Dashboard Visualizations ..................................................................................................................................... 152 2.7.1. Polystats ..................................................................................................................................................... 153 2.7.2. Service maps .............................................................................................................................................. 155 2.7.3. Histograms ................................................................................................................................................. 156 2.7.4. Heatmaps ................................................................................................................................................... 156 2.7.5. Time Series ................................................................................................................................................. 157 2.7.6. Flame graphs .............................................................................................................................................. 158 2.7.7. State Timelines ........................................................................................................................................... 158 2.7.8. Stats ........................................................................................................................................................... 160 Table of ConTenTs

Page 7

vii 2.7.9. Traces ......................................................................................................................................................... 160 2.7.10. Annotations ............................................................................................................................................... 160 2.7.11. Dashboard Development .......................................................................................................................... 161 2.8. Summary .............................................................................................................................................................. 169 Chapter 3: Trace-Centric RCA ����������������������������������������������������������������������������������������������������������171 3.1. Introduction .......................................................................................................................................................... 171 3.2. How Trace Works .................................................................................................................................................. 173 3.2.1. Context ....................................................................................................................................................... 180 3.2.2. Propagators ................................................................................................................................................ 182 3.2.3. Propagating Trace ....................................................................................................................................... 187 3.2.4. Propagating Baggage ................................................................................................................................. 192 3.2.5. OpenTracing Shim ...................................................................................................................................... 194 3.3. Propagating Managed Services ........................................................................................................................... 196 3.3.1. AWS CloudFront Demo ................................................................................................................................ 199 3.3.2. GCP PubSub Demo ..................................................................................................................................... 202 3.3.3. Azure SQS Demo ......................................................................................................................................... 211 3.4. Propagating Message Services ............................................................................................................................ 217 3.4.1. Solace JMS Demo ...................................................................................................................................... 219 3.4.2. TIBCO JMS Demo ........................................................................................................................................ 223 3.4.3. MQTT Demo ................................................................................................................................................ 225 3.4.4. Kafka Demo ................................................................................................................................................ 226 3.4.5. Spring Cloud Stream Demo ........................................................................................................................ 229 3.5. Propagating an EAI Server ................................................................................................................................... 233 3.6. Propagation a Black Box System ......................................................................................................................... 237 3.7. Propagating Server Frameworks ......................................................................................................................... 239 3.7.1. Commercial Observability SDK Trace Demo ............................................................................................... 241 3.7.2. Micrometer Trace Demo ............................................................................................................................. 243 3.7.3. Commercial Observability Demo ................................................................................................................ 245 3.7.4. WebSocket Demo ....................................................................................................................................... 247 3.8. Using OpenTelemetry Extensions ......................................................................................................................... 252 3.8.1. Bytecode Instrumentation Demo ................................................................................................................ 254 3.8.2. Extension Demo .......................................................................................................................................... 257 3.8.3. Debugging Extensions ................................................................................................................................ 263 Table of ConTenTs

Page 8

viii 3.9. Propagating Commercial Observability ................................................................................................................ 264 3.9.1. Automating Commercial Observability ....................................................................................................... 266 3.9.2. Agent Chaos................................................................................................................................................ 272 3.10. Summary ............................................................................................................................................................ 277 Chapter 4: Observability Practices by the Industry �������������������������������������������������������������������������279 4.1. Introduction .......................................................................................................................................................... 279 4.2. Observability in Banking ...................................................................................................................................... 284 4.2.1. The Bank Process ....................................................................................................................................... 284 4.2.2. Bank Legacy Systems ................................................................................................................................ 290 4.2.3. Banking Demo Overview ............................................................................................................................ 316 4.2.4. From RUM to API Server ............................................................................................................................. 323 4.2.5. From API Server to Kafka ........................................................................................................................... 324 4.2.6. From API Server to Microservices .............................................................................................................. 325 4.2.7. From API Server to EAI Server .................................................................................................................... 328 4.2.8. From EAI Server to Legacy System ............................................................................................................ 328 4.2.9. Transfer to Jaeger....................................................................................................................................... 330 4.3. Observability in Telecom ...................................................................................................................................... 334 4.3.1. Telecom Architecture .................................................................................................................................. 341 4.3.2. Order Orchestration .................................................................................................................................... 352 4.3.3. Network Provisioning ................................................................................................................................. 359 4.4. Observability in Online Games ............................................................................................................................. 364 4.4.1. Game Operation .......................................................................................................................................... 365 4.4.2. Game Observability .................................................................................................................................... 370 4.5. Trading Observability ............................................................................................................................................ 374 4.5.1. Designing Ultra-Low Latency...................................................................................................................... 375 4.5.2. Ultra-Low Latency Reference ..................................................................................................................... 389 4.6. Summary .............................................................................................................................................................. 392 Chapter 5: OpenTelemetry Demo �����������������������������������������������������������������������������������������������������393 5.1. Introduction .......................................................................................................................................................... 393 5.2. Overview .............................................................................................................................................................. 394 5.3. Observability Demo .............................................................................................................................................. 398 5.3.1. Profile Support ............................................................................................................................................ 399 5.3.2. OpenTelemetry Agent ................................................................................................................................. 405 5.3.3. SLOs ........................................................................................................................................................... 405 Table of ConTenTs

Page 9

ix 5.3.4. OpenTelemetry Collector ............................................................................................................................ 414 5.3.5. RUM ............................................................................................................................................................ 415 5.3.6. Automated Instrumentation ........................................................................................................................ 418 5.3.7. Commercial Observability Demo ................................................................................................................ 419 5.3.8. Live Debugging ........................................................................................................................................... 421 5.3.9. Baggage Context ........................................................................................................................................ 423 5.3.10. Span Attributes ......................................................................................................................................... 424 5.3.11. Span Annotations...................................................................................................................................... 429 5.3.12. Promscale Kubernetes ............................................................................................................................. 432 5.3.13. Promscale SQL ......................................................................................................................................... 439 5.4. Summary .............................................................................................................................................................. 441 Chapter 6: Infrastructure RCA ���������������������������������������������������������������������������������������������������������443 6.1. Introduction .......................................................................................................................................................... 443 6.2. System Traces ...................................................................................................................................................... 448 6.2.1. KUtrace ....................................................................................................................................................... 452 6.2.2. ftrace .......................................................................................................................................................... 471 6.2.3. Kubeshark .................................................................................................................................................. 476 6.2.4. System Utilities ........................................................................................................................................... 476 6.2.5. How the Kernel Works ................................................................................................................................ 479 6.2.6. Kernel Development ................................................................................................................................... 499 6.3. eBPF ..................................................................................................................................................................... 504 6.3.1. BCC and bpftrace ........................................................................................................................................ 505 6.3.2. PCP ............................................................................................................................................................. 521 6.4. Chaos Engineering ............................................................................................................................................... 528 6.4.1. Overview..................................................................................................................................................... 528 6.4.2. Demo .......................................................................................................................................................... 532 6.5. Network Observability .......................................................................................................................................... 541 6.5.1. Introduction ................................................................................................................................................ 543 6.5.2. Metrics ....................................................................................................................................................... 543 6.5.3. Hubble ........................................................................................................................................................ 545 6.5.4. The Grafana Plugin ..................................................................................................................................... 548 6.6. The Cilium L7 Network ......................................................................................................................................... 549 6.6.1. Introduction ................................................................................................................................................ 549 6.6.2. Cilium Architecture ..................................................................................................................................... 551 6.6.3. Demo .......................................................................................................................................................... 552 Table of ConTenTs

Page 10

x 6.7. Cilium Add-ons ..................................................................................................................................................... 560 6.7.1. Network Policies ......................................................................................................................................... 561 6.7.2. Multi-Cluster ............................................................................................................................................... 566 6.7.3. Service Meshes .......................................................................................................................................... 567 6.7.4. Ingress ........................................................................................................................................................ 568 6.8. Summary .............................................................................................................................................................. 569 Chapter 7: Anomaly Detection ���������������������������������������������������������������������������������������������������������571 7.1. SQL Anomaly Detection ........................................................................................................................................ 574 7.1.1. Anomaly Detection Method......................................................................................................................... 575 7.1.2. Anomaly Types ............................................................................................................................................ 577 7.2. Machine Learning Anomaly Detection .................................................................................................................. 578 7.2.1. Example 1: Internet of Things ..................................................................................................................... 581 7.2.2. Example 2: Security .................................................................................................................................... 581 7.2.3. Example 3: IT Operations ............................................................................................................................ 581 7.2.4. Ratio Analysis ............................................................................................................................................. 585 7.2.5. Category Analysis ....................................................................................................................................... 587 7.2.6. Analyzing Populations ................................................................................................................................ 588 7.2.7. Pattern Analytics ........................................................................................................................................ 590 7.2.8. Bucket Analysis .......................................................................................................................................... 591 7.3. Analyzing the Results ........................................................................................................................................... 592 7.3.1. How to Analyze Your Results ...................................................................................................................... 593 7.3.2. Results Analytics API .................................................................................................................................. 594 7.4. Configuring Anomaly Detection ............................................................................................................................ 599 7.4.1. Configuring OpenSearch Anomaly Detection .............................................................................................. 599 7.4.2. Detector Considerations ............................................................................................................................. 603 7.5. Summary .............................................................................................................................................................. 604 Chapter 8: Analyze RCA Data �����������������������������������������������������������������������������������������������������������607 8.1. Analyzing SQL Data .............................................................................................................................................. 611 8.1.1. Time Windows ............................................................................................................................................ 612 8.1.2. Cumulative Values ...................................................................................................................................... 613 8.1.3. Seasonality Analysis ................................................................................................................................... 613 8.2. Using Promscale .................................................................................................................................................. 614 8.2.1. Promscale Features .................................................................................................................................... 616 8.2.2. The Promscale Method ............................................................................................................................... 617 Table of ConTenTs

Page 11

xi 8.3. Promscale Demo .................................................................................................................................................. 618 8.3.1. Metric SQL .................................................................................................................................................. 619 8.3.2. Metric SQL Demo ........................................................................................................................................ 620 8.3.3. Trace SQL .................................................................................................................................................... 625 8.3.4. Trace SQL Demo ......................................................................................................................................... 626 8.4. Summary .............................................................................................................................................................. 633 Chapter 9: Aggregate RCA Data �������������������������������������������������������������������������������������������������������635 9.1. The Presto Case Study ......................................................................................................................................... 635 9.1.1. Analytics Frameworks ................................................................................................................................ 638 9.1.2. OpenTelemetry Data Model ........................................................................................................................ 639 9.1.3. Presto’s Features ........................................................................................................................................ 643 9.1.4. Presto’s Configuration ................................................................................................................................ 646 9.2. The Apache Druid Case Study .............................................................................................................................. 650 9.2.1. Schema-Less Searches .............................................................................................................................. 652 9.2.2. Index Methods ............................................................................................................................................ 652 9.2.3. Rollups ........................................................................................................................................................ 652 9.2.4. Druid Components ...................................................................................................................................... 652 9.2.5. Performance Improvements ....................................................................................................................... 656 9.2.6. Druid Observability ..................................................................................................................................... 663 9.3. Java Observability ................................................................................................................................................ 670 9.3.1. Java CPU Profile ......................................................................................................................................... 671 9.3.2. The Java Virtual Machine ............................................................................................................................ 683 9.3.3. Coroutines .................................................................................................................................................. 684 9.3.4. Implementing Threads ................................................................................................................................ 687 9.3.5. The Virtual Thread Demo ............................................................................................................................ 689 9.3.6. Spring WebFlux Demo ................................................................................................................................ 703 9.3.7. Non-Blocking Reactor Demo ...................................................................................................................... 704 9.4. Summary .............................................................................................................................................................. 711 Chapter 10: AIOps RCA ��������������������������������������������������������������������������������������������������������������������713 10.1. AIOps Limitations ............................................................................................................................................... 714 10.1.1. Example 1 ................................................................................................................................................. 714 10.1.2. Example 2 ................................................................................................................................................. 715 10.1.3. Example 3 ................................................................................................................................................. 716 Table of ConTenTs

Page 12

xii 10.2. AIOps Correlation ................................................................................................................................................ 720 10.2.1. AIOps Correlation Demo ............................................................................................................................ 724 10.3. IT Operation Data ............................................................................................................................................... 729 10.3.1. Using CMDB .............................................................................................................................................. 737 10.3.2. Autoscaling ............................................................................................................................................... 742 10.4. The Failure List Data Model ................................................................................................................................ 755 10.4.1. System Resource Failures ........................................................................................................................ 759 10.4.2. Cluster Failures ........................................................................................................................................ 763 10.4.3. Server Framework Failures ...................................................................................................................... 765 10.4.4. Legacy Middleware Failures ..................................................................................................................... 767 10.4.5. Data Pipeline Failures ............................................................................................................................... 773 10.4.6. Data Consistency Failures ........................................................................................................................ 774 10.4.7. Microservices Failures ............................................................................................................................. 776 10.5. The Observability Failure Case ........................................................................................................................... 777 10.6. Retrieval-Augmented Generation (RAG) ............................................................................................................. 782 10.6.1. LangChain ................................................................................................................................................. 783 10.6.2. Searching Observability Data ................................................................................................................... 788 10.6.3. OpenSearch AIOps .................................................................................................................................... 797 10.6.4. RAG’s Advanced Features ......................................................................................................................... 833 10.6.5. RCA Agents ............................................................................................................................................... 844 10.7. Summary ............................................................................................................................................................ 887 Index ������������������������������������������������������������������������������������������������������������������������������������������������889 Table of ConTenTs

Page 13

xiii About the Author Hyen Seuk Jeong has worked as a systems engineer at IBM, Accenture, and Oracle. He is an engineer who enjoys solving complex problems and developing solutions. As an SRE in complex banking and telecommunications, he has experienced many failures with observability. He recently began working on root cause analysis and automation through AI using agents and RAG.

Page 14

xv Introduction Writing this book was a challenge. The banking and telecom industries are technically constrained and demanding in terms of compliance and security. Rerunning many legacy applications and preparing demos from long ago is a challenge. I initially thought that E2E trace would not be technically possible, but the process of solving it provided a great opportunity to learn new skills. I believe that this book will be helpful to someone in a position similar to mine. Although observability has matured and a variety of technologies are available, many observability and AI projects still fail. • The banking and telecom industries have a lot of legacy, black box, and complicated processes that make it difficult for them to implement E2E observability. They often spend a lot, yet don’t get the results they expect. • The banking and telecom industries use AI technology to configure anomaly detection and AIOps, but the results are inaccurate and noisy, making them impractical. • The data required for IT operations is missing, broken, or lacking relationships and structure, making it impossible to analyze and automate. • There is a lot of unnecessary work for IT operations, including unnecessary data pipelines, API interfaces, and manual tagging, and technical debt continues to grow. • AI agents and RAG are being applied to observability and IT operations, but no one has successfully implemented them yet. If my project hadn’t failed, I wouldn’t have had the opportunity to publish this book. However, the reality is that most projects don’t work out. What This Book Covers This book explains how to successfully configure complex E2E trace, minimize MTTR, analyze root cause in nanoseconds, and automate IT operations with AI agents and RAG. • Using OpenTelemetry and commercial observability, the book provides a variety of demos. The demos run on Kubernetes to give readers a better understanding of observability.

Page 15

xvi • This book explains the limitations of entities such as resources, and covers service and signals, such as traces, events, and logs. It describes 14 correlations between signals and entities to overcome limitations. • The book explains the data models needed for observability and IT operations, including CMDB, to guide cost savings and build successful IT operations. • It analyzes nearly 40 legacy systems, including IBM CICS, SAP, Siebel, and Tuxedo, and develops an OpenTelemetry agent and extension to apply E2E trace. • It designs a data model for root cause analysis by correlating 12 signals. It analyzes approximately 400 failure cases and uses RAG to analyze root causes with AI. • It identifies nanosecond-level waiting, interference, contention, and saturation in the CPU, network, and kernel, and quickly analyzes the root cause. It also creates 5,000 spans on a single transaction and visualizes them in a dashboard. It describes bytecode instrumentation and various kernel instrumentation methods. • It demonstrates how to apply observability to diverse legacy and complex processes and automate operations using the AI agents and RAG technology. Who This Book Is For This book is ideal for developers, data engineers, SREs, architects, operators, and DevOps engineers. It’s also appropriate for system developers, infrastructure engineers, system architects, and Java developers who are enthusiastic about observability and want to implement it with legacy technology. Potential readers include the following. Developers Developers who want to understand the internal operation of their systems and the root cause of a problem. It often happens that something fails, but you don’t understand why. Due to time and manpower constraints, the application is restarted and they move on without understanding the root cause. Willing or unwilling, developers and engineers cannot analyze root causes if they do not have knowledge of how internal systems process, or if the signals within their observability are not accurately collected. The purpose of this book is to help engineers understand the internal operation of their systems, collect signals, and analyze root causes. That’s why I go into great detail about kernel, VM, instrumentation, CPU, IO, trace, stack trace, and more. This book is also for developers who are curious about observability best practices at tech companies and large enterprises. InTroduCTIon

Page 16

xvii Developers love Google’s SRE culture and practices and have a certain amount of admiration for SRE at tech companies. At my current job, my manager, team leader, and teammates are from Google SRE, so I have a great opportunity to understand and learn about SRE from the perspective of people from tech companies. What I’ve learned from working with them is that, like everything, there are pros and cons. It’s true that legacy is the most important core system, so there are many constraints to implementing observability. The difference is the complexity of the domain and business processes that we don’t have in tech companies. Data Engineers Data engineers who want to automate observability and understand AIOps. From a data perspective, observability is big data, and it’s a great place to apply AI. It’s a transitional time, and both open source and commercial observability offerings have limited capabilities in terms of data utilization. At the same time, companies are investing heavily in emerging technologies like AIOps. My experience with commercial AIOps deployments is that they are not effective in terms of analyzing root causes and reducing noise in operations. AI, as it is currently evolving, is still far from being cost- effective, but given the evolution of the technology, it is worth staying interested and learning. Analyzing and aggregating observability data is something developers need to understand. The RAG, AI agents, and ML described in this book are highly accurate and successfully produce results. SREs Site reliability engineers (SREs) who want to quickly identify root causes and understand solutions. An important part of operations is automating systems and resolving issues quickly. The ability to identify and resolve problems is especially important for SREs. It is not easy to recognize failures quickly and come up with solutions in a situation where development languages and types of problems vary and propagate. Recently published SRE books have focused more on the macro view than on the practical details that help solve problems, and it would be helpful for many SREs to have a book that explains approaches and solutions to the various errors encountered in practice. To understand and solve the problem, traces are discussed in detail. Traces consist of two types: distributed traces and system traces. My approach is to start with traces, as they are the most basic signals and the starting point for observability, and then gradually expand to other signals. Despite the complexity and difficulty of traces, it is necessary to configure distributed traces across the board and system traces at the system-lease level to troubleshoot. InTroduCTIon

Page 17

xviii Operators in Banking and Telecom Operators in banking, telecom, and other organizations with a lot of legacy who want to bring in new observability. It’s true that SRE cultures are often introduced in the context of tech companies, but tech companies are the minority of organizations, with more observability examples coming from traditional banking, telecom, and manufacturing companies. It’s important to consider whether the SRE culture of tech companies is applicable to larger organizations and industries with more legacy. Organizing observability as a cross-cutting concern is difficult and progress is slow. This is because legacy breaks the propagation of traces, and agents are often difficult to configure and not technically supported. This book describes a variety of middleware. Since legacy is often connected through EAI servers and message servers, this book shows you how to instrument this middleware to configure E2E observability. It describes various legacy applications, such as SAP ERP and Tuxedo, and explains how to configure observability in legacy. Architects Architects who want to technically advance observability and understand its business value. In this book, I divided observability into three phases: • Configure application and infrastructure observability and leverage it for root cause analysis. • Automate the analysis and aggregation of observability data and use AIOps to analyze root causes and predict failures. • Deliver reliable services to customers to minimize losses from failures and use observability to improve the business and create opportunities. Looking at observability as a tool for business improvement and opportunity is new. This is because observability has only been considered in the realm of systems—applications, kernels, and VM. Gone are the days when observability was viewed solely from a technical and cost perspective. In the future, observability can be utilized as a tool to further support and align with the business and increase revenue. I explain how to move beyond root cause analysis and operational automation to observability that can lead the business and deliver value to product owners and executives. In the future, expect observability to expand, supporting both business and technology. This book will help you see what’s possible. DevOps Engineers DevOps engineers torn between open source and commercial observability. InTroduCTIon

Page 18

xix While I personally prefer open source, many companies are paying tens of billions of dollars for commercial observability. They want to minimize failures and resolve issues quickly, even at a high cost, because failures can be costly to their business. They’re paying a lot of money, so they need to prove value. Observability is not just a tool for SRE, it’s a tool used across the organization with developers, and they are looking for ways to reduce costs and increase developer productivity. Open source is difficult to support productively and externally, so internal SRE capabilities are critical. The main disadvantage is that it is difficult to support the rapid growth of the business and the time-to- market schedule. It takes time and patience from many people to develop internal technical capabilities, know-how, and trial and error. It’s a mistake to choose open source observability just to save money. There are pros and cons. InTroduCTIon

Page 19

1 © Hyen Seuk Jeong 2025 H. S. Jeong, Observability For Legacy Systems, https://doi.org/10.1007/979-8-8688-1688-8_1 CHAPTER 1 RCA Essentials 1.1. Introduction This book explains how to analyze root causes and uses observability and AIOps. There are differences in terminology: • Root cause analysis is the process of identifying problems, understanding their causes, and proposing solutions. It focuses on the problem. • Observability is the process of correctly defining signals, collecting, transforming, storing, and analyzing signals, and the series of processes used by the SRE (site reliability engineer). It focuses on signals. • AIOps refers to the use of AI based on data to automate IT operations. It focus on automation. Although there are some terminological differences, the goal is the same—to improve the reliability of services and reduce costs. More specifically, to: • Reduce losses from failures with observability. • Improve and tune performance to reduce costs. • Optimize observability to reduce costs. • Automate IT operations to reduce operational costs. • Improve reliability to provide stable services. • Leverage collected events for marketing and increased sales. This book describes the various errors and failures you will encounter in IT operations, including commercial legacy, Linux kernel, system resources, and large cluster issues. You will learn to accurately match the relationships between business processes, applications, and system resources and clearly identify any problems. For example, a one-second delay experienced by a user is composed of multiple microsecond delays occurring in the application. What are multiple microsecond delays?

Page 20

2 Microsecond delays in an application are composed of multiple nanosecond delays occurring in the system resources. Multiple nanosecond delays can be clearly identified. The book is organized into four topics: observability theory, application observability, infrastructure observability, and utilizing observability data. Part 1, “RCA theory,” includes the following: • Chapter 1, RCA Essentials: This chapter analyzes the limitations and problems of existing observability and the root cause analysis methods. It explains a successful root cause analysis definition. • Chapter 2, RCA Approach: Most of the observability dashboards and alerts are inaccurate and fail. This chapter explains the causes of failure and solutions. Part 2, “Application RCA,” includes the following: • Chapter 3, Trace-Centric RCA: When building observability, there is a problem where the trace is broken or logs are conflicted in the server framework and message server. This chapter analyzes a practical correlation example in Grafana and OpenSearch. • Chapter 4, Observability Practices in the Industry: This chapter explains complex trace reference in banking, telecom, trading, and gaming. Through practical examples, you learn about the advantages and direction of observability. • Chapter 5, OpenTelemetry Demo: This chapter introduces new techniques and augments the observability described so far. Part 3, “Infrastructure RCA,” includes the following: • Chapter 6, Infrastructure RCA: Traditional infrastructure observability cannot analyze the root cause. This chapter explains how to understand the existing problem and configure new system traces and profiles for successful root cause analysis. • Chapter 7, Anomaly Detection: Traditional anomaly detection is mostly unsuccessful because it has low accuracy and includes a lot of noise. The chapter explains the existing anomaly detection problem and explains how to configure successful anomaly detection. This book focuses more on infrastructure than on applications. Part 4, “Leveraging Observability Data,” includes the following: • Chapter 8, Analyze RCA Data: The advantage of commercial observability is that observability data can be analyzed in detail. This chapter explains how to analyze observability data. ChapTer 1 rCa essenTIals