Troubleshooting Java (Laurentiu Spilca) (Z-Library)

📄 File Format: PDF

💾 File Size: 7.9 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

M A N N I N G Laurenţiu Spilcă Foreword by Ben Evans SECOND EDITION Java TROUBLESHOOTING

📄 Page 2

A comparison between the memory usage for a healthy app versus an app suffering from a memory leak. The GC frees unneeded data from memory for a healthy app, and the allocated space never fills up. An app with a memory leak prevents the GC from removing enough data. At some point, the memory fills up completely, generating an OutOfMemoryError. In an app that behaves normally, you will see this pattern. The memory fills, and at a certain point, the GC cleans the unneeded data, freeing up the memory. These are moments when the GC cleaned the unneeded data, making space for new data to be added in memory. When an app has a memory leak, the used memory continuously grows. The GC attempts to free the memory but can’t deallocate enough objects since the app holds the references for most of them. Normal behavior Abnormal behavior

📄 Page 3

Praise for the first edition Definitely a must read for every Java developer, who needs to tune performance of the Java production system. —Amrah Umudlu, Software Engineer, Azericard LLC A compendium of immense value for java programmers—finally a book that illustrates how to effectively troubleshoot and reason about your Java apps in production! This distilled wisdom could save the day! —Atul Shriniwas Khot, Software Architect, SquareOne Insights Regardless of whether you are a seasoned engineer or a Junior Developer, you should read this book if you want to have an in-depth understanding of the debugging processes, principles and techniques, within Java applications. Highly recommended. —Giorgi Tsiklauri, Independent Software Consultant, Engineer, Lecturer This is a must-read for all Software Engineers regardless of experience level, who want to be well- equipped to troubleshoot and resolve Java applications issues with severity ranging from simple to extremely complex, like multithreading or memory leaks. —Latif Benzzine, Lead Software Engineer, Cognizant The most comprehensive guide to understanding and debugging Java applications to date. —Peter Szabó, Senior Software Engineer, Tesco Technology The book that demystifies debugging. A daring work of scholarship and exploration into the depths of Java codebases, distilling concrete knowledge in the most accessible way possible. —Michael Kolesidis, Software Engineer, Okto Laurențiu’s book is packed with extremely valuable tips and tricks for every level. —Alex Gout, Senior Data Engineer, Shopify

📄 Page 4

(This page has no text content)

📄 Page 5

MANN I NG Shelter ISland LaurenȚiu SpilcĂ Troubleshooting Java Second Edition Foreword by Ben Evans

📄 Page 6

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com © 2026 Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid- free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. ∞ Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 ISBN 9781633435575 Printed in the United States of America The author and publisher have made every effort to ensure that the information in this book was correct at press time. The author and publisher do not assume and hereby disclaim any liability to any party for any loss, damage, or disruption caused by errors or omissions, whether such errors or omissions result from negligence, accident, or any other cause, or from any usage of the information herein. Development editor: Marina Michaels Technical editor: Nicholas A Watts Review editor: Dunja NikitoviÊ Production editor: Andy Marinkovich Copy editor: Lana Todorovic-Arndt Proofreader: Keri Hales Typesetter: Tamara ŠveliÊ SabljiÊ Cover designer: Marija Tudor

📄 Page 7

v brief contents Part 1 Revisiting the foundation for code investigation 1 1 ■ Starting to know your apps 3 2 ■ Understanding your app’s logic through debugging techniques 25 3 ■ Finding problem root causes using advanced debugging techniques 53 4 ■ Making the most of logs: Auditing an app’s behavior 76 Part 2 Deep diagnosing an app’s execution ...................... 103 5 ■ Identifying resource consumption problems using profiling techniques 105 6 ■ Finding hidden problems using profiling techniques 129 7 ■ Investigating locks in multithreaded architectures 157 8 ■ Investigating deadlocks with thread dumps 180 Part 3 Diagnosing memory-related problems ................. 199 9 ■ Profiling memory-related problems 201 10 ■ Investigating memory problems with heap dumps 211 11 ■ Analyzing potential JVM problems with GC logs 228

📄 Page 8

vi brief contents Part 4 Finding problems in large systems ........................ 251 12 ■ Uncovering system-level failures and service communication problems 253 13 ■ Measuring data consistency and transactions 276 appendix A ■ Tools you’ll need 298 appendix B ■ Opening a project 299 appendix C ■ Recommended further reading 301 appendix D ■ Understanding Java threads 303 appendix E ■ Memory management in Java apps 319

📄 Page 9

vii contents foreword xii preface xiv acknowledgments xvi about this book xvii about the author xxi about the cover illustration xxii Part 1 Revisiting the foundation for code ............ investigation ................................................1 1 Starting to know your apps 3 1.1 How to more easily understand your app 5 1.2 Typical scenarios for using investigation techniques 10 Demystifying the unexpected output 11  ■  Getting familiar with your external libraries 15  ■  Clarifying slowness 16 Understanding app crashes 18 1.3 AI as a game changer in troubleshooting apps 19 1.4 What you will learn in this book 23 2 Understanding your app’s logic through debugging techniques 25 2.1 When analyzing code is not enough 27

📄 Page 10

viii contents 2.2 Investigating code with a debugger 31 What is the execution stack trace, and how do I use it? 37 Navigating code with the debugger 43 2.3 When using the debugger might not be enough 50 3 Finding problem root causes using advanced debugging techniques 53 3.1 Minimizing investigation time with conditional breakpoints 54 3.2 Using breakpoints that don’t pause the execution 62 3.3 Dynamically altering the investigation scenario 65 3.4 Rewinding the investigation case 69 4 Making the most of logs: Auditing an app’s behavior 76 4.1 Investigating issues with logs 80 Using logs to identify exceptions 84  ■  Using exception stack traces to identify what calls a method 86  ■  Measuring time spent to execute a given instruction 87  ■  Investigating problems in multithreaded architectures 88 4.2 Implementing logging 89 Persisting logs 89  ■  Defining logging levels and using logging frameworks 90  ■  Problems caused by logging and how to avoid them 97 Part 2 Deep diagnosing an app’s execution ....... 103 5 Identifying resource consumption problems using profiling techniques 105 5.1 Where would a profiler be useful? 106 Identifying abnormal usage of resources 106  ■  Finding out what code executes 107  ■  Identifying slowness in an app’s execution 108 5.2 Using a profiler 108 Installing and configuring VisualVM 109  ■  Observing the CPU and memory usage 111  ■  Identifying memory leaks 121 5.3 Using AI assistance 126

📄 Page 11

ixcontents 6 Finding hidden problems using profiling techniques 129 6.1 Sampling to observe executing code 130 6.2 Profiling to learn how many times a method is executed 138 6.3 Using a profiler to identify SQL queries an app executes 142 Using a profiler to retrieve SQL queries not generated by a framework 143  ■  Using the profiler to get the SQL queries generated by a framework 148  ■  Using the profiler to get programmatically generated SQL queries 151 7 Investigating locks in multithreaded architectures 157 7.1 Monitoring threads for locks 158 7.2 Analyzing thread locks 163 7.3 Analyzing waiting threads 171 8 Investigating deadlocks with thread dumps 180 8.1 Getting a thread dump 181 Getting a thread dump using a profiler 182  ■  Generating a thread dump from the command line 183 8.2 Reading thread dumps 187 Reading plain-text thread dumps 188  ■  Using tools to better grasp thread dumps 194 Part 3 Diagnosing memory-related problems .. 199 9 Profiling memory-related problems 201 9.1 Sampling to identify memory allocation problems 202 9.2 Profiling to find the culprit 207 10 Investigating memory problems with heap dumps 211 10.1 Obtaining a heap dump 212 Configuring an app to generate a heap dump when it encounters a memory problem 212  ■  Obtaining a heap dump using a profiler 214  ■  Obtaining a heap dump with the command line 214 10.2 Reading a heap dump 216 10.3 Using the OQL console to query a heap dump 221

📄 Page 12

x contents 11 Analyzing potential JVM problems with GC logs 228 11.1 Enabling GC logs 229 11.2 Storing GC logs in files 235 11.3 Particular configurations for storing GC logs 239 11.4 Analyzing GC logs 241 Troubleshooting performance lags with GC pause times 241 Identifying memory leaks with heap usage logs 243 Identifying insufficient memory with full GC events 244 Tuning parallelism in GC 247 Part 4 Finding problems in large systems ......... 251 12 Uncovering system-level failures and service communication problems 253 12.1 Troubleshooting communication patterns: RPC and messaging 254 Working with trace IDs and spans 255  ■  OpenTelemetry, Jaeger, Zipkin, and other utilities 259 12.2 Serialization mismatches and versioning problems 262 12.3 Understanding systemic failure modes 265 Cascading failures 265  ■  Retry storms 269 Timeout mismatches 271 13 Measuring data consistency and transactions 276 13.1 Troubleshooting inconsistencies across services 277 Inspecting time-based anomalies in event flows 277 Applying domain invariants to identify invalid states 282 13.2 Tracking and correlating multistep transactions 284 Reviewing audit logs to reconstruct transaction steps 285 Replaying events or examining event logs for missing messages 289 13.3 Measuring and monitoring consistency guarantees 292 Verifying data integrity using checksums or hashes 293 13.4 Running reconciliation jobs to compare expected vs. actual state 296

📄 Page 13

xicontents appendix A Tools you’ll need 298 appendix B Opening a project 299 appendix C Recommended further reading 301 appendix D Understanding Java threads 303 appendix E Memory management in Java apps 319 references 331 index 332

📄 Page 14

xii foreword You may already be familiar with Laurențiu Spilcă from his books Spring Security in Action and Spring Start Here, but if not, Troubleshooting Java provides a great introduction to his unique style of writing and pervasive humor. If you’re a newcomer to Java devel- opment, you may well find his careful coverage of concepts extremely useful, especially when combined with thorough coverage of the available (free) tools and how to use them. In addition, the book includes many examples and demonstrations of what can go wrong, to help you build experience and confidence in using the tools. Part 1 focuses on debuggers and covers common tools like the IntelliJ CE debugger— both their capabilities and their limitations. Unlike many other resources, Laurențiu covers several debugger techniques that go well beyond the basics, such as conditional breakpoints and nonpausing breakpoints. There’s also solid coverage of the basics of troubleshooting Java applications using logs. In part 2, the key topic of resource consumption and its central role in troubleshoot- ing is front and center. The free VisualVM tool serves as a primary means of under- standing this area and its related concepts. There is in-depth coverage of important topics such as CPU sampling and instrumentation and how to use the tools effectively. One essential, but often overlooked, area—the handling of external dependencies—is neatly showcased via the example of an SQL database using Hibernate. The subject of multithreaded programming, and especially troubleshooting locks, follows naturally from CPU profiling. Part 3 is devoted to memory-related issues, including tracking down memory leaks with sampling and profiling techniques, as well as creating and navigating heap dumps. Practical techniques, such as filtering out JDK types and using the capabilities of VisualVM to consider live objects, are also covered, as well as the use of OQL to query

📄 Page 15

xiiiforeword heap dumps. Just as in part 1, the use of logs (in this case, GC logs) forms the subject of the last chapter in part 3. To conclude, Laurențiu takes us on a quick tour through larger-scale systems, start- ing with an introduction to distributed tracing as a key technique, before tying it all together by discussing distributed transactions across heterogeneous systems. Through- out the book, Laurențiu makes careful and tactical use of AI tools, stressing the lack of magic bullets and focusing on practical use cases. While these tools cannot replace a human user, they can make a proficient engineer more productive and able to concen- trate on the higher-level concerns. The end result is a book that is suitable for newcom- ers but is also forward-looking and recognizes how modern tooling can complement and enhance the intuition, experience, and problem-solving insight of working soft- ware engineers. —Ben Evans Java Champion and author of The Well-Grounded Java Developer

📄 Page 16

xiv preface What does a software developer do for a living? “Implement software” is most likely the answer many would give. But what does implementing software really mean? Is it only writing code? Not quite. While the code is the visible result of a developer’s work, the activity of writing it takes only a small fraction of the time. Most of a developer’s day is spent designing solutions, reading existing code to understand how it behaves, and learning new concepts. Writing code is simply the outcome of successfully doing all of these. That’s why programmers often spend far more time reading code than writing it. Clean coding as a discipline grew out of this realization: it is more efficient to write solutions from the start in a way that makes them easier to read later. Still, not all code is clean, and not all systems are simple. You will always face situations where you need to dig into an unfamiliar solution and uncover how code really works. The truth is, a software developer spends much of their career investigating how applications behave. We trace through our own code and third-party dependencies to understand why something doesn’t work as expected. Sometimes, we do it to fix prob- lems and other times just to learn. Often, reading code isn’t enough. We need to go deeper—using debugging, profiling, or log analysis—to understand what is happen- ing inside the JVM or how the environment affects the application. Knowing the right techniques, and when to apply them, can save enormous amounts of time. Optimizing this investigative activity is one of skills with the greatest strategic effect a developer can build. The goal of this book is to help you optimize the way you investigate software sys- tems. You will find relevant techniques illustrated with practical examples—debugging, profiling, log analysis, and how to combine them effectively. Along the way, I share tips

📄 Page 17

xvpreface and practices that will help you become faster and more confident in tackling difficult problems. In this second edition, we also explore a new and increasingly important partner in troubleshooting: artificial intelligence. AI tools have become part of the developer’s daily toolkit, capable of analyzing logs, suggesting hypotheses, or even pointing out suspicious code paths. They can accelerate investigation dramatically. But AI does not replace your judgment. Just like a detective uses assistants and tools but still solves the case, you remain the expert who interprets the evidence and makes the decisions. Used wisely, AI can help you focus on the most meaningful parts of the investigation and reach insights faster. My goal remains the same: to make you more efficient as a developer. But in today’s landscape, efficiency also means knowing how to use AI without becoming dependent on it. With the techniques and mindset in this book, you will be better equipped to find root causes quickly, learn continuously, and solve even the toughest problems with confidence. I hope this edition brings you both practical value and inspiration for your daily work.

📄 Page 18

xvi acknowledgments This book wouldn’t be possible without the large number of smart, professional, and friendly people who helped me out throughout its development process. A heartful thank you goes to my wife Daniela, who was there for me, helped with valu- able opinions, and continuously supported and encouraged me. I’d like to thank the entire Manning team for their huge help in making this a valu- able resource. I specifically want to mention Marina Michels and Nick Watts for being incredibly supportive and professional. Their advice brought great value to this book. I also thank my friend Ioana Göz for the drawings she created for the book. She did a great job turning my thoughts into the cartoons you’ll discover here and there in the book. Next, I thank everyone who reviewed the manuscript and provided useful feedback. To Alok Ranjan, Andrew Oswald, Becky Huett, Burkhard Nestmann, Chris Allan, Cur- tis Krauskopf, Faiz Gouri, Frank Beutelschiess, German Gonzalez-Morris, Jason Clark, Jitender Jain, Lars Opitz, Max Loukianov, Mebin Jacob, Naga Rishyendar Panguluri, Nancy Al Kalach, Nicolas Bievre, Prashant Gupta, Purushotham Krishnegowda, Ravi Laudya, Sachin Handiekar, Sanjay Belaturu Krishnegowda, Sathiesh Veera, Sumit Bhat- nagar, Vamsi Kavuri, William Brawner, Heinz Kabuz, and Vlad Mihalcea, your feedback helped me improve the content of this book. Finally, thanks to Ben Evans for taking the time to review the book and write its foreword.

📄 Page 19

xvii about this book Because you opened this book, I assume you are a developer working with a JVM lan- guage. You might use Java, but you could just as well be coding in Kotlin or Scala. Regardless of the JVM language, you will find the content valuable. This book teaches practical investigation techniques that help you identify the root causes of problems and also learn new technologies more effectively. As a software developer, you’ve probably noticed how much of your time is spent understanding what your application actually does. Whether you are reading code, debugging, analyzing logs, or checking performance, chances are you spend far more time on these activities than on writing new code. That’s why becoming more efficient at investigating and analyzing application behavior pays off so quickly—it helps you solve problems faster and with greater confidence. This second edition also includes a new dimension: how to take advantage of AI as a partner in troubleshooting. AI can process large volumes of logs, highlight unusual pat- terns, or suggest possible causes. It won’t replace your skills, but it can accelerate your work and allow you to focus on the insights that matter. This book discusses and illustrates through examples topics such as ¡ Simple and advanced debugging techniques ¡ Efficient use of logs to understand application behavior ¡ Profiling CPU and memory resource consumption ¡ Profiling to locate executing code ¡ Profiling to understand how an app interacts with persisted data ¡ Analyzing communication between services

📄 Page 20

xviii about this book ¡ Monitoring system events ¡ Using AI to assist in log analysis, root-cause identification, and knowledge discovery Who should read this book? This book is for any developer working with Java or another JVM language such as Kotlin or Scala. Regardless of your level of experience, you will find value—whether you are learning investigation techniques for the first time or refreshing skills you already use. Beginners and intermediate developers are likely to benefit the most, but even seasoned engineers may discover new tips, tools, or perspectives to sharpen their troubleshooting practice. While the book is written with JVM developers in mind, many of the techniques— such as debugging strategies, log analysis, and profiling—apply broadly to software development in other languages as well. This edition also introduces ways to incorpo- rate AI into troubleshooting, which is useful for any developer looking to work more efficiently in modern environments. The only prerequisite for this book is a basic understanding of the Java language. All examples are presented in Java (for consistency), but the ideas can be applied in any JVM language. If you are comfortable with fundamental concepts such as classes, methods, variables, and basic control flow (loops and conditionals), you will have no difficulty following the discussions. How this book is organized: A road map This book is divided into four parts, each with its own focus. Think of them as stages in your journey from local detective work to system-wide investigations. You don’t have to read them in order (though that’s how I recommend it), but the sequence is designed to build your skills step by step. ¡ Part 1—We start at the beginning: the everyday techniques you’ll use most often. Debugging and logging are your bread and butter as a developer. Here you’ll learn not just how to set a breakpoint, but how to use advanced debugging tricks (such as conditional and nonblocking breakpoints) and how to make logs work for you instead of against you. This part is about building your investigative reflexes—the same way a detective first learns how to look for fingerprints. ¡ Part 2—In this part, we move to profiling. Profiling sounds fancy, but it’s really just asking, What’s eating my CPU? What’s this app actually doing when I’m not looking? You’ll learn how to track resource consumption, find hidden perfor- mance bottlenecks, and even spot suspicious SQL queries. We’ll also use these chapters to introduce AI assistance—because sometimes you really do want a sidekick who can sift through mountains of data while you focus on the bigger picture. ¡ Part 3—Memory is where things get subtle (and sometimes sneaky). Here we dive into heap dumps, GC logs, and techniques for finding leaks or tuning memory