THE BOOK OF DEBUGGING A Systematic Workflow for Finding and Fixing Bugs (Johannes Kuhlmann)（Z-Library）

Page 1

(This page has no text content)

Page 2

CONTENTS IN DETAIL TITLE PAGE COPYRIGHT ABOUT THE AUTHOR AND THE TECHNICAL REVIEWER ACKNOWLEDGMENTS INTRODUCTION What Is a Bug? Why We Should Fix Bugs Debugging Is Inevitable Debugging Is Your Job Who This Book Is For Prerequisites How to Use This Book Manage Expectations THE WORKFLOW Step 1: Reproduce Step 2: Probe Step 3: Examine Step 4: Fix Handle Bugs as a Team Common Root Causes Bad Data Bad Pointers Bad Logic Edge Cases Floating-Point Inaccuracies Stale Code or Assets Wrong Timing or Order Unclear or Bad Requirements User Experience Issues Summary

Page 3

PART I: REPRODUCE What Makes a Great Bug Report? 1. Reproduce the Bug Yourself Re-create the User Experience Emulate the Reporter Introduce Variations Beware of Networking Complexity 2. Shorten the Feedback Loop Reduce Build Times Reduce Deployment Times Reduce Startup Times Jump to the Correct Location 3. Automate the Reproduction Whether to Automate How to Automate When You Can't Automate Summary PART II: PROBE 4. Try Shortcuts Make Educated Guesses Focus on Suspicious Systems Ignore Irrelevant Systems Know When to Change Your Strategy 5. Search the Internet Don't Scroll Too Far Try Different Search Engines Use Advanced Search Engine Features Search for Source Code Don't Forget Your Intranet 6. Check Version Histories Find a Working Version Decide What to Test Identify the Offending Change Look into the Future Look into Branches and Forks 7. Attach the Debugger by Default Inspect Your Program's Behavior Use Exception Handling Use Breakpoints to Inspect Your Code's Execution Set Up Debuggers on Difficult Platforms Handle Cases Where the Debugger Interferes 8. Instrument the Program to Report Problems Use Assertions Handle Errors Signal Errors to the Caller Use Obvious Test Patterns

Page 4

Don't Break the Reproduction with Modifications Make the Instrumentation Optional 9. Remove Randomness Create Deterministic Programs Disable Potential Sources of Randomness Put Determinism to Work Consider Rare Hardware Issues 10. Ask Colleagues for Help Ask Your Team Ask for External Support 11. Search in All Files Choose and Configure Your Search Tool Choose Useful Search Terms Search Beyond the Source Code Peek into Nontext Files Look for Strings Composed in Code Use Searchable Patterns in Code 12. Collect Crash Dumps Trigger a Crash Dump Get Information from a Crash Dump Implement Crash Dumps Turn Crash Dumps into Actionable Data Use Third-Party Tools to Simplify Crash Reporting Summary PART III: EXAMINE 13. Step Through the Code Choose the Right Build Configuration Start the Debugging Process Use Advanced Debugging Features 14. Read the Code Act Like a Virtual Machine Strive for Readable Code Record What You Find Know Your Programming Languages 15. Follow the Data Find the Original Data Spot the Symptoms of Bad Data Find Where the Data Becomes Bad Compare Good and Bad Data Follow Good Data 16. Question Everything Find the Cause That Must Exist Don't Trust Blindly Investigate Assumptions and Dependencies Verify Your Assumptions Learn About the Context

Page 5

17. Experiment Conduct Exploratory Code Modifications Refine and Test Hypotheses Restore Your Code to Its Original State 18. Keep a Debugging Log Track Your Progress Avoid Distractions Use Documentation Tools Keep an Archive Use Chats as a Log 19. Build Your Own Tools Implementation Recommended Features for Debugging Tools Promotion and Documentation Release Considerations 20. Build a Minimal Repro Choose a Minimization Method Minimize the Input Share the Example Prevent Regressions 21. Debug with printf Avoid printf Pitfalls Know When to Use printf Implement printfs Leverage Advanced Logging Clean Up 22. Use Off-the-Shelf Tools Consider Availability Know When to Reach for a Specialized Tool 23. Talk It Through Define Your Goal Choose Who You Talk To Plan the Logistics Decide What to Cover Summary PART IV: FIX What Makes a Great Bug Fix Make It Revertible Address the Root Cause Verify 24. Just Fix It Remove the Root Cause Change the Original Sources Test for Side Effects 25. Share What You've Learned Document the Fix

Page 6

Find What Produced the Root Cause Prevent the Bug from Happening Again 26. Search for Related Problems Understand Patterns and Dependencies Make Bulk Changes with Caution 27. Fix Early Assign Reasonable Priorities Leverage Context Prepare for Attrition Avoid Dependent Behavior Take Ownership 28. Deploy a Quick Fix Undo the Offending Change Implement a Workaround Communicate Workarounds Don't Stop at the Workaround 29. Start Over Rewrite the Offending Functionality Test the New Implementation Compare the Results Stub the Implementation 30. Find a Creative Solution Declare It a Feature Remove the Feature Remember to Communicate Just Let It Be Summary CONCLUSION INDEX

Page 7

THE BOOK OF DEBUGGING A Systematic Workflow for Finding and Fixing Bugs by Johannes Kuhlmann

Page 8

THE BOOK OF DEBUGGING. Copyright © 2027 by Johannes Kuhlmann. All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. First printing 30 29 28 27 26 1 2 3 4 5 ISBN-13: 978-1-7185-0406-6 (print) ISBN-13: 978-1-7185-0407-3 (ebook) Published by No Starch Press®, Inc. 245 8th Street, San Francisco, CA 94103 phone: +1.415.863.9900 www.nostarch.com; info@nostarch.com Publisher: William Pollock Managing Editor: Jill Franklin Production Manager: Sabrina Plomitallo-González Developmental Editors: Eva Morrow, Abigail Schott-Rosenfield, and Rebecca Senninger Cover Illustrator: Rob Fiore Interior Design: Octopod Studios Technical Reviewer: Steve Rabin Copyeditor: Isabel Kunkle Proofreaders: Cindy Snyder and Michele Kessel Library of Congress Control Number: 2026024610 For customer service inquiries, please contact info@nostarch.com. For information on distribution, bulk sales, corporate sales, or translations: sales@nostarch.com. For permission to translate this work: rights@nostarch.com. To report counterfeit copies or piracy: counterfeit@nostarch.com. The authorized representative in the EU for product safety and compliance is EU Compliance Partner, Pärnu mnt. 139b-14, 11317 Tallinn, Estonia, hello@eucompliancepartner.com, +3375690241. No Starch Press and the No Starch Press iron logo are registered trademarks of No Starch Press, Inc. Other product and company names mentioned herein may be the trademarks of their respective owners. Rather than use a trademark symbol with every occurrence of a trademarked name, we are using the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark.

Page 9

The information in this book is distributed on an "As Is" basis, without warranty. While every precaution has been taken in the preparation of this work, neither the author nor No Starch Press, Inc. shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in it.

Page 10

About the Author Johannes Kuhlmann is the studio technical director at Fishlabs and has more than 15 years of experience as a programmer and technical lead. Kuhlmann has worked on a wide range of video games, including Chorus, Goat Simulator 3, Valheim, and titles from the Saints Row and LittleBigPlanet franchises. He holds a Diplom in computer science from the University of Hamburg. Having learned debugging by trial and error, he has prevented innumerable bugs from reaching users. Feel free to share your own debugging strategies with him via email at debugging@johanneskuhlmann.de. About the Technical Reviewer Steve Rabin is a principal software engineer at Electronic Arts, where he specializes in optimizing games and developing performance tools as part of the Performance Technology Group. He was previously a principal software engineer at Nintendo Technology Development, where he oversaw UX design for developer tools on the Nintendo Switch and Nintendo Switch 2, as well as architecting tools such as the Nintendo CPU Profiler for the Wii, Wii U, 3DS, and Nintendo Switch. Rabin is an inventor with 16 issued patents covering data visualization, profiling, augmented reality, and game controllers. He is an advisor and founder of the GDC Game AI Summit, the editor of the Game AI Pro book series, and a computer science instructor at DigiPen Institute of Technology. He earned a BS in computer engineering and an MS in computer science, both from the University of Washington.

Page 11

ACKNOWLEDGMENTS Writing this book has been a lifelong goal come true. It was only possible to achieve with the direct and indirect help of many people, and I will just name a few here. First, I must thank Insa, who helped me sort out many of my thoughts, strategized with me on how to get the book done, and supported me spending all this time on it. I'd also like to thank my children, Eik, Mats, and Jarno, who were always encouraging and who I hope one day will read this book and learn a thing or two from it. My buddy Chip has been involved with the book in some way or another since the beginning. I've discussed many of the concepts with him, and he was always a great source of feedback. My editors, Abigail, Eva, and Rebecca, have been instrumental in getting the book into shape. They made it readable and suggested many, many great improvements. Steve, my technical reviewer, not only provided a lot of fruitful feedback but also helped me with further discussions about details and terms. Lastly, I must thank all the people I've worked with over the years. They gave me a lot of input, were great sounding boards, and enabled me to test my ideas in practice.

Page 12

INTRODUCTION When I was starting out in software development, I was working on an educational game that had a built-in dictionary. To provide a more responsive feel, I multithreaded the search so that users could keep typing while the results loaded. On very rare occasions, the program crashed when users closed the dictionary after searching for a word. I was young and confident in my skills, so I got out my debugger and tried to reproduce the problem. I also tried finding a reliable reproduction with concrete steps and went through the code to verify that I hadn't made any mistakes. Nothing. I ultimately automated the reproduction. I changed the game to go into the dictionary module, search for some words, leave the dictionary, and repeat. I was sure if I let the automated test run over the weekend, I'd find a nice, clean call stack in the debugger when I returned on Monday, telling me exactly where the problem was. When I came in on Monday? Nothing. The game was still happily trying to reproduce the bug. No crash.

Page 13

I went through the code again and added some mutexes for safety (even though I was sure I wouldn't need them). The bug became even rarer, but it still happened occasionally. We ended up shipping the program in this state. I didn't have a workflow. I had a debugger, some intuition, and a willingness to try things, but I didn't have a systematic way to find a bug that didn't want to be found. This book is the workflow I wish I'd had then: a sequence of steps and a set of strategies for working through bugs that resist the obvious approaches. We can't avoid bugs entirely. Programming languages, frameworks, libraries, tools, operating systems, hardware, and requirements are too complex for any single person to understand completely. Developers, educators, and researchers are working to create fail-safe languages and tools to prevent bugs in the first place, but that's not what this book is about. This book is about what to do when, despite everyone's best efforts, the bugs are still there. Fixing difficult bugs is like a huge puzzle: If you keep finding more and more pieces that fit together, you'll have the complete picture at some point. Even when there might be too many pieces and too little time to get all of them sorted and identified, you must remind yourself that it's fundamentally possible. To be honest, I enjoy debugging quite a bit. It's such a great feeling when you achieve the seemingly impossible and fix that super-hard bug that was keeping people from using your program effectively — or at all! What Is a Bug? Talking about bugs and debugging raises the obvious question of what a bug actually is. We could go to some lengths to find the one correct and official definition that is true for all software, perhaps by diving into the relevant ISO standard (ISO/IEC/IEEE 24765:2017, a reference of common vocabulary in systems and software engineering). However, this doesn't necessarily cover

Page 14

every kind of bug. For example, is a feature not feeling entirely right something that would fit the formal definition of a fault in a software product? Bugs can manifest in many ways: crashes, slowness, broken features, visual glitches, audio problems, data leaks, and more. Even if a program has perfect technical implementation, bugs can reduce its fidelity and usefulness. For this book, let's keep it simple. We just want our software to be great, so we'll call anything that's not behaving as intended a bug. Why We Should Fix Bugs There are many reasons to fix bugs: Quality matters While annoying little bugs might not directly affect a program's usability, they can make it look cheap or low effort. Bugs in a program don't instill trust in its abilities. Even if there are only minor issues, like text kerning being off or rare, random crashes, your users might quickly look for an alternative. If your program doesn't deliver on its core functionality, that's even worse. You must provide a quality product right out of the gate to get noticed and make a difference in a crowded market. Bugs slow down development A bug in an internal tool that developers use frequently can severely hinder the development process even if users never see it. One that slows down testing will hurt the performance of anyone who is testing the program or executing it as part of their development workflow. Bugs can spread Even though computer bugs don't propagate by themselves, they can still spread through reuse or copy-and-paste of the offending code or asset. This even happens across projects, as programmers reference existing code or reuse small pieces from other software. Such code can provide simple functionality like getting the current time, generating a random number, or sanitizing user input; or it

Page 15

can be much more project-specific. Fixing bugs earlier can prevent this from happening. Aging bugs increase complexity The more recently a bug was added, the easier it is to fix. The programmer who wrote the code is most likely still around and remembers the change, so it's a lot easier for them to pick it up again and fix it. Other programs might start depending on the buggy code, making it harder to fix without breaking dependencies. Bugs can expose private data When programs deal with personal data, developers must follow established best practices to protect it, even when breaches don't directly affect a program's quality. Leaking personal data means losing users' trust, maybe permanently, and might also mean fines in some parts of the world. High usage magnifies bugs Let's say a bug takes a single tester one day and 100 tries in total to reproduce; that is, it has a 1 percent reproduction rate. This incidence seems very low. However, if your program turns out to be successful and you have 100,000 concurrent users, then 1,000 will be exposed to this particular bug every day. Debugging Is Inevitable Debugging is something that you have to do constantly while developing software. It is part of the development process while you iterate on the functionality, it comes up as testers or users report problems, and it takes up a lot of programmers' time. Debugging has been a substantial part of programmers' work since the very beginning. In Memoirs of a Computer Pioneer (MIT Press, 1985), Maurice Wilkes writes: By June 1949 people had begun to realize that it was not so easy to get a program right as had at one time appeared. [...] The realization came over me with full force that a good part of the remainder of my life was going to be spent in finding errors in my own programs. In The Elements of Programming Style (McGraw Hill, 1978), Brian W. Kernighan and P.J. Plauger point out the difficulty of

Page 16

debugging: Everyone knows that debugging is twice as hard as writing a program in the first place. So if you're as clever as you can be when you write it, how will you ever debug it? How much time you spend debugging depends heavily on your project, your role, and what stage of the development process you're in. For a little-used program where you're writing new features in a relatively safe language, you can likely get by without much debugging at all. If you're modifying a popular program that wasn't especially bug-free to begin with, or porting one to a new platform, you can easily burn a lot of time fixing both new and existing bugs. If you're in a senior leadership or mentor position, you're likely involved in fixing bugs that more junior developers find hard to address, splitting time between other non-programming tasks and helping the team sort out bugs. In the beginning stages of a project, you'll focus on writing new code, and while that involves making sure it works correctly, you're probably spending most of your time implementing features. Toward the end of the project, while you're polishing the program, you'll use almost all your time fixing the most critical bugs. Debugging Is Your Job Let's say somehow you happen upon a bug. You may have found it yourself, your quality assurance (QA) team may have reported it, your users may have sounded the alarm, or the bug may have taken one of a thousand other possible routes to you. Regardless, it's hopefully fair to assume that you have some kind of description of the bug (even if it's just in your head) and a vague idea of the circumstances it appears in. What should you do next? First, accept that you're now responsible for fixing this bug. It's almost too easy to blame problems on something or someone else. For example, you might want to attribute the bug

Page 17

to user error, the compiler, or some other team member's code, but it's more likely that code written for a specific project is faulty. A third-party system that others have tested and used is less likely to be broken. That's why you must assume that a bug is your (or your team's) fault and step up to fix it. Another common mistake is to push back on dealing with a bug because, for example, you don't feel like fixing it, you don't see enough detail in the description, you aren't comfortable performing the inspection, or you even assume that, surely, this functionality simply can't be broken. However, if a bug was sighted, it's probably real. You may not have all the information you wish for, but you have to live with that to keep improving your program. In practice, go even further. As a routine, keep an eye on the bug tracker and request assignments to bugs that are related to your work or that you feel best suited for investigating. Take ownership of what you've implemented and what your team has built.

Page 18

Who This Book Is For This book is meant to fill the gap in existing educational material about debugging for software developers. Even though debugging takes up much of a developer's time and is imperative to shipping a successful program, minimal effort goes into training, coaching, and educating professional or aspiring developers. While the underlying assumption is that you're a programmer, it's not a hard requirement, as quite a few strategies will be useful for non-programmers. To follow along with this book, however, it's helpful if you have an idea of how programming works. Developers with a background in other disciplines will find helpful techniques that are applicable even if you work in a different field like art production, interface design, or QA. Debugging involves systematically solving a complex problem, which is frequently possible without touching the source code. You have to collect information on the issue and analyze it in the context of the bug to identify the underlying root cause, or the factor that, if eliminated, would prevent the bug from occurring. When we think about debugging, we might first imagine a traditional programming bug that needs to be solved by a programmer modifying the source code. However, as frameworks and tools become increasingly capable, they put more power — and greater responsibility — in the hands of artists, designers, and people from other disciplines. If you have more power to do things by yourself, you're also responsible for doing them right. Prerequisites You should be comfortable using a version control system (Git, Perforce, or Subversion) and a project management tool with bug tracking built in (there are many options, including Jira, Asana, Redmine, and ClickUp). The version control system tracks

Page 19

your project's history so you can see what changed and when. The bug tracker holds the description, reproduction steps, status, and other data you'll need to resolve each bug. If you're not already using these, get them. This book will help you a lot more if you do. How to Use This Book You can read this book straight through or use it as a reference whenever you're stuck. Experiment and keep the advice that resonates with you. The strategies presented here are generally platform- and language-agnostic. In some cases, I'll go into more detail about standard technologies used in particular programs, but that doesn't mean the method in question won't work in other situations. Even the more technical strategies don't go into very low-level details. For example, analyzing assembly code is rarely necessary. It's usually more beneficial to reason about a bug on a higher level and figure out where the logic is flawed. This book is not about preventing bugs or improving your programs. It addresses situations where a bug is present and needs to be found and fixed, using a structured workflow. This book is organized as follows: The Workflow Covers the general workflow and discrete steps you can use as a guiding framework to approach new bugs. Additionally, this chapter provides a rundown of common root causes or categories of bugs to keep in mind while you're progressing through the workflow. Part I: Reproduce Discusses how to reproduce a bug and what information to include in a report. This part also describes ways to make reproduction easier and faster than simply using your program. Part II: Probe Shows how to find a suitable entry point for a detailed investigation. Methods include using your experience to identify shortcuts, searching the internet,

Page 20

discussing the problem with other people, and incorporating a debugger into your efforts. Part III: Examine Covers how to examine your program and its sources. The debugger comes into play here as well, as do strategies like creating minimal versions of the program, using external tools, and analyzing the code. It's important that you question all of your assumptions during this stage. Part IV: Fix Explains how to fix the bug that you've investigated, and whether you need to do so urgently or can take your time to produce sustainable results. If a clean solution isn't possible, you can implement a hacky workaround, reimplement the functionality from scratch, disable the whole thing, or just live with the bug. Manage Expectations Before we venture deeper into the realm of debugging, a warning is in order. Working on bugs typically involves a lot of stress and pressure. Aside from the rare occasions when you encounter a trivial bug or make a lucky guess, the repair process also takes quite a bit of time and persistence. Users may have unrealistic expectations of how easily and quickly you can fix an issue, which adds to the pressure, as estimating resolution time is notoriously difficult. Manage your coworkers' and users' expectations accordingly: It's always better to under-promise and over-deliver. You'll need to manage your own expectations as well. Working through the bug systematically, using an established workflow and taking your time, will help keep you from overlooking anything. Make sure you have an environment where you can focus, since debugging has a very high cognitive load. If you feel stuck, it may be time to call it a day and continue after a good night's sleep.