Lean Software Systems Engineering for Developers Achieving Predictable Outcomes Through a System for Software Development (Doug Durham, Chad Michel)（Z-Library）

Page 1

Lean Sof tware Systems Engineering for Developers Achieving Predictable Outcomes Through a System for Software Development — Second Edition — Doug Durham Chad Michel

Page 2

Lean Software Systems Engineering for Developers Achieving Predictable Outcomes Through a System for Software Development Second Edition Doug Durham Chad Michel

Page 3

Lean Software Systems Engineering for Developers: Achieving Predictable Outcomes Through a System for Software Development, Second Edition ISBN-13 (pbk): 979-8-8688-2064-9 ISBN-13 (electronic): 979-8-8688-2065-6 https://doi.org/10.1007/979-8-8688-2065-6 Copyright © 2025 by Doug Durham and Chad Michel This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Managing Director, Apress Media LLC: Welmoed Spahr Acquisitions Editor: Ryan Byrnes Coordinating Editor: Gryffin Winkler Cover image by Pexels.com Distributed to the book trade worldwide by Springer Science+Business Media New York, 1 New York Plaza, New York, NY 10004. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com, or visit www.springeronline.com. Apress Media, LLC is a Delaware LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please e-mail booktranslations@springernature.com; for reprint, paperback, or audio rights, please e-mail bookpermissions@springernature.com. Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales web page at http://www.apress.com/bulk-sales. Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub (https://github.com/Apress). For more detailed information, please visit https://www. apress.com/gp/services/source-code. If disposing of this product, please recycle the paper Doug Durham Lincoln, NE, USA Chad Michel Lincoln, NE, USA

Page 4

To my wife, Shana, who continually encourages me and fills my life with love, laughter, and happiness, and my father, Howard Durham, who showed me the importance of doing things the right way. —Doug To my wife Lisa, my son Sam, and my daughter Eva for all the support and encouragement during this endeavor. —Chad

Page 5

(This page has no text content)

Page 6

(This page has no text content)

Page 7

(This page has no text content)

Page 8

(This page has no text content)

Page 9

(This page has no text content)

Page 10

(This page has no text content)

Page 11

(This page has no text content)

Page 12

xiii About the Authors Doug Durham is Managing Partner of Don’t Panic Labs, a firm that helps companies innovate through the design and development of software technologies. He is also the co-founder of Nebraska Global, a pioneer in the startup landscape in Nebraska. Doug has almost four decades of software engineering and development experience in aerospace and defense, healthcare, manufacturing, ecommerce, consumer web applications, and Internet network services. He is passionate about the process of solving problems through software and the application of sound engineering principles and patterns to these efforts. Doug has taught software engineering courses at the University of Nebraska-Lincoln and serves on the School of Computing advisory board. He often speaks at industry conferences on the topic of software engineering and is a frequent guest lecturer. Chad Michel is CTO for Don’t Panic Labs with more than 20 years of software development and engineering experience. He holds a bachelor’s degree in computer engineering and a master’s degree in computer science. At Don’t Panic Labs, he works with clients to solve problems through innovative software solutions. Chad has worked for several companies in Lincoln, helping build a practice management application for lawyers, developing key features for an ecommerce application, and wrangling an Internet content delivery system into a stable platform. He regularly speaks at technical meetups hosted by Don’t Panic Labs, with significant contributions to the company blog. He also enjoys contributing to technical conferences and groups. Chad teaches a Cloud Architecture course at the University of Nebraska-Lincoln. This course covers how to design and build maintainable cloud solutions. Chad enjoys combat sports and frequently trains in taekwondo and Brazilian jiu-jitsu.

Page 13

xv About the Technical Reviewer Vadym Semeniuk is a Senior Software Engineer with over a decade of experience specializing in high- performance data processing, large-scale distributed systems, and performance optimization to reduce network, computational, and infrastructure costs. He has designed platforms capable of processing tens of billions of events daily and has worked across industries from startups to multinational corporations, gaining broad technical and business expertise. Vadym has authored peer- reviewed technical articles and created widely used open-source libraries, which have garnered hundreds of thousands of downloads. He has judged international hackathons and contributes to the software engineering community through mentoring in architecture design and performance tuning.

Page 14

xvii This second edition is a reflection of our continued exploration and understanding of what is involved in creating a system for developing software in this age of accelerating complexity and technological advancement. None of this could happen without the ability to explore, test, and revise these understandings in real-world environments. Special thanks to the whole Don’t Panic Labs crew for enabling us to create a special company that has become an incredible laboratory for experimenting with and refining lean software systems engineering practices, processes, and methods. It is a joy to work with these wonderful people, and we feel lucky every day to do what we do. Acknowledgments

Page 15

xix Introduction One need only look at the rapid turnover in the companies making up the S&P 500 and the introduction of generative and agentic AI technology for evidence that we are living in times of rapid change, innovation, and disruption. Whether you are a small software company or a large enterprise, your ability to be successful is directly related to your development teams’ ability to rapidly respond to change. The responsibility of our software development teams is to enable this business agility. Our ability to do this will play a significant role in the success of our organizations. Making this task more challenging is the fact that everything about the problems we are trying to solve today is becoming more and more complex – the requirements, the solution, the technology, the hosting, the support, the pace of change, etc. In March of 2021, we watched as NASA’s Jet Propulsion Laboratory (JPL) successfully landed their latest Mars rover, Perseverance.1 It is the second time they have successfully landed a vehicle, roughly the size and weight of a car (10-feet long, 9-feet wide, and weighing over 2200 lbs), using a complex, multistage landing sequence that concludes with a final stage that gently lowers the vehicle to the surface using something they call a sky crane. This entire landing process is fully automated, which means software plays a large role in the process. It is another example of the amazing things we can achieve with software. There is an enormous amount of complexity in landing a “car” on the surface of Mars. NASA’s level of success is a result of their staff of scientists and engineers at JPL leaving nothing to chance. They manage this complexity by managing every aspect of the entire program. They understand that when details and decisions are left to chance, then outcomes can be negatively impacted. The same is true when managing the complexity of any modern software system. Lack of structure and poor attention to detail result in increased errors in judgment and poor quality. These problems accumulate over time to create the software entropy or rot 1 https://mars.nasa.gov/mars2020/

Page 16

xx that we have all experienced. It is simply not enough to adopt agile methods or cherry- pick tools. We must look at the entire spectrum of the software development process and ensure we leave nothing to chance. Anything less adds risk to our projects. It is time our industry recognizes that we must approach software development as an engineering discipline to effectively manage both requirements and solution complexity. Those companies that make this transition will survive and flourish, but those that don’t adapt will become less relevant. This is the difference between becoming a professional software engineer and remaining a programmer. We have spent the last 20 years discovering how to integrate lean/agile practices within an engineering discipline that reduces errors in judgment and provides predictable outcomes, while still maintaining a high degree of agility and the ability to respond to change. Over the last 15 years, our small team has completed dozens of successful greenfield projects and product re-inventions. This allowed us to learn rapidly and refined our comprehensive approach that integrates best practices and modern techniques. This book provides the reader with an in-depth view into what we learned and how we approached this challenge. Software engineering is still emerging as a mature form of engineering, and we are still learning how to engineer software systems. It feels like we are a long way from there being well-established knowledge, patterns, and practices. This book, first and foremost, is intended to inspire the reader to think more deeply about what is required to achieve predictable outcomes in software development. We have shared our journey up to this point to act as a foundation for the reader to pursue their own journey. We would consider it a great success if the result was that the reader considered what we have shared, tested some of our methods and ideas, and built upon this foundation to achieve results, rigor, and outcomes beyond what we have experienced ourselves. We look forward to the possibility of hearing from readers how they have improved upon our methods and approaches, and we also look forward to sharing the results of our future efforts to continue to refine our methods in pursuit of “leaving nothing to chance.” Why We Wrote This Book While the body of knowledge around best practices and patterns has matured over the last five decades, many companies and software development teams continue to struggle with adopting these best practices and patterns within their organizations. This InTroduCTIon

Page 17

xxi has led to continued struggles to create and maintain software systems that provide the reliability, supportability, and extensibility required to effectively address the ever- increasing complexity of problems we need to solve via software. We feel there are several reasons that contribute to this challenge. First, while many developers are familiar with the concepts that are considered best practices, many of them have had no first-hand experience with being part of a team that has successfully adopted them. Second, many of the techniques, patterns, and practices have previously been presented by industry thought leaders. But they were presented in ways that make it difficult for developers of all skill levels to confidently advocate for them with management and implement them with a high degree of success. In addition, many of these sources are addressing only one portion of the problem, leaving the reader with an incomplete picture. Third, the current university curriculum for training software developers continues to lack sufficient commitment to ensuring graduates have the necessary depth of understanding and experience to bring these patterns and practices to the industry. In our experience, teams that have adopted these best practices and patterns have relied on one or more individuals who took an interest in developing this understanding on their own and had the capability to effectively advocate and ensure the successful adoption within their work environment. The problem is that these people tend to be unicorns. Who This Book Is For This book is for software developers and team leaders who have struggled to implement design and development best practices due to a lack of in-depth knowledge or experience. It’s designed to provide the confidence and foundational skills needed to achieve success. How This Book Is Organized This book is divided into two parts. The first part, containing Chapters 1–7, outlines the motivation for focusing on targeted outcomes of development and then discusses these outcomes and how to achieve them. The second part, Chapters 8–10, covers the InTroduCTIon

Page 18

xxii leadership required and planning considerations to achieve these desired outcomes and introduces a path toward achieving professionalism in software engineering. The following is a brief summary of the chapters. Chapter 1: Focusing on Software Development Outcomes Instead of Outputs Chapter 1 is dedicated to establishing the importance of focusing on desired outcomes as the goal of our engineering patterns and processes. It covers what it means to be a software engineer and highlights the core reasons for failures in our development processes: errors in judgment. Chapter 2: Managing the Dimensions of Complexity in Software Development Chapter 2 discusses the multiple dimensions of complexity and why it is important for us to effectively manage this complexity. It also introduces the key challenges we face in being able to design solutions that allow us to effectively manage complexity throughout the product life cycle. Chapter 3: Improving How We Learn and Iterate Throughout the Project Chapter 3 explores how we iterate and learn in software development and discusses why it is important for us to develop a shared understanding around the complexity within the systems we are building. It then introduces specific techniques that will increase the shared understanding to achieve the predictable outcomes we are seeking. Chapter 4: Validation of User Experience Chapter 4 discusses the nature of modern user experiences and the challenges we face in developing user experiences that satisfy the needs of the end user. It includes a demonstration of tools and techniques that can be used to reduce the risk of significant rework, resulting from changes in requirements for the UI of a system. Chapter 5: Designing Software Systems That Age Well and Adapt to Change Chapter 5 covers the nature of software rot and software entropy, its causes, and how we can approach decomposing a complex system to avoid this decay and enable sustainable business agility. Case studies are used to provide context to these concepts and to do a deep dive into the principle of information hiding. InTroduCTIon

Page 19

xxiii Chapter 6: Developers “Falling into the Pit of Success” Chapter 6 discusses what happens when we do not put guard rails in place for our developers to ensure that the decisions they make will result in them “falling into the pit of success” as opposed to the “pit of failure.” Several tools and techniques are demonstrated that will make it easier for a developer to do the right thing and not the wrong thing. Chapter 7: Institutionalized Quality Chapter 7 is dedicated to enforcing the concept that making quality a part of your culture requires that your processes and practices reflect a culture that values quality. This chapter reviews how this looks in an organization that has a culture that values quality. Chapter 8: Rethinking the Roles, Interactions, and Accountabilities on Your Teams Chapter 8 focuses on the importance of having strong technical leadership within an organization to help nurture and sustain the design and development of our software systems. This chapter outlines the recommended key responsibilities and accountabilities for a variety of roles within an organization. Chapter 9: Bringing It All Together – Creating an Action Plan Chapter 9 synthesizes the previous chapters into a series of considerations and approaches to develop an executable plan for transforming your organization based upon the most significant pain points. It ends with a discussion of what the developer experience will be like in this new organization. Chapter 10: Moving Toward a Standard of Care for Software Development Chapter 10 provides a history of the motivation for and establishment of professional engineering and relates this to the ongoing need for professional software engineers. It then introduces a framework for enabling individuals and our industry to be better prepared for the day when professional engineering certification and licensing becomes a requirement for our industry. —Doug Durham Lincoln, Nebraska August 12, 2025 InTroduCTIon

Page 20

1 © Doug Durham and Chad Michel 2025 D. Durham and C. Michel, Lean Software Systems Engineering for Developers, https://doi.org/10.1007/979-8-8688-2065-6_1 CHAPTER 1 Focusing on Software Development Outcomes Instead of Outputs Introduction Imagine landscaping your yard is your hobby, and you would like to have a small storage building to keep all of your garden and yard tools organized and stored. You want to be the envy of your neighbors, so you take the opportunity to create something that will stand out. You spend an hour or two drawing something that demonstrates the style of the garden shed and its dimensions. Maybe something like the plans shown in Figure 1-1.