Code as Data Governing and Evolving Large-Scale Codebases in the Era of AI (early release) (Bryan Friedman, Pat Johnson, Olga Kundzich)（Z-Library）

📄 File Format: PDF

💾 File Size: 2.0 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

(This page has no text content)

📄 Page 2

Moderne

📄 Page 3

(This page has no text content)

📄 Page 4

Code as Data Governing and Evolving Large-Scale Codebases in the Era of AI With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles. Bryan Friedman, Pat Johnson, and Olga Kundzich

📄 Page 5

Code as Data by Bryan Friedman, Pat Johnson, and Olga Kundzich Copyright © 2026 O’Reilly Media, Inc. All rights reserved. Published by O’Reilly Media, Inc., 141 Stony Circle, Suite 195, Santa Rosa, CA 95401. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (https://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Louise Corrigan Development Editor: Gary O’Brien Cover Designer: Ellie Volckhausen Cover Illustrator: Ellie Volckhausen Interior Designer: David Futato Interior Illustrator: Kate Dullea July 2026: First Edition Revision History for the Early Release 2026-04-23: First Release The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Code as Data, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors and do not represent the publisher’s views. While the publisher and the authors have

📄 Page 6

used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. This work is part of a collaboration between O’Reilly and Moderne. See our statement of editorial independence. 979-8-341-67422-6 [LSI]

📄 Page 7

Brief Table of Contents (Not Yet Final) Chapter 1: Managing Software in the AI Era (available) Chapter 2: A New Operating Model for Software (available) Chapter 3: From Repositories to a Unified Codebase View (unavailable) Chapter 4: Understanding Software Beyond the SBOM (unavailable) Chapter 5: App Security Through Semantic Program Analysis (unavailable) Chapter 6: Governing AI Models, Agents, and Generated Code (unavailable) Chapter 7: Turning Code as Data into Automated Action (unavailable)

📄 Page 8

Chapter 1. Managing Enterprise Software in the AI Era A NOTE FOR EARLY RELEASE READERS With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles. This will be the 1st chapter of the final report. Before the integrated development environment (IDE) arrived around the year 2000, working in an object-oriented codebase meant carrying most of its structure in your head. Developers wrote code in editors like vim and Emacs and navigated it with grep (a search tool written in the 1970s). Parsers like Tree-sitter improved on this, giving editors a structural view of code. But modern languages like Java became too complex for text and syntax tools to follow. The IDE changed this by doing something fundamentally different: it compiled the code, built a model of its types and relationships, and made that model available in real time. Rename a method and every call site updated. Navigate to a definition and it was there. The structure beneath the text became queryable. And developers continued to use their trusty grep. For many years, that was enough. The repository was the natural boundary of most systems, and the tools built around it reflected that assumption. Today’s software systems no longer live inside a single repository. Over the past two decades, software has expanded outward into collections of services, shared frameworks, infrastructure definitions, generated artifacts, and dependency graphs that evolve independently but operate

📄 Page 9

together. The structure that makes these systems understandable is distributed across the estate and across multiple layers of abstraction rather than contained within any one project. The result is a gap between what organizations need to know about their software and what their tools were designed to show them. This chapter is about that gap. The Unknowable Multi-repo Software Estate A mid-sized company may manage thousands of repositories; a large enterprise can have tens of thousands of repos with billions of lines of code. These codebases comprise millions of dependencies—direct and transitive —spanning languages, frameworks, and generations of architectural change. In fact, the average enterprise application depends on hundreds of open source packages, each with its own dependency tree. This complexity did not arise accidentally. Open source accelerated composition. Cloud services abstracted infrastructure. Microservices architectures increased team autonomy. Continuous delivery compressed release cycles from months to hours. Each was a genuine advance. Together, they produced software estates whose scale and interconnectedness now exceed the cognitive capacity of any single team to fully understand. Teams reached for multi-repository tools to help manage the complexity and make these large codebases more navigable: Search infrastructure indexed source across repositories. Static analysis mapped structural relationships across codebases. Software composition analysis (SCA) tracked dependency inventories at scale. Architecture registries recorded ownership and service boundaries.

📄 Page 10

Spreadsheets (still!) tagged repositories with ownership, framework versions, known dependencies, compliance flags. These tools extended visibility across the estate and are still useful today. But they come with a persistent tax: false positives, alert fatigue, and old data. Their outputs are only as complete as the code representations they rely on, which are inherently incomplete: Text representation of code is stripped of executable meaning like types or symbols; storable and readable, but lacking actionable precision. Abstract syntax trees describe syntax without fully resolving types and call targets across repositories. Dependency inventories show what is declared without revealing what is reachable or exercised. Architecture catalogs depend on manual maintenance and drift as systems evolve. These limitations are not simply coordination problems. They reflect the boundaries of the models themselves. The representations widely used to analyze software today were never designed to support precise reasoning across an entire estate. We’ll discuss what a different kind of model—one that’s compiler-accurate—makes possible in Chapter 2. Impossible Questions About Your Codebase The limits of today’s representations are not theoretical. They appear whenever organizations try to answer routine questions about how their systems are built and how changes will affect them. These are not unusual edge cases. They are the kinds of questions security teams, platform teams, and architecture groups ask every day—and often struggle to answer with confidence.

📄 Page 11

Which services in your estate use a given library—and which of those actually invoke the vulnerable method rather than merely declaring the dependency? Which teams are still running Java 8 or .NET 6? Which of their services cannot be upgraded without first resolving transitive dependency constraints? Which services process user-supplied data, and where does that data cross trust boundaries inside your system? How many distinct AI models, libraries, and agents are embedded in your production systems—and which teams introduced them? Which methods have no test coverage and high cyclomatic complexity in your codebase? These are the ones an agent is most likely to break when it modifies them. You might be tempted to assume agents change this calculus—and in some ways they do. Ask one to trace a data flow or identify a vulnerable call site and it will often move faster than a team of engineers, approximating understanding along the way. But approximation at agent speed is expensive. Without authoritative, structured, continuously maintained knowledge of the software estate to work from, agents are reasoning in the dark—quickly, confidently, and sometimes wrong. TIP Use this list as a self-assessment. If your team cannot answer three or more of these questions within an hour, your organization has a code intelligence gap. AI Agents Inherit the Same Problem Coding agents are operators working directly inside the system. They plan tasks, navigate codebases, modify files, introduce dependencies, and open

📄 Page 12

pull requests. And like every developer before them, they are hungry for context to complete their tasks. Some connect to language-server tooling or IDEs for symbol resolution within a single repository. Much of their work, however, still relies on searching code as text—reaching for grep, reading files to locate usages, unpacking library archives, and following import statements from one module to the next—basically building a working model of the code from the outside in. What they can’t know, they infer. This is a rational response to an incomplete information environment, not a flaw. But inference is inconsistent by nature: assembled differently each time, constrained by what the agent happened to discover, and disconnected from any authoritative representation of how the system actually fits together. It is also expensive to repeat at scale. Cost accumulates as every agent reconstructs the same understanding from scratch across the same repositories again and again. Agent-driven development can be unrelenting. It’s producing more change faster than any point in the history of software development. A team’s sprint becomes an afternoon for a fleet of agents. Increasingly, those agents operate in coordinated fleets—factories of agents—working across repositories simultaneously. Change is coming faster than existing review processes, governance frameworks, and institutional oversight were designed to absorb—faster than any shared understanding of the system can keep up. When agents validate the work of agents without a current, shared structural representation of the estate, uncertainty scales with the velocity. The implication is not that agents should slow down. It is that the systems they operate within must improve. Expanding what agents can know— reducing the need for inference and grounding their decisions in an accurate, shared representation of the software estate—becomes essential as their autonomy increases. Organizations must also maintain the ability to

📄 Page 13

see what is happening across the estate, understand the cumulative effect of thousands of agent-driven decisions, and intervene when the system drifts. The challenge of the AI era is building the infrastructure that enables: agent efficiency and accuracy, as well as meaningful oversight of the systems that AI is changing.

📄 Page 14

What This Book is About What would it look like to actually know your code definitively? It’s not access to more dashboards, inventories, or scans. It’s not agents reconstructing context with every repository. It’s a representation of software systems with enough fidelity to describe how they actually behave across repositories. The more accurate the model of the software estate, the more reliably both teams and their agents can understand what they are changing and why. This book is written for engineering leaders, architects, and senior engineers responsible for large software estates who need to think clearly about what the AI era demands of them. We will cover the semantic models and analysis techniques that make it possible to reason accurately about large codebases—and how they unlock automated transformation at scale, advanced program analysis, and new visibility into code quality and testing gaps. We’ll also share how this same intelligence is what will supercharge coding agents, addressing the foundational challenge that has always limited what engineering organizations can achieve: the gap between what they build and what they actually understand. The pace of change in AI means some details may age quickly. The underlying shift in how software is understood, analyzed, and changed at scale will not.

📄 Page 15

Chapter 2. A New Operating Model for Software A NOTE FOR EARLY RELEASE READERS With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles. This will be the 2nd chapter of the final report. The challenge of the AI era is not writing code faster. It is maintaining an accurate understanding of code in motion at the speed and scale that agents make possible. Treating the codebase as data—structured, semantic, scaled, and queryable—is the way forward, and in this chapter we’ll introduce you to the code representation that makes it possible: the Lossless Semantic Tree (LST). The Lossless Semantic Tree Code Model The LST, which is the technology that underpins OpenRewrite as well as Moderne’s platform and agent tools, is built for working on code. It helps developers and coding agents understand and improve code more accurately and safely by preserving all the important details about how the code is organized and how its parts interact. The name is its specification: lossless, meaning nothing is discarded; semantic, meaning types and symbols are fully resolved; and tree, meaning it’s structured and traversable programmatically.

📄 Page 16

Most static analysis tools resolve structure. Some may resolve symbols— binding names to their declarations within a project. The LST goes further: it achieves compiler-accurate type resolution, meaning it resolves types, generics, and dependencies across the full classpath with the same fidelity the compiler brings to producing bytecode. This is the level at which meaning becomes unambiguous. What this looks like in practice: every node in the LST carries not just its syntactic identity but its full semantic context as shown in Figure 2-1. A method call node knows which library’s method it refers to, including the fully-qualified type that owns it, the resolved return type in this specific calling context, and the resolved types of every argument passed to it. A class declaration node knows every interface it implements, its full supertype chain, and every member it declares. Figure 2-1. The LST’s three layers: markers for annotation, syntax for structure, and type attribution for fully-resolved meaning—illustrated with a simple logger.info call Formatting—whitespace, comments, style—is also preserved alongside every node, so that when a recipe modifies the tree and prints it back to source, the output is indistinguishable from the original source.

📄 Page 17

Returning to the vulnerability question in Chapter 1: when the LST encounters logger.info("Payment received"), it knows exactly which library’s Logger class owns that method—SLF4J, Log4j, Logback, or Java Commons Logging. They all expose a method named info. Without type attribution, they are indistinguishable as text. With it, every call site carries its full identity. This is the difference between flagging every service that mentions a library and identifying the specific call sites that invoke the vulnerable method—then applying a safe fix for each one. One LST property worth naming explicitly: it represents the limits of its knowledge honestly. When a type cannot be resolved—say because a dependency is genuinely missing from the classpath—the LST holds a placeholder rather than fabricating an answer. When an entire file cannot be parsed, it acknowledges the file’s existence, which can signal a potential issue. Knowing precisely what you don’t know is as useful as knowing what you do: it tells you exactly where the gaps are, so they can be addressed deliberately. Now let’s see how the LST is put into practice. The LST at Work: From Data to Action The LST is built for two realities of modern software: enterprise scale and agent-driven development. Understanding how it works explains why it is fit for both. Figure 2-2 provides an overview from building to using LSTs.

📄 Page 18

(This page has no text content)

📄 Page 19

Figure 2-2. Putting LSTs to work for your codebase: from build to knowledge and transformation Building LST Artifacts An LST artifact reflects the full contents of a repository—produced by guiding the compiler through the first two phases of compilation, syntactic and semantic analysis, to produce full type attribution. The result is a complete, compiler-accurate model of the code: every type resolved, every dependency traversed, every formatting detail preserved. LST files can be built in bulk as well as incrementally to support real-time coding. Incremental builds enable only what changed to be re-parsed and re- resolved, without invalidating the rest of the stored model. A commit that modifies three files in one service triggers re-resolution of those files and the affected portions of the semantic graph—the rest of the portfolio remains available for querying immediately. Crucially, the LST supports random access of files during the build itself: the portions of the model already built are available for search, analysis, and transformation before the full build completes. Serializing the LST for Portfolio-Scale Work LSTs are built per repository, but they become far more useful when they are serialized to disk—storing them as a byproduct of the normal compilation process, alongside compiled bytecode—versioned and retrievable in your artifact repository. This transforms LSTs from ephemeral build artifacts into persistent, queryable assets. It’s the difference between a database that exists only while a query runs and one that persists between queries. Consider a shared internal library with an API change. Which consuming services—spread across dozens of repositories owned by different teams— are affected, and which call sites need to change? No single repository’s LST can answer that, but the serialized portfolio of LSTs can, in minutes. This is what makes portfolio-scale analysis fast: parse once, query many times.

📄 Page 20

The Recipe: LST Query & Transformation Language A recipe is a reusable program that traverses the LST, visits nodes matching a pattern, and either marks them (in the case of a search) or transforms them (in the case of a remediation). Unlike ad hoc scripts or probabilistic AI suggestions, recipes are deterministic: the same input always produces the same output. Recipes can run across multiple LSTs at once, searching, analyzing, and transforming code simultaneously across repositories. Recipes can be simple: find every usage of a deprecated method, bump a dependency version, or enforce a naming convention across a repository. And they are composable, combining recipes into complex workflows. A framework migration that requires thousands of coordinated changes across dozens of repositories is expressed as a sequence of smaller, testable transformations rather than a single monolithic script. For example, a Spring Boot 4 migration chains together recipes that update dependencies, rename packages, replace deprecated APIs, and adjust configuration files— each independently testable, together complete. Recipes also emit data tables—structured outputs produced as a byproduct of traversing LSTs. A recipe searching for vulnerable dependencies, for example, simultaneously emits a table of every affected call site, its severity, the minimum fixed version, and its location across the portfolio. These tables are exportable in standard formats and consumable directly by analytics tools, dashboards, and AI agents. From build to query, the LST closes the gap between knowing your codebase in theory and knowing it in practice—continuously, across the entire estate, as it evolves. Code as Data: What the LST Exposes The codebase contains at least seven categories of structured data, each answering different questions—though in practice they overlap and reinforce each other. Let’s take a tour of what becomes answerable once representation is solved.