The Power of Go Tests (John Arundel) (z-library.sk, 1lib.sk, z-lib.sk)

The Power of Go: Tests John Arundel Bitfield Consulting February 13, 2026 © 2022 John Arundel

The Power of Go: Tests Praise for The Power of Go: Tests Introduction 1. Programming with confidence Self-testing code The adventure begins Verifying the test Running tests with go test Using cmp.Diff to compare results New behaviour? New test. Test cases Adding cases one at a time Quelling a panic Refactoring Well, that was easy Sounds good, now what? 2. Tools for testing Go’s built-in testing facilities Writing and running tests Interpreting test output The “magic package”: testing functions that don’t exist Validating our mental models Concurrent tests with t.Parallel Failures: t.Error and t.Errorf Abandoning the test with t.Fatal Writing debug output with t.Log Test flags: -v and -run Assistants: t.Helper t.TempDir and t.Cleanup Tests are for failing Detecting useless implementations Feeble tests Comparisons: cmp.Equal and cmp.Diff 3. Communicating with tests

Tests capture intent Test names should be sentences Failures are a message to the future Are we testing all important behaviours? The power of combining tests Reading tests as docs, with gotestdox Definitions: “does”, not “should” A sentence is about the right size for a unit Keeping behaviour simple and focused Shallow abstractions: “Is this even worth it?” Don’t worry about long test names Crafting informative failure messages Exercising failure messages Executable examples 4. Errors expected Ignoring errors is a mistake Unexpected errors should stop the test Error behaviour is part of your API Simulating errors Testing that an error is not nil String matching on errors is fragile Sentinel errors lose useful information Detecting sentinel errors with errors.Is Wrapping sentinel errors with dynamic information Custom error types and errors.As Conclusions 5. Users shouldn’t do that Exhaustive testing Constructing effective test inputs User testing Crafting bespoke bug detectors Table tests and subtests Table tests group together similar cases Using dummy names to mark irrelevant test inputs Outsourcing test data to variables or functions Loading test data from files Readers and writers versus files

The filesystem abstraction Using t.TempDir for test output with cleanup Managing golden files Dealing with cross-platform line endings What about the inputs you didn’t think of? 6. Fuzzy thinking Generating random test inputs Randomly permuting a set of known inputs Property-based testing Fuzz testing The fuzz target Running tests in fuzzing mode Failing inputs become static test cases Fuzzing a risky function Adding training data with f.Add A more sophisticated fuzz target Using the fuzzer to detect a panic Detecting more subtle bugs What to do with fuzz-generated test cases Fixing the implementation 7. Wandering mutants What is test coverage? Coverage profiling with the go tool Coverage is a signal, not a target Using “bebugging” to discover feeble tests Detecting unnecessary or unreachable code Automated mutation testing Finding a bug with mutation testing Running go-mutesting Introducing a deliberate bug Interpreting go-mutesting results Revealing a subtle test feebleness Fixing up the function Mutation testing is worth your while 8. Testing the untestable Building a “walking skeleton” The first test

Solving big problems Designing with tests Unexported functions Concurrency Concurrency safety Long-running tasks User interaction Command-line interfaces 9. Flipping the script Introducing testscript Running programs with exec Interpreting testscript output The testscript language Negating assertions with the ! prefix Passing arguments to programs Testing CLI tools with testscript Checking the test coverage of scripts Comparing output with files using cmp More matching: exists, grep, and -count The txtar format: constructing test data files Supplying input to programs using stdin File operations Differences from shell scripts Comments and phases Conditions Setting environment variables with env Passing values to scripts via environment variables Running programs in background with & The standalone testscript runner Test scripts as issue repros Test scripts as… tests Conclusion 10. Dependence day Just don’t write untestable functions Reduce the scope of the dependency Be suspicious of dependency injection Avoid test-induced damage

“Chunk” behaviour into subcomponents Reassemble the tested chunks Extract and isolate the key logic Isolate by using an adapter Example: a database adapter Fakes, mocks, stubs, doubles, and spies Don’t be tempted to write mocks Turn time into data 11. Suite smells No tests Legacy code Insufficient tests Ineffective code review Optimistic tests Persnickety tests Over-precise comparisons Too many tests Test frameworks Flaky tests Shared worlds Failing tests Slow tests A fragrant future About this book Cover photo Who wrote this? Feedback Get my free newsletter Free updates to future editions The Deeper Love of Go The Power of Go: Tools Know Go Further reading Credits Acknowledgements

Praise for The Power of Go: Tests Brilliant. I read it with genuine pleasure. Well written, clear, concise, and effective. —Giuseppe Maxia I’d happily pay three times the price for John’s books. —Jakub Jarosz The writing style is engaging, friendly, informative, and snappy. —Kevin Cunningham John’s books are so packed with information, I learn something new every time I re-read them. —Miloš Žižić A great read—it’s a treasure trove of knowledge on not just testing, but software design in general. Best of all, it’s completely language- agnostic. —Joanna Liana I really enjoyed this book. The humour makes learning Go a lot more fun. —Sean Burgoyne This could get someone fired! —David Bailey The best introduction to mutation testing. John’s writing style made me smirk, smile and vigorously nod my head as I was reading. —Manoj Kumar

Introduction When given the bug report, the developer responded, shaking his head vigorously, “Oh. That’s not a bug.” “What do you mean, ‘that’s not a bug’? It crashes.” “Look,” shrugged the developer. “It can crash or it can corrupt your data. Take your pick.” —Gerald Weinberg, “Perfect Software: And Other Illusions About Testing” Programming is fun, even if it can sometimes feel a bit painful, like pounding nails into your head—and if it does, don’t worry, it’s not just you. We all feel that way from time to time, and it’s not because we’re bad programmers. It’s usually because we’re trying to understand or untangle some complicated code that isn’t working properly, and that’s one of the hardest jobs a programmer will ever do. How do you avoid getting into a tangle like this in the first place?

Another not-so-fun thing is building large and complex projects from scratch, when we’re running around trying to design the thing, figure out the requirements, and write the code, all without turning it into a big pile of spaghetti. How the heck do you do that? The two most difficult problems to solve in software engineering are verification—did I build the system right?—and validation—did I build the right system? If we can’t figure out how to correctly answer these two questions, we’ve no business producing software in the first place. But we’ve learned a trick or two over the years. We’ve developed a way of programming that helps us organise our thinking about the product, guides us through the tricky early stages, leads us towards good designs, gives us confidence in the correctness of what we’re doing, catches bugs before users see them, and makes it easier and safer for us to make changes in the future. It’s a way of programming that’s more enjoyable, less stressful, and more productive than any other that I’ve come across. It also produces much better results. I’ll share some of these insider secrets with you in this book. Before we start, though, please take a moment to subscribe to my newsletter —it’s free, and it’s also the best way for you to stay up to date with my latest books, tutorials, and blog posts: https://bitfieldconsulting.com/subscribe

1. Programming with confidence It seemed that for any piece of software I wrote, after a couple of years I started hating it, because it became increasingly brittle and terrifying. Looking back in the rear-view, I’m thinking I was reacting to the experience, common with untested code, of small changes unexpectedly causing large breakages for reasons that are hard to understand. —Tim Bray, “Testing in the Twenties” When you launch yourself on a software engineering career, or even just a new project, what goes through your mind? What are your hopes and dreams for the software you’re going to write? And when you look back on it after a few years, how will you feel about it? There are lots of qualities we associate with good software, but undoubtedly the most important is that it be correct. If it doesn’t do what it’s supposed to, then almost nothing else about it matters.

Self-testing code How do we know that the software we write is correct? And, even if it starts out that way, how do we know that the minor changes we make to it aren’t introducing bugs? One thing that can help give us confidence about the correctness of software is to write tests for it. While tests are useful whenever we write them, it turns out that they’re especially useful when we write them first. Why? The most important reason to write tests first is that, to do that, we need to have a clear idea of how the program should behave, from the user’s point of view. There’s some thinking involved in that, and the best time to do it is before we’ve written any code. Why? Because trying to write code before we have a clear idea of what it should do is simply a waste of time. It’s almost bound to be wrong in important ways. We’re also likely to end up with a design which might be convenient from the point of view of the implementer, but that doesn’t necessarily suit the needs of users at all. Working test-first encourages us to develop the system in small increments, which helps prevent us from heading too far down the wrong path. Focusing on small, simple chunks of user-visible behaviour also means that everything we do to the program is about making it more valuable to users. Tests can also guide us toward a good design, partly because they give us some experience of using our own APIs, and partly because breaking a big program up into small, independent, well-specified modules makes it much easier to understand and work on. What we aim to end up with is self-testing code: You have self-testing code when you can run a series of automated tests against the code base and be confident that, should the tests pass, your code is free of any substantial defects.

One way I think of it is that as well as building your software system, you simultaneously build a bug detector that’s able to detect any faults inside the system. Should anyone in the team accidentally introduce a bug, the detector goes off. —Martin Fowler, “Self-Testing Code” This isn’t just because well-tested code is more reliable, though that’s important too. The real power of tests is that they make developers happier, less stressed, and more productive as a result. Tests are the Programmer’s Stone, transmuting fear into boredom. “No, I didn’t break anything. The tests are all still green.” The more stress I feel, the more I run the tests. Running the tests immediately gives me a good feeling and reduces the number of errors I make, which further reduces the stress I feel. —Kent Beck, “Test-Driven Development by Example” The adventure begins Let’s see what writing a function test-first looks like in Go. Suppose we’re writing an old-fashioned text adventure game, like Zork, and we want the player to see something like this: Attic The attics, full of low beams and awkward angles, begin here in a relatively tidy area which extends north, south and east. You can see here a battery, a key, and a tourist map. Adventure games usually contain lots of different locations and items, but one thing that’s common to every location is that we’d like to be able to list its contents in the form of a sentence: You can see here a battery, a key, and a tourist map. Suppose we’re storing these items as strings, something like this: a battery a key

a tourist map How can we take a bunch of strings like this and list them in a sentence, separated by commas, and with a concluding “and”? It sounds like a job for a function; let’s call it ListItems. What kind of test could we write for such a function? You might like to pause and think about this a little. One way would be to call the function with some specific inputs (like the strings in our example), and see what it returns. We can predict what it should return when it’s working properly, so we can compare that prediction against the actual result. Here’s one way to write that in Go, using the built-in testing package: (Listing game/1) Don’t worry too much about the details for now; we’ll deal with them later. The gist of this test is as follows: func TestListItems_GivesCorrectResultForInput(t *testing.T) { t.Parallel() input := []string{ "a battery", "a key", "a tourist map", } want := "You can see here a battery, a key, and a tourist map." got := game.ListItems(input) if want != got { t.Errorf("want %q, got %q", want, got) } }

1. We call the function game.ListItems with our test inputs. 2. We check the result against the expected string. 3. If they’re not the same, we call t.Errorf, which causes the test to fail. Note that we’ve written this code as though the game.ListItems function already exists. It doesn’t. This test is, at the moment, an exercise in imagination. It’s saying if this function existed, here’s what we think it should return, given this input. But it’s also interesting that we’ve nevertheless made a number of important design decisions as an unavoidable part of writing this test. First, we have to call the function, so we’ve decided its name (ListItems), and what package it’s part of (game). We’ve also decided that its parameter is a slice of strings, and (implicitly) that it returns a single result that is a string. Finally, we’ve encoded the exact behaviour of the function into the test (at least, for the given inputs), by specifying exactly what the function should produce as a result. The original description of test-driven development was in an ancient book about programming. It said you take the input tape, manually type in the output tape you expect, then program until the actual output tape matches the expected output. When describing this to older programmers, I often hear, “Of course. How else could you program?” —Kent Beck Naming something and deciding its inputs, outputs, and behaviour are usually the hardest decisions to make about any software component, so even though we haven’t yet written a single line of code for ListItems, we’ve actually done some pretty important thinking about it. And the mere fact of writing the test has also had a significant influence on the design of ListItems, even if it’s not very visible. For example, if we’d just gone ahead and written ListItems first, we might well have made it print the result to the terminal. That’s fine for the real game, but it would be difficult to test.

Testing a function like ListItems requires decoupling it from some specific output device, and making it instead a pure function: that is, a function whose result is deterministic, depends on nothing but its inputs, and has no side-effects. Functions that behave like this tend to make a system easier to understand and reason about, and it turns out that there’s a deep synergy between testability and good design, which we’ll return to later in this book. Verifying the test So what’s the next step? Should we go ahead and implement ListItems now and make sure the test passes? We’ll do that in a moment, but there’s a step we need to take first. We need some feedback on whether the test itself is correct. How could we get that? It’s helpful to think about ways the test could be wrong, and see if we can work out how to catch them. Well, one major way the test could be wrong is that it might not fail when it’s supposed to. Tests in Go pass by default, unless you explicitly make them fail, so a test function with no code at all would always pass, no matter what: That test is so obviously useless that we don’t need to say any more. But there are more subtle ways to accidentally write a useless test. For example, suppose we mistakenly wrote something like this: A value always equals itself, so this if statement will never be true, and the test will never fail. We might spot this just by looking at the code, but then func TestAlwaysPasses(t *testing.T) {} if want != want { t.Errorf("want %q, got %q", want, got) }

again we might not. I’ve noticed that when I teach Go to my students, this is a concept that often gives them trouble. They can readily imagine that the function itself might be wrong. But it’s not so easy for them to encompass the idea that the test could be wrong. Sadly, this is something that happens all too often, even in the best-written programs. Until you’ve seen the test fail as expected, you don’t really have a test. So we can’t be sure that the test doesn’t contain logic bugs unless we’ve seen it fail when it’s supposed to. When should the test fail, then? When ListItems returns the wrong result. Could we arrange that? Certainly we could. That’s the next step, then: write just enough code for ListItems to return the wrong result, and verify that the test fails in that case. If it doesn’t, we’ll know we have a problem with the test that needs fixing. Writing an incorrect function doesn’t sound too difficult, and something like this would be fine: Almost everything here is dictated by the decisions we already made in the test: the function name, its parameter type, its result type. And all of these need to be there in order for us to call this function, even if we’re only going to implement enough of it to return the wrong answer. The only real choice we need to make here, then, is what actual result to return, remembering that we want it to be incorrect. What’s the simplest incorrect string that we could return given the test inputs? Just the empty string, perhaps. Any other string would also be fine, func ListItems(items []string) string { return "" }

provided it’s not the one the test expects, but an empty string is the easiest to type. Running tests with go test Let’s run the test and check that it does fail as we expect it to: go test --- FAIL: TestListItems_GivesCorrectResultForInput (0.00s) game_test.go:18: want "You can see here a battery, a key, and a tourist map.", got "" FAIL exit status 1 FAIL game 0.345s Reassuring. We know the function doesn’t produce the correct result yet, so we expected the test to detect this, and it did. If, on the other hand, the test had passed at this stage, or perhaps failed with some different error, we would know there was a problem. But it seems to be fine, so now we can go ahead and implement ListItems for real. Here’s one rough first attempt: (Listing game/1) I really didn’t think too hard about this, and I’m sure it shows. That’s all right, because we’re not aiming to produce elegant, readable, or efficient code at this stage. Trying to write code from scratch that’s both correct and func ListItems(items []string) string { result := "You can see here" result += strings.Join(items, ", ") result += "." return result }

elegant is pretty hard. Let’s not stack the odds against ourselves by trying to multi-task here. In fact, the only thing we care about right now is getting the code correct. Once we have that, we can always tidy it up later. On the other hand, there’s no point trying to beautify code that doesn’t work yet. The goal right now is not to get the perfect answer but to pass the test. We’ll make our sacrifice at the altar of truth and beauty later. —Kent Beck, “Test-Driven Development by Example” Let’s see how it performs against the test: --- FAIL: TestListItems_GivesCorrectResultForInput (0.00s) game_test.go:18: want "You can see here a battery, a key, and a tourist map.", got "You can see herea battery, a key, a tourist map." Well, that looks close, but clearly not exactly right. In fact, we can improve the test a little bit here, to give us a more helpful failure message. Using cmp.Diff to compare results Since part of the result is correct, but part isn’t, we’d actually like the test to report the difference between want and got, not just print both of them out. There’s a useful third-party package for this, go-cmp. We can use its Diff function to print just the differences between the two strings. Here’s what that looks like in the test: func TestListItems_GivesCorrectResultForInput(t *testing.T) { t.Parallel() input := []string{ "a battery", "a key", "a tourist map", }

(Listing game/2) Here’s the result: --- FAIL: TestListItems_GivesCorrectResultForInput (0.00s) game_test.go:20: strings.Join({ "You can see here", - " ", "a battery, a key,", - " and", " a tourist map.", }, "") When two strings differ, cmp.Diff shows which parts are the same, which parts are only in the first string, and which are only in the second string. According to this output, the first part of the two strings is the same: "You can see here", But now comes some text that’s only in the first string (want). It’s preceded by a minus sign, to indicate that it’s missing from the second string, and the exact text is just a space, shown in quotes: - " ", So that’s one thing that’s wrong with ListItems, as detected by the test. It’s not including a space between the word “here” and the first item. The next part, though, ListItems got right, because it’s the same in both want and got: want := "You can see here a battery, a key, and a tourist map." got := game.ListItems(input) if want != got { t.Error(cmp.Diff(want, got)) } }