The Power of Go Tools (John Arundel) (z-library.sk, 1lib.sk, z-lib.sk)

The Power of Go: Tools John Arundel Bitfield Consulting February 13, 2026 © 2023 John Arundel

The Power of Go: Tools Praise for ‘The Power of Go: Tools’ Introduction Who is this book for? What should I know before reading it? How do I install Go? Where to find the code examples What you’ll learn 1. Packages The universal library Packages are a force multiplier The universal Go library is huge Sharing our code benefits us all Writing packages, not programs Command-line tools Zen mountaineering Guided by tests Building a hello package The structure of a test Tests are bug detectors So when should a test fail? Where does fmt.Println print to? Meet bytes.Buffer, an off-the-shelf io.Writer Failure standards Implementing hello, guided by tests We start with a failing test Creating a module One folder, one package A null implementation The real implementation The naming of tests Refactoring to use our new package Going further 2. Paperwork

Making our package more flexible Mandatory arguments are annoying What if we allow users to pass nil? Maybe some global variable instead? A struct is the answer A convenience wrapper with defaults A simple line counter Focus on behaviour, not implementation One possible first version What about configuration? Config structs don’t solve the problem An elegant option API Okay, so what’s an “option”? Options are functions “Always valid” fields Some internal paperwork Handling internal errors A line counter with options Methodical options Going further 3. Arguments Designing the behaviour Deciding the scope Testing CLI arguments A first attempt Test data files Creating the test data Using the data in a test The failing test Implementing file-reading Empty slice checking Testing the “no args” behaviour Closing files and other resources Updating the user interface Some user testing Test scripts Running the program as a binary

Introducing testscript Invoking test scripts Defining custom commands The delegate Main function The stdout assertion Testing arguments Going further 4. Flags Commands The Unix way Multiple main packages A words command A test for word counting Updating the test scripts Flags Introducing flags Adding a -lines flag Implementing the behaviour Help and usage information Going further 5. Files Writing to files The art of judicious logging Testing a WriteToFile function Designing errors out of existence Looking for inspiration What are we really testing? The go-cmp module Implementing a WriteToFile function File permissions When the directory doesn’t exist A disastrous bug Using t.TempDir Finishing the job It’s clobbering time Ensuring permissions What’s the worst that could happen?

A security leak Going further 6. Filesystems Files and filesystems What even is a file? Organising files A simple file finder Handling folders recursively Filesystems and io/fs Matching files by name Walking the tree A file-finding tree-walker Starting at the top The fs.FS abstraction Any path-value map is a “filesystem” The fstest.MapFS filesystem Adding a filesystem to our API Timing potentially slow operations Writing benchmark functions Taking fs.FS makes APIs more flexible Going further 7. Commands The exec package What even is a process? Managing command output When not to use exec When to use exec Migrating from shell scripts to Go Why use Go to run commands? A command wrapper in Go The pmset command What can we test? Breaking down behaviour into chunks Parsing command output Testing the parsing function Parsing command output Clarifying the problem

Writing a regular expression Using the regexp Integration tests Why isolate integration tests? Build tags Testing the command runner Running the command Capturing output When to import a third-party package Going further 8. Shells A simple shell Defining some behaviour Identifying the first test Comparing the incomparable Parsing user input Prototyping A pseudocode outline Main-driven development Feedback from user testing A stateful shell session What would we like to write? What object would make sense? Designing the Session object Running a session Testing the Run method Dependencies on external commands A dry-run mode Implementing Run What’s still missing Globbing Redirection Piping Quoting Going further 9. Pipelines A realistic operations task

Matching and counting log lines A quick shell spell Solving the problem with Go In what language would this be easy? Programs as pipelines A fluent API Errors in sequenced operations An error-safe writer Putting the pieces together How does data flow from one method to another? Testing the end of the pipeline Breaking ground Getting the test passing Adding error safety Obviousness-oriented programming Trying it out Hello, world The world strikes back Setting defaults with a constructor Reading data from files Another pipeline explosion Filtering data Extracting columns A String sink Testing Column Validating arguments Implementing Column The script package Was this a waste of time? Introducing script Some simple one-liners More sophisticated programs Concurrent pipeline stages Custom filter functions Solving problems The landscape of simple programs Going further

10. Data Marshalling Serialisation The gob package A file-based data store Testing data persistence Setters and getters A sensible key-value API Testing the key-value machinery Implementing the store Adding persistence What are we really testing here? An end-to-end persistence test Saving and loading Tightening up the tests JSON: a text data format The json package JSON is a useful auxiliary language A JSON output option Adding JSON to the battery package Producing JSON strings What could go wrong? Testing ToJSON Implementing ToJSON Pretty-printing with indentation Querying JSON output YAML: a less verbose JSON Parsing YAML in Go Designing the API with a test Defining our types The go-yaml package Decoding When unmarshalling doesn’t work The format of struct tags Setting defaults Eliminating config structs Going further

11. Clients A simple weather client What do we need? Kicking the tyres Environmental credentials Making a GET request Initial user testing A second pass Parsing JSON responses Testing ParseResponse To decode or to unmarshal? A temporary struct type The response struct Implementing ParseResponse What could go wrong? Other kinds of invalid data What are we really testing here? Back to a running program Constructing the request URL A FormatURL function Getting the location as input Refactoring the remaining code A paperwork-reducing GetWeather function Testing against a local HTTP server A simple httptest example A trustful TLS client A weather client object A familiar pattern Refactoring the tests to use our client Writing the client constructor Refactoring GetWeather as a client method Testing GetWeather Implementing GetWeather A convenience wrapper Adding a Main function Adding temperature support Parsing temperature data

More user testing Handling quantities with units Presenting temperatures in Celsius Defining a Temperature type Tackling more complex APIs Request data “CRUD” methods Last words Going further About this book Who wrote this? Feedback Get my free newsletter Free updates to future editions The Deeper Love of Go The Power of Go: Tests Know Go Further reading Credits Acknowledgements A sneak preview 1. Programming with confidence Self-testing code The adventure begins Verifying the test Running tests with go test Using cmp.Diff to compare results New behaviour? New test. Test cases Adding cases one at a time Quelling a panic Refactoring Well, that was easy Sounds good, now what?

Praise for ‘The Power of Go: Tools’ Curse you for derailing my day with another fascinating book! The content is absolutely awesome. —Peter Nunn Superb and well written: all the examples worked and were very helpful. —Lee Gibson It’s fantastic! I really like the approach. —Sal DiStefano The book does a great job of teaching what to do with Go. I really liked the emphasis on testing. —Pedro Sandoval Exactly the book I was looking for next! I love it. —Elliot Thomas What I really love about this book is it takes the newbie to the next level via real-world, relatable examples. The narrative flow clicks with me. —Rajaseelan Ganeswaran It’s clear, concise, and practical, especially for folks like me who enjoy the art of making end-user tools with Go. It’s really made me think when I write code about how simple and easy I can make it for folks to use, without, as John likes to say, ‘a lot of paperwork’. —Josh Feierman Everywhere I go, I’m asked if I think the university stifles writers. My opinion is that they don’t stifle enough of them. —Flannery O’Connor

Introduction The guiding motto in the life of every natural philosopher should be, “Seek simplicity and distrust it.” —Alfred North Whitehead, “The Concept of Nature” Hello, and welcome to the book! It’s great to have you here.

Who is this book for? There are lots of books that will teach you Go, but not many that will show you what to do with it. In other words, once you’ve learned how to write Go code, what code should you write? This book is aimed at those who have a little experience with Go (or even a lot), and would now like to learn how to build good software with it. What is “good” software anyway? What would it look like in Go? And how do we get there from here? If software engineering is a craft, which it surely is, then how do we go about mastering it? It’s all very well to say “just write programs”, but how? How do we take some problem and start designing a program to solve it? How can we incorporate tests into the design? What are we even aiming to do here? I hope you’ll find at least some useful answers to these questions in this book, which focuses on developing command-line tools, but most of it applies to any kind of Go program. Before we start, though, please take a moment to subscribe to my newsletter —it’s free, and it’s also the best way for you to stay up to date with my latest books, tutorials, and blog posts: https://bitfieldconsulting.com/subscribe What should I know before reading it? While you don’t need to be a confident or expert Go programmer, I’ll assume in this book that you’re familiar with at least the basics: compiling and running Go programs, how structs and slices work, what functions do, and so on. If you’re entirely new to Go, or even to programming in general, I recommend you read my previous book, “The Deeper Love of Go”, first. Once you’ve read it, you’ll be in the ideal place to start reading this book. Go ahead! I’ll wait right here until you come back.

How do I install Go? If Go isn’t yet available as a package in your operating system distribution, you can build it from source or download a suitable binary package from the Go website directly: https://go.dev/learn/ Where to find the code examples There are dozens of challenges for you to solve throughout the book, each designed to help you test your understanding of the concepts you’ve just learned. Throughout the book, you’ll see a number of code goals for you to achieve, marked with the word GOAL, like this: GOAL: Get the test passing. When you see this, stop reading at that point and see if you can figure out how to solve the problem. You can try to write the code and check it against the tests, or just think about what you would do. If you reckon you have the answer, or alternatively if you’ve got a bit stuck and aren’t sure what to do, you can then read on for a HINT, or read on even further for step-by-step instructions on how to construct the SOLUTION. If you run into trouble, or just want to check your code, each challenge is accompanied by a complete sample solution, with tests. All these solutions are also available in a public GitHub repo here: https://github.com/bitfield/tpg-tools2 Each listing in the book is accompanied by a name and number (for example, listing hello/1), and you’ll find the solution to that exercise in the

corresponding folder of the repo. What you’ll learn By reading through this book and completing the exercises, you’ll learn: How to build reusable packages instead of one-off programs How to design user-friendly APIs and packages, without annoying paperwork How to write robust, testable tools that take command-line flags and arguments How to design Go packages that work with files and other kinds of streaming data How to write flexible tools to operate on trees of files, and more generally path-value databases such as URLs or zip archives How to use Go to drive external commands and provide elegant APIs to abstract their functionality How to write commands that interact extensively with users, such as shells How to sequence operations into simple pipelines that abstract away the details of handling streaming data and errors How to encode and decode data in binary format, and translate Go data to and from JSON and YAML formats How to create robust, reusable client packages for HTTP services and other APIs

1. Packages When it was announced that the Library contained all books, the first reaction was unbounded joy. All… felt themselves the possessors of an intact and secret treasure. There was no personal problem, no world problem, whose eloquent solution did not exist—somewhere… —Jorge Luis Borges, “The Library of Babel” The universal library What’s wrong with this program? package main import ( "fmt" ) func main() { fmt.Println("Hello, world") }

(Listing hello/1) If you looked in vain, unable to spot the bug, I don’t blame you. The fact is, there’s nothing wrong with this program as such. It works, to the extent that it does what the author intended: it prints a message to the terminal, and exits. But there are some limitations on what we can do with this program. The most serious of these is that it’s not an importable package. Let’s talk about why this is a big deal. The earliest computers were single-purpose machines. They could only execute the specific computation they were designed for: the trajectory of an artillery shell, say. To compute something different meant physically re- wiring the machine. It took a leap of insight to realise that a much more useful kind of computer would be one that could execute any computation, without being re-wired. That is to say, it would be programmable. Packages are a force multiplier A further significant advance was the idea that we don’t need to write every program from scratch every time. Instead, we could create reusable “routines”. For example, the code to calculate square roots is complicated and easy to get wrong, but it only has to be written once. We can then copy and use that routine in any program that needs to take a square root. Nowadays, we would call such independent chunks of software modules, or packages, and they’re fundamental to all modern programming. Without packages we would always have to instruct the computer about every detail of what we want to do. It would be hard to write useful programs in Go without the packages in the standard library, for example. Packages, in other words, are an immensely powerful force multiplier. When we’re programming in a language like Go that has a rich ecosystem of importable packages, we never have to reinvent the wheel.

If we can figure out how to break down our unsolved problems into a bunch of mini-problems that have already been solved by existing packages, then we’re 90% done. The universal Go library is huge The Go language itself is pretty small, in the sense that there aren’t many keywords and there’s not a vast amount of syntax to learn. And that’s great news for those of us learning it. But since this little language ships with a big standard library, full of all sorts of useful and well-designed packages, we can use it to construct some really powerful programs right away. That’s nice, but are we restricted to importing only packages from the standard library? Certainly not. We can import any published package in the world, and there are plenty available. We might call this wider ecosystem of importable Go packages the universal library. A quick search on GitHub reveals something close to half a million packages in this universal library (and there are more published in other places). If you can imagine it, in other words, there’s probably a Go package that provides it. So many, indeed, that simply finding the package you want can be a challenge. The pkg.go.dev site lets you search and browse the whole universal library: https://pkg.go.dev If you’re looking for a package to tackle some specific problem, a good place to check first is awesome-go, a carefully-curated list of a couple of thousand or so of the very best Go packages, by subject area: https://github.com/avelino/awesome-go This is one of the many reasons that Go is such a popular choice for developing software nowadays. In many cases, all we need to do to create a

particular program is to figure out the right way to connect up the various packages that we need. We can then make our program a package that other people can use, and the process continues like a chain reaction. It’s packages all the way down! Sharing our code benefits us all No program is an island, in other words. But that’s what’s wrong with the “hello, world” program in listing hello/1: it is isolated, breaking the chain of importability. The syntax rules of Go mean that all code has to be in some package, and it happens that ours is in package main. But there’s something special about the main package in Go: it can’t be imported. That makes sense, because it could only be imported into some program that already has a main package, so we’d have a namespace conflict. That means no one else can benefit from our wonderful code. What a shame! We’re taking from the universal package ecosystem, as it were, but not giving anything back. That’s just rude. If our program is worth writing, it’s worth sharing with the millions of other Gophers who also benefit from the universal library. This isn’t just wide-eyed idealism, though there’s nothing wrong with that: it also makes good commercial sense. For us to be able to write software at the scale we need today, it has to be an industrial process using standardised, modular components. Today’s software developer is like the colonial-era gunsmith, lovingly handcrafting every nut and screw. We haven’t yet made an industrial revolution leap from “filing away at software like gunsmiths at iron bars” to “commercially robust repositories of trusted, stable components”. —Scott Rosenberg, “Dreaming in Code” Good programmers, then, are always thinking in terms of writing importable packages, not mere dead-end programs. And it turns out that this