Everyday Golang - Your Fast Track for Golang (Alex Ellis) (Z-Library)

📄 File Format: PDF

💾 File Size: 2.5 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

EVERYDAY GO BY ALEX ELLIS Learn tools, techniques and patterns from real tools used in production. YOUR FAST TRACK FOR GOLANG

📄 Page 2

Everyday Golang - The Fast Track Alex Ellis 2024

📄 Page 3

Contents 1 Introduction 4 How to use the book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Help the book to evolve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Changelog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Why Go? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Getting started with Go . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 A note for Windows users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Direct installation on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Packages and paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Vendoring and Go modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2 Create your first Go program 12 Add an external dependency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Make use of flags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Define a separate package or library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3 Cross-compile for different operating systems 21 Go build arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 CGO and portability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 4 Fetch JSON from a HTTP endpoint 23 Get started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Learn to parse JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Retrieve the JSON via HTTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Extracting additional data from the JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 5 Five ways to write an awesome CLI 30 Pick Go . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Parse flags & arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Automate everything . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Integrate with package managers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Accept contributions and gather feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Now, Go and write your CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 6 Writing unit-tests in Golang 35 Testing opinions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Testing the Sum method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Adding a test table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Party tricks for “go test” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Statement coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Generating an HTML coverage report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Benchmarking your code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Go doesn’t ship your tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Stress testing your tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Running tests in parallel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 1

📄 Page 4

Compile the tests to a binary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Isolating dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Example of isolating dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Unit-testing a HTTP server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 7 Work concurrently work with Goroutines 55 A simple example with Goroutines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 Avoid a common mistake with Goroutines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 Error groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 Limiting requests to expensive resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Sharing data between Goroutines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 Using channels for synchronisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Using a context to handle timeouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Limiting concurrent work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 8 Store data with a database 76 9 Write your own config files in YAML 81 A practical example of YAML in action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Read and write your own YAML schema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Merging objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 10 Inject version information into your binaries 86 Capture useful variable(s) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 Prepare your code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 Do it with Docker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Examples in production . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 11 Embed data in your application 91 12 Create dynamic content with templates 93 Advanced templates: ranges and custom functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Real-world templates - arkade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 13 Write a HTTP server 99 Reading the request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Advanced route handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 Writing HTTP middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 14 Add Prometheus metrics to your microservices 105 Tracking HTTP metrics for our microservice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 Adding custom metrics to your Prometheus data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 The Prometheus Collector interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 15 Building and releasing Go binaries 119 The release-it example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Releasing your binaries with a GitHub Action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 16 Releasing a Docker container image 122 Going multi-arch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Shipping a Docker container from GitHub Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 17 Your onward journey 126 Giving feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 Contributing to Open Source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 My other content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 18 OpenFaaS and Go 127 Topics covered . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 2

📄 Page 5

Serverless use-cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 Introduction to OpenFaaS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 The value vs cloud functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 Pros and Cons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 Where can you run OpenFaaS functions? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 The Go template for OpenFaaS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 Your first function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Building your function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Adding unit tests to your function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 Iterating locally . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 Split code into multiple files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 Introduce a sub-package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 Working with secrets and databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Create the todo-fn example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Exploring the request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 The complete example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 Managing config between multiple environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Futher tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Custom domains and routes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Where to go next . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 3

📄 Page 6

Chapter 1 Introduction “Everyday Go” is the fast way to learn tools, techniques and patterns from real tools used in production. About the author: Alex is a software engineer and an avid technical writer. There’s nothing he likes more than learning a new technology, and then writing it up in a clear, practical way. He’s well known for his blog posts on containers, Kubernetes, cloud computing and Go. He founded the OpenFaaS project in 2016, and since then has gone on to publish training courses, eBooks and dozens of other tools for developers. How to use the book This book is a compilation of practical examples, lessons and techniques for Go developers. They are the kind of things that you may need to learn and apply in your everyday journey. The topics cover the software lifecycle from learning the fundamentals, to software testing, to distribution and monitoring. If you feel like you may have seen some of these techniques before, you would be right. I’ve been writing about Go since 2015 and wanted to take everything that I’ve needed on a regular basis, and to lay it out for you to benefit from. It’s not a reference book or a complete guide to the language, but if you like learning by example and aren’t afraid to get your feet wet, then this style is probably for you. The chapter “Writing unit-tests in Go” was originally part of my top ranking and performing blog post. It was even featured in the Kubernetes documentation to help new Go developers get up to speed. In Everyday Go, I’ve rewritten, updated and extended the tutorial with more content and techniques that you can use in your work. You could say that this is the fast track to get the results you’ll need in your everyday writing of applications in Go. • How to use the book • Why Go? • Getting started with Go • Managing packages and paths • Understand Go modules and vendoring • Creating your first program in Go • Adding external dependencies to your program • Cross-compile your code for different Operating Systems • Inject version information into your binaries • Merge Go objects together into one • Deal with HTTP and JSON parsing • Learn how to use concurrency with Goroutines, channels, Mutexes, error groups and WaitGroups • Avoid a common mistake with Goroutines • Write unit tests and understand everything that’s available to you • Learn party tricks of the testing in Go • Isolate dependencies and unit test a HTTP server • Embed data and files into your application • Create dynamic content with templates 4

📄 Page 7

• Build a CLI that you’ll enjoy using • Load and write to config files in YAML • Write your own HTTP microservices • Integrate with Prometheus for metrics • Learn to build and release Go binaries • Build Docker container images • Release your code on GitHub • Get set for your onward journey Help the book to evolve Just as my knowledge of Go has evolved over time, this book will also evolve with updates and new examples. This book should be a place that you can come to for a pattern or sample that you can use as a basis of a new program or feature. Check the final chapter for how to submit comments, suggestions or corrections. Changelog • 12 Dec 2021 - New “full” edition released with OpenFaaS & Go chapters added • 11 Dec 2021 - Updates for breaking changes in Go 1.17 • 5 August 2021 - Clarify usage of RWMutex and locking behaviour • 29 July 2021 - Fix typo on in a test table example. • 19 July 2021 - Fix typo on X509 certificates. • 14 July 2021 - Correct git clone URL, explain Marshal and Unmarshal, remove an import. • 28 June 2021 - Update wording on concurrency vs parallelism, introduce new locking example and Mu- tex/RWMutex. • 27 June 2021 - Update to Go modules section and clarification of paths in package cache directory • 24 June 2021 - MIT license added for example apps • 24 June 2021 - Add section on multiple packages and new example app “multiple-packages” • 23 June 2021 - Add new section on Go modules usage and vendoring • 22 June 2021 - Update Go release date in intro, add “merge-objects” example app • 8 Nov 2021 - Add line-wrapping for epub version • 23 May 2022 - Added “cd” commands to examples, updated output from hash example • 6 June 2022 - Added “Custom domains and routes” section and example to OpenFaaS supplement • 5 July 2022 - Update Insiders’ Portal domain • 20 July 2022 - Added Singleflight example for concurrency limiting and protecting against the “thundering herd” problem • 13 August 2022 - Added example for Prometheus Collector interface • 9 Jan 2023 - Updated GitHub Actions examples for publishing containers and using GHCR.io • 19 Jan 2023 - Updated version in go.mod files and removed ioutil.ReadFile/WriteFile due to deprecations. Con- verted direct Linux installation to arkade so the version in the URL is always up to date. • 28 Jan 2023 - Removed -installsuffix from go build examples, which is no longer required. Updated an example to Go 1.19. • 16May 2024 - Updated Go version to 1.22 andmin version in examples to go 1.21, update alexellis/upload-assets version to 0.4.1. • 29 Dec 2024 - Updated Go version to 1.23 in eBook and in examples, clarified use of external HMAC library. Fixed the rendering of a list “When done well, a CLI:”. Updated Docker’s GitHub Actions versions. Why Go? According to Wikipedia, Go (or Golang) is a programming language that was developed by a team at Google and first appeared in 2009. It’s synonymous with server-side development and tooling for infrastructure projects. “Go is an open source programming language that enables the production of simple, efficient and reliable software at scale” 5

📄 Page 8

Figure 1.1: Go’s new logo • Go branding guidelines Here are some projects that come to mind, which are written entirely (or have major portions) in Go: • Docker - a container engine written by Docker Inc • Kubernetes - a clustering system for containers built by Google and open source contributors • Traefik - a load-balancer built by Traefik Labs • Istio - a service mesh to encrypt traffic between Kubernetes Pods • Consul, Terraform - tools for managing infrastructure built by Hashicorp • Caddy - a high performance HTTP server written by Matt Holt • OpenFaaS - serverless framework for hosting your own functions, built by myself and various other contributors These are just a few of hundreds of thousands of projects which have appeared on GitHub’s trending page for Go. As you can see from the above, Go is very popular for server-side projects, but it is also very popular forwriting Command Line Interfaces (CLIs) because the binaries can be statically linked. Statically-linked Go binaries are both fast and small, and do not require an additional runtime making them a very popular choice. My personal highlights for Go are: • portability - static compilation for easy distribution • cross-compilation - being able to build binaries for Linux, Windows, MacOS Operating Systems (OS) and Arm computers with a single command (go build) • concurrency model - Go has built-in mechanisms for concurrency that make it well suited to tasks both small and large • ecosystem - knowing Go means being able to contribute to almost any other open source Go project or product • tooling - Go includes its own unit-testing runner and profiler, both of which are simple and easy to use • opinions - Go has strong opinions on formatting, style and typing that make it easy to move between codebases Above everything else, Go has an impressive set of packages available within its standard libraries or “stdlib”. There are many useful packages and utilities that would need to be installed separately in a programming language like JavaScript, Ruby or Java. • crypto - libraries for X.509 certificates and cryptography • compress - work with zip files and archives • http - a very powerful and simple HTTP client and server package including things like reverse proxies • net - work directly with sockets, URLS and DNS • encoding/json - work directly with JSON files • text/template - a powerful templating engine for replacing tokens in files, and generating text and HTML 6

📄 Page 9

• os - work with low level OS primitives You can find a complete package list here. Through the next few chapters we will explore some of the above highlights and also solve various practical problems. Even if you’re a seasoned Go developer, I’d recommend going through each of the exercises, because even after 11 years, I myself am still learning new approaches and styles. Go itself is a very stable language, which seldom sees deprecations or breaking changes, for that reason the latest version of Go has little bearing on the version mentioned in this material. The content is updated periodically in line with major changes. Getting started with Go There are three ways to install Go. • Install Go using a system package - such as an msi (Windows) or pkg (MacOS) • Install Go using a package manager - such as apt-get (Debian), pacman (Arch Linux), brew (MacOS) • Download a tar archive and unpack it to a set location (Linux only) If you’re a new user and running on Windows or MacOS, then go ahead with the first option and download Go from the homepage. The version changes frequently, so download the latest stable version. If you like to use package managers, you should be aware that the version they offer often lags behind. You may be installing a very old version of Go, and some features may not be available. A note for Windows users Whilst the Go commands can be run cmd.exe and PowerShell, some of the example commands expect a UNIX-style shell which is present on MacOS and Linux hosts. For ease of use, it’s recommended that you try the content using Windows Subsystem for Linux (WSL), if it is available for your system. Or if not, make use of Git Bash. Git bash provides a terminal where you can run bash, have access to the git command and various other UNIX-style commands such as export, ls, mkdir and curl. Direct installation on Linux As a Linux user, I often need to install Go on a server, within a Docker image, or on my Raspberry Pi. My arkade project provides a quick and efficient way to download the latest version without having to construct a curl statement or run tar. Download and install arkade from: https://github.com/alexellis/arkade Then run: sudo arkade system install go Installing Go to /usr/local/ Installing version: go1.23.4 for: amd64 Downloading from: https://go.dev/dl/go1.23.4.linux-amd64.tar.gz 142.05 MiB / 142.05 MiB [----------------------------------------------] 100.00% Downloaded to: /tmp/go1.23.4.linux-amd64.tar.gz Unpacking Go to: /usr/local/go export PATH=$PATH:/usr/local/go/bin:$HOME/go/bin export GOPATH=$HOME/go/ 7

📄 Page 10

After the installation, you will need to set your PATH and GOPATH variables in your bash file, so that they load every time you log in. echo "export PATH=\$PATH:/usr/local/go/bin/" | tee -a ~/.bash_profile echo "export GOPATH=\$HOME/go" | tee -a ~/.bash_profile Now disconnect from your server or Raspberry Pi, or close the terminal and connect again. pi@raspberrypi:~$ go version go version go1.23 linux/arm arkade also supports a --version argument if you’d like to install an earlier version. Packages and paths Go is usually installed to /usr/local/go/, with the bin folder containing the main binary and any other built-in com- mands like gofmt. You should also make yourself familiar with the GOPATH variable. Whenever packages are fetched, they will usually be downloaded there. In the section on Go modules, we will clarify its use further. Whenever you run go install path/to/project@version, you’ll find the binaries there also. Your $GOPATH will usually be $HOME/go/ • $GOPATH/pkg/ - Go modules downloaded as dependencies • $GOPATH/bin/ - any CLIs that you build or install via go install • $GOPATH/src/ - the default location for source code When the $GOPATH is not set, you can add it to your shell by editing $HOME/.bash_profile and adding a line: export GOPATH=$HOME/go/. To make any binaries installed via go install appear on your command line, you can move them to /usr/local/bin/ or add $GOPATH/bin/ to your PATH variable instead. Most of the time when you work on your own projects, you will store them in a public or private Git repository, perhaps on GitHub. So your first step may be to clone that repository, and open an IDE. $ mkdir -p $GOPATH/src/github.com/alexellis/hash-browns $ cd $GOPATH/src/github.com/alexellis/hash-browns $ go build # Installation of Go binaries involves moving the # the binary to your "bin" directory $ sudo cp ./hash-browns /usr/local/bin/ $ hash-browns Go has a utility method go install for downloading, building and moving a binary to $GOPATH/bin/. I don’t tend to use it, but you will see it in project README files as an installation method. Let’s download an example HTTP server that I wrote called hash-browns. It will generate a hash from the HTTP body posted to it. Using go install we can download and install a specific version of it: $ go install github.com/alexellis/hash-browns@v1.3.0 The text after the @ symbol is the version number, which could also be a Git commit SHA. To get the latest version, omit @v1.3.0 or change it to @latest. The project’s source code and its dependencieswill be downloaded to your Gomodules cache under $GOPATH/pkg/mod: $ ls $GOPATH/pkg/mod/github.com/ 8

📄 Page 11

total 0 drwxr-xr-x 3 alex staff 96 12 Dec 09:04 alexellis drwxr-xr-x 3 alex staff 96 12 Dec 09:04 beorn7 drwxr-xr-x 3 alex staff 96 12 Dec 09:04 golang drwxr-xr-x 3 alex staff 96 12 Dec 09:04 gorilla drwxr-xr-x 3 alex staff 96 12 Dec 09:04 matttproud drwxr-xr-x 6 alex staff 192 12 Dec 09:04 prometheus The binary for hash-browns will be in $GOPATH/bin/: $ ls $GOPATH/bin/ $ $GOPATH/bin/hash-browns hash-browns by Alex Ellis Version: Commit: 2021/12/12 09:05:24 Listening on port: 8080 You may also see the go get command mentioned on project README files. As of Go 1.17, the go get com- mand will be deprecated for installing binaries. It can still be used for fetching individual modules for use within your projects. For instance, the hash-browns project uses the Prometheus SDK, so if you were starting a project from scratch, you’d probably type in go get or go get github.com/prometheus/client_golang or go get github.com/prometheus/client_golang@v0.8.0. Note: Some projects you find on GitHub may not have moved to Go modules yet. So you will see an rather odd looking version like this in your go.mod file: golang.org/x/sync v0.0.0-20210220032951- 036812b2e83c. The final part of the string is a shortened Git commit SHA 036812b2e83c. Since Go 1.17 turns Go modules on by default (GO111MODULE=on), the rest of the book will assume that you also have it enabled. Note: In older Go versions, whenever GO111MODULE is set to off, then the code is downloaded to $GOPATH/src/ instead of $GOPATH/pkg. Building without Go modules may be disabled completely in a future release of Go. Vendoring and Go modules In the early days of the Go ecosystem, the source code of any dependencies being used by your program was placed a vendor folder. The vendor folder would be committed alongside your application code, giving you a copy and back-up of anything required to build your application. In the previous example, hash-browns, the github.com/prometheus/procfs module was required, which would be cloned into the source tree as follows: $GOPATH/src/github.com/alexellis/hash-browns $GOPATH/src/github.com/alexellis/hash-browns/vendor/github.com/prometheus/procfs Various vendoring tools such as depwere used to download andmanage these dependencies. They are nowdeprecated. One file would specify which packages were required, and another “a lock file” would specify the versions. So at any time you could rm -rf the vendor folder and run something like dep ensure to get a fresh copy of all the various files restored. In Go 1.13 the concept of Go modules was introduced which largely mirrors vendoring, with one big change. The source for dependencies no longer needed to be cloned and committed to source control with your application. The transition for OpenFaaS’ 40+ projects was relatively smooth, with just a few projects causing headaches and needing to be deferred. 9

📄 Page 12

Another change with Go modules, was the ability to compile source code outside of the $GOPATH/src convention. For instance, you could have your source code at $HOME/dev/ instead of $HOME/go/src/ or wherever you’d set it. Whenever you create a new Go project, it’s best to run go mod init first. If your code falls within the conventional GOPATH structure, then it will have its name determined automatically, but if not, you can pass it as an argument. Compare the following: export GH_USERNAME=alexellis $ mkdir -p $GOPATH/src/github.com/$GH_USERNAME/hash-gen $ cd $GOPATH/src/github.com/$GH_USERNAME/hash-gen $ go mod init go: creating new go.mod: module github.com/alexellis/hash-gen $ cat go.mod module github.com/alexellis/hash-gen go 1.16 And when run outside of GOPATH: $ mkdir /tmp/hash-gen $ cd /tmp/hash-gen $ go mod init go: cannot determine module path for source directory /tmp/hash-gen (outside GOPATH, module path must be specified)↪ Example usage: 'go mod init example.com/m' to initialize a v0 or v1 module 'go mod init example.com/m/v2' to initialize a v2 module Run 'go help mod init' for more information. Try one of the following instead: $ go mod init hash-gen $ go mod init github.com/alexellis/hash-gen It’s also possible to use Go modules and vendoring at the same time: $ cd /tmp/ $ git clone https://github.com/alexellis/hash-browns $ cd hash-brown $ rm -rf vendor $ go mod vendor $ ls vendor/ Now see that the vendor folder has been created. You can ask Go to build using the vendor folder instead of its local packages cache at $GOPATH/pkg by adding -mod vendor to the go build command. $ go build -mod vendor So which approach should you use? Vendoring has the advantage of keeping all your source and dependencies in one place, it also doesn’t require anything 10

📄 Page 13

to be downloaded to execute a build. With some projects such as Kubernetes controllers this means saving time and bandwidth. If youhave any private dependencies, vendoring canwork out as the easiest option, especiallywhenbuilding inside a container or CI pipeline. Using modules without vendoring seems to be trendy, with many projects removing their vendor folder. It has the advantage ofmaking code reviews easier since there are far fewer files that change in a pull request or commit. Working with private Go modules can be a source of contention, so you have been warned. See also: Go modules reference docs 11

📄 Page 14

Chapter 2 Create your first Go program You can create a Go program from memory, but I find it more useful to use an IDE like Visual Studio Code. Visual Studio Code has a number of plugins that it can install to format your code, tidy it up and add any paths for modules or packages that you import. The automation is not as comprehensive as a paid IDE like Jetbrains’ GoLand, but is fast and free. Create a folder first-go then, save the following as main.go: package main func main() { } Now you have two ways to run the program. 1) Run go run main.go 2) Run go build and then run the resulting binary ./first-go Now print a console statement using fmt.Printf. This will import the fmt package at the top of the file, and VSCode will automatically run a number of Go plugins to update the main.go file for you. package main import ( "fmt" "os" ) func main() { name := os.Getenv("USER") fmt.Printf("Well done %s for having your first Go\n", name) } One of the programs run by VScode is gofmt and it’s an example of how Go’s opinionated design makes it easy to move between code-bases without learning new conventions. You can run it against a text file using gofmt -w -s main.go. Note the use of the os.Getenv call to look-up your username from the USER environment variable. Note for Windows users: try replacing “USER” with “USERNAME” Now, run the program again and you should see output as per above. 12

📄 Page 15

Figure 2.1: A screenshot of Visual Studio Code and our first-go app Add an external dependency Whilst the standard library is extensive, it’s likely that you will need to use an external dependency. Dependencies in Go are made available as source code that can be linked to your program, which differs from C# or Java where compiled assets (DLLs or JARs) are downloaded. For this reason, you’ll usually see that a Go package has a path to where it can be downloaded with git. You can use webhooks to receive events from third-party APIs such as GitHub.com or Stripe. Most of the time the webhooks are sent with a digest in the header, that you can use to verify that the message hasn’t been tampered with. HMAC uses a symmetric key that both sender/receiver share ahead of time. The sender will generate a hash when wanting to transmit amessage - this data is sent along with the payload. The recipient will then sign the payload with the shared key again. And if the hash matches then the payload is assumed to be from the sender. Quote from alexellis/hmac Let’s use my library to create a digest of our own message, and then show how to verify it after. You can of course use the standard library directly, but the point of this exercise is to show: how to reference an external dependency. You can also browse the code for the dependency to see how to define your own. To begin with, we should initialise a Go module for our application. The Go modules file go.mod lists external depen- dencies that we want our program to depend upon. export GH_USERNAME="alexellis" mkdir -p $GOPATH/src/github.com/$GH_USERNAME/validate-hmac/ cd $GOPATH/src/github.com/$GH_USERNAME/validate-hmac/ go mod init go: creating new go.mod: module github.com/alexellis/validate-hmac Review the contents of go.mod: 13

📄 Page 16

module github.com/alexellis/validate-hmac go 1.23 It shows the path or name of this module, along with the minimum Go version required. In the past all code needed to be placed with the $GOPATH i.e. $HOME/go/src/github.com/alexellis/validate-hmac, but with go modules, the go.mod file can make this explicit, instead of implicit (read from the path). Since the original publication date in 2021, the SHA1 algorithm was broadly replaced by SHA256, and a v2 version of the hmac library was released. The examples now reference the new version of the library. Create main.go: package main import ( "fmt" "crypto/sha256" hmac "github.com/alexellis/hmac/v2" ) func main() { input := []byte(`input message from API`) secret := []byte(`so secret`) digest := hmac.Sign(input, secret, sha256.New) fmt.Printf("Digest: %x\n", digest) } According to the Go docs, %x used in fmt.Printf() renders base16 string, with lower-case letters for a-f, and is a way that we can share the raw computed bytes in a human readable format. Now run the program: $ go get go: added github.com/alexellis/hmac/v2 v2.0.0 $ go run main.go Digest: aee2adbab7eb4c1d0891e553391efd187feafccd54b86288d313c2c0a60e6849 The digest 17074131772d763bc4a360a6e4cb1a5ad1a98764 was printed which is a hash that can be sent with the original input. Any user with the secret can compute another hash and if they match, that user will know the message was from us. You’ll see that the Go module is resolved, then downloaded from GitHub, and updated in go.sum and go.mod: $ cat go.mod module github.com/alexellis/validate-hmac go 1.23 require github.com/alexellis/hmac/v2 v2.0.0 // indirect $ cat go.sum github.com/alexellis/hmac/v2 v2.0.0 h1:/sH/UJxDXPpJorUeg2DudeKSeUrWPF32Yamw2TiDoOQ= 14

📄 Page 17

github.com/alexellis/hmac/v2 v2.0.0/go.mod h1:O7hZZgTfh5fp5+vAamzodZPlbw+aQK+nnrrJNHsEvL0= The go.sum pins the specific version of the module for later use. Now let’s verify the digest we received using the library: package main import ( "fmt" "crypto/sha256" hmac "github.com/alexellis/hmac/v2" ) func main() { input := []byte(`input message from API`) secret := []byte(`so secret`) digest := hmac.Sign(input, secret, sha256.New) fmt.Printf("Digest: %x\n", digest) err := hmac.Validate(input, fmt.Sprintf("sha256=%x", digest), string(secret)) if err != nil { panic(err) } fmt.Printf("Digest validated.\n") } Run the program with go run main.go and you’ll see it pass as expected. Make use of flags Go has a built-in flag parsing library. We will explore an alternative library, Cobra, in a later chapter. I’ve used Cobra in several projects but even now, I always start with the simplest thing that will work. Then later, if it is warranted, I’ll move to a more complex, but powerful library than the standard library’s flags package. export GH_USERNAME="alexellis" mkdir -p $GOPATH/src/github.com/$GH_USERNAME/validate-hmac-flags/ cd $GOPATH/src/github.com/$GH_USERNAME/validate-hmac-flags/ go mod init go: creating new go.mod: module github.com/alexellis/validate-hmac-flags In our previous example we created hashes by hard-coding the input and secret, but that isn’t going to be useful for our users. Let’s use flags instead, so that we can run: go run main.go -message="my message" \ -secret="terces os" Adapt the solution to use flag.StringVar to define each flag, followed by flag.Parse: 15

📄 Page 18

package main import ( "flag" "fmt" ) func main() { var inputVar string var secretVar string flag.StringVar(&inputVar, "message", "", "message to create a digest from") flag.StringVar(&secretVar, "secret", "", "secret for the digest") flag.Parse() fmt.Printf("Computing hash for: %q\nSecret: %q\n", inputVar, secretVar) } This is a good time to let you know about the %q variable, which adds quotes around any string we print. I use it to make sure there are no extra lines or whitespace within a value. $ go run main.go -message="my message" -secret="terces os" Computing hash for: "my message" Secret: "terces os" Now let’s take the value from the flag, and pass it into the hash generation code. digest := hmac.Sign([]byte(inputVar), []byte(secretVar)) fmt.Printf("Digest: %x\n", digest) Then add "github.com/alexellis/hmac" to the imports section, and run go get followed by go mod tidy tomake sure the dependency is saved to go.mod. Run the whole program and see the digest generated: $ go run main.go -message="my message" -secret="terces os" Computing hash for: "my message" Secret: "terces os" Digest: 2aefe2f570c0e61bc346b9fd3b6fbb3107ebfdd9 Try the same message with a different secret: $ go run main.go -message="my message" -secret="so secret" Computing hash for: "my message" Secret: "so secret" Digest: d82655d25497765dc774e8e9cde8795858cb9f8e Now, what if no secret is given as a flag value? How are you going to validate that? if len(secretVar) == 0 { panic("--secret is required") } Try adding the above, which looks at the length of the string: 16

📄 Page 19

$ go run main.go -message="my message" panic: --secret is required goroutine 1 [running]: main.main() /home/alex/go/src/github.com/alexellis/validate-hmac-flags/main.go:20 +0x3ed exit status 2 But we can get around this by setting the secret to ” ” whitespace, so how do you handle that? if len(strings.TrimSpace(secretVar)) == 0 { panic("--secret is required") } We can simply wrap the value with strings.TrimSpace which will remove any whitespace and handle that case for us. Here’s the complete example: package main import ( "flag" "fmt" "strings" "github.com/alexellis/hmac" ) func main() { var inputVar string var secretVar string flag.StringVar(&inputVar, "message", "", "message to create a digest from") flag.StringVar(&secretVar, "secret", "", "secret for the digest") flag.Parse() if len(strings.TrimSpace(secretVar)) == 0 { panic("--secret is required") } fmt.Printf("Computing hash for: %q\nSecret: %q\n", inputVar, secretVar) digest := hmac.Sign([]byte(inputVar), []byte(secretVar)) fmt.Printf("Digest: %x\n", digest) } As an extra-work task, why don’t you add a mode, so that your CLI can be used for both generating and validating hashes? You could either add a flag with flag.BoolVar such as -generate=true/false, or a string mode like - mode=generate/validate. 17

📄 Page 20

Define a separate package or library Until now, we have written code within a single package called main. The main package is required to make a program runnable on its own, however sometimes you’ll want to write a library so that code can be shared between programs. You may also have a project with a main package and an additional named package for code organisation. Packages can be tested separately. Generally, starting a variable, function or struct with a capital letter means that it will be available outside the package to other consumers. We’ll create a package that can perform a similar task to the dir or ls shell commands. Create a new project: $ export GH_USERNAME="alexellis" $ mkdir -p $GOPATH/src/github.com/$GH_USERNAME/multiple-packages/ $ cd $GOPATH/src/github.com/$GH_USERNAME/multiple-packages/ $ go mod init go: creating new go.mod: module github.com/alexellis/multiple-packages Now create a folder for the package called cmd: $ mkdir -p $GOPATH/src/github.com/$GH_USERNAME/multiple-packages/cmd Then create ls.go inside it: package cmd Now you have defined a package called cmd, but it doesn’t have any functions or structs exported. package cmd func ExecuteLs(path string) (string, error) { return "", nil } The command can then be imported by a new main package using the path github.com/alexellis/multiple- packages/cmd. In some cases, like the HMAC library we used earlier, there is no need for the project to have its own main entrypoint because it will only be consumed by third-parties as a function. Create main.go in the root folder: package main import ( "fmt" "os" "github.com/alexellis/multiple-packages/cmd" ) func main() { wd, err := os.Getwd() if err != nil { fmt.Fprintf(os.Stderr, "unable to get working directory: %s", err.Error()) } 18