Network Programming with Go Learn to Code Secure and Reliable Network Services from Scratch (Adam Woodbeck) (Z-Library)

A D A M W O O D B E C K N E T W O R K P R O G R A M M I N G W I T H G O C O D E S E C U R E A N D R E L I A B L E N E T W O R K S E R V I C E S F R O M S C R A T C H

NETWORK PROGRAMMING WITH GO

N E T W O R K P R O G R A M M I N G W I T H G O C o d e S e c u r e a n d R e l i a b l e N e t w o r k S e r v i c e s f r o m S c r a t c h Adam Woodbeck San Francisco

NETWORK PROGRAMMING WITH GO. © 2021 by Adam Woodbeck All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. ISBN-13: 978-1-7185-0088-4 (print) ISBN-13: 978-1-7185-0089-1 (ebook) Publisher: William Pollock Executive Editor: Barbara Yien Production Editor: Kate Kaminski Developmental Editor: Frances Saux Cover Illustration: Gina Redman Interior Design: Octopod Studios Technical Reviewer: Jeremy Bowers Copyeditor: Sharon Wilkey Compositor: Jeff Lytle, Happenstance Type-O-Rama Proofreader: Paula L. Fleming For information on book distributors or translations, please contact No Starch Press, Inc. directly: No Starch Press, Inc. 245 8th Street, San Francisco, CA 94103 phone: 1-415-863-9900; info@nostarch.com www.nostarch.com Library of Congress Control Number: 2020943331 No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc. Other product and company names mentioned herein may be the trademarks of their respective owners. Rather than use a trademark symbol with every occurrence of a trademarked name, we are using the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The information in this book is distributed on an “As Is” basis, without warranty. While every precaution has been taken in the preparation of this work, neither the author nor No Starch Press, Inc. shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in it.

For my wife, Mandy, and my children, Benjamin and Lilyanna

About the Author Adam Woodbeck is a senior software engineer at Barracuda Networks, where he implemented a distributed cloud environment in Go to supplant the previous cloud infrastructure, profoundly increasing its scalability and performance. He’s since served as the architect for many network-based services in Go. About the Technical Reviewer Jeremy Bowers is a distinguished software architect in the Office of CTO at Barracuda Networks. Equipped with many years of lead developer experi- ence at Barracuda and security startups, especially in network engineering, Jeremy has successfully designed and implemented services that efficiently serve hundreds of thousands of customers worldwide. He holds a bachelor’s and a master’s degree in computer science from Michigan State University.

B R I E F C O N T E N T S Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xvii Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix PART I: NETWORK ARCHITECTURE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Chapter 1: An Overview of Networked Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Chapter 2: Resource Location and Traffic Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 PART II: SOCKET-LEVEL PROGRAMMING . . . . . . . . . . . . . . . . . . . . . . . . 43 Chapter 3: Reliable TCP Data Streams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Chapter 4: Sending TCP Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Chapter 5: Unreliable UDP Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Chapter 6: Ensuring UDP Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 Chapter 7: Unix Domain Sockets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 PART III: APPLICATION-LEVEL PROGRAMMING . . . . . . . . . . . . . . . . . . 163 Chapter 8: Writing HTTP Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 Chapter 9: Building HTTP Services . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 Chapter 10: Caddy: A Contemporary Web Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 Chapter 11: Securing Communications with TLS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 PART IV: SERVICE ARCHITECTURE . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 Chapter 12: Data Serialization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 Chapter 13: Logging and Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295 Chapter 14: Moving to the Cloud . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355

C O N T E N T S I N D E T A I L ACKNOWLEDGMENTS XVII INTRODUCTION XIX Who This Book Is For . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx Installing Go . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xx Recommended Development Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi What’s in This Book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi PART I: NETWORK ARCHITECTURE 1 1 AN OVERVIEW OF NETWORKED SYSTEMS 3 Choosing a Network Topology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Bandwidth vs . Latency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 The Open Systems Interconnection Reference Model . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 The Hierarchal Layers of the OSI Reference Model . . . . . . . . . . . . . . . . . . . . . 8 Sending Traffic by Using Data Encapsulation . . . . . . . . . . . . . . . . . . . . . . . . 10 The TCP/IP Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 The Application Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 The Transport Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 The Internet Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 The Link Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 What You’ve Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2 RESOURCE LOCATION AND TRAFFIC ROUTING 17 The Internet Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 IPv4 Addressing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 Network and Host IDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Subdividing IPv4 Addresses into Subnets . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Ports and Socket Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Network Address Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Unicasting, Multicasting, and Broadcasting . . . . . . . . . . . . . . . . . . . . . . . . . 25 Resolving the MAC Address to a Physical Network Connection . . . . . . . . . . . 26 IPv6 Addressing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Writing IPv6 Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 IPv6 Address Categories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Advantages of IPv6 over IPv4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 The Internet Control Message Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Internet Traffic Routing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Routing Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 The Border Gateway Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

xii   Contents in Detail Name and Address Resolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Domain Name Resource Records . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Multicast DNS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Privacy and Security Considerations of DNS Queries . . . . . . . . . . . . . . . . . . 40 What You’ve Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 PART II: SOCKET-LEVEL PROGRAMMING 43 3 RELIABLE TCP DATA STREAMS 45 What Makes TCP Reliable? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Working with TCP Sessions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Establishing a Session with the TCP Handshake . . . . . . . . . . . . . . . . . . . . . . 47 Acknowledging Receipt of Packets by Using Their Sequence Numbers . . . . . . . 47 Receive Buffers and Window Sizes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 Gracefully Terminating TCP Sessions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Handling Less Graceful Terminations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Establishing a TCP Connection by Using Go’s Standard Library . . . . . . . . . . . . . . . . . . 51 Binding, Listening for, and Accepting Connections . . . . . . . . . . . . . . . . . . . . 51 Establishing a Connection with a Server . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Implementing Deadlines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 What You’ve Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 4 SENDING TCP DATA 73 Using the net .Conn Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Sending and Receiving Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Reading Data into a Fixed Buffer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Delimited Reading by Using a Scanner . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 Dynamically Allocating the Buffer Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Handling Errors While Reading and Writing Data . . . . . . . . . . . . . . . . . . . . 86 Creating Robust Network Applications by Using the io Package . . . . . . . . . . . . . . . . . 87 Proxying Data Between Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 Monitoring a Network Connection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Pinging a Host in ICMP-Filtered Environments . . . . . . . . . . . . . . . . . . . . . . . . 96 Exploring Go’s TCPConn Object . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 Controlling Keepalive Messages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Handling Pending Data on Close . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Overriding Default Receive and Send Buffers . . . . . . . . . . . . . . . . . . . . . . . 100 Solving Common Go TCP Network Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 Zero Window Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 Sockets Stuck in the CLOSE_WAIT State . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 What You’ve Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 5 UNRELIABLE UDP COMMUNICATION 105 Using UDP: Simple and Unreliable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 Sending and Receiving UDP Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Using a UDP Echo Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

Contents in Detail   xiii Receiving Data from the Echo Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Every UDP Connection Is a Listener . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 Using net .Conn in UDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Avoiding Fragmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 What You’ve Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 6 ENSURING UDP RELIABILITY 119 Reliable File Transfers Using TFTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 TFTP Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 Read Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Data Packets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 Handling Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 The TFTP Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Writing the Server Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Handling Read Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 Starting the Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 Downloading Files over UDP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 What You’ve Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 7 UNIX DOMAIN SOCKETS 141 What Are Unix Domain Sockets? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 Binding to Unix Domain Socket Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Changing a Socket File’s Ownership and Permissions . . . . . . . . . . . . . . . . . 143 Understanding Unix Domain Socket Types . . . . . . . . . . . . . . . . . . . . . . . . . 144 Writing a Service That Authenticates Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Requesting Peer Credentials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Writing the Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156 Testing the Service with Netcat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 What You’ve Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 PART III: APPLICATION-LEVEL PROGRAMMING 163 8 WRITING HTTP CLIENTS 165 Understanding the Basics of HTTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 Uniform Resource Locators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166 Client Resource Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 Server Responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 From Request to Rendered Page . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172 Retrieving Web Resources in Go . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 Using Go’s Default HTTP Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 Closing the Response Body . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 Implementing Time-outs and Cancellations . . . . . . . . . . . . . . . . . . . . . . . . . 176 Disabling Persistent TCP Connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

xiv   Contents in Detail Posting Data over HTTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 Posting JSON to a Web Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 Posting a Multipart Form with Attached Files . . . . . . . . . . . . . . . . . . . . . . . . 181 What You’ve Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184 9 BUILDING HTTP SERVICES 187 The Anatomy of a Go HTTP Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 Clients Don’t Respect Your Time . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 Adding TLS Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 Handlers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 Test Your Handlers with httptest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 How You Write the Response Matters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 Any Type Can Be a Handler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 Injecting Dependencies into Handlers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200 Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202 Timing Out Slow Clients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 Protecting Sensitive Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204 Multiplexers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 HTTP/2 Server Pushes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 Pushing Resources to the Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 Don’t Be Too Pushy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 What You’ve Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 10 CADDY: A CONTEMPORARY WEB SERVER 217 What Is Caddy? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 Let’s Encrypt Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 How Does Caddy Fit into the Equation? . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 Retrieving Caddy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 Downloading Caddy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 Building Caddy from Source Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 Running and Configuring Caddy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 Modifying Caddy’s Configuration in Real Time . . . . . . . . . . . . . . . . . . . . . . 222 Storing the Configuration in a File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 Extending Caddy with Modules and Adapters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224 Writing a Configuration Adapter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 Writing a Restrict Prefix Middleware Module . . . . . . . . . . . . . . . . . . . . . . . 226 Injecting Your Module into Caddy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 Reverse-Proxying Requests to a Backend Web Service . . . . . . . . . . . . . . . . . . . . . . . 232 Creating a Simple Backend Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . 232 Setting Up Caddy’s Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 Adding a Reverse-Proxy to Your Service . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 Serving Static Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 Checking Your Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236 Adding Automatic HTTPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 What You’ve Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238

Contents in Detail   xv 11 SECURING COMMUNICATIONS WITH TLS 241 A Closer Look at Transport Layer Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242 Forward Secrecy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 In Certificate Authorities We Trust . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 How to Compromise TLS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244 Protecting Data in Transit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 Client-side TLS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 TLS over TCP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 Server-side TLS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 Certificate Pinning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252 Mutual TLS Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 Generating Certificates for Authentication . . . . . . . . . . . . . . . . . . . . . . . . . 256 Implementing Mutual TLS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 What You’ve Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 PART IV: SERVICE ARCHITECTURE 267 12 DATA SERIALIZATION 269 Serializing Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270 JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276 Gob . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278 Protocol Buffers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280 Transmitting Serialized Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 Connecting Services with gRPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 Creating a TLS-Enabled gRPC Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 Creating a gRPC Client to Test the Server . . . . . . . . . . . . . . . . . . . . . . . . . . 289 What You’ve Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294 13 LOGGING AND METRICS 295 Event Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296 The log Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 Leveled Log Entries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300 Structured Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 Scaling Up with Wide Event Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312 Log Rotation with Lumberjack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 Instrumenting Your Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 316 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 Counters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 Gauges . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318 Histograms and Summaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319 Instrumenting a Basic HTTP Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320 What You’ve Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326

xvi   Contents in Detail 14 MOVING TO THE CLOUD 329 Laying Some Groundwork . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330 AWS Lambda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333 Installing the AWS Command Line Interface . . . . . . . . . . . . . . . . . . . . . . . . 333 Configuring the CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333 Creating a Role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 Defining an AWS Lambda Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336 Compiling, Packaging, and Deploying Your Function . . . . . . . . . . . . . . . . . . 339 Testing Your AWS Lambda Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 340 Google Cloud Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341 Installing the Google Cloud Software Development Kit . . . . . . . . . . . . . . . . . 341 Initializing the Google Cloud SDK . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341 Enable Billing and Cloud Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 Defining a Cloud Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 342 Deploying Your Cloud Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 344 Testing Your Google Cloud Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345 Azure Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 346 Installing the Azure Command Line Interface . . . . . . . . . . . . . . . . . . . . . . . 346 Configuring the Azure CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 Installing Azure Functions Core Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347 Creating a Custom Handler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 348 Defining a Custom Handler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349 Locally Testing the Custom Handler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 350 Deploying the Custom Handler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351 Testing the Custom Handler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 What You’ve Learned . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353 INDEX 355

A C K N O W L E D G M E N T S I’ve never played in a rock band, but I imagine writing this book is a bit like that. I may have been the singer-songwriter, but this book would have been noticeably inferior had it not been for the exceptional talents and support of the following people. Jeremy Bowers is one of the most talented engineers and enjoyable human beings I’ve had the pleasure of knowing. The depth and breadth of his knowledge considerably eased my impostor syndrome, knowing that he staked his reputation and likely his career on the success of this book. He reviewed every paragraph and line of code to ensure their coherence and accuracy. But as with any large pull request, the responsibility for any bugs lies with me, and despite Jeremy’s best efforts, I’ve been known to write some profoundly clever bugs. Thank you, Jeremy, for contributing your technical expertise to this book. I don’t know how much editing my writing required compared to the average author, but judging by the red markup on my drafts, Frances Saux is a saint. She shepherded me through this process and was an outstanding advocate for the reader. I could rely on her to keep my writing conversa- tional and ask pointed questions that prompted me to elaborate on topics I take for granted. Thank you, Frances, for your patience and consistency throughout the writing process. This book certainly wouldn’t be the same without your extensive efforts.

xviii   Acknowledgments I would also like to thank Bill Pollock for giving this book his blessing; Barbara Yien for supervising it; Sharon Wilkey and Kate Kaminski for their copyediting and production expertise, respectively; Paula Fleming for proof- reading; Derek Yee for the book’s cover and interior design; Gina Redman for the cover illustration; and Matt Holt for reviewing Chapter 10 for techni- cal accuracy. Most of all, I’d like to thank my wife, Amandalyn; my son, Benjamin; and my daughter, Lilyanna. As with any extracurricular endeavor, the research and writing process consumed a lot of our family time. But I’m thankful for your patience while I strived to find a balance that worked for our family during this undertaking. Your love and support allowed me to step outside my comfort zone. Hopefully, my example will encourage you to do the same.

I N T R O D U C T I O N With the advent of the internet came an ever-increasing demand for network engi- neers and developers. Today, personal comput- ers, tablets, phones, televisions, watches, gaming systems, vehicles, common household items, and even doorbells communicate over the internet. Network pro- gramming makes all this possible. And secure network programming makes it trustworthy, driving increasing numbers of people to adopt these services. This book will teach you how to write contemporary network software using Go’s asynchronous features.

xx   Introduction Google created the Go programming language in 2007 to increase the productivity of developers working with large code bases. Since then, Go has earned a reputation as a fast, efficient, and safe language for the devel- opment and deployment of software at some of the largest companies in the world. Go is easy to learn and has a rich standard library, well suited for tak- ing advantage of multicore, networked systems. This book details the basics of network programming with an emphasis on security. You will learn socket-level programming including TCP, UDP, and Unix sockets, interact with application-level protocols like HTTPS and HTTP/2, serialize data with formats like Gob, JSON, XML, and protocol buffers, perform authentication and authorization for your network services, create streams and asynchronous data transfers, write gRPC microservices, perform structured logging and instrumentation, and deploy your applications to the cloud. At the end of our journey, you should feel comfortable using Go, its standard library, and popular third-party packages to design and imple- ment secure network applications and microservices. Every chapter uses best practices and includes nuggets of wisdom that will help you avoid potential pitfalls. Who This Book Is For If you’d like to learn how to securely share data over a network using standard protocols, all the while writing Go code that is stable, secure, and effective, this book is for you. The target reader is a security-conscious developer or system adminis- trator who wishes to take a deep dive into network programming and has a working knowledge of Go and Go’s module support. That said, the first few chapters introduce basic networking concepts, so networking newcomers are welcome. Staying abreast of contemporary protocols, standards, and best practices when designing and developing network applications can be difficult. That’s why, as you work through this book, you’ll be given increased responsibil- ity. You’ll also be introduced to tools and tech that will make your workload manageable. Installing Go To follow along with the code in this book, install the latest stable version of Go available at https://golang.org/. For most programs in this book, you’ll need at least Go 1.12. That said, certain programs in this book are compatible with only Go 1.14 or newer. The book calls out the use of this code. Keep in mind that the Go version available in your operating system’s package manager may be several versions behind the latest stable version.