Mastering Bitcoin Programming the Open Blockchain, 3rd Edition (Andreas M. Antonopoulos, David A. Harding) (Z-Library)

📄 File Format: PDF

💾 File Size: 6.3 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

Andreas M. Antonopoulos & David A. Harding Third Edition Mastering Bitcoin Programming the Open Blockchain

📄 Page 2

DATA “Nearly a decade after the initial publishing, the third edition of Mastering Bitcoin cements the book’s role as the go-to source of technical Bitcoin educational content. No other book is as comprehensive or up-to-date.” —Olaoluwa Osuntokun CTO at Lightning Labs “A comprehensive overview of what goes on under Bitcoin’s hood and how things fit together. ” —Mark “Murch” Erhardt Bitcoin engineer at Chaincode Labs Mastering Bitcoin Twitter: @oreillymedia linkedin.com/company/oreilly-media youtube.com/oreillymedia Join the technological revolution that’s taking the financial world by storm. Mastering Bitcoin is your guide through the seemingly complex world of Bitcoin, providing the knowledge you need to participate in the internet of money. Whether you’re building the next killer app, investing in a startup, or simply curious about the technology, this revised and expanded third edition provides essential detail to get you started. Bitcoin, the first successful decentralized digital currency, has already spawned a multibillion-dollar global economy open to anyone with the knowledge and passion to participate. Mastering Bitcoin provides the knowledge. You supply the passion. The third edition includes: • A broad introduction to Bitcoin and its underlying blockchain—ideal for nontechnical users, investors, and business executives • An explanation of Bitcoin’s technical foundation and cryptographic currency for developers, engineers, and software and systems architects • Details of the Bitcoin decentralized network, peer-to-peer architecture, transaction lifecycle, and security principles • New developments such as Taproot, Tapscript, Schnorr signatures, and the Lightning Network • A deep dive into Bitcoin applications, including how to combine the building blocks offered by this platform into powerful new tools • User stories, analogies, examples, and code snippets illustrating key technical concepts Andreas M. Antonopoulos is an expert in Bitcoin and open blockchain technologies. David A. Harding is coauthor of the Bitcoin Optech weekly newsletter. US $69.99 CAN $87.99 ISBN: 978-1-098-15009-9

📄 Page 3

Andreas M. Antonopoulos and David A. Harding Mastering Bitcoin Programming the Open Blockchain THIRD EDITION Boston Farnham Sebastopol TokyoBeijing

📄 Page 4

978-1-098-15009-9 [LSI] Mastering Bitcoin by Andreas M. Antonopoulos and David A. Harding Copyright © 2024 David Harding. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (https://oreilly.com). For more information, contact our corporate/institu‐ tional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Michelle Smith Development Editor: Angela Rufino Production Editor: Clare Laylock Copyeditor: Kim Cofer Proofreader: Heather Walley Indexer: nSight, Inc. Interior Designer: David Futato Cover Designer: Randy Comer Illustrator: Kate Dullea December 2014: First Edition June 2017: Second Edition November 2023: Third Edition Revision History for the Third Edition 2023-11-03: First Release See https://oreilly.com/catalog/errata.csp?isbn=9781098150099 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Mastering Bitcoin, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

📄 Page 5

Dedicated to my mum, Theresa (1946–2017) She taught me to love books and question authority Thank you, mum —Andreas For Amanda It wasn’t until I met you that I actually began living in paradise —Dave

📄 Page 6

(This page has no text content)

📄 Page 7

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii 1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 History of Bitcoin 4 Getting Started 5 Choosing a Bitcoin Wallet 5 Quick Start 7 Recovery Codes 8 Bitcoin Addresses 9 Receiving Bitcoin 10 Getting Your First Bitcoin 11 Finding the Current Price of Bitcoin 12 Sending and Receiving Bitcoin 12 2. How Bitcoin Works. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Bitcoin Overview 15 Buying from an Online Store 16 Bitcoin Transactions 18 Transaction Inputs and Outputs 18 Transaction Chains 19 Making Change 20 Coin Selection 20 Common Transaction Forms 21 Constructing a Transaction 22 Getting the Right Inputs 22 Creating the Outputs 23 Adding the Transaction to the Blockchain 23 Bitcoin Mining 24 Spending the Transaction 28 v

📄 Page 8

3. Bitcoin Core: The Reference Implementation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 From Bitcoin to Bitcoin Core 29 Bitcoin Development Environment 31 Compiling Bitcoin Core from the Source Code 31 Selecting a Bitcoin Core Release 32 Configuring the Bitcoin Core Build 33 Building the Bitcoin Core Executables 35 Running a Bitcoin Core Node 36 Configuring the Bitcoin Core Node 37 Bitcoin Core API 41 Getting Information on Bitcoin Core’s Status 42 Exploring and Decoding Transactions 43 Exploring Blocks 45 Using Bitcoin Core’s Programmatic Interface 46 Alternative Clients, Libraries, and Toolkits 50 C/C++ 50 JavaScript 50 Java 51 Python 51 Go 51 Rust 51 Scala 51 C# 51 4. Keys and Addresses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Public Key Cryptography 54 Private Keys 55 Elliptic Curve Cryptography Explained 56 Public Keys 59 Output and Input Scripts 61 IP Addresses: The Original Address for Bitcoin (P2PK) 62 Legacy Addresses for P2PKH 63 Base58check Encoding 66 Compressed Public Keys 69 Legacy Pay to Script Hash (P2SH) 71 Bech32 Addresses 74 Problems with Bech32 Addresses 76 Bech32m 77 Private Key Formats 81 Compressed Private Keys 82 Advanced Keys and Addresses 83 Vanity Addresses 83 Paper Wallets 86 vi | Table of Contents

📄 Page 9

5. Wallet Recovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Independent Key Generation 89 Deterministic Key Generation 90 Public Child Key Derivation 92 Hierarchical Deterministic (HD) Key Generation (BIP32) 93 Seeds and Recovery Codes 94 Backing Up Nonkey Data 97 Backing Up Key Derivation Paths 99 A Wallet Technology Stack in Detail 101 BIP39 Recovery Codes 101 Creating an HD Wallet from the Seed 108 Using an Extended Public Key on a Web Store 114 6. Transactions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 A Serialized Bitcoin Transaction 119 Version 121 Extended Marker and Flag 122 Inputs 123 Length of Transaction Input List 123 Outpoint 124 Input Script 127 Sequence 127 Outputs 130 Outputs Count 131 Amount 131 Output Scripts 132 Witness Structure 133 Circular Dependencies 134 Third-Party Transaction Malleability 135 Second-Party Transaction Malleability 136 Segregated Witness 137 Witness Structure Serialization 138 Lock Time 139 Coinbase Transactions 139 Weight and Vbytes 141 Legacy Serialization 142 7. Authorization and Authentication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Transaction Scripts and Script Language 143 Turing Incompleteness 144 Stateless Verification 144 Script Construction 145 Pay to Public Key Hash 148 Table of Contents | vii

📄 Page 10

Scripted Multisignatures 150 An Oddity in CHECKMULTISIG Execution 152 Pay to Script Hash 153 P2SH Addresses 155 Benefits of P2SH 155 Redeem Script and Validation 156 Data Recording Output (OP_RETURN) 156 Transaction Lock Time Limitations 158 Check Lock Time Verify (OP_CLTV) 158 Relative Timelocks 160 Relative Timelocks with OP_CSV 161 Scripts with Flow Control (Conditional Clauses) 162 Conditional Clauses with VERIFY Opcodes 163 Using Flow Control in Scripts 164 Complex Script Example 165 Segregated Witness Output and Transaction Examples 166 Upgrading to Segregated Witness 170 Merklized Alternative Script Trees (MAST) 172 Pay to Contract (P2C) 176 Scriptless Multisignatures and Threshold Signatures 177 Taproot 178 Tapscript 180 8. Digital Signatures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 How Digital Signatures Work 184 Creating a Digital Signature 184 Verifying the Signature 184 Signature Hash Types (SIGHASH) 185 Schnorr Signatures 187 Serialization of Schnorr Signatures 193 Schnorr-based Scriptless Multisignatures 193 Schnorr-based Scriptless Threshold Signatures 195 ECDSA Signatures 197 ECDSA Algorithm 198 Serialization of ECDSA Signatures (DER) 199 The Importance of Randomness in Signatures 200 Segregated Witness’s New Signing Algorithm 201 9. Transaction Fees. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 Who Pays the Transaction Fee? 204 Fees and Fee Rates 205 Estimating Appropriate Fee Rates 206 Replace By Fee (RBF) Fee Bumping 207 viii | Table of Contents

📄 Page 11

Child Pays for Parent (CPFP) Fee Bumping 210 Package Relay 211 Transaction Pinning 212 CPFP Carve Out and Anchor Outputs 213 Adding Fees to Transactions 214 Timelock Defense Against Fee Sniping 215 10. The Bitcoin Network. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 Node Types and Roles 218 The Network 218 Compact Block Relay 219 Private Block Relay Networks 221 Network Discovery 223 Full Nodes 227 Exchanging “Inventory” 227 Lightweight Clients 228 Bloom Filters 231 How Bloom Filters Work 231 How Lightweight Clients Use Bloom Filters 235 Compact Block Filters 237 Golomb-Rice Coded Sets (GCS) 237 What Data to Include in a Block Filter 239 Downloading Block Filters from Multiple Peers 240 Reducing Bandwidth with Lossy Encoding 241 Using Compact Block Filters 242 Lightweight Clients and Privacy 243 Encrypted and Authenticated Connections 243 Mempools and Orphan Pools 244 11. The Blockchain. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 Structure of a Block 246 Block Header 247 Block Identifiers: Block Header Hash and Block Height 247 The Genesis Block 248 Linking Blocks in the Blockchain 249 Merkle Trees 252 Merkle Trees and Lightweight Clients 256 Bitcoin’s Test Blockchains 257 Testnet: Bitcoin’s Testing Playground 257 Signet: The Proof of Authority Testnet 259 Regtest: The Local Blockchain 260 Using Test Blockchains for Development 261 Table of Contents | ix

📄 Page 12

12. Mining and Consensus. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 Bitcoin Economics and Currency Creation 265 Decentralized Consensus 267 Independent Verification of Transactions 268 Mining Nodes 269 The Coinbase Transaction 270 Coinbase Reward and Fees 270 Structure of the Coinbase Transaction 271 Coinbase Data 272 Constructing the Block Header 273 Mining the Block 275 Proof-of-Work Algorithm 275 Target Representation 277 Retargeting to Adjust Difficulty 278 Median Time Past (MTP) 280 Successfully Mining the Block 281 Validating a New Block 281 Assembling and Selecting Chains of Blocks 282 Mining and the Hash Lottery 284 The Extra Nonce Solution 284 Mining Pools 285 Hashrate Attacks 288 Changing the Consensus Rules 291 Hard Forks 291 Soft Forks 295 Consensus Software Development 301 13. Bitcoin Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 Security Principles 303 Developing Bitcoin Systems Securely 304 The Root of Trust 305 User Security Best Practices 306 Physical Bitcoin Storage 307 Hardware Signing Devices 307 Ensuring Your Access 307 Diversifying Risk 308 Multisig and Governance 308 Survivability 308 14. Second-Layer Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 Building Blocks (Primitives) 311 Applications from Building Blocks 313 Colored Coins 314 x | Table of Contents

📄 Page 13

Single-Use Seals 315 Pay to Contract (P2C) 315 Client-Side Validation 316 RGB 316 Taproot Assets 317 Payment Channels and State Channels 318 State Channels—Basic Concepts and Terminology 319 Simple Payment Channel Example 321 Making Trustless Channels 323 Asymmetric Revocable Commitments 327 Hash Time Lock Contracts (HTLC) 331 Routed Payment Channels (Lightning Network) 332 Basic Lightning Network Example 333 Lightning Network Transport and Pathfinding 336 Lightning Network Benefits 337 A. The Bitcoin Whitepaper by Satoshi Nakamoto. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 B. Errata to the Bitcoin Whitepaper. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351 C. Bitcoin Improvement Proposals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363 Table of Contents | xi

📄 Page 14

(This page has no text content)

📄 Page 15

Preface Writing the Bitcoin Book I (Andreas) first stumbled upon Bitcoin in mid-2011. My immediate reaction was more or less “Pfft! Nerd money!” and I ignored it for another six months, failing to grasp its importance. This is a reaction that I have seen repeated among many of the smartest people I know, which gives me some consolation. The second time I came across Bitcoin, in a mailing list discussion, I decided to read the whitepaper written by Satoshi Nakamoto and see what it was all about. I still remember the moment I finished reading those nine pages, when I realized that Bitcoin was not simply a digital currency, but a network of trust that could also provide the basis for so much more than just currencies. The realization that “this isn’t money, it’s a decentralized trust network,” started me on a four-month journey to devour every scrap of information about Bitcoin I could find. I became obsessed and enthralled, spending 12 or more hours each day glued to a screen, reading, writing, coding, and learning as much as I could. I emerged from this state of fugue, more than 20 pounds lighter from lack of consistent meals, determined to dedicate myself to working on Bitcoin. Two years later, after creating a number of small startups to explore various Bitcoin- related services and products, I decided that it was time to write my first book. Bitcoin was the topic that had driven me into a frenzy of creativity and consumed my thoughts; it was the most exciting technology I had encountered since the internet. It was now time to share my passion about this amazing technology with a broader audience. Intended Audience This book is mostly intended for coders. If you can use a programming language, this book will teach you how cryptographic currencies work, how to use them, and how to develop software that works with them. The first few chapters are also suitable as xiii

📄 Page 16

an in-depth introduction to Bitcoin for noncoders—those trying to understand the inner workings of Bitcoin and cryptocurrencies. Why Are There Bugs on the Cover? The leafcutter ant is a species that exhibits highly complex behavior in a colony super-organism, but each individual ant operates on a set of simple rules driven by social interaction and the exchange of chemical scents (pheromones). Per Wikipedia: “Next to humans, leafcutter ants form the largest and most complex animal societies on Earth.” Leafcutter ants don’t actually eat leaves, but rather use them to farm a fungus, which is the central food source for the colony. Get that? These ants are farming! Although ants form a caste-based society and have a queen for producing offspring, there is no central authority or leader in an ant colony. The highly intelligent and sophisticated behavior exhibited by a multimillion-member colony is an emergent property from the interaction of the individuals in a social network. Nature demonstrates that decentralized systems can be resilient and can produce emergent complexity and incredible sophistication without the need for a central authority, hierarchy, or complex parts. Bitcoin is a highly sophisticated decentralized trust network that can support myriad financial processes. Yet, each node in the Bitcoin network follows a few simple rules. The interaction between many nodes is what leads to the emergence of the sophisticated behavior, not any inherent complexity or trust in any single node. Like an ant colony, the Bitcoin network is a resilient network of simple nodes following simple rules that together can do amazing things without any central coordination. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. xiv | Preface

📄 Page 17

Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. Code Examples All the code snippets can be replicated on most operating systems with a minimal installation of compilers and interpreters for the corresponding languages. Where necessary, we provide basic installation instructions and step-by-step examples of the output of those instructions. Some of the code snippets and code output have been reformatted for print. In all such cases, the lines have been split by a backslash (\) character, followed by a newline character. When transcribing the examples, remove those two characters and join the lines again and you should see identical results as shown in the example. All the code snippets use real values and calculations where possible, so that you can build from example to example and see the same results in any code you write to calculate the same values. Using Code Examples This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book Preface | xv

📄 Page 18

and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Mastering Bitcoin, 3rd ed., by Andreas M. Antonopoulos and David A. Harding (O’Reilly). Copyright 2024 David Harding, ISBN 978-1-098-15009-9.” Some editions of this book are offered under an open source license, such as CC-BY- NC, in which case the terms of that license apply. If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. Changes Since the Previous Edition A particular focus in the third edition has been modernizing the 2017 second edition text and the remaining 2014 first edition text. In addition, many concepts that are relevant to contempory Bitcoin development in 2023 have been added: Chapter 4 We rearranged the address info so that we work through everything in historical order, adding a new section with P2PK (where “address” was “IP address”), refreshed the previous P2PKH and P2SH sections, and then added new sections for segwit/bech32 and taproot/bech32m. Old Chapters 6 and 7 Text from previous versions of Chapter 6, “Transactions,” and Chapter 7, “Advanced Transactions,” has been rearranged and expanded across four new chapters: Chapter 6, “Transactions” (the structure of transactions), Chapter 7, “Authorization and Authentication”, Chapter 8, “Digital Signatures”, and Chap‐ ter 9, “Transaction Fees”. Chapter 6 We added almost entirely new text describing the structure of a transaction. Chapter 7 We added new text about MAST, P2C, scriptless multisignatures, taproot, and tapscript. Chapter 8 We revised the ECDSA text and added new text about schnorr signatures, multi‐ signatures, and threshold signatures. xvi | Preface

📄 Page 19

Chapter 9 We added almost entirely new text about fees, RBF and CPFP fee bumping, transaction pinning, package relay, and CPFP carve-out. Chapter 10 We added text about compact block relay, added a significant update to bloom filters that better describes their privacy problems, and new text about compact block filters. Chapter 11 We added text about signet. Chapter 12 We added text about BIP8 and speedy trial. Appendixes We removed library-specific appendixes. After the appendix containing the orig‐ inal whitepaper, we added a new appendix describing how the implementation and properties of Bitcoin differ from those proposed in the whitepaper. Bitcoin Addresses and Transactions in This Book The Bitcoin addresses, transactions, keys, QR codes, and blockchain data used in this book are, for the most part, real. That means you can browse the blockchain, look at the transactions offered as examples, retrieve them with your own scripts or programs, etc. However, note that the private keys used to construct addresses are either printed in this book or have been “burned.” That means if you send money to any of these addresses, the money will either be lost forever, or in some cases everyone who can read the book can take it using the private keys printed in here. DO NOT SEND MONEY TO ANY OF THE ADDRESSES IN THIS BOOK. Your money will be taken by another reader or lost forever. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Preface | xvii

📄 Page 20

Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-889-8969 (in the United States or Canada) 707-829-7019 (international or local) 707-829-0104 (fax) support@oreilly.com https://www.oreilly.com/about/contact.html We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/MasteringBitcoin3e. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly-media. Follow us on Twitter: https://twitter.com/oreillymedia. Watch us on YouTube: https://youtube.com/oreillymedia. Contacting the Authors You can contact Andreas M. Antonopoulos on his personal site: https://antonopoulos.com. Follow Andreas on Facebook: https://facebook.com/AndreasMAntonopoulos. Follow Andreas on Twitter: https://twitter.com/aantonop. Follow Andreas on LinkedIn: https://linkedin.com/company/aantonop. Many thanks to all of Andreas’s patrons who support his work through monthly donations. You can follow his Patreon page here: https://patreon.com/aantonop. Information about Mastering Bitcoin, as well as Andreas’s Open Edition and transla‐ tions, is available on https://bitcoinbook.info. You can contact David A. Harding on his personal site: https://dtrt.org. xviii | Preface