Foundations of Scalable Systems (Eighth Early Release) (Ian Gorton) (Z-Library)

📄 File Format: PDF

💾 File Size: 4.1 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

(This page has no text content)

📄 Page 2

Foundations of Scalable Systems Designing Distributed Architectures With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles. Ian Gorton

📄 Page 3

Foundations of Scalable Systems by Ian Gorton Copyright © 2022 Ian Gorton. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Melissa Duffield Development Editor: Virginia Wilson Production Editor: Jonathon Owen Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea August 2022: First Edition Revision History for the Early Release 2021-05-10: First Release 2021-06-22: Second Release 2021-09-08: Third Release 2021-10-04: Fourth Release

📄 Page 4

2021-11-19: Fifth Release 2021-12-08: Sixth Release 2022-02-24: Seventh Release 2022-03-30: Eighth Release See http://oreilly.com/catalog/errata.csp?isbn=9781098106065 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Foundations of Scalable Systems, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-098-10599-0 [LSI]

📄 Page 5

Preface A NOTE FOR EARLY RELEASE READERS With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles. If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at vwilson@oreilly.com. This book is built around the thesis that the ability of software systems to operate at scale is increasingly a driving system quality. As our world becomes more interconnected, this characteristic will only accelerate. Hence the goal of the book is to provide the reader with the core knowledge of distributed and concurrent systems. It will also introduce a collection of software architecture approaches and distributed technologies that can be used to build scalable systems. Why Scalability? The pace of change in our world is daunting. Innovations appear daily, creating new capabilities for us all to interact with, conduct business, be entertained, end pandemics. The fuel for much of this innovation is software, written by veritable armies of developers in major internet companies, crack small teams in startups, and all shapes and sizes of teams in between. Delivering software systems that are responsive to user needs is difficult enough, but it becomes an order of magnitude more difficult to do for

📄 Page 6

systems at scale. We all know of systems that fail suddenly when exposed to unexpected high loads - such situations are minimally bad publicity for organizations, and at worst can lose jobs and destroy companies. Software is unlike physical systems in that it’s amorphous—its physical form (1’s and 0’s) bears no resemblance to its actual capabilities. We’d never expect to transform a small village of 500 people into a city of 10 million overnight. But we sometimes expect our software systems to suddenly handle 1000x the number of requests they were designed for. Not surprisingly, the outcomes are rarely pretty.

📄 Page 7

Who This Book Is For The major target audience for this book is software engineers and architects who have no or limited experience with distributed, concurrent systems. They need to deepen both their theoretical and practical design knowledge in order to meet the challenges of building larger scale, typically Internet- facing applications. Much of the content of this book has been developed in the context of an advanced undergraduate/graduate course at Northeastern University. It has proven a very popular and effective approach for equipping students with the knowledge and skills needed to launch their careers with major Internet companies. Additional materials on the book web site are available to support educators who wish to use the book for their course. What You Will Learn This book covers the landscape of concurrent and distributed systems through the lens of scalability. While it’s impossible to totally divorce scalability from other architectural qualities, scalability is the main focus of discussion. Of course, other qualities necessarily come in to play, with performance, availability and consistency regularly raising their heads. Building distributed systems requires some fundamental understanding of distribution and concurrency - this knowledge is a recurrent theme throughout this book. It’s needed because at their core, there are two problems in distributed systems that make them complex, as I describe below. First, although systems operate perfectly correctly nearly all the time, an individual part of the system may fail at any time. When a component fails (hardware crash, network down, bug in server), we have to employ techniques that enable the system as a whole to continue operations and recover from failures. And any distributed system will experience component failure, often in weird and mysterious and unanticipated ways.

📄 Page 8

Second, creating a scalable distributed system requires the coordination of multiple moving parts. Each component of the system needs to keep its part of the bargain and process requests as quickly as possible. If just one component causes requests to be delayed, the whole system may perform poorly and even eventually crash. To deal with these problems there is a rich deep body of literature available to draw on. And luckily for us engineers, there’s a rich, extensive collection of technologies that are designed to help us build distributed systems that are tolerant to fail and scalable. These technologies embody theoretical approaches and complex algorithms that are incredibly hard to build correctly. Using these platform level, widely applicable technologies, our applications can stand on the shoulders of giants, enabling us to build sophisticated business solutions. Specifically, readers of this book will learn: The fundamental characteristics of distributed systems, including state management, time coordination, concurrency, communications and coordination Architectural approaches and supporting technologies for building scalable, robust services How distributed databases operate and can be used to build scalable distributed systems Architectures and technologies such as Apache Kafka and Flink for building streaming, event-based systems

📄 Page 9

Part I. Scalability in Modern Software Systems A NOTE FOR EARLY RELEASE READERS With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles. If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at vwilson@oreilly.com. The first four chapters in Part 1 of this book motivate the need for scalability as a key architectural attribute in modern software systems. The chapters provide broad coverage of the basic mechanisms for achieving scalability, the fundamental characteristics of distributed systems, and an introduction to concurrent programming. This knowledge lays the foundation for what follows, and if you are new to the areas of distributed, concurrent systems, you’ll need to spend some time on these four chapters. They will make the rest of the book much easier to digest.

📄 Page 10

Chapter 1. Introduction to Scalable Systems A NOTE FOR EARLY RELEASE READERS With Early Release ebooks, you get books in their earliest form—the author’s raw and unedited content as they write—so you can take advantage of these technologies long before the official release of these titles. This will be the 1st chapter of the final book. If you have comments about how we might improve the content and/or examples in this book, or if you notice missing material within this chapter, please reach out to the editor at vwilson@oreilly.com. The last 20 years have seen unprecedented growth in the size, complexity and capacity of software systems. This rate of growth is hardly likely to slow in the next 20 years – what these future systems will look like is close to unimaginable right now. The one thing we can guarantee is that more and more software systems will need to be built with constant growth - more requests, more data, more analysis - as a primary design driver. Scalable is the term used in software engineering to describe software systems that can accommodate growth. In this chapter I’ll explore what precisely is meant by the ability to scale – known, not surprisingly, as scalability. I’ll also describe a few examples that put hard numbers on the capabilities and characteristics of contemporary applications and give a brief history of the origins of the massive systems we routinely build today. Finally, I’ll describe two general principles for achieving scalability, namely replication and optimization, that will recur in various forms throughout the

📄 Page 11

rest of this book and examine the indelible link between scalability and other software architecture quality attributes. What is Scalability? Intuitively, scalability is a pretty straightforward concept. If we ask Wikipedia for a definition, it tells us “scalability is the property of a system to handle a growing amount of work by adding resources to the system.” We all know how we scale a highway system – we add more traffic lanes so it can handle a greater number of vehicles. Some of my favorite people know how to scale beer production – they add more capacity in terms of the number and size of brewing vessels, the number of staff to perform and manage the brewing process, and the number of kegs they can fill with tasty fresh brews. Think of any physical system – a transit system, an airport, elevators in a building – and how we increase capacity is pretty obvious. Unlike physical systems, software systems are somewhat amorphous. They are not something you can point at, see, touch, feel, and get a sense of how it behaves internally from external observation. It’s a digital artifact. At its core, the stream of 1’s and 0’s that make up executable code and data are hard for anyone to tell apart. So, what does scalability mean in terms of a software system? Put very simply, and without getting into definition wars, scalability defines a software system’s capability to handle growth in some dimension of its operations. Examples of operational dimensions are: The number of simultaneous user or external (e.g. sensor) requests a system can process The amount of data a system can effectively process and manage The value that can be derived from the data a system stores The ability to maintain a stable. consistent response time as the request load grows

📄 Page 12

For example, imagine a major supermarket chain is rapidly opening new stores and increasing the number of self-checkout kiosks in every store. This requires the core supermarket software systems to: Handle increased volume from item scanning without decreased response time. Instantaneous responses to item scans are necessary to keep customers happy. Process and store the greater data volumes generated from increased sales. This data is needed for inventory management, accounting, planning and likely many other functions. Derive ‘real-time’ (e.g. hourly) sales data summaries from each store, region and country and compare to historical trends. This trend data can help highlight unusual events in regions (e.g. unexpected weather conditions, large crowds at events, etc.) and help the stores affected quickly respond. Evolve the stock ordering prediction subsystem to be able to correctly anticipate sales (and hence the need for stock reordering) as the number of stores and customers grow These dimensions are effectively the scalability requirements of the system. If, over a year, the supermarket chain opens 100 new stores and grows sales by 400 times (some of the new stores are big!), then the software system needs to scale to provide the necessary processing capacity to enable the supermarket to operate efficiently. If the systems don’t scale, we could lose sales as customers are unhappy. We might hold stock that will not be sold quickly, increasing costs. We might miss opportunities to increase sales by responding to local circumstances with special offerings. All these reduce customer satisfaction and profits. None are good for business. Successfully scaling is therefore crucial for our imaginary supermarket’s business growth, and likewise is in fact the lifeblood of many modern internet applications. But for most business and Government systems, scalability is not a primary quality requirement in the early stages of development and deployment. New features to enhance usability and utility

📄 Page 13

become the drivers of our development cycles. As long as performance is adequate under normal loads, we keep adding user-facing features to enhance the system’s business value. In fact, introducing some of the sophisticated distributed technologies I’ll describe in this book before there is a clear requirement can actually handicap a project, with the additional complexity causing development inertia. Still, it’s not uncommon for systems to evolve into a state where enhanced performance and scalability become a matter of urgency, or even survival. Attractive features and high utility breed success, which brings more requests to handle and more data to manage. This often heralds a tipping point, where design decisions that made sense under light loads are now suddenly technical debt. External trigger events often cause these tipping points – look in the March/April 2020 media for the many reports of Government Unemployment and supermarket online ordering sites crashing under demand caused by the coronavirus pandemic. Increasing a systems’ capacity in some dimension by increasing resources is called scaling up or scaling out – I’ll explore the difference between these later. In addition, unlike physical systems, it is often equally important to be able to scale down the capacity of a system to reduce costs. The canonical example of this is Netflix, which has a predictable regional diurnal load that it needs to process. Simply, a lot more people are watching Netflix in any geographical region at 9pm than are at 5am. This enables Netflix to reduce its processing resources during times of lower load. This saves the cost of running the processing nodes that are used in the Amazon cloud, as well as societally worthy things such as reducing data center power consumption. Compare this to a highway. At night when few cars are on the road, we don’t retract lanes (except for repairs). The full road capacity is available for the few drivers to go as fast as they like. In software systems, we can expand and contract our processing capacity in a matter of seconds to meet instantaneous load. Compared to physical systems, the strategies we deploy are very, very different.

📄 Page 14

There’s a lot more to consider about scalability in software systems, but let’s come back to these issues after examining the scale of some contemporary software systems circa 2021. System scale in early 2020’s: Examples Looking ahead in this technology game is always fraught with danger. In 2008 I wrote [1]: While petabyte datasets and gigabit data streams are today’s frontiers for data-intensive applications, no doubt 10 years from now we’ll fondly reminisce about problems of this scale and be worrying about the difficulties that looming exascale applications are posing. Reasonable sentiments, it is true, but exascale? That’s almost commonplace in today’s world. Google reported multiple exabytes of Gmail in 2014, and by now, do all Google services manage a yottabyte or more? I don’t know. I’m not even sure I know what a yottabyte is! Google won’t tell us about their storage, but I wouldn’t bet against it. Similarly, how much data does Amazon store in the various AWS data stores for their clients. And how many requests does, say, DynamoDB process per second collectively, for all client applications supported? Think about these things for too long and your head will explode. A great source of information that sometimes gives insights into contemporary operational scales are the major Internet company’s technical blogs. There are also Web sites analyzing Internet traffic that are highly illustrative of traffic volumes. Let’s take a couple of ‘point in time’ examples to illustrate a few things we do know today. Bear in mind these will look almost quaint in a year or four. Facebook’s engineering blog describes Scribe, their solution for collecting, aggregating, and delivering petabytes of log data per hour, with low latency and high throughput. Facebook’s computing infrastructure comprises millions of machines, each of which generates log files that capture important events relating to system and

📄 Page 15

application health. Processing these log files, for example from a Web server, can give development teams insights into their application’s behavior and performance, and support fault finding. Scribe is a custom buffered queuing solution that can transport logs from servers at a rate of several terabytes per second and deliver them to downstream analysis and data warehousing systems. That, my friends, is a lot of data! You can see live Internet traffic for numerous services at www.internetlivestats.com. Dig around and you’ll find statistics like Google handles around 3.5 billion search requests a day, Instagram uploads about 65 million photos per day, and there is something like 1.7 billion web sites. It is a fun site with lots of information to amaze you. Note the data is not really ‘live’, just estimates based on statistical analyses of multiple data sources. In 2016 Google published a paper describing the characteristics of their code base. Amongst the many startling facts reported is: “The repository contains 86TBs of data, including approximately two billion lines of code in nine million unique source files.” Remember, this was 2016. Still, real, concrete data on the scale of the services provided by major Internet sites remain shrouded in commercial-in-confidence secrecy. Luckily, we can get some deep insights into the request and data volumes handled at Internet scale through the annual usage report from one tech company. You can browse their incredibly detailed usage statistics from 2019. It’s a fascinating glimpse into the capabilities of massive scale systems. Beware though, this is Pornhub.com. How Did We Get Here? A Brief History of System Growth I am sure many readers will have trouble believing there was civilized life without Internet search, YouTube and social media. In fact, the first video 1

📄 Page 16

upload to YouTube occurred in 2005. Yep, it is hard for even me to believe. So, let’s take a brief look back in time at how we arrived at the scale of today’s systems. Below are some historical milestones of note: 1980s An age dominated by timeshared mainframe and minicomputers. PCs emerged in the early 1980s but were rarely networked. By the end of the 1980s, development labs, universities and increasingly businesses had email and access to primitive Internet resources. 1990-95 Networks became more pervasive, creating an environment ripe for the creation of the World Wide Web (WWW) with HTTP/HTML technology that had been pioneered at CERN by Tim Berners-Lee during the 1980s. By 1995, the number of web sites was tiny, but the seeds of the future were planted with companies like Yahoo! in 1994 and Amazon and eBay in 1995 1996-2000 The number of web sites grew from around 10,000 to 10 million, a truly explosive growth period. Networking bandwidth and access also grew rapidly. Companies like Amazon, eBay, Google, Yahoo! and the like were pioneering many of the design principles and early versions of advanced technologies for highly scalable systems that we know and use today. Everyday businesses rushed to exploit the new opportunities that e-business offered, and this brought system scalability to prominence, as explained in the sidebar. 2000-2006 The number of web sites grew from around 10 to 80 million during this period, and new service and business models emerged. In 2005, YouTube was launched. 2006 saw Facebook become available to the public. In the same year, Amazon Web Services, which had low key beginnings in 2004, relaunched with its S3 and EC2 services.

📄 Page 17

2007-today We now live in a world with around 2 billion web sites, of which about 20% are active. There are something like 4 billion Internet users. Huge data centers operated by public cloud operators like AWS, GCP and Azure, along with a myriad of private data centers, for example Twitter’s operational infrastructure, are scattered around the planet. Clouds host millions of applications, with engineers provisioning and operating their computational and data storage systems using sophisticated cloud management portals. Powerful cloud services make it possible for us to build, deploy and scale our systems literally with a few clicks of a mouse. All you do is pay your cloud provider bill at the end of the month. This is the world that this book targets. A world where our applications need to exploit the key principles for building scalable systems and leverage highly scalable infrastructure platforms. Bear in mind, in modern applications, most of the code executed is not written by your organization. It is part of the containers, databases, messaging systems and other components that you compose into your application through API calls and build directives. This makes the selection and use of these components at least as important as the design and development of your own business logic. They are architectural decisions that are not easy to change.

📄 Page 18

HOW SCALE IMPACTED BUSINESS SYSTEMS The surge of users with Internet access in the 1990s brought new online money making opportunities for businesses. There was a huge rush to expose business functions - sales, services - to users through a Web browser. This heralded a profound change in how we had to think about building systems. Take for example a retail bank. Before providing online services, it was possible to accurately predict the loads the bank’s business systems would experience. You knew how many people worked in the bank and used the internal systems, how many terminals/PCs were connected to the bank’s networks, how many ATMs you had to support, and the number and nature of connections to other financial institutions. Armed with this knowledge, we could build systems that support, say, a maximum of say 3000 concurrent users, safe in the knowledge that this number could not be exceeded. Growth would also be relatively slow, and probably most of the time (eg outside business hours) the load would be a lot less than the peak. This made our software design decisions and hardware provisioning a lot easier. Now imagine our retail bank decides to let all customers have Internet banking access. And the bank has 5 million customers. What is our maximum load now? How will load be dispersed during a business day? When are the peak periods? What happens if we run a limited time promotion to try and sign up new customers? Suddenly our relatively simple and constrained business systems environment is disrupted by the higher average and peak loads and unpredictability you see from Internet-based user populations. Scalability Basic Design Principles The basic aim of scaling a system is to increase its capacity in some application-specific dimension. A common dimension is increasing the

📄 Page 19

number of requests that a system can process in a given time period. This is known as the system’s throughput. Let’s use an analogy to explore two basic principles we have available to us for scaling our systems and increasing throughput: replication and optimization. In 1932, one of the world’s great icons, the Sydney Harbor Bridge, was opened. Now it is a fairly safe assumption that traffic volumes in 2021 are somewhat higher than in 1932. If by any chance you have driven over the bridge at peak hour in the last 30 years, then you know that its capacity is exceeded considerably every day. So how do we increase throughput on physical infrastructures such as bridges? This issue became very prominent in Sydney in the 1980s, when it was realized that the capacity of the harbor crossing had to be increased. The solution was the rather less iconic Sydney Harbor tunnel, which essentially follows the same route underneath the harbor. This provides 4 more lanes of traffic, and hence added roughly 1/3rd more capacity to harbor crossings. In not too far away Auckland, their harbor bridge also had a capacity problem as it was built in 1959 with only 4 lanes. In essence, they adopted the same solution as Sydney, namely, to increase capacity. But rather than build a tunnel, they ingeniously doubled the number of lanes by expanding the bridge with the hilariously named 'Nippon Clipons', which widened the bridge on each side. These examples illustrate the first strategy we have in software systems to increase capacity. We basically replicate the software processing resources to provide more capacity to handle requests and thus increase throughput, as shown in Figure 1-1. These replicated processing resources are analogous to the traffic lanes on bridges, providing a mostly independent processing pathway for a stream of arriving requests. Luckily, in cloud- based software systems, replication can be achieved at the click of a mouse, and we can effectively replicate our processing resources thousands of times. We have it a lot easier than bridge builders in that respect. Still. we need to take care to replicate resources in order to alleviate bottlenecks, otherwise our resources will simply cause needless costs and give no scalability benefit.

📄 Page 20

Figure 1-1. Increasing Capacity through Replication The second strategy for scalability can also be illustrated with our bridge example. In Sydney, some observant person realized that in the mornings a lot more vehicles cross the bridge from north to south, and in the afternoon we see the reverse pattern. A smart solution was therefore devised – allocate more of the lanes to the high demand direction in the morning, and sometime in the afternoon, switch this around. This effectively increased the capacity of the bridge without allocating any new resources – we optimized the resources we already had available. We can follow this same approach in software to scale our systems. If we can somehow optimize our processing, by maybe using more efficient algorithms, adding extra indexes in our databases to speed up queries, or even rewriting our server in a faster programming language, we can increase our capacity without increasing our resources. The canonical example of this is Facebook’s creation of (the now discontinued) HipHop for PHP, which increased the speed of Facebook’s web page generation by up to 6 times by compiling PHP code to C++. I’ll revisit these two design principles – namely replication and optimization - many times in the remainder of this book. You will see that there are many complex implications of adopting these principles that arise from the fact that we are building distributed systems. Distributed systems have properties that make building scalable systems ‘interesting’, where interesting in this context has both positive and negative connotations.