Page
1
(This page has no text content)
Page
2
Kotlin Coroutines Deep Dive MarcinMoskała This book is for sale at http://leanpub.com/coroutines This versionwas published on 2023-06-26 ISBN 978-83-963958-1-8 This is a Leanpub book. Leanpub empowers authors and publishers with the Lean Publishing process. Lean Publishing is the act of publishing an in-progress ebook using lightweight tools andmany iterations to get reader feedback, pivot until you have the right book and build traction once you do. © 2021 - 2023MarcinMoskała
Page
3
Tweet This Book! Please helpMarcinMoskała by spreading theword about this book onTwitter! The suggested tweet for this book is: I just bought Kotlin Coroutines byMarcinMoskała The suggested hashtag for this book is #KotlinCoroutines. Find outwhat other people are saying about the book by clicking on this link to search for this hashtag onTwitter: #KotlinCoroutines
Page
4
Formy belovedwife,Maja.
Page
5
Contents Introduction 1 Part 1: UnderstandingKotlinCoroutines 7 WhyKotlin Coroutines? 8 Sequence builder 21 Howdoes suspensionwork? 27 Coroutines under the hood 39 Coroutines: built-in support vs library 57 Part 2:KotlinCoroutines library 59 Coroutine builders 60 Coroutine context 76 Jobs and awaiting children 88 Cancellation 101 Exception handling 116 Coroutine scope functions 127 Dispatchers 147 Constructing a coroutine scope 167 The problemwith shared state 177 Testing Kotlin Coroutines 192 Part 3: Channel andFlow 228 Channel 229 Select 251 Hot and cold data sources 257 Flow introduction 265 Understanding Flow 277 Flow building 289 Flow lifecycle functions 301 Flow processing 312 SharedFlow and StateFlow 339 Testing flow 353
Page
6
CONTENTS Part 4:KotlinCoroutines in practice 370 Commonuse cases 371 Recipes 396 Using coroutines fromother languages 411 Launching coroutines vs. suspending functions 424 Best practices 428 TheEnd 444
Page
7
Introduction Why do you want to learn about Kotlin Coroutines? I often ask my workshop attendees this question. “Because they’re cool” and “Be- cause everyone is talking about them” are common answers. Then, when I dig deeper, I hear responses like “because they are lighter threads”, “because they are easier than RxJava”, or “because they enable concurrency while maintaining an imperative code style.” However, coroutines are much more than that. They are the holy grail of concurrency. As a concept, they have been known in Com- puterSciencesince the1960s,but inmainstreamprogramming, they have only been used in a very limited form (such as async/await). This changed with Golang, which introduced much more general- purpose coroutines. Kotlin built on that, creating what I believe is currently the most powerful and practical implementation of this idea. The importance of concurrency is growing, but traditional tech- niques are not enough. Current trends suggest that coroutines are the direction in which our industry is clearly heading, and Kotlin Coroutines represent a significant step forward. Let me introduce them to you, alongwith examples of howwell they handle common use cases. I hope youwill have a lot of fun reading this book. Who is this book for? Asadeveloperexperienced inbothbackendandAndroid, Iprimarily focus on these two perspectives in this book. These are currently the two major industry applications of Kotlin, and it’s evident that coroutineswere largelydesigned to suit theseuse caseswell¹. So, you might say that this book ismainlydesigned forAndroidandbackend developers, but it should be just as useful for other developers using Kotlin. This book assumes that readers know Kotlin well. If you do not, I recommend startingwithmy other book,Kotlin Essentials. ¹Google’s Android team cooperated in designing and creating some featureswewill present in this book. 1
Page
8
Introduction 2 The structure of this book The book is divided into the following parts: • Part 1: Understanding Kotlin Coroutines - dedicated to ex- plainingwhatKotlinCoroutines are andhow they reallywork. • Part2:KotlinCoroutines library - explaining themost impor- tant concepts from the kotlinx.coroutines library and how to use themwell. • Part 3: Channel andFlow - focused onChannel and Flow from the kotlinx.coroutines library. • Part4:KotlinCoroutines inpractice - reviewing commonuse cases and the most important best practices for using Kotlin Coroutines. What will be covered? This book is based on a workshop I conduct. During its iterations I havebeenable toobservewhat interestedattendeesandwhatdidnot. These are the elements that aremost oftenmentioned by attendees: • Howdo coroutines reallywork? (Part 1) • Howtouse coroutines in practice? (Part 2, 3 and especially 4) • What are the best practices? (Part 2, 3 and especially 4) • Testing Kotlin coroutines (Testing Kotlin Coroutines in Part 2 and Testing Flow in Part 3) • What is Flowandhowdoes itwork? (Part 3) The Kotlin for Developers series This book is a part of a series of books called Kotlin for Developers, which includes the following books: • Kotlin Essentials, which covers all the basic Kotlin features. • Functional Kotlin, which is dedicated to functional Kotlin features, including function types, lambda expressions, collec- tion processing, DSLs, and scope functions. • Kotlin Coroutines, which covers all the features of Kotlin Coroutines, including how to use and test them, using flow, best practices, and themost commonmistakes.
Page
9
Introduction 3 • Advanced Kotlin, which is dedicated to advanced Kotlin fea- tures, including generic variance modifiers, delegation, mul- tiplatform programming, annotation processing, KSP, and compiler plugins. • Effective Kotlin, which is dedicated to the best practices of Kotlin programming. Conventions When Iuse a concrete element fromcode, Iwill use code-font.When naming a concept, I will capitalize the word. To reference an arbi- trary element of some type, I will use lowercase. For example: • Flow is a type or an interface, it is printed with code-font (like in “Function needs to return Flow”); • Flow represents a concept, so it startswith an uppercase letter (like in “This explains the essential difference between Chan- nel and Flow”); • a flow is an instance, like a list or a set, therefore it is in lowercase (like in “Every flow consists of a few elements”). Another example:Listmeansan interfaceora type (“The typeof l is List”); List represents a concept (a data structure), while a list is one ofmany lists (“the list variable holds a list”). I have used American English in the book, except for the spelling of “cancellation/cancelled”, which I chose due to the spelling of the “Cancelled” coroutine state. Code conventions Most of the presented snippets are executable code with no import statements. Some chapters of this book are published as articles on the Kt. Academywebsite, wheremost snippets can be executed, and readers can playwith the code. The source code of all the snippets is published in the following repository: https://github.com/MarcinMoskala/coroutines_sources Snippet results are presented using the println function. The result will oftenbeplacedat theendof these snippets incomments. If there is a delay between output lines, it will be shown in brackets. Here is an example:
Page
10
Introduction 4 suspend fun main(): Unit = coroutineScope { launch { delay(1000L) println("World!") } println("Hello,") } // Hello, // (1 sec) // World! Sometimes, someparts of code or a result are shortenedwith .... In such cases, you can read it as “there shouldbemorehere, but it is not relevant to the example”. launch(CoroutineName("Name1")) { ... } launch(CoroutineName("Name2") + Job()) { ... } Insomecases, Iwill showcommentsnext tothe linethatprints them. I do this when the order is clear: suspend fun main(): Unit = coroutineScope { println("Hello,") // Hello, delay(1000L) // (1 sec) println("World!") // World! } In a few snippets I have added a number after the line tomore easily explain the snippet’s behavior. This is what itmight look like: suspend fun main(): Unit = coroutineScope { println("Hello,") // 1 delay(1000L) // 2 println("World!") // 3 } At 1 we print “Hello,”, then we wait for a second because line 2 contains delay, andwe print “World!” at line 3.
Page
11
Introduction 5 Acknowledgments This bookwould not be so goodwithout the reviewers’ great sugges- tions and comments. I would like to thank all of them. Here is the whole list of reviewers, starting from themost active ones. Nicola Corti - a Google Developer Expert for Kotlin.Hehasbeenworkingwiththe language since before version 1.0, and he is the main- tainer of several open-source libraries and tools for mobile developers (Detekt, Chucker, AppIntro).He’s currentlyworking in theReact Native core team at Meta, building one of the most popular cross-platform mobile frameworks. Furthermore, he is an active member of the developer community. His involvement goes from speaking at international conferences to being a member of CFP committees and supporting developer communities across Europe. In his free time, he also loves baking, podcasting, and run- ning. Garima Jain - a Google Developer Expert in Android from India. She is also knownaround the community as @ragdroid. Garima works as a Principal Android Engineer at GoDaddy. She is also an international speaker and an active technical blogger. She enjoys interact- ing with other people from the community and sharing her thoughts with them. In her leisure time, she loves watching television shows, playing TT, and basketball. Due to her love for fiction and coding, she loves tomix technology with fiction and then shares her ideaswith others through talks and blog posts. Ilmir Usmanov - a software developer at Jet- Brains, working on coroutine support in the Kotlin compiler since 2017. Was responsible for stabilization and implementation of the coroutines design. Since then, he has moved to other features, namely inline classes. Cur- rently, his work with coroutines is limited to bug fixing and optimization, since coroutines as a language feature is complete and stable and does not requiremuch attention.
Page
12
Introduction 6 Sean McQuillan - a Developer Advocate at Google. With a decade of experience at Twilio and other San Francisco startups, he is an ex- pert at building apps that scale. Sean is pas- sionate about using great tooling to build high- quality apps quickly. When he is not working on Android, you can find him fiddling on the piano or crocheting hats. Igor Wojda - a passionate engineer with over a decade of software development experience. He is deeply interested in Android application architecture and the Kotlin language, and he is an active member of the open-source com- munity. Igor is a conference speaker, technical proofreader for the ‘Kotlin In Action’ book, and author of the ‘Android Development with Kotlin’ book. Igor enjoys sharing his passion for codingwith other developers. JanaJarolimova - anAndroid developer at Avast. She started her ca- reer teaching Java classes at Prague City University, before moving on to mobile development, which inevitably led to Kotlin and her love thereof. RichardSchielek - an experienced developer and an early adopter of Kotlin and coroutines, using both in production before they became stable.Worked in the European space industry for several years. VsevolodTolstopyatov - a team lead of theKotlin Libraries team.He worksatJetBrainsand is interested inAPIdesign, concurrency,JVM internals, performance tuning andmethodologies. Lukas Lechner, Ibrahim Yilmaz, Dean Djermanović and Dan O’Neill. I would also like to thank Michael Timberlake, our language re- viewer, for his excellent corrections to thewhole book.
Page
13
Part 1: Understanding Kotlin Coroutines Before we begin our adventure with the Kotlin Coroutines library, let’s startwith somemorebasic concepts.Whatare coroutines?How does suspensionwork?Whatdoes it all look likeunder thehood?We will explore all this while learning some useful tools and practices. 7
Page
14
Part 1: Understanding Kotlin Coroutines 8 Why Kotlin Coroutines? Why do we need to learn Kotlin Coroutines? We already have well- established JVM libraries like RxJava or Reactor. Moreover, Java itselfhas support formultithreading,whilemanypeoplealso choose to just use plain old callbacks instead. Clearly,we alreadyhavemany options for performing asynchronous operations. Kotlin Coroutines offer much more than that. They are an imple- mentation of a concept that was first described in 1963² but waited years for a proper industry-ready implementation³. Kotlin Corou- tines connects powerful capabilities presented by half-century-old papers to a library that is designed to perfectly help in real-life use cases. What is more, Kotlin Coroutines are multiplatform, which means they can be used across all Kotlin platforms (like JVM, JS, iOS, and also in the common modules). Finally, they do not change the code structure drastically. We can use most Kotlin coroutines’ capabilitiesnearly effortlessly (whichwecannot say aboutRxJavaor callbacks). Thismakes thembeginner-friendly⁴. Let’s see it in practice. We will explore how different common use cases are solved by coroutines and other well-known approaches. I will show two typical use cases: Android and backend business logic implementation. Let’s start with the first one. ²Conway,MelvinE. (July1963).“DesignofaSeparableTransition- diagram Compiler”. Communications of the ACM. ACM. 6 (7): 396– 408. doi:10.1145/366663.366704. ISSN 0001-0782. S2CID 10559786 ³I believe that the first industry-ready and universal coroutines were introduced by Go in 2009. However, it is worth mentioning that coroutineswere also implemented in someolder languages, like Lisp, but they didn’t become popular. I believe this is because their implementationwasn’t designed to support real-life cases. Lisp (just likeHaskell) wasmostly treated as a playground for scientists rather than as a language for professionals. ⁴This does not change the fact that we should understand corou- tines to use themwell.
Page
15
Part 1: Understanding Kotlin Coroutines 9 Coroutines on Android (and other frontend platforms) When you implement application logic on the frontend, what you most often need to do is: 1. get some data from one or many sources (API, view element, database, preferences, another application); 2. process this data; 3. do somethingwith this data (display it in the view, store it in a database, send it to an API). To make our discussion more practical, let’s first assume we are developing an Android application.Wewill start with a situation in whichwe need to get news from an API, sort it, and display it on the screen. This is a direct representation ofwhatwewant our function to do: fun onCreate() { val news = getNewsFromApi() val sortedNews = news .sortedByDescending { it.publishedAt } view.showNews(sortedNews) } Sadly, this cannot be done so easily. On Android, each application has only one thread that canmodify the view.This thread is very im- portant and shouldneverbeblocked.That iswhy theabove function cannot be implemented in this way. If it were started on the main thread, getNewsFromApi would block it, and our application would crash. Ifwestarted itonanother thread,ourapplicationwouldcrash whenwe call showNews because it needs to run on themain thread. Thread switching Wecouldsolve theseproblemsbyswitching threads.First toa thread that can be blocked, and then to themain thread.
Page
16
Part 1: Understanding Kotlin Coroutines 10 fun onCreate() { thread { val news = getNewsFromApi() val sortedNews = news .sortedByDescending { it.publishedAt } runOnUiThread { view.showNews(sortedNews) } } } Such thread switching can still be found in some applications, but it is known for being problematic for several reasons: • There is no mechanism here to cancel these threads, so we often facememory leaks. • Making somany threads is costly. • Frequently switching threads is confusing and hard to man- age. • The codewill unnecessarily get bigger andmore complicated. To see those problems clearly, imagine the following situation: You open and quickly close a view. While opening, you might have started multiple threads that fetch and process data. Without cancelling them, they will still be doing their job and trying to modify a view that no longer exists. This means unnecessary work for your device, possible exceptions in the background, and who knowswhat other unexpected results. Considering all these problems, let’s look for a better solution. Callbacks Callbacks are another pattern that might be used to solve our prob- lems. The idea is that we make our functions non-blocking, but we pass to them a function that should be executed once the process started by the callback function has finished. This is how our func- tionmight look if we use this pattern:
Page
17
Part 1: Understanding Kotlin Coroutines 11 fun onCreate() { getNewsFromApi { news -> val sortedNews = news .sortedByDescending { it.publishedAt } view.showNews(sortedNews) } } Notice that this implementation does not support cancellation. We might make cancellable callback functions, but it is not easy. Not only does each callback function need to be specially implemented for cancellation, but to cancel themweneed to collect all the objects separately. fun onCreate() { startedCallbacks += getNewsFromApi { news -> val sortedNews = news .sortedByDescending { it.publishedAt } view.showNews(sortedNews) } } Callback architecture solves this simple problem, but it has many downsides. To explore them, let’s discuss a more complex case in whichwe need to get data from three endpoints: fun showNews() { getConfigFromApi { config -> getNewsFromApi(config) { news -> getUserFromApi { user -> view.showNews(user, news) } } } } This code is far fromperfect for several reasons: • Getting news and user data might be parallelized, but our current callback architecture doesn’t support that (itwouldbe hard to achieve this with callbacks).
Page
18
Part 1: Understanding Kotlin Coroutines 12 • Asmentioned before, supporting cancellationwould require a lot of additional effort. • The increasingnumber of indentationsmake this codehard to read (code with multiple callbacks is often considered highly unreadable). Such a situation is called “callback hell”, which can be found especially in some older Node.JS projects: • Whenweusecallbacks, it ishard tocontrolwhathappensafter what. The following way of showing a progress indicator will notwork: fun onCreate() { showProgressBar() showNews() hideProgressBar() // Wrong }
Page
19
Part 1: Understanding Kotlin Coroutines 13 The progress bar will be hidden just after starting the process of showing news, so practically immediately after it has been shown. To make this work, we would need to make showNews a callback function aswell. fun onCreate() { showProgressBar() showNews { hideProgressBar() } } That’s why the callback architecture is far from perfect for non- trivial cases. Let’s take a look at another approach:RxJava andother reactive streams. RxJava and other reactive streams Analternative approach that is popular in Java (both inAndroid and backend) is using reactive streams (or Reactive Extensions): RxJava or its successor Reactor. With this approach, all operations happen inside a stream of data that can be started, processed, and observed. These streamssupport thread-switchingandconcurrentprocessing, so they are often used to parallelize processing in applications. This is howwemight solve our problemusing RxJava: fun onCreate() { disposables += getNewsFromApi() .subscribeOn(Schedulers.io()) .observeOn(AndroidSchedulers.mainThread()) .map { news -> news.sortedByDescending { it.publishedAt } } .subscribe { sortedNews -> view.showNews(sortedNews) } } The disposables in the above example are needed to cancel this stream if (for example) the user exits the screen.
Page
20
Part 1: Understanding Kotlin Coroutines 14 This is definitely a better solution than callbacks: nomemory leaks, cancellation is supported, proper use of threads. The only problem is that it is complicated. If you compare itwith the “ideal” code from thebeginning (also shownbelow),you’ll see that theyhavevery little in common. fun onCreate() { val news = getNewsFromApi() val sortedNews = news .sortedByDescending { it.publishedAt } view.showNews(sortedNews) } All these functions, like subscribeOn, observeOn, map, or subscribe, need to be learned. Cancelling needs to be explicit. Functions need to return objects wrapped inside Observable or Single classes. In practice,whenwe introduceRxJava,weneed to reorganize our code a lot. fun getNewsFromApi(): Single<List<News>> Think of the second problem, for which we need to call three end- points before showing the data. This can be solved properly with RxJava, but it is evenmore complicated. fun showNews() { disposables += Observable.zip( getConfigFromApi().flatMap { getNewsFromApi(it) }, getUserFromApi(), Function2 { news: List<News>, config: Config -> Pair(news, config) .subscribeOn(Schedulers.io()) .observeOn(AndroidSchedulers.mainThread()) .subscribe { (news, config) -> view.showNews(news, config) } }