Succeeding with AI How to Make AI Work for Your Business (Veljko Krunic) (Z-Library)

M A N N I N G Veljko Krunic How to make AI work for your business

Succeeding with AI

ii

Succeeding with AI HOW TO MAKE AI WORK FOR YOUR BUSINESS VELJKO KRUNIC M A N N I N G SHELTER ISLAND

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com ©2020 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. Manning Publications Co. Acquisitions editor: Mike Stephens 20 Baldwin Road Development editor: Marina Michaels PO Box 761 and Jennifer Stout Shelter Island, NY 11964 Technical development editor: Al Krinker Review editor: Ivan Martinović Production editor: Anthony Calcara Copy editor: Carl Quesnel ESL editor: Frances Buran Proofreader: Keri Hales Typesetter and cover designer: Marija Tudor Neither Manning nor the Author make any warranty regarding the completeness, accuracy, timeliness, or other fitness for use nor the results obtained from the use of the contents herein and accept no liability for any decision or action taken in reliance on the information in this book nor for any damages resulting from this work or its application. ISBN 9781617296932 Printed in the United States of America

brief contents 1 ■ Introduction 1 2 ■ How to use AI in your business 26 3 ■ Choosing your first AI project 53 4 ■ Linking business and technology 82 5 ■ What is an ML pipeline, and how does it affect an AI project? 112 6 ■ Analyzing an ML pipeline 135 7 ■ Guiding an AI project to success 165 8 ■ AI trends that may affect you 195v

BRIEF CONTENTSvi

contents preface xiii acknowledgments xv about this book xvii about the author xxi about the cover illustration xxii 1 Introduction 1 1.1 Whom is this book for? 2 1.2 AI and the Age of Implementation 4 1.3 How do you make money with AI? 6 1.4 What matters for your project to succeed? 7 1.5 Machine learning from 10,000 feet 8 1.6 Start by understanding the possible business actions 11 1.7 Don’t fish for “something in the data” 13 1.8 AI finds correlations, not causes! 15 1.9 Business results must be measurable! 16 1.10 What is CLUE? 19 1.11 Overview of how to select and run AI projects 21 1.12 Exercises 23 True/False questions 24 ■ Longer exercises: Identify the problem 24vii

CONTENTSviii2 How to use AI in your business 26 2.1 What do you need to know about AI? 27 2.2 How is AI used? 29 2.3 What’s new with AI? 31 2.4 Making money with AI 33 AI applied to medical diagnosis 34 ■ General principles for monetizing AI 36 2.5 Finding domain actions 38 AI as part of the decision support system 39 ■ AI as a part of a larger product 40 ■ Using AI to automate part of the business process 42 ■ AI as the product 43 2.6 Overview of AI capabilities 45 2.7 Introducing unicorns 47 Data science unicorns 47 ■ What about data engineers? 48 So where are the unicorns? 49 2.8 Exercises 50 Short answer questions 51 ■ Scenario-based questions 51 3 Choosing your first AI project 53 3.1 Choosing the right projects for a young AI team 54 The look of success 54 ■ The look of failure 57 3.2 Prioritizing AI projects 59 React: Finding business questions for AI to answer 60 Sense/Analyze: AI methods and data 63 ■ Measuring AI project success with business metrics 65 ■ Estimating AI project difficulty 68 3.3 Your first project and first research question 69 Define the research question 70 ■ If you fail, fail fast 74 3.4 Pitfalls to avoid 74 Failing to build a relationship with the business team 75 Using transplants 75 ■ Trying moonshots without the rockets 76 ■ It’s about using advanced tools to look at the sea of data 77 ■ Using your gut feeling instead of CLUE 78 3.5 Exercises 80

CONTENTS ix4 Linking business and technology 82 4.1 A project can’t be stopped midair 83 What constitutes a good recommendation engine? 83 ■ What is gut feeling? 85 4.2 Linking business problems and research questions 85 Introducing the L part of CLUE 86 ■ Do you have the right research question? 87 ■ What questions should a metric be able to answer? 87 ■ Can you make business decisions based on a technical metric? 88 ■ A metric you don’t understand is a poor business metric 91 ■ You need the right business metric 93 4.3 Measuring progress on AI projects 94 4.4 Linking technical progress with a business metric 96 Why do we need technical metrics? 97 ■ What is the profit curve? 97 ■ Constructing a profit curve for bike rentals 99 Why is this not taught in college? 102 ■ Can’t businesses define the profit curve themselves? 103 ■ Understanding technical results in business terms 105 4.5 Organizational considerations 106 Profit curve precision depends on the business problem 106 A profit curve improves over time 107 ■ It’s about learning, not about being right 108 ■ Dealing with information hoarding 108 ■ But we can’t measure that! 109 4.6 Exercises 110 5 What is an ML pipeline, and how does it affect an AI project? 112 5.1 How is an AI project different? 113 The ML pipeline in AI projects 113 ■ Challenges the AI system shares with a traditional software system 117 ■ Challenges amplified in AI projects 117 ■ Ossification of the ML pipeline 118 ■ Example of ossification of an ML pipeline 121 How to address ossification of the ML pipeline 123 5.2 Why we need to analyze the ML pipeline 126 Algorithm improvement: MNIST example 126 ■ Further examples of improving the ML pipeline 127 ■ You must analyze the ML pipeline! 128 5.3 What’s the role of AI methods? 129 5.4 Balancing data, AI methods, and infrastructure 131 5.5 Exercises 133

CONTENTSx6 Analyzing an ML pipeline 135 6.1 Why you should care about analyzing your ML pipeline 136 6.2 Economizing resources: The E part of CLUE 138 6.3 MinMax analysis: Do you have the right ML pipeline? 140 6.4 How to interpret MinMax analysis results 142 Scenario: The ML pipeline for a smart parking meter 142 What if your ML pipeline needs improvement? 146 ■ Rules for interpreting the results of MinMax analysis 147 6.5 How to perform an analysis of the ML pipeline 147 Performing the Min part of MinMax analysis 149 ■ Performing the Max part of MinMax analysis 149 ■ Estimates and safety factors in MinMax analysis 152 ■ Categories of profit curves 154 ■ Dealing with complex profit curves 157 6.6 FAQs about MinMax analysis 159 Should MinMax be the first analysis of the ML pipeline? 160 Which analysis should you perform first? Min or Max? 160 ■ Should a small company or small team skip the MinMax analysis? 161 ■ Why do you use the term MinMax analysis? 161 6.7 Exercises 162 7 Guiding an AI project to success 165 7.1 Improving your ML pipeline with sensitivity analysis 166 Performing local sensitivity analysis 167 ■ Global sensitivity analysis 170 ■ Example of using sensitivity analysis results 171 7.2 We’ve completed CLUE 172 7.3 Advanced methods for sensitivity analysis 175 Is local sensitivity analysis appropriate for your ML pipeline? 176 ■ How to address the interactions between ML pipeline stages 179 ■ Should I use design of experiments? 180 ■ One common objection you might encounter 181 ■ How to analyze the stage that produces data 184 ■ What types of sensitivity analysis apply to my project? 184 7.4 How your AI project evolves through time 186 Time affects your business results 186 ■ Improving the ML pipeline over time 187 ■ Timing diagrams: How business value changes over time 188

CONTENTS xi7.5 Concluding your AI project 190 7.6 Exercises 192 8 AI trends that may affect you 195 8.1 What is AI? 196 8.2 AI in physical systems 198 First, do no harm 198 ■ IoT devices and AI systems must play well together 201 ■ The security of AI is an emerging topic 202 8.3 AI doesn’t learn causality, only correlations 203 8.4 Not all data is created equal 206 8.5 How are AI errors different from human mistakes? 207 The actuarial view 208 ■ Domesticating AI 210 8.6 AutoML is approaching 211 8.7 What you’ve learned isn’t limited to AI 213 8.8 Guiding AI to business results 214 8.9 Exercises 216 appendix A Glossary of terms 219 appendix B Exercise solutions 225 appendix C Bibliography 244 index 257

CONTENTSxii

preface Many AI projects are in progress today, and many of them will fail. This book will help you avoid starting an AI project that’s doomed to failure and will guide you toward the projects that can succeed. I wrote this book to help you get concrete business results, and to help you influ- ence how AI is used in industry today. Current discussions about AI focus on algo- rithms and case studies of successful applications. What’s lost in this discussion is the human element of AI. We see algorithms, and we know what large organizations have done with them, but what we don’t hear about is the leadership needed for an AI proj- ect to achieve business success and the principles applicable to our own organizations for leading AI projects. That causes us to have unrealistic expectations of what AI can do, and when paired with only a vague understanding of the actions leaders must take for their AI projects to succeed, the result is that many of the AI projects we have in progress will fail. This book addresses what differentiates the AI projects that will succeed from the ones that will fail. In one word, it is agency—a capacity that AI lacks. Conventional wis- dom tells us that the determinant of success or failure of an AI project is the project team’s in-depth knowledge of AI technology. Believing that success with AI is deter- mined solely by technical prowess confounds an enabler with a capability. Although you do need to have technical skills on your team for your AI project to succeed techni- cally, to implement AI in your business, you also need to link technical success with business results. The very definition of high tech is that it refers to a newly emerging technology. As a corollary, the best practices of that technology are only developed later, once the experiences and early application of technology are understood and systematized.xiii

PREFACExivThis book introduces the best practices for using AI. It guides you through the treach- erous waters of running an AI project in 2020 and beyond. This book shows you how to lead an AI project toward business success, measure technical progress in business terms, and run your projects economically. You’ll learn how to determine which AI projects are likely to give you actionable results, and how to get those results. Finally, this book teaches you how to analyze your technical solu- tions to help you find the investment opportunities with the greatest business impact.

acknowledgments I want to thank my wife, Helen Stella, for love, support, patience, advice, and encour- agement during the process of writing this book. You’re there when I’m winning, and you’re there while there are still challenges I have yet to conquer. Helen, I am lucky to have you in my life! I’d also like to thank Dr. Jeffrey Luftig, from whom I learned that the key to busi- ness success is bringing together business competence and strong technical profi- ciency in scientific methods you’re using. His work and teaching had a profound impact on my thinking (for example, his book with Steve Ouellette [1], his paper “TOTAL Asset Utilization” [2], and his course content [3]). Jeff taught me how to align business and technology, and I learned from him how to apply Peter Drucker’s dictum that it’s more important to be effective (do the right things) than to be effi- cient (to do things right) [4].1 I would also like to thank Steve Ouellette. Several years ago, I started writing a book quite different from this one, and Steve reviewed my early writing. This is, for all practical purposes, a very different book. Nevertheless, Steve’s thoughts on my previ- ous writing helped me write a better book. Most importantly, I would like to thank all the early adopters of AI and other novel technologies. You’re willing to take chances on the latest technologies, whose poten- tial you have the vision to see, as opposed to playing it safe and opting for “what every- one else is using.” Without people like you, software, high tech, and progress in general can’t exist. You’re the unsung heroes of technology revolutions. 1 Peter Drucker, from “Managing for Business Effectiveness” [4]: “It is fundamentally the confusion between effectiveness and efficiency that stands between doing the right things and doing things right. There is surely nothing quite so useless as doing with great efficiency what should not be done at all.”xv

ACKNOWLEDGMENTSxvi No book is the product of the author alone, and I would like to thank the Manning team. Associate Publisher Michael Stephens had the vision to understand what this book could become. All he had to work with at that time was a proposal for a book approaching AI projects from a different angle than any other book on the market. His ongoing help and guidance made this book possible. My copy editor Carl Quesnel has supplied a lot of invaluable suggestions regarding the style and the flow of the writing in this book, and the book is much better for his involvement. I would also like to thank my technical development editor, Al Krinker, for his technical review of this book and for pointing out many technical details to include in the text. In addition, I would like to thank my ESL editor, Frances Buran, who had the job of proofreading my initial drafts and correcting many spelling, grammatical, and typographical errors in them. I would also like to thank the reviewers of this book. Our whole community owes gratitude to people like them. These reviewers are presented with texts and ideas still in draft form. They donate their considerable knowledge and experience to read and evaluate writing full of rough edges and then help authors to revise their rough ideas so that the whole community can benefit. I’m embarrassed to realize how rarely I think about the work of reviewers when I’m a reader of the finished book. As an author, I came to appreciate their role, help, and guidance, and I did my best to incor- porate their advice in this book. Those reviewers were Andrea Paciolla; Ayon Roy; Craig Henderson; David Goldfarb; David Paccoud; Eric Cantuba; Ishan Khurana; James J. Byleckie, PhD; Jason Rendel; Jousef Murad; Madhavan Ramani; Manjula Iyer; Miguel Eduardo Gil Biraud; Nikos Kanakaris; Sara Khan; Simona Russo; Sune Lom- holt; Teresa Fontanella De Santis; and Zarak Mahmud. Although the whole team did their best to help me write a flawless book, I’m afraid that any published book will still have some errors and typos. My name is on the cover, and the buck stops with me. While I’m grateful to share the credit for many things that went well with this book, I invite readers to assign full credit for all errors, typos, and imperfections in this book to me.

about this book The purpose for writing Succeeding with AI: How to Make AI Work for Your Business was to help you lead an AI project toward business success. This book starts by showing you how to select AI projects that can become a business success, and then how to run those projects in a way that will achieve it. Who should read this book I wrote this book for the business leader who’s tasked with delivering results with AI and views technology as a vehicle to deliver those results. I’ve also written it for the leadership team that is working with and advising such a business leader. As a prerequisite, the reader of this book should have experience on the leader- ship team of a successful software project and should understand the business basics of their organization. Although an engineering background or deep knowledge about AI isn’t required, an open mind and a willingness to facilitate conversations between people with technical and business backgrounds are. I also wrote this book for leadership-focused and business-focused data scientists and data scientists who want to learn more about the business applications of AI meth- ods. I purposely don’t focus on specific technologies in AI, so if you’re interested solely in the technical side of AI, this is not the book for you. How this book is organized This book is organized into eight chapters:  Chapter 1 is an introduction to the AI project landscape today. It introduces you to the critical versus nice-to-have elements of a successful AI project and helps you understand business actions you can take based on AI project results.xvii

ABOUT THIS BOOKxviiiIt also provides a high-level overview of the process that a successful AI project should use.  Chapter 2 introduces you to topics project leaders must know about AI. It helps you find which business problems benefit from the use of AI and match AI capabilities with the business problems you need to solve. It also helps you uncover any data science skill gaps on your team that might affect your project.  Chapter 3 helps you select your first AI project and formulate a research ques- tion directed at your business problem. It also presents pitfalls to avoid when selecting AI projects, as well as best practices of such projects.  Chapter 4 shows you how to link business and technology metrics and how to measure technical progress in business terms. It also shows you how to over- come organizational obstacles that you will typically encounter at the start of your first AI project.  Chapter 5 helps you understand an ML pipeline and how it would evolve throughout the project life cycle. It shows you how to balance attention between business questions you are asking, the data you need, and AI algorithms you should use.  Chapter 6 shows you how to determine if you’re using the right ML pipeline for your AI project. It introduces you to the technique called MinMax analysis and shows you how to both perform it and interpret its results.  Chapter 7 shows you how to correctly choose the right parts of your ML pipe- line to improve for optimal business results. It also introduces the technique of sensitivity analysis and demonstrates how to interpret its results, as well as how to account for the passage of time in a long-running AI project.  Chapter 8 focuses on trends in AI and how they’ll affect you. This chapter intro- duces you to trends such as AutoML (automation of the work that data scien- tists do in AI) and explores how AI relates to causality and Internet of Things (IoT) systems. It also contrasts AI system errors with the typical errors humans make and shows you how to account for those differences in your project. Some further comments about the organization of the book:  The material in this book is multidisciplinary and requires a combination of both theory and practice to understand. Each chapter in this book combines the use of concrete examples illustrating general concepts and a detailed explanation of those concepts. The exercises at the end of each chapter will help you apply what you’ve learned in the chapter in the context of new business problems.  Executives should make sure they read and understand both the content and details of the first four chapters and the last chapter. The business-focused exercises in those chapters will help every reader, up to and including the level of business-focused executives. Even if you prefer to skip the exercises, I recommend you still carefully review the answers provided in appendix B, “Exercise solutions.” Business-focused readers should understand chapters 5, 6,

ABOUT THIS BOOK xixand 7 broadly, while technically-focused readers should understand those chap- ters in detail.  Some concepts discussed in this book are complicated. Instead of overwhelm- ing you with every part and particle related to a concept the very first time you encounter it, I start with a high-level description of the idea. After you’ve mas- tered the basics of a concept, later chapters refer to the concept you already know and explore the finer points of its applications. If you ever wonder “Hey, didn’t you already cover that concept in a previous chapter?” I certainly did, and now we’re applying that concept in a brand-new context.  Speaking of examples, I use examples from many different business verticals. I encourage you to scrutinize even more the examples of verticals with which you are not familiar. They’re chosen to be small, self-contained, and described so that you can easily understand them in a business sense. I then show you how to apply the technical concepts you’re learning in this book to these busi- ness examples. This is the position in which you will find yourself when apply- ing AI to a new problem in your own business. No two business problems are identical, so you should already be used to comprehending the simple business concepts that come with new problems, even when they’re in an unfamiliar business domain.  The methods described in this book are independent from any underlying technical infrastructure. That infrastructure is evolving rapidly and consists of cloud or on-premise big data systems, development frameworks, and program- ming languages. I focus this book on the mechanisms of how to tie AI and busi- ness together, and I hope that the material in this book will serve you well years from now. I stay technology-neutral and leave it for other books to discuss the characteristics and tradeoffs of various infrastructure products marketed today.  As in any other business book, the audience and readers for this book come from diverse backgrounds. Business and AI are broad topics, but most leaders of AI projects are already familiar with most of the terms I’m using. If you find a term you’re not familiar with, please consult appendix A, “Glossary of terms,” which contains the definitions of these terms.  This book covers a wide range of topics and builds on the work of many other people. You will find many citations of other works, such as “[4].” The citation style used is Vancouver style notation, and [4] is an example of a citation. You can find the reference corresponding to [4] in appendix C, “Bibliography.” In addition to giving credit where credit is due, the references cited direct you to where you can find more in-depth information about topics discussed in this book. Those references range from popular texts intended for a wider audi- ence, to books focused toward practicing management professionals, to aca- demic business publications, to technical and academic references requiring an in-depth knowledge of theoretical aspects of data science. I hope that the refer- ence list will be of interest to everyone on your team.