Machine Learning Pocket Reference Working with Structured Data in Python (Matt Harrison) (z-library.sk, 1lib.sk, z-lib.sk)

📄 File Format: PDF

💾 File Size: 25.2 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

Matt Harrison Machine Learning Working with Structured Data in Python Pocket Reference

📄 Page 2

(This page has no text content)

📄 Page 3

Matt Harrison Machine Learning Pocket Reference Working with Structured Data in Python

📄 Page 4

978-1-492-04754-4 [LSI] Machine Learning Pocket Reference by Matt Harrison Copyright © 2019 Matt Harrison. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promo‐ tional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Rachel Roumeliotis Development Editor: Nicole Tache Production Editor: Christopher Faucher Copyeditor: Sonia Saruba Proofreader: Christina Edwards Indexer: WordCo Indexing Services, Inc. Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Rebecca Demarest September 2019: First Edition Revision History for the First Edition 2019-08-27: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492047544 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Machine Learning Pocket Reference, the cover image, and related trade dress are trade‐ marks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages result‐ ing from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

📄 Page 5

Table of Contents Preface ix Chapter 1: Introduction 1 Libraries Used 2 Installation with Pip 5 Installation with Conda 6 Chapter 2: Overview of the Machine Learning Process 9 Chapter 3: Classification Walkthrough: Titanic Dataset 11 Project Layout Suggestion 11 Imports 12 Ask a Question 13 Terms for Data 13 Gather Data 15 Clean Data 16 Create Features 23 Sample Data 25 iii

📄 Page 6

Impute Data 25 Normalize Data 27 Refactor 27 Baseline Model 29 Various Families 29 Stacking 31 Create Model 32 Evaluate Model 33 Optimize Model 34 Confusion Matrix 35 ROC Curve 36 Learning Curve 38 Deploy Model 39 Chapter 4: Missing Data 41 Examining Missing Data 42 Dropping Missing Data 47 Imputing Data 47 Adding Indicator Columns 49 Chapter 5: Cleaning Data 51 Column Names 51 Replacing Missing Values 52 Chapter 6: Exploring 55 Data Size 55 Summary Stats 56 Histogram 58 Scatter Plot 59 Joint Plot 60 iv | Table of Contents

📄 Page 7

Pair Grid 63 Box and Violin Plots 64 Comparing Two Ordinal Values 65 Correlation 67 RadViz 71 Parallel Coordinates 73 Chapter 7: Preprocess Data 77 Standardize 77 Scale to Range 79 Dummy Variables 80 Label Encoder 81 Frequency Encoding 82 Pulling Categories from Strings 82 Other Categorical Encoding 84 Date Feature Engineering 86 Add col_na Feature 87 Manual Feature Engineering 88 Chapter 8: Feature Selection 89 Collinear Columns 90 Lasso Regression 92 Recursive Feature Elimination 94 Mutual Information 96 Principal Component Analysis 97 Feature Importance 97 Chapter 9: Imbalanced Classes 99 Use a Different Metric 99 Tree-based Algorithms and Ensembles 99 Table of Contents | v

📄 Page 8

Penalize Models 100 Upsampling Minority 100 Generate Minority Data 101 Downsampling Majority 101 Upsampling Then Downsampling 103 Chapter 10: Classification 105 Logistic Regression 106 Naive Bayes 111 Support Vector Machine 113 K-Nearest Neighbor 116 Decision Tree 119 Random Forest 127 XGBoost 133 Gradient Boosted with LightGBM 143 TPOT 148 Chapter 11: Model Selection 153 Validation Curve 153 Learning Curve 156 Chapter 12: Metrics and Classification Evaluation 159 Confusion Matrix 159 Metrics 162 Accuracy 164 Recall 164 Precision 164 F1 165 Classification Report 165 ROC 166 vi | Table of Contents

📄 Page 9

Precision-Recall Curve 167 Cumulative Gains Plot 169 Lift Curve 171 Class Balance 172 Class Prediction Error 173 Discrimination Threshold 175 Chapter 13: Explaining Models 177 Regression Coefficients 177 Feature Importance 178 LIME 178 Tree Interpretation 180 Partial Dependence Plots 181 Surrogate Models 185 Shapley 186 Chapter 14: Regression 191 Baseline Model 193 Linear Regression 194 SVMs 198 K-Nearest Neighbor 200 Decision Tree 202 Random Forest 208 XGBoost Regression 211 LightGBM Regression 218 Chapter 15: Metrics and Regression Evaluation 223 Metrics 223 Residuals Plot 226 Heteroscedasticity 227 Table of Contents | vii

📄 Page 10

Normal Residuals 228 Prediction Error Plot 230 Chapter 16: Explaining Regression Models 233 Shapley 233 Chapter 17: Dimensionality Reduction 239 PCA 239 UMAP 259 t-SNE 264 PHATE 268 Chapter 18: Clustering 273 K-Means 273 Agglomerative (Hierarchical) Clustering 280 Understanding Clusters 283 Chapter 19: Pipelines 289 Classification Pipeline 289 Regression Pipeline 292 PCA Pipeline 293 Index 295 viii | Table of Contents

📄 Page 11

Preface Machine learning and data science are very popular right now and are fast-moving targets. I have worked with Python and data for most of my career and wanted to have a physical book that could provide a reference for the common methods that I have been using in industry and teaching during workshops to solve structured machine learning problems. This book is what I believe is the best collection of resources and examples for attacking a predictive modeling task if you have structured data. There are many libraries that perform a portion of the tasks required and I have tried to incorporate those that I have found useful as I have applied these techni‐ ques in consulting or industry work. Many may lament the lack of deep learning techniques. Those could be a book by themselves. I also prefer simpler techniques and others in industry seem to agree. Deep learning for unstructured data (video, audio, images), and powerful tools like XGBoost for structured data. I hope this book serves as a useful reference for you to solve pressing problems. ix

📄 Page 12

What to Expect This book gives in-depth examples of solving common struc‐ tured data problems. It walks through various libraries and models, their trade-offs, how to tune them, and how to inter‐ pret them. The code snippets are meant to be sized such that you can use and adapt them in your own projects. Who This Book Is For If you are just learning machine learning, or have worked with it for years, this book should serve as a valuable reference. It assumes some knowledge of Python, and doesn’t delve at all into syntax. Rather it shows how to use various libraries to solve real-world problems. This will not replace an in-depth course, but should serve as a reference of what an applied machine learning course might cover. (Note: The author uses it as a reference for the data ana‐ lytics and machine learning courses he teaches.) Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, state‐ ments, and keywords. x | Preface

📄 Page 13

TIP This element signifies a tip or suggestion. NOTE This element signifies a general note. WARNING This element indicates a warning or caution. Using Code Examples Supplemental material (code examples, exercises, etc.) is avail‐ able at https://github.com/mattharrison/ml_pocket_reference. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permis‐ sion. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Machine Learning Pocket Reference by Matt Harrison (O’Reilly). Copyright 2019 Matt Harrison, 978-1-492-04754-4.” Preface | xi

📄 Page 14

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For almost 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, conferences, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, please visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, exam‐ ples, and any additional information. You can access this page at http://www.oreilly.com/catalog/9781492047544. To comment or ask technical questions about this book, send email to bookquestions@oreilly.com. xii | Preface

📄 Page 15

For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Acknowledgments Much thanks to my wife and family for their support. I’m grateful to the Python community for providing a wonderful language and toolset to work with. Nicole Tache has been lovely to work with and provided excellent feedback. My tech‐ nical reviewers, Mikio Braun, Natalino Busa, and Justin Fran‐ cis, kept me honest. Thanks! Preface | xiii

📄 Page 16

(This page has no text content)

📄 Page 17

CHAPTER 1 Introduction This is not so much an instructional manual, but rather notes, tables, and examples for machine learning. It was created by the author as an additional resource during training, meant to be distributed as a physical notebook. Participants (who favor the physical characteristics of dead-tree material) could add their own notes and thoughts and have a valuable reference of cura‐ ted examples. We will walk through classification with structured data. Other common machine learning applications include predicting a continuous value (regression), creating clusters, or trying to reduce dimensionality, among others. This book does not dis‐ cuss deep learning techniques. While those techniques work well for unstructured data, most recommend the techniques in this book for structured data. We assume knowledge and familiarity with Python. Learning how to manipulate data using the pandas library is useful. We have many examples using pandas, and it is an excellent tool for dealing with structured data. However, some of the index‐ ing operations may be confusing if you are not familiar with numpy. Full coverage of pandas could be a book in itself. 1

📄 Page 18

Libraries Used This book uses many libraries. This can be a good thing and a bad thing. Some of these libraries may be hard to install or con‐ flict with other library versions. Do not feel like you need to install all of these libraries. Use “JIT installation” and only install the libraries that you want to use as you need them. >>> import autosklearn, catboost, category_encoders, dtreeviz, eli5, fancyimpute, fastai, featuretools, glmnet_py, graphviz, hdbscan, imblearn, janitor, lime, matplotlib, missingno, mlxtend, numpy, pandas, pdpbox, phate, pydotplus, rfpimp, scikitplot, scipy, seaborn, shap, sklearn, statsmodels, tpot, treeinterpreter, umap, xgbfir, xgboost, yellowbrick >>> for lib in [ ... autosklearn, ... catboost, ... category_encoders, ... dtreeviz, ... eli5, ... fancyimpute, ... fastai, ... featuretools, ... glmnet_py, ... graphviz, ... hdbscan, ... imblearn, ... lime, ... janitor, ... matplotlib, ... missingno, ... mlxtend, ... numpy, ... pandas, ... pandas_profiling, ... pdpbox, ... phate, 2 | Chapter 1: Introduction

📄 Page 19

... pydotplus, ... rfpimp, ... scikitplot, ... scipy, ... seaborn, ... shap, ... sklearn, ... statsmodels, ... tpot, ... treeinterpreter, ... umap, ... xgbfir, ... xgboost, ... yellowbrick, ... ]: ... try: ... print(lib.__name__, lib.__version__) ... except: ... print("Missing", lib.__name__) catboost 0.11.1 category_encoders 2.0.0 Missing dtreeviz eli5 0.8.2 fancyimpute 0.4.2 fastai 1.0.28 featuretools 0.4.0 Missing glmnet_py graphviz 0.10.1 hdbscan 0.8.22 imblearn 0.4.3 janitor 0.16.6 Missing lime matplotlib 2.2.3 missingno 0.4.1 mlxtend 0.14.0 numpy 1.15.2 pandas 0.23.4 Missing pandas_profiling pdpbox 0.2.0 phate 0.4.2 Libraries Used | 3

📄 Page 20

Missing pydotplus rfpimp scikitplot 0.3.7 scipy 1.1.0 seaborn 0.9.0 shap 0.25.2 sklearn 0.21.1 statsmodels 0.9.0 tpot 0.9.5 treeinterpreter 0.1.0 umap 0.3.8 xgboost 0.81 yellowbrick 0.9 NOTE Most of these libraries are easily installed with pip or conda. With fastai I need to use pip install --no-deps fastai. The umap library is installed with pip install umap-learn. The janitor library is installed with pip install pyjanitor. The autosklearn library is installed with pip install auto-sklearn. I usually use Jupyter for doing an analysis. You can use other notebook tools as well. Note that some, like Google Colab, have preinstalled many of the libraries (though they may be outdated versions). There are two main options for installing libraries in Python. One is to use pip (an acronym for Pip Installs Python), a tool that comes with Python. The other option is to use Anaconda. We will introduce both. 4 | Chapter 1: Introduction