Python for Data Analysis (Wes McKinney) (Z-Library)

Python for Data Analysis Data Wrangling with pandas, NumPy & Jupyter Wes McKinney Third Edition powered by

DATA “With this new edition, Wes has updated his book to ensure it remains the go-to resource for all things related to data analysis with Python and pandas. I cannot recommend this book highly enough.” —Paul Barry Lecturer and author of O’Reilly’s Head First Python Python for Data Analysis 9 781098 104030 5 6 9 9 9 US $69.99 CAN $87.99 ISBN: 978-1-098-10403-0 Twitter: @oreillymedia linkedin.com/company/oreilly-media youtube.com/oreillymedia Get the definitive handbook for manipulating, processing, cleaning, and crunching datasets in Python. Updated for Python 3.10 and pandas 1.4, the third edition of this hands- on guide is packed with practical case studies that show you how to solve a broad set of data analysis problems effectively. You’ll learn the latest versions of pandas, NumPy, and Jupyter in the process. Written by Wes McKinney, the creator of the Python pandas project, this book is a practical, modern introduction to data science tools in Python. It’s ideal for analysts new to Python and for Python programmers new to data science and scientific computing. Data files and related material are available on GitHub. • Use the Jupyter notebook and the IPython shell for exploratory computing • Learn basic and advanced features in NumPy • Get started with data analysis tools in the pandas library • Use flexible tools to load, clean, transform, merge, and reshape data • Create informative visualizations with matplotlib • Apply the pandas groupBy facility to slice, dice, and summarize datasets • Analyze and manipulate regular and irregular time series data • Learn how to solve real-world data analysis problems with thorough, detailed examples Wes McKinney, cofounder and chief technology officer of Voltron Data, is an active member of the Python data community and an advocate for Python use in data analysis, finance, and statistical computing applications. A graduate of MIT, he’s also a member of the project management committees for the Apache Software Foundation’s Apache Arrow and Apache Parquet projects.

Wes McKinney Python for Data Analysis Data Wrangling with pandas, NumPy, and Jupyter THIRD EDITION Boston Farnham Sebastopol TokyoBeijing

978-1-098-10403-0 [LSI] Python for Data Analysis by Wes McKinney Copyright © 2022 Wesley McKinney. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Jessica Haberman Development Editor: Angela Rufino Production Editor: Christopher Faucher Copyeditor: Sonia Saruba Proofreader: Piper Editorial Consulting, LLC Indexer: Sue Klefstad Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea October 2012: First Edition October 2017: Second Edition August 2022: Third Edition Revision History for the Third Edition 2022-08-12: First Release See https://www.oreilly.com/catalog/errata.csp?isbn=0636920519829 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Python for Data Analysis, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi 1. Preliminaries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.1 What Is This Book About? 1 What Kinds of Data? 1 1.2 Why Python for Data Analysis? 2 Python as Glue 3 Solving the “Two-Language” Problem 3 Why Not Python? 3 1.3 Essential Python Libraries 4 NumPy 4 pandas 5 matplotlib 6 IPython and Jupyter 6 SciPy 7 scikit-learn 8 statsmodels 8 Other Packages 9 1.4 Installation and Setup 9 Miniconda on Windows 9 GNU/Linux 10 Miniconda on macOS 11 Installing Necessary Packages 11 Integrated Development Environments and Text Editors 12 1.5 Community and Conferences 13 1.6 Navigating This Book 14 Code Examples 15 iii

Data for Examples 15 Import Conventions 16 2. Python Language Basics, IPython, and Jupyter Notebooks. . . . . . . . . . . . . . . . . . . . . . . . 17 2.1 The Python Interpreter 18 2.2 IPython Basics 19 Running the IPython Shell 19 Running the Jupyter Notebook 20 Tab Completion 23 Introspection 25 2.3 Python Language Basics 26 Language Semantics 26 Scalar Types 34 Control Flow 42 2.4 Conclusion 45 3. Built-In Data Structures, Functions, and Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.1 Data Structures and Sequences 47 Tuple 47 List 51 Dictionary 55 Set 59 Built-In Sequence Functions 62 List, Set, and Dictionary Comprehensions 63 3.2 Functions 65 Namespaces, Scope, and Local Functions 67 Returning Multiple Values 68 Functions Are Objects 69 Anonymous (Lambda) Functions 70 Generators 71 Errors and Exception Handling 74 3.3 Files and the Operating System 76 Bytes and Unicode with Files 80 3.4 Conclusion 82 4. NumPy Basics: Arrays and Vectorized Computation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 4.1 The NumPy ndarray: A Multidimensional Array Object 85 Creating ndarrays 86 Data Types for ndarrays 88 Arithmetic with NumPy Arrays 91 Basic Indexing and Slicing 92 iv | Table of Contents

Boolean Indexing 97 Fancy Indexing 100 Transposing Arrays and Swapping Axes 102 4.2 Pseudorandom Number Generation 103 4.3 Universal Functions: Fast Element-Wise Array Functions 105 4.4 Array-Oriented Programming with Arrays 108 Expressing Conditional Logic as Array Operations 110 Mathematical and Statistical Methods 111 Methods for Boolean Arrays 113 Sorting 114 Unique and Other Set Logic 115 4.5 File Input and Output with Arrays 116 4.6 Linear Algebra 116 4.7 Example: Random Walks 118 Simulating Many Random Walks at Once 120 4.8 Conclusion 121 5. Getting Started with pandas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 5.1 Introduction to pandas Data Structures 124 Series 124 DataFrame 129 Index Objects 136 5.2 Essential Functionality 138 Reindexing 138 Dropping Entries from an Axis 141 Indexing, Selection, and Filtering 142 Arithmetic and Data Alignment 152 Function Application and Mapping 158 Sorting and Ranking 160 Axis Indexes with Duplicate Labels 164 5.3 Summarizing and Computing Descriptive Statistics 165 Correlation and Covariance 168 Unique Values, Value Counts, and Membership 170 5.4 Conclusion 173 6. Data Loading, Storage, and File Formats. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 6.1 Reading and Writing Data in Text Format 175 Reading Text Files in Pieces 182 Writing Data to Text Format 184 Working with Other Delimited Formats 185 JSON Data 187 Table of Contents | v

XML and HTML: Web Scraping 189 6.2 Binary Data Formats 193 Reading Microsoft Excel Files 194 Using HDF5 Format 195 6.3 Interacting with Web APIs 197 6.4 Interacting with Databases 199 6.5 Conclusion 201 7. Data Cleaning and Preparation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203 7.1 Handling Missing Data 203 Filtering Out Missing Data 205 Filling In Missing Data 207 7.2 Data Transformation 209 Removing Duplicates 209 Transforming Data Using a Function or Mapping 211 Replacing Values 212 Renaming Axis Indexes 214 Discretization and Binning 215 Detecting and Filtering Outliers 217 Permutation and Random Sampling 219 Computing Indicator/Dummy Variables 221 7.3 Extension Data Types 224 7.4 String Manipulation 227 Python Built-In String Object Methods 227 Regular Expressions 229 String Functions in pandas 232 7.5 Categorical Data 235 Background and Motivation 236 Categorical Extension Type in pandas 237 Computations with Categoricals 240 Categorical Methods 242 7.6 Conclusion 245 8. Data Wrangling: Join, Combine, and Reshape. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 8.1 Hierarchical Indexing 247 Reordering and Sorting Levels 250 Summary Statistics by Level 251 Indexing with a DataFrame’s columns 252 8.2 Combining and Merging Datasets 253 Database-Style DataFrame Joins 254 Merging on Index 259 vi | Table of Contents

Concatenating Along an Axis 263 Combining Data with Overlap 268 8.3 Reshaping and Pivoting 270 Reshaping with Hierarchical Indexing 270 Pivoting “Long” to “Wide” Format 273 Pivoting “Wide” to “Long” Format 277 8.4 Conclusion 279 9. Plotting and Visualization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281 9.1 A Brief matplotlib API Primer 282 Figures and Subplots 283 Colors, Markers, and Line Styles 288 Ticks, Labels, and Legends 290 Annotations and Drawing on a Subplot 294 Saving Plots to File 296 matplotlib Configuration 297 9.2 Plotting with pandas and seaborn 298 Line Plots 298 Bar Plots 301 Histograms and Density Plots 309 Scatter or Point Plots 311 Facet Grids and Categorical Data 314 9.3 Other Python Visualization Tools 317 9.4 Conclusion 317 10. Data Aggregation and Group Operations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 319 10.1 How to Think About Group Operations 320 Iterating over Groups 324 Selecting a Column or Subset of Columns 326 Grouping with Dictionaries and Series 327 Grouping with Functions 328 Grouping by Index Levels 328 10.2 Data Aggregation 329 Column-Wise and Multiple Function Application 331 Returning Aggregated Data Without Row Indexes 335 10.3 Apply: General split-apply-combine 335 Suppressing the Group Keys 338 Quantile and Bucket Analysis 338 Example: Filling Missing Values with Group-Specific Values 340 Example: Random Sampling and Permutation 343 Example: Group Weighted Average and Correlation 344 Table of Contents | vii

Example: Group-Wise Linear Regression 347 10.4 Group Transforms and “Unwrapped” GroupBys 347 10.5 Pivot Tables and Cross-Tabulation 351 Cross-Tabulations: Crosstab 354 10.6 Conclusion 355 11. Time Series. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 11.1 Date and Time Data Types and Tools 358 Converting Between String and Datetime 359 11.2 Time Series Basics 361 Indexing, Selection, Subsetting 363 Time Series with Duplicate Indices 365 11.3 Date Ranges, Frequencies, and Shifting 366 Generating Date Ranges 367 Frequencies and Date Offsets 370 Shifting (Leading and Lagging) Data 371 11.4 Time Zone Handling 374 Time Zone Localization and Conversion 375 Operations with Time Zone-Aware Timestamp Objects 377 Operations Between Different Time Zones 378 11.5 Periods and Period Arithmetic 379 Period Frequency Conversion 380 Quarterly Period Frequencies 382 Converting Timestamps to Periods (and Back) 384 Creating a PeriodIndex from Arrays 385 11.6 Resampling and Frequency Conversion 387 Downsampling 388 Upsampling and Interpolation 391 Resampling with Periods 392 Grouped Time Resampling 394 11.7 Moving Window Functions 396 Exponentially Weighted Functions 399 Binary Moving Window Functions 401 User-Defined Moving Window Functions 402 11.8 Conclusion 403 12. Introduction to Modeling Libraries in Python. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405 12.1 Interfacing Between pandas and Model Code 405 12.2 Creating Model Descriptions with Patsy 408 Data Transformations in Patsy Formulas 410 Categorical Data and Patsy 412 viii | Table of Contents

12.3 Introduction to statsmodels 415 Estimating Linear Models 415 Estimating Time Series Processes 419 12.4 Introduction to scikit-learn 420 12.5 Conclusion 423 13. Data Analysis Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425 13.1 Bitly Data from 1.USA.gov 425 Counting Time Zones in Pure Python 426 Counting Time Zones with pandas 428 13.2 MovieLens 1M Dataset 435 Measuring Rating Disagreement 439 13.3 US Baby Names 1880–2010 443 Analyzing Naming Trends 448 13.4 USDA Food Database 457 13.5 2012 Federal Election Commission Database 463 Donation Statistics by Occupation and Employer 466 Bucketing Donation Amounts 469 Donation Statistics by State 471 13.6 Conclusion 472 A. Advanced NumPy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473 A.1 ndarray Object Internals 473 NumPy Data Type Hierarchy 474 A.2 Advanced Array Manipulation 476 Reshaping Arrays 476 C Versus FORTRAN Order 478 Concatenating and Splitting Arrays 479 Repeating Elements: tile and repeat 481 Fancy Indexing Equivalents: take and put 483 A.3 Broadcasting 484 Broadcasting over Other Axes 487 Setting Array Values by Broadcasting 489 A.4 Advanced ufunc Usage 490 ufunc Instance Methods 490 Writing New ufuncs in Python 493 A.5 Structured and Record Arrays 493 Nested Data Types and Multidimensional Fields 494 Why Use Structured Arrays? 495 A.6 More About Sorting 495 Indirect Sorts: argsort and lexsort 497 Table of Contents | ix

Alternative Sort Algorithms 498 Partially Sorting Arrays 499 numpy.searchsorted: Finding Elements in a Sorted Array 500 A.7 Writing Fast NumPy Functions with Numba 501 Creating Custom numpy.ufunc Objects with Numba 502 A.8 Advanced Array Input and Output 503 Memory-Mapped Files 503 HDF5 and Other Array Storage Options 504 A.9 Performance Tips 505 The Importance of Contiguous Memory 505 B. More on the IPython System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 509 B.1 Terminal Keyboard Shortcuts 509 B.2 About Magic Commands 510 The %run Command 512 Executing Code from the Clipboard 513 B.3 Using the Command History 514 Searching and Reusing the Command History 514 Input and Output Variables 515 B.4 Interacting with the Operating System 516 Shell Commands and Aliases 517 Directory Bookmark System 518 B.5 Software Development Tools 519 Interactive Debugger 519 Timing Code: %time and %timeit 523 Basic Profiling: %prun and %run -p 525 Profiling a Function Line by Line 527 B.6 Tips for Productive Code Development Using IPython 529 Reloading Module Dependencies 529 Code Design Tips 530 B.7 Advanced IPython Features 532 Profiles and Configuration 532 B.8 Conclusion 533 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 535 x | Table of Contents

Preface The first edition of this book was published in 2012, during a time when open source data analysis libraries for Python, especially pandas, were very new and developing rapidly. When the time came to write the second edition in 2016 and 2017, I needed to update the book not only for Python 3.6 (the first edition used Python 2.7) but also for the many changes in pandas that had occurred over the previous five years. Now in 2022, there are fewer Python language changes (we are now at Python 3.10, with 3.11 coming out at the end of 2022), but pandas has continued to evolve. In this third edition, my goal is to bring the content up to date with current versions of Python, NumPy, pandas, and other projects, while also remaining relatively con‐ servative about discussing newer Python projects that have appeared in the last few years. Since this book has become an important resource for many university courses and working professionals, I will try to avoid topics that are at risk of falling out of date within a year or two. That way paper copies won’t be too difficult to follow in 2023 or 2024 or beyond. A new feature of the third edition is the open access online version hosted on my website at https://wesmckinney.com/book, to serve as a resource and convenience for owners of the print and digital editions. I intend to keep the content reasonably up to date there, so if you own the paper book and run into something that doesn’t work properly, you should check there for the latest content changes. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. xi

Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. Using Code Examples You can find data files and related material for each chapter in this book’s GitHub repository at https://github.com/wesm/pydata-book, which is mirrored to Gitee (for those who cannot access GitHub) at https://gitee.com/wesmckinn/pydata-book. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. xii | Preface

We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Python for Data Analysis by Wes McKinney (O’Reilly). Copyright 2022 Wes McKinney, 978-1-098-10403-0.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/python-data-analysis-3e. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For news and information about our books and courses, visit http://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly-media. Follow us on Twitter: http://twitter.com/oreillymedia. Watch us on YouTube: http://youtube.com/oreillymedia. Preface | xiii

Acknowledgments This work is the product of many years of fruitful discussions and collaborations with, and assistance from many people around the world. I’d like to thank a few of them. In Memoriam: John D. Hunter (1968–2012) Our dear friend and colleague John D. Hunter passed away after a battle with colon cancer on August 28, 2012. This was only a short time after I’d completed the final manuscript for this book’s first edition. John’s impact and legacy in the Python scientific and data communities would be hard to overstate. In addition to developing matplotlib in the early 2000s (a time when Python was not nearly so popular), he helped shape the culture of a critical generation of open source developers who’ve become pillars of the Python ecosystem that we now often take for granted. I was lucky enough to connect with John early in my open source career in January 2010, just after releasing pandas 0.1. His inspiration and mentorship helped me push forward, even in the darkest of times, with my vision for pandas and Python as a first-class data analysis language. John was very close with Fernando Pérez and Brian Granger, pioneers of IPython, Jupyter, and many other initiatives in the Python community. We had hoped to work on a book together, the four of us, but I ended up being the one with the most free time. I am sure he would be proud of what we’ve accomplished, as individuals and as a community, over the last nine years. Acknowledgments for the Third Edition (2022) It has more than a decade since I started writing the first edition of this book and more than 15 years since I originally started my journey as a Python prorammer. A lot has changed since then! Python has evolved from a relatively niche language for data analysis to the most popular and most widely used language powering the plurality (if not the majority!) of data science, machine learning, and artificial intelligence work. I have not been an active contributor to the pandas open source project since 2013, but its worldwide developer community has continued to thrive, serving as a model of community-centric open source software development. Many “next-generation” Python projects that deal with tabular data are modeling their user interfaces directly after pandas, so the project has proved to have an enduring influence on the future trajectory of the Python data science ecosystem. xiv | Preface

I hope that this book continues to serve as a valuable resource for students and individuals who want to learn about working with data in Python. I’m especially thankful to O’Reilly for allowing me to publish an “open access” version of this book on my website at https://wesmckinney.com/book, where I hope it will reach even more people and help expand opportunity in the world of data analysis. J.J. Allaire was a lifesaver in making this possible by helping me “port” the book from Docbook XML to Quarto, a wonderful new scientific and technical publishing system for print and web. Special thanks to my technical reviewers Paul Barry, Jean-Christophe Leyder, Abdul‐ lah Karasan, and William Jamir, whose thorough feedback has greatly improved the readability, clarity, and understandability of the content. Acknowledgments for the Second Edition (2017) It has been five years almost to the day since I completed the manuscript for this book’s first edition in July 2012. A lot has changed. The Python community has grown immensely, and the ecosystem of open source software around it has flourished. This new edition of the book would not exist if not for the tireless efforts of the pandas core developers, who have grown the project and its user community into one of the cornerstones of the Python data science ecosystem. These include, but are not limited to, Tom Augspurger, Joris van den Bossche, Chris Bartak, Phillip Cloud, gfyoung, Andy Hayden, Masaaki Horikoshi, Stephan Hoyer, Adam Klein, Wouter Overmeire, Jeff Reback, Chang She, Skipper Seabold, Jeff Tratner, and y-p. On the actual writing of this second edition, I would like to thank the O’Reilly staff who helped me patiently with the writing process. This includes Marie Beaugureau, Ben Lorica, and Colleen Toporek. I again had outstanding technical reviewers with Tom Augspurger, Paul Barry, Hugh Brown, Jonathan Coe, and Andreas Müller con‐ tributing. Thank you. This book’s first edition has been translated into many foreign languages, including Chinese, French, German, Japanese, Korean, and Russian. Translating all this content and making it available to a broader audience is a huge and often thankless effort. Thank you for helping more people in the world learn how to program and use data analysis tools. I am also lucky to have had support for my continued open source development efforts from Cloudera and Two Sigma Investments over the last few years. With open source software projects more thinly resourced than ever relative to the size of user bases, it is becoming increasingly important for businesses to provide support for development of key open source projects. It’s the right thing to do. Preface | xv

Acknowledgments for the First Edition (2012) It would have been difficult for me to write this book without the support of a large number of people. On the O’Reilly staff, I’m very grateful for my editors, Meghan Blanchette and Julie Steele, who guided me through the process. Mike Loukides also worked with me in the proposal stages and helped make the book a reality. I received a wealth of technical review from a large cast of characters. In particu‐ lar, Martin Blais and Hugh Brown were incredibly helpful in improving the book’s examples, clarity, and organization from cover to cover. James Long, Drew Conway, Fernando Pérez, Brian Granger, Thomas Kluyver, Adam Klein, Josh Klein, Chang She, and Stéfan van der Walt each reviewed one or more chapters, providing pointed feedback from many different perspectives. I got many great ideas for examples and datasets from friends and colleagues in the data community, among them: Mike Dewar, Jeff Hammerbacher, James Johndrow, Kristian Lum, Adam Klein, Hilary Mason, Chang She, and Ashley Williams. I am of course indebted to the many leaders in the open source scientific Python community who’ve built the foundation for my development work and gave encour‐ agement while I was writing this book: the IPython core team (Fernando Pérez, Brian Granger, Min Ragan-Kelly, Thomas Kluyver, and others), John Hunter, Skipper Seabold, Travis Oliphant, Peter Wang, Eric Jones, Robert Kern, Josef Perktold, Fran‐ cesc Alted, Chris Fonnesbeck, and too many others to mention. Several other people provided a great deal of support, ideas, and encouragement along the way: Drew Conway, Sean Taylor, Giuseppe Paleologo, Jared Lander, David Epstein, John Krowas, Joshua Bloom, Den Pilsworth, John Myles-White, and many others I’ve forgotten. I’d also like to thank a number of people from my formative years. First, my former AQR colleagues who’ve cheered me on in my pandas work over the years: Alex Reyf‐ man, Michael Wong, Tim Sargen, Oktay Kurbanov, Matthew Tschantz, Roni Israelov, Michael Katz, Ari Levine, Chris Uga, Prasad Ramanan, Ted Square, and Hoon Kim. Lastly, my academic advisors Haynes Miller (MIT) and Mike West (Duke). I received significant help from Phillip Cloud and Joris van den Bossche in 2014 to update the book’s code examples and fix some other inaccuracies due to changes in pandas. On the personal side, Casey provided invaluable day-to-day support during the writing process, tolerating my highs and lows as I hacked together the final draft on top of an already overcommitted schedule. Lastly, my parents, Bill and Kim, taught me to always follow my dreams and to never settle for less. xvi | Preface

CHAPTER 1 Preliminaries 1.1 What Is This Book About? This book is concerned with the nuts and bolts of manipulating, processing, cleaning, and crunching data in Python. My goal is to offer a guide to the parts of the Python programming language and its data-oriented library ecosystem and tools that will equip you to become an effective data analyst. While “data analysis” is in the title of the book, the focus is specifically on Python programming, libraries, and tools as opposed to data analysis methodology. This is the Python programming you need for data analysis. Sometime after I originally published this book in 2012, people started using the term data science as an umbrella description for everything from simple descriptive statistics to more advanced statistical analysis and machine learning. The Python open source ecosystem for doing data analysis (or data science) has also expanded significantly since then. There are now many other books which focus specifically on these more advanced methodologies. My hope is that this book serves as adequate preparation to enable you to move on to a more domain-specific resource. Some might characterize much of the content of the book as “data manipulation” as opposed to “data analysis.” We also use the terms wrangling or munging to refer to data manipulation. What Kinds of Data? When I say “data,” what am I referring to exactly? The primary focus is on structured data, a deliberately vague term that encompasses many different common forms of data, such as: 1

• Tabular or spreadsheet-like data in which each column may be a different type (string, numeric, date, or otherwise). This includes most kinds of data commonly stored in relational databases or tab- or comma-delimited text files. • Multidimensional arrays (matrices). • Multiple tables of data interrelated by key columns (what would be primary or foreign keys for a SQL user). • Evenly or unevenly spaced time series. This is by no means a complete list. Even though it may not always be obvious, a large percentage of datasets can be transformed into a structured form that is more suitable for analysis and modeling. If not, it may be possible to extract features from a dataset into a structured form. As an example, a collection of news articles could be processed into a word frequency table, which could then be used to perform sentiment analysis. Most users of spreadsheet programs like Microsoft Excel, perhaps the most widely used data analysis tool in the world, will not be strangers to these kinds of data. 1.2 Why Python for Data Analysis? For many people, the Python programming language has strong appeal. Since its first appearance in 1991, Python has become one of the most popular interpreted programming languages, along with Perl, Ruby, and others. Python and Ruby have become especially popular since 2005 or so for building websites using their numer‐ ous web frameworks, like Rails (Ruby) and Django (Python). Such languages are often called scripting languages, as they can be used to quickly write small programs, or scripts to automate other tasks. I don’t like the term “scripting languages,” as it carries a connotation that they cannot be used for building serious software. Among interpreted languages, for various historical and cultural reasons, Python has devel‐ oped a large and active scientific computing and data analysis community. In the last 20 years, Python has gone from a bleeding-edge or “at your own risk” scientific com‐ puting language to one of the most important languages for data science, machine learning, and general software development in academia and industry. For data analysis and interactive computing and data visualization, Python will inevi‐ tably draw comparisons with other open source and commercial programming lan‐ guages and tools in wide use, such as R, MATLAB, SAS, Stata, and others. In recent years, Python’s improved open source libraries (such as pandas and scikit-learn) have made it a popular choice for data analysis tasks. Combined with Python’s overall strength for general-purpose software engineering, it is an excellent option as a primary language for building data applications. 2 | Chapter 1: Preliminaries