Share E-Book

Python Data Analysis Perform data collection, data processing, wrangling, visualization, and model building using Python (Avinash Navlani, Armando Fandango etc.)(Z-Library)

Author

,
,

Data
Language English

Understand data analysis pipelines using machine learning algorithms and techniques with this practical guideKey Features* Prepare and clean your data to use it for exploratory analysis, data manipulation, and data wrangling* Discover supervised, unsupervised, probabilistic, and Bayesian machine learning methods* Get to grips with graph processing and sentiment analysisBook DescriptionData analysis enables you to generate value from small and big data by discovering new patterns and trends, and Python is one of the most popular tools for analyzing a wide variety of data. With this book, you'll get up and running using Python for data analysis by exploring the different phases and methodologies used in data analysis and learning how to use modern libraries from the Python ecosystem to create efficient data pipelines.Starting with the essential statistical and data analysis fundamentals using Python, you'll perform complex data analysis and modeling, data manipulation, data cleaning, and data visualization using easy-to-follow examples. You'll then understand how to conduct time series analysis and signal processing using ARMA models. As you advance, you'll get to grips with smart processing and data analytics using machine learning algorithms such as regression, classification, Principal Component Analysis (PCA), and clustering. In the concluding chapters, you'll work on real-world examples to analyze textual and image data using natural language processing (NLP) and image analytics techniques, respectively. Finally, the book will demonstrate parallel computing using Dask.By the end of this data analysis book, you'll be equipped with the skills you need to prepare data for analysis and create meaningful data visualizations for forecasting values from data.

Format PDF
Size 14.0 MB
8
Views
0
Downloads
0.00
Total Donations
(First 20 pages)

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

Page 1
(This page has no text content)
Page 2
Python Data Analysis Third Edition Copyright © 2021 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Group Product Manager: Kunal Parikh Publishing Product Manager: Ali Abidi Content Development Editor: Joseph Sunil Senior Editor: Roshan Kumar Technical Editor: Sonam Pandey Copy Editor: Safis Editing Project Coordinator: Aishwarya Mohan Proofreader: Safis Editing Indexer: Rekha Nair Production Designer: Roshan Kawale First published: October 2014 Second edition: March 2017 Third Edition: February 2021 Production reference: 1070121 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78995-524-8 www.packt.com
Page 3
Packt.com Subscribe to our online digital library for full access to over 7,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website. Why subscribe? Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals Improve your learning with Skill Plans built especially for you Get a free eBook or video every month Fully searchable for easy access to vital information Copy and paste, print, and bookmark content Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at customercare@packtpub.com for more details. At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.
Page 4
Contributors About the authors Avinash Navlani has over 8 years of experience working in data science and AI. Currently, he is working as a senior data scientist, improving products and services for customers by using advanced analytics, deploying big data analytical tools, creating and maintaining models, and onboarding compelling new datasets. Previously, he was a university lecturer, where he trained and educated people in data science subjects such as Python for analytics, data mining, machine learning, database management, and NoSQL. Avinash has been involved in research activities in data science and has been a keynote speaker at many conferences in India. Armando Fandango creates AI-empowered products by leveraging his expertise in deep learning, machine learning, distributed computing, and computational methods and has provided thought leadership roles as the chief data scientist and director at start-ups and large enterprises. He has advised high-tech AI-based start-ups. Armando has authored books such as Python Data Analysis - Second Edition and Mastering TensorFlow, Packt Publishing. He has also published research in international journals and conferences. Ivan Idris has an MSc in experimental physics. His graduation thesis had a strong emphasis on applied computer science. After graduating, he worked for several companies as a Java developer, data warehouse developer, and QA analyst. His main professional interests are business intelligence, big data, and cloud computing. Ivan Idris enjoys writing clean, testable code and interesting technical articles. Ivan Idris is the author of NumPy 1.5 Beginner's Guide and NumPy Cookbook by Packt Publishing. You can find more information and a blog with a few NumPy examples at ivanidris.net.
Page 5
About the reviewers Greg Walters has been involved with computers and computer programming since 1972. He is well versed in Visual Basic, Visual Basic .NET, Python, and SQL and is an accomplished user of MySQL, SQLite, Microsoft SQL Server, Oracle, C++, Delphi, Modula-2, Pascal, C, 80x86 Assembler, COBOL, and Fortran. He is a programming trainer and has trained numerous people on many pieces of computer software, including MySQL, Open Database Connectivity, Quattro Pro, Corel Draw!, Paradox, Microsoft Word, Excel, DOS, Windows 3.11, Windows for Workgroups, Windows 95, Windows NT, Windows 2000, Windows XP, and Linux. He is semi-retired and has written over 100 articles for Full Circle Magazine. He is also a musician and loves to cook. He is open to working as a freelancer on various projects. Alistair McMaster is currently employed as a Software Engineer and Quantitative Strategist at a major financial services firm. He graduated from the University of Cambridge in 2016 with a B.A. (Hons) in Natural Sciences specializing in Astrophysics. His broader career interests include applications of data science to relationship networks and supporting social causes. Alistair is an active contributor to pandas and a strong advocate of open-source software. In his spare time, he enjoys distance running, cycling, rock climbing, and walks with his family and friends on weekends. Packt is searching for authors like you If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.
Page 6
Table of Contents Preface 1 Section 1: Foundation for Data Analysis Chapter 1: Getting Started with Python Libraries 7 Understanding data analysis 8 The standard process of data analysis 9 The KDD process 10 SEMMA 11 CRISP-DM 12 Comparing data analysis and data science 14 The roles of data analysts and data scientists 14 The skillsets of data analysts and data scientists 15 Installing Python 3 17 Python installation and setup on Windows 17 Python installation and setup on Linux 18 Python installation and setup on Mac OS X with a GUI installer 18 Python installation and setup on Mac OS X with brew 18 Software used in this book 19 Using IPython as a shell 20 Reading manual pages 23 Where to find help and references to Python data analysis libraries 24 Using JupyterLab 24 Using Jupyter Notebooks 26 Advanced features of Jupyter Notebooks 27 Keyboard shortcuts 28 Installing other kernels 29 Running shell commands 30 Extensions for Notebook 30 Summary 36 Chapter 2: NumPy and pandas 37 Technical requirements 38 Understanding NumPy arrays 38 Array features 41 Selecting array elements 42 NumPy array numerical data types 43 dtype objects 45 Data type character codes 46
Page 7
Table of Contents [ ii ] dtype constructors 47 dtype attributes 47 Manipulating array shapes 48 The stacking of NumPy arrays 50 Partitioning NumPy arrays 53 Changing the data type of NumPy arrays 55 Creating NumPy views and copies 56 Slicing NumPy arrays 58 Boolean and fancy indexing 60 Broadcasting arrays 61 Creating pandas DataFrames 63 Understanding pandas Series 65 Reading and querying the Quandl data 68 Describing pandas DataFrames 72 Grouping and joining pandas DataFrame 75 Working with missing values 79 Creating pivot tables 81 Dealing with dates 83 Summary 85 References 85 Chapter 3: Statistics 86 Technical requirements 87 Understanding attributes and their types 87 Types of attributes 87 Discrete and continuous attributes 89 Measuring central tendency 89 Mean 89 Mode 90 Median 91 Measuring dispersion 91 Skewness and kurtosis 95 Understanding relationships using covariance and correlation coefficients 96 Pearson's correlation coefficient 97 Spearman's rank correlation coefficient 98 Kendall's rank correlation coefficient 98 Central limit theorem 98 Collecting samples 100 Performing parametric tests 101 Performing non-parametric tests 107 Summary 113 Chapter 4: Linear Algebra 114
Page 8
Table of Contents [ iii ] Technical requirements 115 Fitting to polynomials with NumPy 115 Determinant 117 Finding the rank of a matrix 117 Matrix inverse using NumPy 118 Solving linear equations using NumPy 119 Decomposing a matrix using SVD 120 Eigenvectors and Eigenvalues using NumPy 122 Generating random numbers 123 Binomial distribution 124 Normal distribution 126 Testing normality of data using SciPy 127 Creating a masked array using the numpy.ma subpackage 131 Summary 133 Section 2: Exploratory Data Analysis and Data Cleaning Chapter 5: Data Visualization 135 Technical requirements 135 Visualization using Matplotlib 136 Accessories for charts 137 Scatter plot 139 Line plot 140 Pie plot 142 Bar plot 143 Histogram plot 144 Bubble plot 146 pandas plotting 148 Advanced visualization using the Seaborn package 150 lm plots 151 Bar plots 154 Distribution plots 155 Box plots 156 KDE plots 157 Violin plots 158 Count plots 159 Joint plots 161 Heatmaps 162 Pair plots 164 Interactive visualization with Bokeh 166 Plotting a simple graph 166 Glyphs 168 Layouts 169
Page 9
Table of Contents [ iv ] Nested layout using row and column layouts 173 Multiple plots 175 Interactions 177 Hide click policy 177 Mute click policy 179 Annotations 180 Hover tool 183 Widgets 184 Tab panel 185 Slider 186 Summary 189 Chapter 6: Retrieving, Processing, and Storing Data 190 Technical requirements 191 Reading and writing CSV files with NumPy 191 Reading and writing CSV files with pandas 192 Reading and writing data from Excel 194 Reading and writing data from JSON 195 Reading and writing data from HDF5 196 Reading and writing data from HTML tables 197 Reading and writing data from Parquet 198 Reading and writing data from a pickle pandas object 199 Lightweight access with sqllite3 200 Reading and writing data from MySQL 201 Inserting a whole DataFrame into the database 204 Reading and writing data from MongoDB 205 Reading and writing data from Cassandra 206 Reading and writing data from Redis 207 PonyORM 208 Summary 209 Chapter 7: Cleaning Messy Data 210 Technical requirements 211 Exploring data 211 Filtering data to weed out the noise 214 Column-wise filtration 215 Row-wise filtration 217 Handling missing values 220 Dropping missing values 221 Filling in a missing value 221 Handling outliers 223 Feature encoding techniques 226 One-hot encoding 226 Label encoding 228 Ordinal encoder 229
Page 10
Table of Contents [ v ] Feature scaling 230 Methods for feature scaling 231 Feature transformation 234 Feature splitting 235 Summary 236 Chapter 8: Signal Processing and Time Series 237 Technical requirements 238 The statsmodels modules 238 Moving averages 239 Window functions 242 Defining cointegration 244 STL decomposition 246 Autocorrelation 248 Autoregressive models 250 ARMA models 254 Generating periodic signals 257 Fourier analysis 259 Spectral analysis filtering 262 Summary 264 Section 3: Deep Dive into Machine Learning Chapter 9: Supervised Learning - Regression Analysis 266 Technical requirements 267 Linear regression 267 Multiple linear regression 269 Understanding multicollinearity 269 Removing multicollinearity 270 Dummy variables 272 Developing a linear regression model 274 Evaluating regression model performance 276 R-squared 276 MSE 277 MAE 277 RMSE 278 Fitting polynomial regression 279 Regression models for classification 282 Logistic regression 282 Characteristics of the logistic regression model 284 Types of logistic regression algorithms 285 Advantages and disadvantages of logistic regression 285 Implementing logistic regression using scikit-learn 286 Summary 288
Page 11
Table of Contents [ vi ] Chapter 10: Supervised Learning - Classification Techniques 289 Technical requirements 290 Classification 290 Naive Bayes classification 292 Decision tree classification 296 KNN classification 299 SVM classification 302 Terminology 302 Splitting training and testing sets 305 Holdout 305 K-fold cross-validation 306 Bootstrap method 306 Evaluating the classification model performance 307 Confusion matrix 307 Accuracy 310 Precision 311 Recall 311 F-measure 311 ROC curve and AUC 312 Summary 315 Chapter 11: Unsupervised Learning - PCA and Clustering 316 Technical requirements 317 Unsupervised learning 317 Reducing the dimensionality of data 318 PCA 319 Performing PCA 320 Clustering 323 Finding the number of clusters 324 The elbow method 325 The silhouette method 327 Partitioning data using k-means clustering 329 Hierarchical clustering 332 DBSCAN clustering 336 Spectral clustering 338 Evaluating clustering performance 341 Internal performance evaluation 342 The Davies-Bouldin index 342 The silhouette coefficient 342 External performance evaluation 343 The Rand score 343 The Jaccard score 343 F-Measure or F1-score 344 The Fowlkes-Mallows score 344 Summary 347
Page 12
Table of Contents [ vii ] Section 4: NLP, Image Analytics, and Parallel Computing Chapter 12: Analyzing Textual Data 349 Technical requirements 350 Installing NLTK and SpaCy 350 Text normalization 351 Tokenization 352 Removing stopwords 356 Stemming and lemmatization 358 POS tagging 360 Recognizing entities 361 Dependency parsing 362 Creating a word cloud 363 Bag of Words 365 TF-IDF 366 Sentiment analysis using text classification 367 Classification using BoW 368 Classification using TF-IDF 373 Text similarity 376 Jaccard similarity 377 Cosine similarity 378 Summary 379 Chapter 13: Analyzing Image Data 380 Technical requirements 381 Installing OpenCV 381 Understanding image data 382 Binary images 382 Grayscale images 383 Color images 383 Color models 384 Drawing on images 387 Writing on images 392 Resizing images 393 Flipping images 395 Changing the brightness 398 Blurring an image 399 Face detection 403 Summary 407 Chapter 14: Parallel Computing Using Dask 408 Parallel computing using Dask 409 Dask data types 410
Page 13
Table of Contents [ viii ] Dask Arrays 411 Dask DataFrames 412 DataFrame Indexing 413 Filter data 416 Groupby 417 Converting a pandas DataFrame into a Dask DataFrame 418 Converting a Dask DataFrame into a pandas DataFrame 418 Dask Bags 419 Creating a Dask Bag using Python iterable items 419 Creating a Dask Bag using a text file 420 Storing a Dask Bag in a text file 421 Storing a Dask Bag in a DataFrame 421 Dask Delayed 422 Preprocessing data at scale 424 Feature scaling in Dask 424 Feature encoding in Dask 426 Machine learning at scale 428 Parallel computing using scikit-learn 429 Reimplementing ML algorithms for Dask 431 Logistic regression 431 Clustering 433 Summary 435 Other Books You May Enjoy 437 Index 440
Page 14
Preface Data analysis enables you to generate value from small and big data by discovering new patterns and trends, and Python is one of the most popular tools for analyzing a wide variety of data. With this book, you'll get up and running with using Python for data analysis by exploring the different phases and methodologies used in data analysis, and you'll learn how to use modern libraries from the Python ecosystem to create efficient data pipelines. Starting with the essential statistical and data analysis fundamentals using Python, you'll perform complex data analysis and modeling, data manipulation, data cleaning, and data visualization using easy-to-follow examples. You'll then learn how to conduct time series analysis and signal processing using ARMA models. As you advance, you'll get to grips with smart processing and data analytics using machine learning algorithms such as regression, classification, Principal Component Analysis (PCA), and clustering. In the concluding chapters, you'll work on real-world examples to analyze textual and image data using natural language processing (NLP) and image analytics techniques, respectively. Finally, the book will demonstrate parallel computing using Dask. By the end of this data analysis book, you'll be equipped with the skills you need to prepare data for analysis and create meaningful data visualizations in order to forecast values from data. Who this book is for This book is for data analysts, business analysts, statisticians, and data scientists looking to learn how to use Python for data analysis. Students and academic faculties will also find this book useful for learning and teaching Python data analysis using a hands-on approach. A basic understanding of math and a working knowledge of Python will help you get started with this book. What this book covers Chapter 1, Getting Started with Python Libraries, explains the data analyst process and the successful installation of Python libraries and Anaconda. Also, we will discuss Jupyter Notebook and its advanced features.
Page 15
Preface [ 2 ] Chapter 2, NumPy and Pandas, introduces NumPy and Pandas. This chapter provides a basic overview of NumPy arrays, Pandas DataFrames, and their associated functions. Chapter 3, Statistics, gives a quick overview of descriptive and inferential statistics. Chapter 4, Linear Algebra, gives a quick overview of linear algebra and its associated NumPy and SciPy functions. Chapter 5, Data Visualization, introduces us to the matplotlib, seaborn, Pandas plotting, and bokeh visualization libraries. Chapter 6, Retrieving, Processing, and Storing Data, explains how to read and write various data formats, such as CSV, Excel, JSON, HTML, and Parquet. Also, we will discuss how to acquire data from relational and NoSQL databases. Chapter 7, Cleaning Messy Data, explains how to preprocess raw data and perform feature engineering. Chapter 8, Signal Processing and Time Series, contains time series and signal processing examples using sales, beer production, and sunspot cycle dataset. In this chapter, we will mostly use NumPy, SciPy, and statsmodels. Chapter 9, Supervised Learning – Regression Analysis, explains linear regression and logistic regression in detail with suitable examples using the scikit-learn library. Chapter 10, Supervised Learning – Classification Techniques, explains various classification techniques, such as naive Bayes, decision tree, K-nearest neighbors, and SVM. Also, we will discuss model performance evaluation measures. Chapter 11, Unsupervised Learning – PCA and Clustering, gives a detailed discussion on dimensionality reduction and clustering techniques. Also, we will evaluate the clustering performance. Chapter 12, Analyzing Textual Data, gives a quick overview of text preprocessing, feature engineering, sentiment analysis, and text similarity. This chapter mostly uses the NLTK, SpaCy, and scikit-learn libraries. Chapter 13, Analyzing Image Data, gives a quick overview of image processing operations using OpenCV. Also, we will discuss face detection. Chapter 14, Parallel Computing Using Dask, explains how to perform data preprocessing and machine learning modeling in parallel using Dask.
Page 16
Preface [ 3 ] To get the most out of this book The execution of the code examples provided in this book requires the installation of Python 3.5 or newer on Mac OS X, Linux, or Microsoft Windows. In this book, we will frequently use SciPy, NumPy, Pandas, scikit-learn, statsmodels, matplotlib, and seaborn. Chapter 1, Getting Started with Python Libraries, provides instructions for the installation and advanced tips so that you can work smoothly. Also, the process of installing specific and additional libraries is explained in the respective chapters. Installation of Bokeh is explained in Chapter 5, Data Visualization. Similarly, the installation of NLTK and SpaCy is explained in Chapter 12, Analyzing Textual Data. We can also install any library or package that you want to explore using the pip command. We need to run the following command with admin privileges: $ pip install <library name> We can also install it from our Jupyter Notebook with ! (exclamation mark) before the pip command: !pip install <library name> To uninstall a Python library or package installed with pip, use the following command: $ pip uninstall <library name> If you are using the digital version of this book, we advise you to type the code yourself or access the code via the GitHub repository (link available in the next section). Doing so will help you avoid any potential errors related to the copying and pasting of code. Download the example code files You can download the example code files for this book from GitHub at https:/ ​/​github. com/​PacktPublishing/ ​Python- ​Data- ​Analysis- ​Third- ​Edition. In case there's an update to the code, it will be updated on the existing GitHub repository. We also have other code bundles from our rich catalog of books and videos available at https:/​/​github. ​com/ ​PacktPublishing/ ​. Check them out!
Page 17
Preface [ 4 ] Download the color images We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://static.packt-cdn.com/downloads/9781789955248_ColorImages.pdf. Conventions used In this book, you will find a number of text styles and conventions used throughout this book. Here, we have shown some examples of these styles. Code words in the text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "The other convention the pandas project insists on is the import pandas as pd import statement." A block of code is set as follows: # Creating an array import numpy as np a = np.array([2,4,6,8,10]) print(a) Any command-line input or output is written as follows: $ mkdir $ cd css Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "Select System info from the Administration panel." Warnings or important notes appear like this. Tips and tricks appear like this.
Page 18
Preface [ 5 ] Get in touch Feedback from our readers is always welcome. General feedback: If you have questions about any aspect of this book, mention the book title in the subject of your message and email us at customercare@packtpub.com. Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit www.packtpub.com/support/errata, selecting your book, clicking on the Errata Submission Form link, and entering the details. Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at copyright@packt.com with a link to the material. If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit authors.packtpub.com. Reviews Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you! For more information about Packt, please visit packt.com.
Page 19
1 Section 1: Foundation for Data Analysis The main objective of this section is to build fundamental data analysis skills for the learner. These skills involve the Jupyter Notebook, and basic Python libraries such as NumPy, Pandas, Scipy, and statsmodels. Also, this section focuses on subjective knowledge of statistics and linear algebra to build math capabilities. This section includes the following chapters: Chapter 1, Getting Started with Python Libraries Chapter 2, NumPy and pandas Chapter 3, Statistics Chapter 4, Linear Algebra
Page 20
1 Getting Started with Python Libraries As you already know, Python has become one of the most popular, standard languages and is a complete package for data science-based operations. Python offers numerous libraries, such as NumPy, Pandas, SciPy, Scikit-Learn, Matplotlib, Seaborn, and Plotly. These libraries provide a complete ecosystem for data analysis that is used by data analysts, data scientists, and business analysts. Python also offers other features, such as flexibility, being easy to learn, faster development, a large active community, and the ability to work on complex numeric, scientific, and research applications. All these features make it the first choice for data analysis. In this chapter, we will focus on various data analysis processes, such as KDD, SEMMA, and CRISP-DM. After this, we will provide a comparison between data analysis and data science, as well as the roles and different skillsets for data analysts and data scientists. Finally, we will shift our focus and start installing various Python libraries, IPython, Jupyter Lab, and Jupyter Notebook. We will also look at various advanced features of Jupyter Notebooks. In this introductory chapter, we will cover the following topics: Understanding data analysis The standard process of data analysis The KDD process SEMMA CRISP-DM Comparing data analysis and data science The skillsets of data analysts and data scientists Installing Python 3 Software used in this book Using IPython as a shell
The above is a preview of the first 20 pages. Register to read the complete e-book.

Recommended for You

Loading recommended books...
Failed to load, please try again later

Tip the Site

Scan the WeChat Pay or Alipay code to tip. No login required.

WeChat Pay
Alipay
Back to List