Hands-on Deep Learning. A Guide to Deep Learning with Projects and Applications (Harsh Bhasin) (Z-Library)

📄 File Format: PDF

💾 File Size: 29.4 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

I Hands-on Deep Learning A Guide to Deep Learning with Projects and Applications Harsh Bhasin Apress8

📄 Page 2

Hands-on Deep Learning A Guide to Deep Learning with Projects and Applications Harsh Bhasin Apress

📄 Page 3

Hands-on Deep Learning: A Guide to Deep Learning with Projects and Applications Harsh Bhasin Faridabad, Haryana, India ISBN-13 (pbk): 979-8-8688-1034-3 ISBN-13 (electronic): 979-8-8688-1035-0 https://doi.org/10.1007/979-8-8688-1035-0 Copyright © 2024 by Harsh Bhasin This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Managing Director, Apress Media LLC: Welmoed Spahr Acquisitions Editor: Celestin Suresh John Development Editor: Laura Berendson Coordinating Editor: Gryffin Winkler Cover designed by eStudioCalamar Cover image designed by Freepik (www.freepik.com) Distributed to the book trade worldwide by Apress Media, LLC, 1 New York Plaza, New York, NY 10004, U.S.A. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com, or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please e-mail booktranslations@springernature.com; for reprint, paperback, or audio rights, please e-mail bookpermissions@springernature.com. Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales web page at http://www.apress.com/bulk-sales. Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub (https://github.com/Apress). For more detailed information, please visit https://www. apress.com/gp/services/source-code. If disposing of this product, please recycle the paper

📄 Page 4

To My Mother ...

📄 Page 5

Table of Contents About the Author ............................................................................................. xiii About the Technical Reviewers ..........................................................................xv Acknowledgments ............................................................................................xix Chapter 1: Revisiting Machine Learning.............................................................. 1 Machine Learning: Brief History, Definition, and Applications.................................................... 3 Types of Machine Learning: Task (T) ........................................................................................ 6 Performance (P) ...................................................................................................................... 7 Conventional Machine Learning Pipeline ................................................................................ 11 Regression ............................................................................................................................ 12 Feature Selection .................................................................................................................. 14 Filter Method ................................................................................................................... 14 Wrapper Method.............................................................................................................. 18 Filter vs. Wrapper Methods............................................................................................... 19 Feature Extraction................................................................................................................. 19 Gray-Level Co-occurrence Matrix..................................................................................... 20 Local Binary Pattern ......................................................................................................... 21 Histogram of Oriented Gradients ...................................................................................... 24 Principal Component Analysis................................................................................................ 24 Bias-Variance Trade-off..........................................................................................................28 Overfitting and Underfitting .............................................................................................. 28 Bias and Variance ............................................................................................................ 29 Application: Classification of Handwritten Digits Using a Conventional Machine Learning Pipeline .................................................................................................................. 30 v

📄 Page 6

TABLE OF CONTENTS Conclusion ............................................................................................................................ 38 Exercises............................................................................................................................... 39 Multiple-Choice Questions ............................................................................................... 39 Applications..................................................................................................................... 41 References ............................................................................................................................ 42 Chapter 2: Introduction to Deep Learning......................................................... 43 Neurons................................................................................................................................. 43 From Perceptron to the Winter of Artificial Intelligence ........................................................... 45 Imagery and Convolutional Neural Networks.......................................................................... 47 What’s New ........................................................................................................................... 49 Sequences ............................................................................................................................ 50 The Definition........................................................................................................................ 51 Generate Data Using Deep Learning ...................................................................................... 52 Conclusion ............................................................................................................................ 55 Exercises............................................................................................................................... 56 Multiple-Choice Questions ............................................................................................... 56 Activity............................................................................................................................. 57 References ............................................................................................................................ 58 Chapter 3: Neural Networks ............................................................................ 59 Objectives ............................................................................................................................. 59 Introduction........................................................................................................................... 59 Single-Layer Perceptron ........................................................................................................ 62 Implementation of a SLP........................................................................................................ 64 XOR Problem ......................................................................................................................... 75 Activation Functions.............................................................................................................. 76 1. Sigmoid ..................................................................................................................... 76 2. Tanh .......................................................................................................................... 77 3. Rectified Linear Unit (ReLU)........................................................................................ 78 4. Softmax ..................................................................................................................... 79 vi

📄 Page 7

TABLE OF CONTENTS Multi-layer Perceptron ........................................................................................................... 80 Solving the XOR Problem Using Multi-layer Perceptron ..................................................... 80 Architecture of MLP and Forward Pass ............................................................................. 82 Gradient Descent................................................................................................................... 84 Backpropagation................................................................................................................... 86 Implementation ..................................................................................................................... 87 Conclusion .......................................................................................................................... 104 Exercises............................................................................................................................. 105 Multiple-Choice Questions ............................................................................................. 105 Theory ........................................................................................................................... 108 Numerical ...................................................................................................................... 109 References .......................................................................................................................... 109 Chapter 4: Training Deep Networks ............................................................... 111 Introduction......................................................................................................................... 111 Train-Test Split.................................................................................................................... 111 Train-Validation-Test Split....................................................................................................112 K-Fold Split ......................................................................................................................... 112 Batch, Stochastic, and Mini-batch Gradient Descent ............................................................ 113 Batch Gradient Descent.................................................................................................. 114 Stochastic Gradient Descent .......................................................................................... 114 Mini-batch Gradient Descent.......................................................................................... 114 RMSprop ............................................................................................................................. 116 Adam Optimizer .................................................................................................................. 118 Conclusion .......................................................................................................................... 126 Exercises............................................................................................................................. 127 Multiple-Choice Questions ............................................................................................. 127 Theory ........................................................................................................................... 129 Experiments................................................................................................................... 130 References .......................................................................................................................... 130 vii

📄 Page 8

TABLE OF CONTENTS Chapter 5: Hyperparameter Tuning................................................................. 133 Introduction......................................................................................................................... 133 Bias-Variance Revisited........................................................................................................134 Hyperparameter Tuning........................................................................................................ 137 Experiments: Hyperparameter Tuning................................................................................... 142 Conclusion .......................................................................................................................... 150 Exercises............................................................................................................................. 150 Multiple-Choice Questions ............................................................................................. 150 Experiments................................................................................................................... 154 References .......................................................................................................................... 155 Chapter 6: Convolutional Neural Networks: I ................................................. 157 Convolutional Layer............................................................................................................. 159 Implementing Convolution ................................................................................................... 161 Padding ............................................................................................................................... 165 Stride and Other Layers ....................................................................................................... 167 Stride ............................................................................................................................. 167 Pooling........................................................................................................................... 168 Normalization ................................................................................................................ 169 Fully Connected Layer .................................................................................................... 170 Importance of Kernels.......................................................................................................... 170 Architecture of LeNet .......................................................................................................... 177 Conclusion .......................................................................................................................... 180 Exercises............................................................................................................................. 182 Multiple-Choice Questions ............................................................................................. 182 Numerical ...................................................................................................................... 184 Applications................................................................................................................... 184 Chapter 7: Convolutional Neural Network: II .................................................. 185 Sequential Model ................................................................................................................ 186 Creating the Model ......................................................................................................... 186 Adding Layers in the Model ............................................................................................ 187 viii

📄 Page 9

TABLE OF CONTENTS Removing the Last Layer from the Model........................................................................ 187 Initializing Weights ......................................................................................................... 188 Summary ....................................................................................................................... 188 Keras Layers ....................................................................................................................... 189 1. Dense Layer............................................................................................................. 189 2. Conv2D Layer .......................................................................................................... 190 3. Pooling..................................................................................................................... 190 4. Activations ............................................................................................................... 190 5. Initializing Weights ................................................................................................... 191 6. Miscellaneous.......................................................................................................... 191 MNIST Dataset Classification Using LeNet: Prerequisite........................................................ 192 LeNet................................................................................................................................... 192 Structure........................................................................................................................ 192 Implementation ............................................................................................................. 194 AlexNet................................................................................................................................ 198 Some More Architectures..................................................................................................... 201 GoogLeNet ..................................................................................................................... 201 ResNet ........................................................................................................................... 201 DenseNet ....................................................................................................................... 202 Conclusion .......................................................................................................................... 202 Exercises............................................................................................................................. 202 Multiple-Choice Questions ............................................................................................. 202 Implementations............................................................................................................ 205 References .......................................................................................................................... 205 Chapter 8: Transfer Learning ......................................................................... 207 Introduction......................................................................................................................... 207 Idea .................................................................................................................................... 207 VGG 16 and VGG 19 for Binary Classification ........................................................................ 208 Types and Strategies........................................................................................................... 217 ix

📄 Page 10

TABLE OF CONTENTS Limitations and Applications of Transfer Learning ................................................................ 219 Conclusion .......................................................................................................................... 220 Exercises............................................................................................................................. 220 Multiple-Choice Questions ............................................................................................. 220 Application..................................................................................................................... 222 References .......................................................................................................................... 222 Chapter 9: Recurrent Neural Network ............................................................ 225 Introduction......................................................................................................................... 225 Why Neural Networks Cannot Infer Sequences .................................................................... 226 Idea .................................................................................................................................... 228 Backpropagation Through Time ........................................................................................... 229 Types of RNN....................................................................................................................... 230 Applications ........................................................................................................................ 234 Sentiment Classification................................................................................................. 234 Parts of Speech Tagging ................................................................................................. 241 Handwritten Text Recognition......................................................................................... 249 Speech to Text ............................................................................................................... 250 Conclusion .......................................................................................................................... 251 Exercises............................................................................................................................. 251 Multiple-Choice Questions ............................................................................................. 251 Theory ........................................................................................................................... 254 Image Captioning........................................................................................................... 254 References .......................................................................................................................... 255 Chapter 10: Gated Recurrent Unit and Long Short-Term Memory .................... 257 Introduction......................................................................................................................... 257 GRU .................................................................................................................................... 258 Long Short-Term Memory .................................................................................................... 260 Named Entity Recognition.................................................................................................... 262 Sentiment Classification ...................................................................................................... 273 Conclusion .......................................................................................................................... 282 x

📄 Page 11

TABLE OF CONTENTS Exercises............................................................................................................................. 283 Multiple-Choice Questions ............................................................................................. 283 Theory ........................................................................................................................... 285 Application-Based Questions ......................................................................................... 285 References .......................................................................................................................... 286 Chapter 11: Autoencoders ............................................................................. 287 Introduction......................................................................................................................... 287 Concept and Types .............................................................................................................. 287 The Math ....................................................................................................................... 288 Types of Autoencoders ................................................................................................... 288 Autoencoder and Principal Component Analysis................................................................... 290 Training of an Autoencoder .................................................................................................. 291 Latent Representation Using Autoencoders .......................................................................... 293 Experiment 1 ................................................................................................................. 293 Experiment 2 ................................................................................................................. 297 Finding Latent Representation Using Multiple Layers........................................................... 299 Variants of Autoencoders ..................................................................................................... 302 Sparse Autoencoder ....................................................................................................... 302 Denoising Autoencoder................................................................................................... 303 Variational Autoencoder ................................................................................................. 303 Conclusion .......................................................................................................................... 303 Exercises............................................................................................................................. 304 Multiple-Choice Questions ............................................................................................. 304 Theory ........................................................................................................................... 306 Applications................................................................................................................... 306 Chapter 12: Introduction to Generative Models................................................ 307 Introduction......................................................................................................................... 307 Hopfield Networks .............................................................................................................. 307 Boltzmann Machines........................................................................................................... 310 A Gentle Introduction to Transformers .................................................................................. 314 xi

📄 Page 12

TABLE OF CONTENTS An Introduction to Self-Attention .................................................................................... 315 The Transformer ............................................................................................................ 317 Conclusion .......................................................................................................................... 318 Exercise .............................................................................................................................. 318 Multiple-Choice Questions ............................................................................................. 318 Theory ........................................................................................................................... 320 References ......................................................................................................................... 321 Appendix A: Classifying The Simpsons Characters ......................................... 323 Appendix B: Face Detection........................................................................... 331 Appendix C: Sentiment Classification Revisited .............................................. 335 Appendix D: Predicting Next Word................................................................. 343 Appendix E: COVID Classification................................................................... 347 Appendix F: Alzheimer’s Classification .......................................................... 351 Appendix G: Music Genre Classification Using MFCC and Convolutional Neural Network.............................................................................................. 355 Index.............................................................................................................. 359 xii

📄 Page 13

About the Author Harsh Bhasin is a researcher and practitioner. He has completed his PhD in “Diagnosis and Conversion Prediction of Mild Cognitive Impairment Using Machine Learning” from Jawaharlal Nehru University, New Delhi. He worked as a Deep Learning consultant for various firms and taught at various universities, including Jamia Hamdard and Delhi Technological University (DTU). He is currently associated with Bennett University. He has authored 11 books including Programming in C#, Oxford University Press, 2014, and Algorithms, Oxford University Press, 2015. He has authored more than 40 papers that have been published in international conferences and renowned journals, including Alzheimer’s & Dementia, Soft Computing, Springer Nature, BMC Medical Informatics and Decision Making, AI & Society, etc. He is the reviewer of a few renowned journals and has been the editor of a few special issues. He has been a recipient of Visvesvaraya Fellowship, Ministry of Electronics and Information Technology. His areas of expertise include Deep Learning, algorithms, and medical imaging. Apart from his professional endeavors, he is deeply interested in Hindi poetry: the progressive era and Hindustani classical music: percussion instruments. xiii

📄 Page 14

About the Technical Reviewers Karanbir Singh is an accomplished engineering leader with over 7 years of experience leading AI/ML engineering, distributed systems, and microservices projects across diverse industries, including fintech and automotive. Currently working as a Senior Software Engineer at Salesforce, he focuses on backend technologies as well as AI. His career has been marked by a commitment to building high-performing teams, driving technological innovation, and delivering impactful solutions that enhance business outcomes. At TrueML, as an engineering manager, he managed a critical team to develop and deploy Machine Learning models in production. He successfully expanded and led engineering teams, significantly improving feature development velocity and client engagement through strategic collaboration and mentorship. His leadership directly contributed to increased revenue, client retention, and substantial cost savings through innovative internal solutions. His role involved not only steering technical projects but also shaping the company’s roadmap in partnership with data science, product management, and platform teams. Previously, at Lucid Motors and Poynt, he developed critical components and integrations that advanced product capabilities and strengthened industry partnerships. His technical expertise spans across AI/ML, cloud computing, and software architecture, and he is adept at utilizing cutting-edge technologies and methodologies to drive results. Karanbir holds a master’s degree in Computer Software Engineering from San Jose State University and has been recognized for his innovative contributions, including winning the Silicon Valley Innovation Challenge. He is passionate about mentoring and coaching emerging talent and thrives in environments where he can leverage his skills to solve complex problems and advance technological initiatives. xv

📄 Page 15

ABOUT THE TECHNICAL REVIEWERS Prashanth Josyula, a dynamic force in the tech world whose journey is marked by an unyielding passion for innovation and an extraordinary depth of expertise in both technical literature and software engineering. As a Principal Member of Technical Staff (PMTS) at Salesforce, Prashanth doesn’t just meet expectations—he consistently exceeds them, pushing the boundaries of what’s possible in technology. With over 16 years of robust experience in the IT industry, Prashanth has mastered a multitude of programming languages and technologies, establishing himself as a true polyglot programmer. His proficiency spans across Java, Python, Scala, Kotlin, JavaScript, TypeScript, Shell Scripting, SQL, and an array of open source solutions. Since beginning his professional journey in 2008, he has delved into various domains, each time leaving a mark of excellence. In the realm of Java/Java EE and Spring, Prashanth has been instrumental in designing and building resilient, scalable backend systems that power critical applications across industries. His deep understanding of these technologies ensures robust and high-performance solutions tailored to meet complex business needs. Prashanth’s expertise in UI technologies is equally impressive. He has crafted intuitive, responsive user interfaces using frameworks like ExtJS, JQuery, DOJO, Angular, and React. His commitment to creating seamless user experiences shines through in every project, bridging the gap between complex backend processes and user-friendly frontend interfaces. Venturing into big data, Prashanth has leveraged platforms like Hadoop, Spark, Hive, Oozie, and Pig to transform massive datasets into valuable insights, driving strategic decisions and innovations. His ability to harness the power of big data showcases his analytical mindset and his knack for tackling large-scale data challenges. In the field of microservices and infrastructure, Prashanth has been a pioneer, in engineering robust and scalable solutions with cutting-edge tools like Kubernetes, Helm, Terraform, and Spinnaker. His contributions to open source projects reflect his commitment to collaborative innovation and continuous improvement. Moreover, Prashanth is at the forefront of AI and Machine Learning, exploring and advancing the capabilities of these transformative technologies. His work in this area is characterized by a fearless approach to experimentation and a relentless pursuit of knowledge. xvi

📄 Page 16

ABOUT THE TECHNICAL REVIEWERS Each day for Prashanth is an exciting adventure, filled with opportunities to learn, innovate, and lead. His career is a testament to his dedication to advancing technology, not just for the sake of progress, but to truly make a difference. With his unparalleled skills and a visionary mindset, Prashanth continues to inspire peers and push the envelope of technological possibility. xvii

📄 Page 17

Acknowledgments Knowledge is in the end based on acknowledgement. —Ludwig Wittgenstein I have been lucky enough to have met people who inspired me to learn. First of all, I would like to thank Professor Moinuddin, former Pro-Vice Chancellor, Delhi Technological University, for his unconditional support. He has deposed his faith in me in my formative years and helped me grow. I would also like to thank the late Professor A. K. Sharma, former Dean and Chairperson, Department of Computer Science, YMCA, Faridabad, for his constant encouragement. I have been able to write this book, author papers, and work on projects only because of the encouragement provided by him. I am also thankful to the following academicians and professionals for their encouragement and providing unconditional support to me: • Professor I. K. Bhat, Vice Chancellor, MRU, India • Professor Prashant Jha, King’s College London • Professor Tapas Kumar, Associate Dean, SET, MRIIRS, India • Professor Ranjit Biswas, former Dean, Faculty of Engineering, Jamia Hamdard • Professor Naresh Chauhan, Department of Computer Science, YMCA University of Science and Technology I am thankful to my student Nishant Kumar, NorthCap University, for editing the chapters. I would also like to acknowledge the help of the students’ team, Amit Thakur, Ankit Singh, and Jai Mishra, for their help. I would like to express my sincere gratitude to my mother, Vanita Bhasin; sister, Swati Bhasin; and rest of the family, including my pets, late Zoe and Xena, and friends for their unconditional support to me. xix

📄 Page 18

ACKNOWLEDGMENTS I am extremely grateful to the team at Springer for their insightful guidance and unwavering support. I am also thankful to the team and reviewers for their editorial feedback, design inputs, and constant reviews, which have transformed this manuscript into an informative and interesting book. I would be glad to receive your comments or suggestions, which can be incorporated in the future editions of the book. You can reach me at i_harsh_bhasin@yahoo.com. —Dr. Harsh Bhasin xx

📄 Page 19

CHAPTER 1 Revisiting Machine Learning Imagine being transported back to the late 1990s in the United States, where the authorities discover your expertise in Machine Learning (ML). They reach out to seek your assistance in the automation of a time-consuming task: reading pin code on letters. Supposedly there are 500 such employees in various post offices across the country, and each employee was being paid a sum of $2000 per month to perform this task. This accumulates to a monthly expenditure of $1000000, resulting in an annual cost of $12 million, or a staggering $60 million nationwide, over the next five years. To assist the government in saving valuable exchequer funds, you are tasked with designing a program that can efficiently read and interpret pin codes on the letters. This solution will not only help in cost savings but also will greatly augment accuracy and accelerate the process. Can you think of an algorithm to accomplish this task? It turns out that it is not very easy to write such an algorithm. Let’s see why! To understand the problem, let us start with an algorithm that recognizes “1” in a 28-pixel x 28-pixel image. Ideally, the pixels around the central vertical may be considered for identifying if the image contains “1.” However, the number is handwritten, and therefore it can be written in many ways, in terms of scale, orientation, style, etc. Figure 1-1 shows some pictures of handwritten “1”s obtained from the popular MNIST dataset containing images of handwritten digits. If recognizing “1” is difficult, then imagine recognizing all the digits and alphabets and processing these, in general. © Harsh Bhasin 2024 H. Bhasin, Hands-on Deep Learning, https://doi.org/10.1007/979-8-8688-1035-0_1 1

📄 Page 20

CHAPTER 1 REVISITING MACHINE LEARNING Fig. 1(a): Fig. 1(d): Figure 1-1. Some pictures of “1”s obtained from the MNIST dataset Recognizing handwritten digits is an easy task for human beings, but it is difficult to come out with a set of rules or algorithms that recognizes the digit in a given picture. So we need some system that can imitate human beings to accomplish this task. Here Machine Learning (ML) can help us. Informally ML can be defined as follows: Machine Learning is a subset of Artificial Intelligence, which may be considered as the ability of machines to imitate humans. [1] The formal definition of ML is discussed in the following sections. ML helps us accomplish tasks like disease classification, prediction and forecasting, object recognition, sentiment classification, etc. This chapter briefly introduces Machine Learning and discusses its types, the pipeline and its components, its applications, and the bias-variance trade-offs. This 2