Getting started with Deep Learning for Natural Language Processing Learn how to build NLP applications with Deep Learning… (Sunil Patel) (Z-Library)

📄 File Format: PDF

💾 File Size: 9.2 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

(This page has no text content)

📄 Page 2

Getting Started with Deep Learning for Natural Language Processing Learn How to Build NLP Applications with Deep Learning Sunil Patel www.bpbonline.com

📄 Page 3

FIRST EDITION 2021 Copyright © BPB Publications, India ISBN: 978-93-89898-11-8 All Rights Reserved. No part of this publication may be reproduced, distributed or transmitted in any form or by any means or stored in a database or retrieval system, without the prior written permission of the publisher with the exception to the program listings which may be entered, stored and executed in a computer system, but they can not be reproduced by the means of publication, photocopy, recording, or by any electronic and mechanical means. LIMITS OF LIABILITY AND DISCLAIMER OF WARRANTY The information contained in this book is true to correct and the best of author’s and publisher’s knowledge. The author has made every effort to ensure the accuracy of these publications, but publisher cannot be held responsible for any loss or damage arising from any information in this book. All trademarks referred to in the book are acknowledged as properties of their respective owners but BPB Publications cannot guarantee the accuracy of this information. Distributors:

📄 Page 4

BPB PUBLICATIONS 20, Ansari Road, Darya Ganj New Delhi-110002 Ph: 23254990/23254991 MICRO MEDIA Shop No. 5, Mahendra Chambers, 150 DN Rd. Next to Capital Cinema, V.T. (C.S.T.) Station, MUMBAI-400 001 Ph: 22078296/22078297 DECCAN AGENCIES 4-3-329, Bank Street, Hyderabad-500195 Ph: 24756967/24756400

📄 Page 5

BPB BOOK CENTRE 376 Old Lajpat Rai Market, Delhi-110006 Ph: 23861747 Published by Manish Jain for BPB Publications, 20 Ansari Road, Darya Ganj, New Delhi-110002 and Printed by him at Repro India Ltd, Mumbai www.bpbonline.com

📄 Page 6

Dedicated to My family

📄 Page 7

About the Author Sunil Patel has completed his Master’s in Information Technology from the Indian Institute of Information Technology-Allahabad, with a thesis focused on investigating 3D protein-protein interactions with deep learning. Sunil has worked with TCS Innovation Labs, Excelra, and Innoplexus before joining Nvidia. The main areas of research were using Deep Learning, Natural language processing in Banking, and healthcare domain. Sunil started experimenting with deep learning by implanting the basic layer used in pipelines and then developing complex pipelines for a real-life problem. Additionally, Sunil has participated in CASP-2014 in collaboration with SCFBIO-IIT Delhi to efficiently predict possible Protein multimer formation and its impact on diseases using Deep Learning. Currently, Sunil works as Data Scientist – III with Nvidia. In Nvidia, Sunil has expanded his area of interest to computer vision and simulated environments, and he extensively works in the banking, defense, and healthcare verticals areas. Sunil is currently focused on using GPUs for high-fidelity physics simulation. He has 3 pending US patents and 4 publications in the Deep Learning domain. To know more about his current research topic and interests, you can check out his LinkedIn profile:

📄 Page 8

About the Reviewer Anurag Punia has 6 years of experience in data science and machine learning, with a special interest in topic modeling, information retrieval, and named entity recognition under the subfield of natural language processing. He has worked and delivered several data science projects across industry verticals, like insurance, asset management, marketing, tourism, and real estate. Currently, he is part of the center of excellence of a leading logistics company in Dubai, UAE. Anurag has a research-focused BS-MS dual degree from IISER Bhopal with a major in physics. He can be reached at anurag.punia@gmail.com or https://www.linkedin.com/in/anurag-punia-data-scientist/

📄 Page 9

Acknowledgements First and foremost, I would like to thank God for giving me the courage to write this book. I would like to thank everyone at BPB Publications for helping me polish it and finally converting my writing to paperback. I would also like to thank my parents, wife, and brother for their endless support and for helping me in numerous ways. Lastly, I would like to thank my critics. Without their criticism, I would never be able to write this book. Sunil Patel

📄 Page 10

Preface “The world’s most valuable resource is no longer oil but its data”. Nowadays, titans and the most valued firm in the world like Amazon, Google, Apple, and Microsoft have similar concerns as were raised for oil a century ago. Data is changing the way we live, and the amount of data generated in the past few years is more than that generated since human beings have existed. The amount of data is expected to grow exponentially with the boom in connected devices, personal assistants, blockchain, and mobile devices. The condition for the storage of data is getting favorable, as storage devices are getting cheaper 3X every 3 years. Hardware giants like Nvidia already claimed to have broken Moore’s law, which also indicates the exponential growth in processing power. Today’s world is highly favorable to the data-centric economy. And that’s exactly why data is the next oil. Unstructured and structured data is increasing at a similar rate. The former comes from a majority of sources, and algorithms are constantly being discovered to store and assimilate such data. Unstructured data can be anything, for example, scientific literature, randomly clicked selfies, chat messages, sensor data from self-driving vehicles, and voice/video over the Internet. It is rich in information, but processing such data and training a machine using such data is challenging. However, advancement has been made in gaining better understanding of unstructured data and using such a pre-trained network for supervised learning

📄 Page 11

in recent years. This technique is popularly known by the term “Transfer Learning”. Transfer learning decreases training time and also requires less amount of training data to achieve state-of-the-art results. Another type of data is structured data, which is majorly manually curated or generated semi-automatically. Actually, structured data is a bar of gold, an asset that costs millions and is capable of paying back in billions. Machine learning is being extensively used in the field of medical diagnostics. Recently, the Food and Drugs Administration (FDA) developed a robot named IDx DR as the first autonomous AI- based diagnostic system. Yet another San Francisco startup developed a text recruit system called Automated Recruitment Interface (ARI), which is capable of holding a two-way conversation with candidates. It is also capable of posting job advertisements and openings, scheduling and conducting interviews, and maintaining all updates along the entire hiring funnel. Startups and firms are developing a system like Artificial Intelligence Virtual Artist (AIVA). Firms like Melodies and Google are generating music using artificial intelligence. In a popular blog by Andrej Karpathy “The Unreasonable Effectiveness of Recurrent Neural Networks,” he demonstrated that LSTM Models can easily generate lyrics. The days are not far when there will be robots making food in the restaurant and serving it while singing beautifully. This generated music will be rated by you and will be instantly sold live in another part of the world based on your ratings. The new wave has been created by

📄 Page 12

Google duplex—an AI engine that can make a call on your behalf to make reservations. Machine learning has an equal number of applications in the fields of vision and text. Vision-related use cases exist in robotics, self-driving cars, self-flying vehicles, optical character readers, surveillance cameras, and security systems. The application of machine learning techniques on text is also known as Natural Language Processing (NLP), which can be applied to applications like text summarization, sentiment analysis, intend analysis, plagiarism detection, language translation, topic extraction, and audio language translation, text to speech and speech to text. In the last 2 years, the GLUE score rose by almost 15 points from 64.7 to 80.4. The General Language Understanding Evaluation (GLUE) benchmark is a collection of resources for training, evaluating, and analyzing natural language understanding systems. Various state-of-the-art models like ELMo, ULMFiT, OpenAi transformer and Brat like models have come up and are constantly shaking up previous state-of-the-art models. This is an ImageNet movement for text. This book is a comprehension of all the resources requires to not only learn NLP but to master it, and it is written keeping a beginner’s skillset in mind. This book covers the entire spectrum, from understanding the basic concept of machine learning to the application of complex networks like generative networks, reinforcement learning, and speech processing in NLP.

📄 Page 13

In chapter we will learn about the basics of machine learning. The chapter includes basic concept like understanding data, when to apply machine learning, understanding various aspects of training a model, the founding principle of machine learning and AI, generalization, and dealing with overfitting and underfitting. This chapter will cover diagnostic concepts like bias-variance tradeoff, training and learning curves, generalization, and regularization concepts. In chapter we will learn basic text processing. This chapter will cover the use as well as the implementation of techniques like stemming, lemmatization, and tokenization. This chapter covers basic operation and network building with Pytorch. Learning about Pytorch helps users quickly compile the network as per the desired thought process. As the scope of this book is focused toward NLP, we will also explore a utility called TorchText. It alleviates many problems related to text processing and also helps easily distribute data to multiple GPUs. Chapter 3 is about converting/ representing our text into vectors so that it can be easily consumed by models. This chapter will cover various vital techniques like TF-IDF and Word2Vec. In addition to traditional techniques, also it will cover character-based vector embedding techniques like FastText. Chapter 4 will cover Recurrent Neural Network (RNN), which is considered a milestone in sequence processing techniques. Going ahead of Vanilla RNN, this chapter will also help readers understand as well as implement the Gated Recurrent Units (GRU) and Long Short-Term Memory (LSTM) Units. This chapter

📄 Page 14

will cover topics like a batch implementation of recurrent networks, attention architecture, and highway networks that enable us to train very deep sequence models. Chapter 5 will take you through Convolution Neural Networks (CNN), which are heavily used in text processing nowadays. In this chapter, we will learn basic convolution operations and the effect of various parameters related to CNN to the accuracy of the concerned task. This chapter also covers concepts like Dropout and batch normalization, which help achieve greater accuracy with CNN. We will also cover advanced architectures like DenseNet. After covering everything required to get going, it’s time to use the generated model in an unsupervised way or by someone else in our task. Chapter 6 will explore vital topics required to apply transfer learning with text. This chapter will cover advanced architectures like ELMo-Bilm, sentence to vector, skip thought and InferSent. All previous chapters make for a good foundation, and now we will apply a combination of all techniques to practical NLP tasks like sentiment analysis, implementing various approaches of topic modeling, text generation, building named entity recognition, building text summarization engine, and building language translation model. Chapter 7 will provide hands-on experience regarding all the listed use cases.

📄 Page 15

Chapter 8 is all about complex networks and very recent techniques. It will take you through Recurrent Convolution Neural Network (RCNN) and Siamese Network. This chapter will cover advanced techniques like Random Multi-Model, Snapshot Ensemble techniques, CTC loss Recognition, and Sentence Piece. It will also explore a wonderful application of RNN and CNN in generating captions from images. Chapter 9 will help you understand the fascinating world of Ian Goodfellow and concepts like Nash Equilibrium, KL-Divergence, KL- Divergence, JS-Divergence and KullbackLeibler Divergence to understand working on the Generative Adversarial Network. We will look at tips and tricks to solve the problem of an unstable gradient in the GAN. Finally, we will understand and code different types of GAN like Variational Autoencoder, and learn the application of GAN in generating images from text. Chapter 10 will walk you through more advanced techniques of speech processing. It will cover how audio signals are captured and stored and look at a small use case of spoken digit recognition with an end-to-end model. This chapter will also cover advance frameworks, like deep speech and deep voice, and their usage is covered. At the end, the book look at how to perform faster training and better deployment by utilizing the latest development in hardware and software. This book covers all the necessary topics from the basics of machine learning to advance NLP techniques. That said, one

📄 Page 16

should know the basic concepts of machine learning to quickly grasp these topics. This book assumes that you have hands-on experience with the basics of machine learning and libraries like Numpy, NLTK, Matplotlib, PIL, and Scikit-Learn. Libraries like PyTorch deal with the differentiation required during the backpropagation of Deep Learning models and keeps users away from the mathematics required in building such models from scratch. We will use PyTorch, but understanding basic algebra, statistics, and vector space will aid easier grasping.

📄 Page 17

Downloading the code bundle and coloured images: Please follow the link to download the Code Bundle and the Coloured Images of the book: https://rebrand.ly/fxrpk Errata We take immense pride in our work at BPB Publications and follow best practices to ensure the accuracy of our content to provide with an indulging reading experience to our subscribers. Our readers are our mirrors, and we use their inputs to reflect and improve upon human errors, if any, that may have occurred during the publishing processes involved. To let us maintain the quality and help us reach out to any readers who might be having difficulties due to any unforeseen errors, please write to us at : errata@bpbonline.com Your support, suggestions and feedbacks are highly appreciated by the BPB Publications’ Family.

📄 Page 18

Did you know that BPB offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.bpbonline.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at business@bpbonline.com for more details. At you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on BPB books and eBooks.

📄 Page 19

BPB is searching for authors like you If you're interested in becoming an author for BPB, please visit www.bpbonline.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea. The code bundle for the book is also hosted on GitHub at In case there's an update to the code, it will be updated on the existing GitHub repository. We also have other code bundles from our rich catalog of books and videos available at Check them out! PIRACY If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at business@bpbonline.com with a link to the material. If you are interested in becoming an author

📄 Page 20

If there is a topic that you have expertise in, and you are interested in either writing or contributing to a book, please visit REVIEWS Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at BPB can understand what you think about our products, and our authors can see your feedback on their book. Thank you! For more information about BPB, please visit