Deep Learning with JavaScript Neural networks in TensorFlow.js (Shanqing Cai, Stanley Bileschi, Eric D. Nielsen etc.) (Z-Library)

📄 File Format: PDF

💾 File Size: 17.6 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

M A N N I N G Shanqing Cai Stanley Bileschi Eric D. Nielsen François Chollet Foreword by Nikhil Thorat and Daniel Smilkov Neural networks in TensorFlow.js

📄 Page 2

Data Ingest data Sect. 6.1, 6.2, 6.3 Clean data Sect. 6.4 Visualize data Sect. 7.1 Augment data Sect. 6.5 Input data type Recommended layer API Reference Numerical data (without sequential order) Dense Chapters 2 and 3 Images or data that can be represented as images (e.g., audio, game board) 2D convolution and pooling Chapters 4 and 5 Sequential data, including text • RNN (LSTM, GRU) • Embedding • 1D convolution • Attentional • Sect. 9.1.2 • Sect. 9.2.3 • Sect. 9.2.4 • Sect. 9.3 Model building 1: Choosing key layer types based on your data Model building 2: Choosing last-layer activation, loss, and metric functions Last-layer activation Loss function Metric(s) Reference Regression (predicting a real number) Linear meanSquaredError meanAbsoluteError (Same as loss) Chapter 2 Sect. 9.1 Binary classification (making a binary decision) Sigmoid binaryCrossentropy Accuracy, precision, recall, sensitivity, TPR, FPR, ROC, AUC Sect. 3.1, 3.2, 9.2 Multi-class classification (deciding among multiple classes) Softmax categoricalCrossentropy Accuracy, confusion matrix Sect. 3.3, 9.3 A mix of the above (for example, numbers plus classes) (Multiple) Custom loss function (multiple) Continues inside back cover Sect. 5.2 Advanced and miscellaneous task types Reference Transfer learning (applying a pretrained model to new data) Chapter 5 Generative learning (generating new examples based on training data) Chapter 10 Reinforcement learning (training an agent to interact with the environment) Chapter 11 Working with data Task type (What are you predicting?)

📄 Page 3

Deep Learning with JavaScript

📄 Page 4

(This page has no text content)

📄 Page 5

Deep Learning with JavaScript NEURAL NETWORKS IN TENSORFLOW.JS SHANQING CAI STANLEY BILESCHI ERIC D. NIELSEN WITH FRANÇOIS CHOLLET FOREWORD BY NIKHIL THORAT DANIEL SMILKOV M A N N I N G SHELTER ISLAND

📄 Page 6

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com © 2020 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. Manning Publications Co. Development editor: Jenny Stout 20 Baldwin Road Technical development editor: Marc-Phillipe Huget PO Box 761 Review editor: Ivan Martinovič Shelter Island, NY 11964 Project editor: Lori Weidert Copy editor: Rebecca Deuel-Gallegos Proofreader: Jason Everett Technical proofreader: Karsten Strøbæck Typesetter: Dottie Marsico Cover designer: Marija Tudor ISBN 9781617296178 Printed in the United States of America

📄 Page 7

brief contents PART 1 MOTIVATION AND BASIC CONCEPTS. .................................1 1 ■ Deep learning and JavaScript 3 PART 2 A GENTLE INTRODUCTION TO TENSORFLOW.JS . ............. 35 2 ■ Getting started: Simple linear regression in TensorFlow.js 37 3 ■ Adding nonlinearity: Beyond weighted sums 79 4 ■ Recognizing images and sounds using convnets 117 5 ■ Transfer learning: Reusing pretrained neural networks 152 PART 3 ADVANCED DEEP LEARNING WITH TENSORFLOW.JS. ....... 199 6 ■ Working with data 201 7 ■ Visualizing data and models 246 8 ■ Underfitting, overfitting, and the universal workflow of machine learning 273 9 ■ Deep learning for sequences and text 292 10 ■ Generative deep learning 334 11 ■ Basics of deep reinforcement learning 371 PART 4 SUMMARY AND CLOSING WORDS . .................................. 415 12 ■ Testing, optimizing, and deploying models 417 13 ■ Summary, conclusions, and beyond 453v

📄 Page 8

(This page has no text content)

📄 Page 9

contents foreword xiii preface xv acknowledgments xvii about this book xix about the authors xxii about the cover illustration xxiii PART 1 MOTIVATION AND BASIC CONCEPTS....................1 1 Deep learning and JavaScript 3 1.1 Artificial intelligence, machine learning, neural networks, and deep learning 6 Artificial intelligence 6 ■ Machine learning: How it differs from traditional programming 7 ■ Neural networks and deep learning 12 ■ Why deep learning? Why now? 16 1.2 Why combine JavaScript and machine learning? 18 Deep learning with Node.js 24 ■ The JavaScript ecosystem 25 1.3 Why TensorFlow.js? 27 A brief history of TensorFlow, Keras, and TensorFlow.js 27 ■ Why TensorFlow.js: A brief comparison with similar libraries 31 ■ How is TensorFlow.js being used by the world? 31 ■ What this book will and will not teach you about TensorFlow.js 32vii

📄 Page 10

CONTENTSviiiPART 2 A GENTLE INTRODUCTION TO TENSORFLOW.JS ..............................................35 2 Getting started: Simple linear regression in TensorFlow.js 37 2.1 Example 1: Predicting the duration of a download using TensorFlow.js 38 Project overview: Duration prediction 38 ■ A note on code listings and console interactions 39 ■ Creating and formatting the data 40 ■ Defining a simple model 43 ■ Fitting the model to the training data 46 ■ Using our trained model to make predictions 48 ■ Summary of our first example 49 2.2 Inside Model.fit(): Dissecting gradient descent from example 1 50 The intuitions behind gradient-descent optimization 50 Backpropagation: Inside gradient descent 56 2.3 Linear regression with multiple input features 59 The Boston Housing Prices dataset 60 ■ Getting and running the Boston-housing project from GitHub 61 ■ Accessing the Boston- housing data 63 ■ Precisely defining the Boston-housing problem 65 ■ A slight diversion into data normalization 66 Linear regression on the Boston-housing data 70 2.4 How to interpret your model 74 Extracting meaning from learned weights 74 ■ Extracting internal weights from the model 75 ■ Caveats on interpretability 77 3 Adding nonlinearity: Beyond weighted sums 79 3.1 Nonlinearity: What it is and what it is good for 80 Building the intuition for nonlinearity in neural networks 82 Hyperparameters and hyperparameter optimization 89 3.2 Nonlinearity at output: Models for classification 92 What is binary classification? 92 ■ Measuring the quality of binary classifiers: Precision, recall, accuracy, and ROC curves 96 The ROC curve: Showing trade-offs in binary classification 99 Binary cross entropy: The loss function for binary classification 103 3.3 Multiclass classification 106 One-hot encoding of categorical data 107 ■ Softmax activation 109 ■ Categorical cross entropy: The loss function for multiclass classification 111 ■ Confusion matrix: Fine-grained analysis of multiclass classification 113

📄 Page 11

CONTENTS ix4 Recognizing images and sounds using convnets 117 4.1 From vectors to tensors: Representing images 118 The MNIST dataset 119 4.2 Your first convnet 120 conv2d layer 122 ■ maxPooling2d layer 126 ■ Repeating motifs of convolution and pooling 127 ■ Flatten and dense layers 128 ■ Training the convnet 130 ■ Using a convnet to make predictions 134 4.3 Beyond browsers: Training models faster using Node.js 137 Dependencies and imports for using tfjs-node 137 ■ Saving the model from Node.js and loading it in the browser 142 4.4 Spoken-word recognition: Applying convnets on audio data 144 Spectrograms: Representing sounds as images 145 5 Transfer learning: Reusing pretrained neural networks 152 5.1 Introduction to transfer learning: Reusing pretrained models 153 Transfer learning based on compatible output shapes: Freezing layers 155 ■ Transfer learning on incompatible output shapes: Creating a new model using outputs from the base model 161 Getting the most out of transfer learning through fine-tuning: An audio example 174 5.2 Object detection through transfer learning on a convnet 185 A simple object-detection problem based on synthesized scenes 186 Deep dive into simple object detection 187 PART 3 ADVANCED DEEP LEARNING WITH TENSORFLOW.JS . ...........................................199 6 Working with data 201 6.1 Using tf.data to manage data 202 The tf.data.Dataset object 203 ■ Creating a tf.data.Dataset 203 Accessing the data in your dataset 209 ■ Manipulating tfjs-data datasets 210 6.2 Training models with model.fitDataset 214

📄 Page 12

CONTENTSx6.3 Common patterns for accessing data 220 Working with CSV format data 220 ■ Accessing video data using tf.data.webcam() 225 ■ Accessing audio data using tf.data.microphone() 228 6.4 Your data is likely flawed: Dealing with problems in your data 230 Theory of data 231 ■ Detecting and cleaning problems with data 235 6.5 Data augmentation 242 7 Visualizing data and models 246 7.1 Data visualization 247 Visualizing data using tfjs-vis 247 ■ An integrative case study: Visualizing weather data with tfjs-vis 255 7.2 Visualizing models after training 260 Visualizing the internal activations of a convnet 262 Visualizing what convolutional layers are sensitive to: Maximally activating images 265 ■ Visual interpretation of a convnet’s classification result 269 8 Underfitting, overfitting, and the universal workflow of machine learning 273 8.1 Formulation of the temperature-prediction problem 274 8.2 Underfitting, overfitting, and countermeasures 278 Underfitting 278 ■ Overfitting 280 ■ Reducing overfitting with weight regularization and visualizing it working 282 8.3 The universal workflow of machine learning 287 9 Deep learning for sequences and text 292 9.1 Second attempt at weather prediction: Introducing RNNs 294 Why dense layers fail to model sequential order 294 ■ How RNNs model sequential order 296 9.2 Building deep-learning models for text 305 How text is represented in machine learning: One-hot and multi-hot encoding 306 ■ First attempt at the sentiment-analysis problem 308 ■ A more efficient representation of text: Word embeddings 310 ■ 1D convnets 312

📄 Page 13

CONTENTS xi9.3 Sequence-to-sequence tasks with attention mechanism 321 Formulation of the sequence-to-sequence task 321 ■ The encoder- decoder architecture and the attention mechanism 324 ■ Deep dive into the attention-based encoder-decoder model 327 10 Generative deep learning 334 10.1 Generating text with LSTM 335 Next-character predictor: A simple way to generate text 335 The LSTM-text-generation example 337 ■ Temperature: Adjustable randomness in the generated text 342 10.2 Variational autoencoders: Finding an efficient and structured vector representation of images 345 Classical autoencoder and VAE: Basic ideas 345 ■ A detailed example of VAE: The Fashion-MNIST example 349 10.3 Image generation with GANs 356 The basic idea behind GANs 357 ■ The building blocks of ACGAN 360 ■ Diving deeper into the training of ACGAN 363 Seeing the MNIST ACGAN training and generation 366 11 Basics of deep reinforcement learning 371 11.1 The formulation of reinforcement-learning problems 373 11.2 Policy networks and policy gradients: The cart-pole example 376 Cart-pole as a reinforcement-learning problem 376 ■ Policy network 378 ■ Training the policy network: The REINFORCE algorithm 381 11.3 Value networks and Q-learning: The snake game example 389 Snake as a reinforcement-learning problem 389 ■ Markov decision process and Q-values 392 ■ Deep Q-network 396 ■ Training the deep Q-network 399 PART 4 SUMMARY AND CLOSING WORDS .....................415 12 Testing, optimizing, and deploying models 417 12.1 Testing TensorFlow.js models 418 Traditional unit testing 419 ■ Testing with golden values 422 Considerations around continuous training 424

📄 Page 14

CONTENTSxii12.2 Model optimization 425 Model-size optimization through post-training weight quantization 426 ■ Inference-speed optimization using GraphModel conversion 434 12.3 Deploying TensorFlow.js models on various platforms and environments 439 Additional considerations when deploying to the web 439 Deployment to cloud serving 440 ■ Deploying to a browser extension, like Chrome Extension 441 ■ Deploying TensorFlow.js models in JavaScript-based mobile applications 443 ■ Deploying TensorFlow.js models in JavaScript-based cross-platform desktop applications 445 ■ Deploying TensorFlow.js models on WeChat and other JavaScript-based mobile app plugin systems 447 Deploying TensorFlow.js models on single-board computers 448 Summary of deployments 450 13 Summary, conclusions, and beyond 453 13.1 Key concepts in review 454 Various approaches to AI 454 ■ What makes deep learning stand out among the subfields of machine learning 455 ■ How to think about deep learning at a high level 455 ■ Key enabling technologies of deep learning 456 ■ Applications and opportunities unlocked by deep learning in JavaScript 457 13.2 Quick overview of the deep-learning workflow and algorithms in TensorFlow.js 458 The universal workflow of supervised deep learning 458 Reviewing model and layer types in TensorFlow.js: A quick reference 460 ■ Using pretrained models from TensorFlow.js 465 The space of possibilities 468 ■ Limitations of deep learning 470 13.3 Trends in deep learning 473 13.4 Pointers for further exploration 474 Practice real-world machine-learning problems on Kaggle 474 Read about the latest developments on arXiv 475 ■ Explore the TensorFlow.js Ecosystem 475 appendix A Installing tfjs-node-gpu and its dependencies 477 appendix B A quick tutorial of tensors and operations in TensorFlow.js 482 glossary 507 index 519

📄 Page 15

foreword When we started TensorFlow.js (TF.js), formerly called deeplearn.js, machine learning (ML) was done mostly in Python. As both JavaScript developers and ML practitioners on the Google Brain team, we quickly realized that there was an opportunity to bridge the two worlds. Today, TF.js has empowered a new set of developers from the extensive JavaScript community to build and deploy ML models and enabled new classes of on- device computation. TF.js would not exist in its form today without Shanqing, Stan, and Eric. Their con- tributions to TensorFlow Python, including the TensorFlow Debugger, eager execu- tion, and build and test infrastructure, uniquely positioned them to tie the Python and JavaScript worlds together. Early on in the development, their team realized the need for a library on top of deeplearn.js that would provide high-level building blocks to develop ML models. Shanqing, Stan, and Eric, among others, built TF.js Layers, allowing conversion of Keras models to JavaScript, which dramatically increased the wealth of available models in the TF.js ecosystem. When TF.js Layers was ready, we released TF.js to the world. To investigate the motivations, hurdles, and desires of software developers, Carrie Cai and Philip Guo deployed a survey to the TF.js website. This book is in direct response to the study’s summary: “Our analysis found that developers’ desires for ML frameworks extended beyond simply wanting help with APIs: more fundamentally, they desired guidance on understanding and applying the conceptual underpinnings of ML itself.”1 Deep Learning with JavaScript contains a mix of deep learning theory as well as real- world examples in JavaScript with TF.js. It is a great resource for JavaScript developers 1 C. Cai and P. Guo, (2019) “Software Developers Learning Machine Learning: Motivations, Hurdles, and Desires,” IEEE Symposium on Visual Languages and Human-Centric Computing, 2019.xiii

📄 Page 16

FOREWORDxivwith no ML experience or formal math background, as well as ML practitioners who would like to extend their work into the JavaScript ecosystem. This book follows the template of Deep Learning with Python, one of the most popular applied-ML texts, writ- ten by the Keras creator, François Chollet. Expanding on Chollet’s work, Deep Learning with JavaScript does an amazing job building on the unique things that JavaScript has to offer: interactivity, portability, and on-device computation. It covers core ML con- cepts, but does not shy away from state-of-the-art ML topics, such as text translation, generative models, and reinforcement learning. It even gives pragmatic advice on deploying ML models into real-world applications written by practitioners who have extensive experience deploying ML to the real world. The examples in this book are backed by interactive demos that demonstrate the unique advantages of the JavaScript ecosystem. All the code is open-sourced, so you can interact with it and fork it online. This book should serve as the authoritative source for readers who want to learn ML and use JavaScript as their main language. Sitting at the forefront of ML and JavaScript, we hope you find the concepts in this book useful and the journey in Java- Script ML a fruitful and exciting one. —NIKHIL THORAT AND DANIEL SMILKOV, inventors of deeplearn.js and technical leads of TensorFlow.js

📄 Page 17

preface The most significant event in the recent history of technology is perhaps the explo- sion in the power of neural networks since 2012. This was when the growth in labeled datasets, increases in computation power, and innovations in algorithms came together and reached a critical mass. Since then, deep neural networks have made previously unachievable tasks achievable and boosted the accuracies in other tasks, pushing them beyond academic research and into practical applications in domains such as speech recognition, image labeling, generative models, and recommendation systems, just to name a few. It was against this backdrop that our team at Google Brain started developing TensorFlow.js. When the project started, many regarded “deep learning in JavaScript” as a novelty, perhaps a gimmick, fun for certain use cases, but not to be pursued with seriousness. While Python already had several well-established and powerful frame- works for deep learning, the JavaScript machine-learning landscape remained splin- tered and incomplete. Of the handful of JavaScript libraries available back then, most only supported deploying models pretrained in other languages (usually in Python). For the few that supported building and training models from scratch, the scope of supported model types was limited. Considering JavaScript’s popular status and its ubiquity that straddles client and server sides, this was a strange situation. TensorFlow.js is the first full-fledged industry-quality library for doing neural net- works in JavaScript. The range of capabilities it provides spans multiple dimensions. First, it supports a wide range of neural-networks layers, suitable for various data types ranging from numeric to text, from audio to images. Second, it provides APIs for load- ing pretrained models for inference, fine-tuning pretrained models, and building and training models from scratch. Third, it provides both a high-level, Keras-like API for practitioners who opt to use well-established layer types, and a low-level, TensorFlow- like API for those who wish to implement more novel algorithms. Finally, it is designedxv

📄 Page 18

PREFACExvito be runnable in a wide selection of environments and hardware types, including the web browser, server side (Node.js), mobile (e.g., React Native and WeChat), and desk- top (electron). Adding to the multidimensional capability of TensorFlow.js is its status as a first-class integrated part of the larger TensorFlow/Keras ecosystem, specifically its API consistency and two-way model-format compatibility with the Python libraries. The book you have in your hands will guide your grand tour through this multi- dimensional space of capabilities. We’ve chosen a path that primarily cuts through the first dimension (modeling tasks), enriched by excursions along the remaining dimen- sions. We start from the relatively simpler task of predicting numbers from numbers (regression) to the more complex ones such as predicting classes from images and sequences, ending our trip on the fascinating topics of using neural networks to gen- erate new images and training agents to make decisions (reinforcement learning). We wrote the book not just as a recipe for how to write code in TensorFlow.js, but as an introductory course in the foundations of machine learning in the native lan- guage of JavaScript and web developers. The field of deep learning is a fast-evolving one. It is our belief that a firm understanding of machine learning is possible without formal mathematical treatment, and this understanding will enable you to keep your- self up-to-date in future evolution of the techniques. With this book you’ve made the first step in becoming a member of the growing community of JavaScript machine-learning practitioners, who’ve already brought about many impactful applications at the intersection between JavaScript and deep learning. It is our sincere hope that this book will kindle your own creativity and inge- nuity in this space. SHANQING CAI, STAN BILESCHI, AND ERIC NIELSEN September 2019 Cambridge, MA

📄 Page 19

acknowledgments This book owes Deep Learning with Python by François Chollet for its overall structure. Despite the fact that the code was rewritten in a different language and much new content was added for the JavaScript ecosystem and to reflect new developments in the field, neither this book nor the entire high-level API of TensorFlow.js would have been a reality without pioneer work on Keras led by François. Our journey to the completion of this book and all the related code was made pleasant and fulfilling thanks to the incredible support from our colleagues on Goo- gle’s TensorFlow.js Team. The seminal and foundational work by Daniel Smilkov and Nikhil Thorat on the low-level WebGL kernels and backpropagation forms a rock- solid foundation for model building and training. The work by Nick Kreeger on the Node.js binding to TensorFlow’s C library is the main reason why we can run neural networks in the browser and Node.js with the same code. The TensorFlow.js data API by David Soergel and Kangyi Zhang makes chapter 6 of the book possible, while chap- ter 7 was enabled by the visualization work by Yannick Assogba. The performance optimization techniques described in chapter 11 wouldn’t be possible without Ping Yu’s work on op-level interface with TensorFlow. The speed of our examples wouldn’t be nearly as fast as it is today without the focused performance optimization work by Ann Yuan. The leadership of Sarah Sirajuddin, Sandeep Gupta, and Brijesh Krishnas- wami is critical to the overall long-term success of the TensorFlow.js project. We would have fallen off the track without the support and encouragement of D. Sculley, who carefully reviewed all the chapters of the book. We’re also immensely grateful for all the encouragement we received from Fernanda Viegas, Martin Watten- berg, Hal Abelson, and many other colleagues of ours at Google. Our writing and content were greatly improved as a result of the detailed review by François Chollet,xvii

📄 Page 20

ACKNOWLEDGMENTSxviiiNikhil Thorat, Daniel Smilkov, Jamie Smith, Brian K. Lee, and Augustus Odena, as well as by in-depth discussion with Suharsh Sivakumar. One of the unique pleasures of working on a project such as TensorFlow.js is the opportunity to work alongside and interact with the worldwide open-source software community. TensorFlow.js was fortunate to have a group of talented and driven con- tributors including Manraj Singh, Kai Sasaki, Josh Gartman, Sasha Illarionov, David Sanders, syt123450@, and many many others, whose tireless work on the library expanded its capability and improved its quality. Manraj Singh also contributed the phishing-detection example used in chapter 3 of the book. We are grateful to our editorial team at Manning Publications. The dedicated and tireless work by Brian Sawyer, Jennifer Stout, Rebecca Rinehart, and Mehmed Pasic, and many others made it possible for we authors to focus on writing the content. Marc-Philip Huget provided extensive and incisive technical review throughout the development process. Special thanks go to our reviewers, Alain Lompo, Andreas Refs- gaard, Buu Nguyen, David DiMaria, Edin Kapic, Edwin Kwok, Eoghan O’Donnell, Evan Wallace, George thomas, Giuliano Bertoti, Jason Hales, Marcio Nicolau, Michael Wall, Paulo Nuin, Pietro Maffi, Polina Keselman, Prabhuti Prakash, Ryan Burrows, Satej Sahu, Suresh Rangarajulu, Ursin Stauss, and Vaijanath Rao, whose suggestions helped make this a better book. We thank our MEAP readers for catching and pointing out quite a few typographi- cal and technical errors. Finally, none of this would be possible without the tremendous understanding and sacrifice on the part of our families. Shanqing Cai would like to express the deepest gratitude to his wife, Wei, as well as his parents and parents-in-law for their help and support during this book’s year-long writing process. Stan Bileschi would like to thank his mother and father, as well as his step-mother and step-father, for providing a foun- dation and direction to build a successful career in science and engineering. He would also like to thank his wife, Constance, for her love and support. Eric Nielsen would like to say to his friends and family, thank you.