Deep Learning on Microcontrollers Learn how to develop embedded AI applications using TinyML (Atul Krishna Gupta etc.) (z-library.sk, 1lib.sk, z-lib.sk)

 i Deep Learning on Microcontrollers Learn how to develop embedded AI applications using TinyML Atul Krishna Gupta Dr. Siva Prasad Nandyala www.bpbonline.com

ii  Copyright © 2023 BPB Online All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor BPB Online or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. BPB Online has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, BPB Online cannot guarantee the accuracy of this information. First published: 2023 Published by BPB Online WeWork 119 Marylebone Road London NW1 5PU UK | UAE | INDIA | SINGAPORE ISBN 978-93-55518-057 www.bpbonline.com

 iii Dedicated to My Parents Mr. Keshava Kumar Gupta and Late. Smt. Lakshmi Devi Gupta and to my wife Richa Gupta and to my children Ananya Gupta and Avi Gupta — Atul Krishna Gupta My Parents Late Mr. Koti Nagaiah Nandyala and Smt. Durgamba Nandyala and to my brother Sambasiva Rao Nandyala and to my wife Sandya Kemisetti — Dr. Sivaprasad Nandyala

iv  About the Authors l Atul Krishna Gupta has held many positions as Research & Development Executive in companies such as Syntiant, Macom, Inphi (now Marvell) and Gennum (now Semtech). He has over 25 years of experience in delivering all aspects of systems from IC design to software support. He has made contributions to various forums such as IEEE, SMPTE and OIF. Two technical Emmy Awards were granted to two companies for the technical work he led in the past. He was awarded with the Employee of the year award and Excellence in R&D award at Gennum. Atul holds over 20 patents. Currently, his research interests are in the field of Battery Management Systems (BMS) where he is finding ways to use AI to make Electrical Vehicles (EV) safer and last longer. He has received his B.Tech degree in Electrical Engineering from Indian Institute of Technology, Kanpur, India and MS degree in Electrical and Computer Engineering from University of Calgary, Canada. l Dr. Sivaprasad Nandyala worked in Eaton Research Labs as Lead Engineer (Data Science) at Eaton India Innovation Center, Pune, India. Prior to Eaton, he worked in companies like Tata Elxsi, Wipro Technologies, Analog Devices & Ikanos Communications in multiple technology areas. Dr. Nandyala has over 35+ research publications, 1 patent grant and 6 patents under review. He obtained his Ph.D. in Speech Processing from NIT Warangal, India. He was an ERASMUS MUNDUS scholarship holder from the European government for his Postdoctoral Research at Politecnico di Milano (POLIMI), Italy.

 v About the Reviewer Dr. Sanjay Boddhu is an experienced Research and Engineering leader with expertise in leading and mentoring geographically distributed teams, in the domains of Computer Vision, Image Processing, Natural Language Processing, Predictive Analytics, and Modelling. He is skilled in using various Machine Learning Ops approaches to design and develop real-world applications in Cloud and at Edge.

vi  Acknowledgements m Atul Krishna Gupta: I am incredibly grateful to Kurt Busch for providing me the opportunity to learn and contribute to the emerging field of Artificial Intelligence (AI). I would like to thank Dr. Jeremy Holleman for their insightful conversation over various topics related to neural networks. I would also like to thank Dr. Stephen Bailey for help in the firmware of the TinyML board. I would not have been able to showcase the TinyML board without guidance from Mallik Maturi and Poupak Khodabandeh. I would like to offer a special thanks to Zack Shelby and Aurelien Lequertier for making available their Machine Learning (ML) platform to the developer community free of cost. The platform enables zero code deployment of production grade AI deployment. I would like to thank my wife and children for their patience and support to finish the book. m Dr. Sivaprasad Nandyala: The writing of a book is never a solo effort, and this “Deep Learning on Microcontrollers” book is no exception. Before anything else, I want to express my deepest gratitude to Mr. Atul Krishna Gupta who had faith in me from the beginning in writing this book. In addition, I would like to thank Dr. Sanjay Boddhu for his feedback and suggestions in reviewing the book. I want to thank the TinyML community for all the great things they have done for the field. My heartfelt thanks go to my family and friends for their unwavering support and understanding during this hard journey.

 vii Preface As the title of the book suggests, this book is intended to enable readers from different backgrounds to make a tangible AI application, which can be deployed on the edge on off-the-shelf platforms such as Arduino or TinyML board. The focus of this book is on the practical aspects of AI deployment. The journey of AI deployment from demo quality to production grade is not easy. We have taken a realistic example to show the pitfalls and given ideas on how to overcome the roadblocks. While the focus of the book is on the practical side, the book also provides a good academic background as well. The field of AI is evolving and it is not practical to have one comprehensive book on all the topics, but we have given insight into some of the advanced topics of the AI field. Deployment of AI on the edge will require some hardware. For cost effective deployment, it is expected that companies will develop their unique hardware. However, for getting started, there are several hardware boards available from websites such as Digikey or Amazon. Readers can buy this type of hardware in the range of $35-$100. This book is divided into 9 chapters. Each chapter description is listed as follows. Chapter 1: Introduction to AI – will show a continuum of traditional code-based solution and Artificial Intelligence based solution. It will show where an AI based solution will be suitable and how to approach the solution. Chapter 2: Traditional ML Lifecycle – will cover how machine learning is different from classical methods, introduction to traditional ML life cycle, performance metrics, and the basics of deep learning (DL) and different DL algorithms. It also covers transfer learning. We will discuss several tools, libraries, and frameworks for developing and deploying ML models on various embedded devices and microcontrollers. We also cover the differences between learning and inference, ML model deployment and inferencing on different hardware platforms and their comparison at various deployment levels.

viii  Chapter 3: TinyML Hardware and Software Platforms – will cover CPUs, GPUs, Raspberry Pi boards, TPUs, and Data Center Servers. We will also look at TinyML compatible microcontrollers and Raspberry Pi boards. We then focus on TinyML's hardware boards and software platforms for machine learning. We will discuss important software platforms, data engineering, and model compression frameworks. Chapter 4: End-to-End TinyML Deployment Phases – will discuss embedded machine learning's (EML) basics, characteristics, and examples. Next, we will also explore EML's building blocks, pros and cons, and how to run an ML model on microcontrollers. We will discuss Edge Impulse and Arduino IDE platforms, their pros and cons, and how to use different hardware boards with them. Data collection from sensors and the different platforms will be covered. We will cover data engineering, model training with Edge Impulse, optimization, and inferencing for model deployment on TinyML hardware platforms. Chapter 5: Real World Use Cases – will cover various use cases of the TinyML deployment. The chapter categorizes these deployments in seven categories. However, many applications overlap multiple categories. These applications just show the tip of the iceberg because we just got started. Over the next few decades, we are expecting an explosion of TinyML deployment. These examples are provided just to ignite the creativity of the reader, so that they can lead innovation and deploy AI solutions which do not exist today. Chapter 6: Practical Experiments with TinyML – will utilize Arduino IDE for TinyML hardware experimentation. We will collect sensor data using the TinyML board, clean the data for the practical experiment (Air Gesture Digit Recognition), upload it to the Edge Impulse platform, train and test the model with Nano RP2040 board sensor data. Finally, we will download the Edge Impulse inference model and test it on the RP2040 using Arduino IDE to evaluate performance. Chapter 7: Advance Implementation with TinyML Board – will deep dive on specific hardware accelerator chips, which provide AI specific computation at a fraction of cost and power relative to microcontroller-based architecture. The development boards are readily available on these hardware accelerator chips where readers can deploy an AI solution. The power of the entire solution can run from batteries months to years. The chapter describes the entire flow of deployment in a few easy steps on the readily available Edge Impulse software platform.

 ix Chapter 8: Continuous Improvement – will cover topics in improving the accuracy of the AI solution. AI is a data driven flow where the accuracy depends on the data. This chapter takes a deeper dive into a keyword detection application to demonstrate how to curate the data and improve the performance to take the solution from demo to production quality. Chapter 9: Conclusion – will provide the conclusion of various aspects learned in the earlier chapters. This is an introduction book on AI and there are many topics which will require many more books. Some of those topics are mentioned in this chapter to ensure that the reader knows there is more to AI than what is covered in the book.

x  Code Bundle and Coloured Images Please follow the link to download the Code Bundle and the Coloured Images of the book: https://rebrand.ly/yt0v6ae The code bundle for the book is also hosted on GitHub at https://github.com/bpbpublications/Deep-Learning-on-Microcontrollers. In case there's an update to the code, it will be updated on the existing GitHub repository. We have code bundles from our rich catalogue of books and videos available at https://github.com/bpbpublications. Check them out! Errata We take immense pride in our work at BPB Publications and follow best practices to ensure the accuracy of our content to provide with an indulging reading experience to our subscribers. Our readers are our mirrors, and we use their inputs to reflect and improve upon human errors, if any, that may have occurred during the publishing processes involved. To let us maintain the quality and help us reach out to any readers who might be having difficulties due to any unforeseen errors, please write to us at : errata@bpbonline.com Your support, suggestions and feedbacks are highly appreciated by the BPB Publications’ Family. Did you know that BPB offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.bpbonline.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at : business@bpbonline.com for more details. At www.bpbonline.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on BPB books and eBooks.

 xi Piracy If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at business@bpbonline.com with a link to the material. If you are interested in becoming an author If there is a topic that you have expertise in, and you are interested in either writing or contributing to a book, please visit www.bpbonline.com. We have worked with thousands of developers and tech professionals, just like you, to help them share their insights with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea. Reviews Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions. We at BPB can understand what you think about our products, and our authors can see your feedback on their book. Thank you! For more information about BPB, please visit www.bpbonline.com. Join our book's Discord space Join the book's Discord Workspace for Latest updates, Offers, Tech happenings around the world, New Release and Sessions with the Authors: https://discord.bpbonline.com

xii  Table of Contents 1. Introduction to AI........................................................................................................ 1 Introduction ............................................................................................................ 1 Structure .................................................................................................................. 2 Objectives ................................................................................................................ 2 Artificial Intelligence ............................................................................................. 3 Continuum of code writing and artificial intelligence ..................................... 3 Exercise .............................................................................................................. 3 Changing the paradigm ........................................................................................ 5 Neural Network ..................................................................................................... 7 Machine Learning ................................................................................................ 11 Intelligent IoT System vs. Cloud based IoT system ........................................ 13 Arduino Nano 33 BLE Sense board ................................................................. 14 Limited compute resources ............................................................................... 15 Battery power limits .............................................................................................. 15 TinyML and Nicla Voice board ........................................................................ 16 >10x parameters .................................................................................................... 18 >200x Power advantage ........................................................................................ 18 >20x Throughput .................................................................................................. 18 TinyML Ecosystem .............................................................................................. 19 Key applications for Intelligent IoT systems ................................................... 19 Smart agriculture ............................................................................................ 20 Smart appliances .............................................................................................. 20 Smart cities ...................................................................................................... 20 Smart health ..................................................................................................... 21 Smart homes .................................................................................................... 21 Smart industry ................................................................................................ 21 Conclusion ............................................................................................................ 21 Key facts ................................................................................................................ 22 Questions .............................................................................................................. 22

 xiii References ............................................................................................................. 23 2. Traditional ML Lifecycle ......................................................................................... 25 Introduction .......................................................................................................... 25 Structure ................................................................................................................ 26 Objectives .............................................................................................................. 26 Traditional methods ............................................................................................ 26 Machine learning landscape .............................................................................. 27 Supervised learning ......................................................................................... 29 Unsupervised learning .................................................................................... 29 Reinforcement Learning (RL) .......................................................................... 30 ML Performance Metrics .................................................................................... 30 Confusion matrix ............................................................................................. 30 Basics of DL and different DL algorithms ........................................................ 32 Transfer Learning ................................................................................................. 35 Tools and Different ML, DL frameworks ........................................................ 35 Python .............................................................................................................. 36 Jupyter Notebooks ............................................................................................ 36 Google Colaboratory ........................................................................................ 36 TensorFlow (TF), TFLite and TensorFlow Lite Micro ..................................... 36 TensorFlow Lite ................................................................................................ 37 TensorFlow Lite Micro ..................................................................................... 38 AI Model Efficiency Toolkit (AIMET) ............................................................. 38 Convolutional Architecture for Fast Feature Embedding (Caffe) ................... 39 CoreML ............................................................................................................ 40 Open Neural Network Exchange (ONNX) ..................................................... 41 Open Visual Inference and Neural network Optimization (OpenVINO) ...... 41 Pytorch and PyTorch Mobile ........................................................................... 42 Embedded Machine Learning (EML) ............................................................... 42 Difference between Learning and Inference .................................................... 43 ML model deployment and inferencing on different platforms ................... 44 Conclusion ............................................................................................................ 46 Key facts ................................................................................................................ 47

xiv  Questions .............................................................................................................. 47 References ............................................................................................................. 48 3. TinyML Hardware and Software Platforms ......................................................... 51 Introduction .......................................................................................................... 51 Structure ................................................................................................................ 52 Objectives .............................................................................................................. 52 Servers at Data Centers: CPUs, GPUs and TPUs ............................................ 52 Mobile CPU, Raspberry Pi board and its types ............................................... 53 Microcontrollers and Microcontroller with AI accelerator ............................ 55 TinyML Hardware Boards .................................................................................. 57 Arduino and Arduino Nano 33 BLE ............................................................... 59 Arduino Nicla Sense ME ................................................................................. 60 Adafruit Feather .............................................................................................. 61 SparkFun Edge ................................................................................................ 62 NVIDIA Jetson Nano ...................................................................................... 62 Google Coral Edge TPU .................................................................................. 63 Qualcomm QCS605 ......................................................................................... 64 NXP i.MX 8M ................................................................................................ 65 STMicroelectronics STM32L4 ........................................................................ 66 Intel Curie ........................................................................................................ 67 Syntiant TinyML ............................................................................................. 68 TinyML Software Suites ...................................................................................... 69 TensorFlow Lite Micro (Google) ...................................................................... 70 uTensor (ARM) ................................................................................................ 71 Arduino Create ................................................................................................ 72 EloquentML ..................................................................................................... 72 EdgeML (Microsoft) ........................................................................................ 72 EON Compiler (Edge Impulse) ....................................................................... 73 STM32Cube.AI and NanoEdge AI Studio (STMicroelectronics) ................... 73 PYNQ .............................................................................................................. 74 OpenMV .......................................................................................................... 76 SensiML ........................................................................................................... 76

 xv Neuton TinyML ............................................................................................... 77 Metavision Intelligence Suite 3.0 (Vision applications) .................................. 78 Data Engineering Frameworks .......................................................................... 78 Edge Impulse .................................................................................................... 78 SensiML ........................................................................................................... 79 Qeexo AutoML ................................................................................................ 81 TinyML Model Compression Frameworks ...................................................... 81 Quantization .................................................................................................... 82 Pruning ............................................................................................................ 83 Low ranked approximation .............................................................................. 83 Knowledge distillation ..................................................................................... 83 TensorFlow Lite ..................................................................................................... 84 STM32 X-CUBE-AI ............................................................................................. 85 QKeras ................................................................................................................... 86 Qualcomm AIMET ............................................................................................... 86 Microsoft NNI ....................................................................................................... 87 CMix-NN .............................................................................................................. 89 OmniML................................................................................................................ 89 Conclusion ............................................................................................................ 90 Key facts ................................................................................................................ 90 Questions .............................................................................................................. 91 References ............................................................................................................. 92 4. End-to-End TinyML Deployment Phases ............................................................. 93 Introduction .......................................................................................................... 93 Structure ................................................................................................................ 94 Objectives .............................................................................................................. 95 Understanding Embedded ML .......................................................................... 95 Introduction to Edge-impulse and Arduino IDE ............................................ 99 Edge-impulse ................................................................................................... 99 Arduino Integrated Development Environment (IDE) ................................. 109 Arduino Driver Installation .................................................................................111 Data collection from multiple sensors ............................................................ 115

xvi  Data collection from an Arduino board ......................................................... 116 Data collection from Syntiant board.............................................................. 117 Data engineering steps for TinyML................................................................. 117 Cleaning ......................................................................................................... 118 Organizing ..................................................................................................... 119 Transformation ............................................................................................... 119 Model Training in TinyML software platforms ............................................. 120 EON Compiler (Edge Impulse) ..................................................................... 120 Model Compression .......................................................................................... 122 Pruning .......................................................................................................... 122 Knowledge distillation ................................................................................... 123 Model conversion .............................................................................................. 124 Quantization .................................................................................................. 124 Inferencing/Prediction of results with test data ........................................... 126 Model Deployment in TinyML Hardware board........................................... 128 Conclusion .......................................................................................................... 130 Key facts .............................................................................................................. 130 Questions ............................................................................................................ 131 References ........................................................................................................... 132 5. Real World Use Cases ............................................................................................. 133 Introduction ........................................................................................................ 133 Structure .............................................................................................................. 133 Objectives ............................................................................................................ 135 Smart agriculture ............................................................................................... 135 Agriculture video analytics ........................................................................... 135 Crop intruder detection ................................................................................. 136 Crop yield prediction and improvement ........................................................ 136 Agribots ......................................................................................................... 136 Insect detection and pesticide reduction ........................................................ 137 Weedicides elimination .................................................................................. 137 Acoustic insect detection ............................................................................... 138 Animal husbandry ......................................................................................... 139

 xvii Smart appliances ................................................................................................ 139 Vision AI for appliances ................................................................................. 139 Audio AI for appliances ................................................................................. 141 Sensors based AI for appliances ..................................................................... 142 Smart cities .......................................................................................................... 142 Safe and secure city ........................................................................................ 142 City maintenance ........................................................................................... 143 Parking enforcement systems ........................................................................ 144 Traffic management ........................................................................................ 144 Maintaining bridges ...................................................................................... 144 Non-Smoking enforcement ............................................................................ 145 Smart health ........................................................................................................ 146 Cataract detection .......................................................................................... 146 Fall detection .................................................................................................. 147 Cough detection ............................................................................................. 148 Boxing Moves Detector ................................................................................. 148 Mosquito detection......................................................................................... 149 Snoring and sleep apnea detection ................................................................. 150 Smart home ......................................................................................................... 151 Person detection at the door ........................................................................... 151 Glassbreak detection ....................................................................................... 151 Smart baby monitoring .................................................................................. 152 Voice recognition for home automation ......................................................... 153 Smart industry ................................................................................................... 153 Railway track defect detection ....................................................................... 153 Telecom towers defect detection ..................................................................... 154 Defect detection in components ..................................................................... 155 Smart automotive .............................................................................................. 156 Drowsy driver alert........................................................................................ 156 Advance collision detection ........................................................................... 156 Conclusion .......................................................................................................... 157 Key facts .............................................................................................................. 157

xviii  Questions ............................................................................................................ 158 References ........................................................................................................... 158 6. Practical Experiments with TinyML .................................................................... 161 Introduction ........................................................................................................ 161 Structure .............................................................................................................. 162 Objectives ............................................................................................................ 162 Introduction to Nano RP2040 TinyML board ................................................ 163 Setting up Arduino IDE and testing the Nano RP2040 Board .................... 163 High level steps involved in the air gesture digit recognition in Edge Impulse platform ................................................................................... 165 Data collection for the air gesture digit recognition ..................................... 166 Loading the dataset in Edge Impulse Platform .............................................. 170 Setting up the development framework and design of neural network classifier ................................................................................ 174 Model training in Edge Impulse platform ..................................................... 177 Model testing with the collected data ............................................................. 183 Model deployment in Nano RP2040 board ................................................... 184 Inferencing/Prediction of results with RP2040 ............................................. 188 Conclusion .......................................................................................................... 192 Key facts .............................................................................................................. 193 Questions ............................................................................................................ 193 References ........................................................................................................... 193 7. Advance Implementation with TinyML Board ................................................. 195 Introduction ........................................................................................................ 195 Structure .............................................................................................................. 195 Objectives ............................................................................................................ 196 NDP101 Architecture ........................................................................................ 196 NDP120 Architecture ........................................................................................ 199 Practical implementation and deployment.................................................... 199 Creating a project ........................................................................................... 199 Uploading Data ............................................................................................. 201 Impulse Design .............................................................................................. 203

 xix Epochs Setting ..................................................................................................... 209 Learning rate setting ........................................................................................... 209 Validation data set setting ................................................................................... 209 Auto balance setting ............................................................................................ 210 Data augmentation .............................................................................................. 210 Neural network architecture ............................................................................... 211 Neural network training ..................................................................................... 212 Model testing ................................................................................................. 216 Deployment .................................................................................................... 217 Conclusion .......................................................................................................... 222 Key facts .............................................................................................................. 222 Questions ............................................................................................................ 222 References ........................................................................................................... 223 8. Continuous Improvement ..................................................................................... 225 Introduction ........................................................................................................ 225 Structure .............................................................................................................. 225 Objectives ............................................................................................................ 226 Expectation gap .................................................................................................. 226 Unique issues about audio application .......................................................... 226 Raw neural network output and softmax transformation ............................. 228 Handling anomalous behavior during target classifier testing .................. 229 Method 1: Running window averaging ........................................................ 230 Method 2: Enriching target classifier ............................................................ 231 Method 3: Enriching open set classifier ......................................................... 232 False Acceptance Rate testing .......................................................................... 238 Optimization of window size in running window averaging .................... 240 Phrase recognition constraints to improve system level performance ...... 246 FRR testing under noisy conditions ................................................................ 249 Improving FRR performance under noisy conditions ................................. 251 Data collection for continuous improvement ................................................ 255 Conclusion .......................................................................................................... 255 Key facts .............................................................................................................. 255

xx  Questions ............................................................................................................ 256 References ........................................................................................................... 256 9. Conclusion ................................................................................................................ 257 Introduction ........................................................................................................ 257 Structure .............................................................................................................. 257 Objectives ............................................................................................................ 258 Review of material covered in this book ........................................................ 258 Chapter 1 ........................................................................................................ 258 Chapter 2 ........................................................................................................ 258 Chapter 3 ........................................................................................................ 259 Chapter 4 ........................................................................................................ 259 Chapter 5 ........................................................................................................ 259 Chapter 6 ........................................................................................................ 259 Chapter 7 ........................................................................................................ 260 Chapter 8 ........................................................................................................ 260 Advanced topics ................................................................................................ 260 Different types of neural networks ................................................................ 261 Neural network optimization ......................................................................... 262 Zero-shot, One-shot or Few-shot learning..................................................... 263 Federated learning ......................................................................................... 264 Transfer learning ................................................................................................. 267 Tuning pretrained networks .......................................................................... 268 MLOps ........................................................................................................... 270 Key facts .............................................................................................................. 270 Questions ............................................................................................................ 270 References ........................................................................................................... 271 Index ...................................................................................................................273-280