Combining DataOps, MLOps and DevOps Outperform Analytics and Software Development with Expert Practices on Process… (Dr. Kalpesh Parikh, Amit Johri) (z-library.sk, 1lib.sk, z-lib.sk)

Combining DataOps, MLOps and DevOps Outperform Analytics and Software Development with Expert Practices on Process Optimization and Automation Dr. Kalpesh Parikh Amit Johri www.bpbonline.com

Copyright © 2022 BPB Online All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor BPB Online or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. BPB Online has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, BPB Online cannot guarantee the accuracy of this information. Group Product Manager: Marianne Conor Publishing Product Manager: Eva Brawn Senior Editor: Connell Content Development Editor: Melissa Monroe Technical Editor: Anne Stokes Copy Editor: Joe Austin Language Support Editor: Justin Baldwin Project Coordinator: Tyler Horan Proofreader: Khloe Styles Indexer: V. Krishnamurthy Production Designer: Malcolm D'Souza Marketing Coordinator: Kristen Kramer First published: June 2022 Published by BPB Online WeWork, 119 Marylebone Road London NW1 5PU UK | UAE | INDIA | SINGAPORE ISBN 978-93-55511-911 www.bpbonline.com

About the Authors Dr. Kalpesh Parikh, a Ph.D in Computer and Information Science offers his expert solutions for business through Product-Process-Technology - Domain-Project Management. He is in the business of a) Embedding Knowledge and Logic in Technology through ‘Think Tech-Know-Logic-ally’ b) Detailing, which is over looked and ignored by others and c) Separating out Black and White out of Grey d) Perfection, Excellence is only tolerated e) Achieving Better than Best through Calibrating Competency and f) Creating Knowledge Product and helping to create knowledge Product of the clients vision since the last 40 years. His core competence and work space is Enterprise Technology Solution through large scale Projects Implementation and Management, Development of Software Products for niche business vertical. He was invited to get enlisted in the world Marquis Who’s Who. He has developed enterprise products of sizable scale like for Construction ERP (600 Man Month), eLearning Product (1200+ Man Month), Consolidation of Financials (300 Man Month), Cloud Computing Product (2000+ Man Month. He has worked in the domains that include Restaurant, FinTech, Construction, Textile, Education, Jewelry, Retail, eLearning, Software, Media, Agro Produces. Institutionalization of Processes - Business, IT and People – through consulting and auditing (as a empanelled auditor with BVQI, BSI, CI-UK) at the local as well as International level by practicing 15 different international standards. He has audited Bank data centre, Large and Complex Product development companies for ISMS – Information Security Management System. He is Lead Appraiser for ISO 27001 (ISMS), ISO 20000(ITSM), ISO 9001 (QMS), SSE-CMM (ISO 21827). He has worked on a very special standard IEEE 12207 – Software Development standards adopted by Department of Defence. He has worked as ISMS consultant in an international project.

Currently, he is working on Cloud Interoperability, Blockchain Technology, Metaverse and Quantum Computing technologies. His book on Blockchain technology (Blockchain Quickstart Guide) is published by BPB. He has delivered various sessions on Blockchain and Metaverse Technology. AWS has carried out global case study on personal name of author for its cloud implementation at GTU. He has also worked on CIP Test Bed project of TSDSI and CCICI. His company also bagged rising star trophy from AWS as advanced consulting partner. He is currently working as panel member of LITD-14 data Panel for scripting Organization Data Maturity Model and ISO SC38/WG5 for cloud interoperability. He had an opportunity to work as expert at UGC, AICTE, INFLIBNET, AMC, GIL, iNDEXTb, CED, GTU and closely associated with couple of universities in different roles such as RDC Member, Advisor, Expert, M.Phil Guide, Ph.D Guide, Ph.D referee. He has also worked as the Director of an Engineering college and Pro-Vice Chancellor of a University. He was also BOA member of GTU’s Graduate School of Smart Cities Development. Some of the large scale project implementations include largest Media SAP Implementation as part of the national core team and RFID implementation as First person responsible for 20000 employees. Amit Johri had a brilliant academic career with qualifications in Business Administration, Commerce including Business Administration, Computer Management, Law and Educational Management. He is a Senior Life Member of the CSI, Member of the IEEE, USA, Life Member of IIPA, New Delhi and Member of BMA. A Systems professional by education, he ventured into Computer System and Software Education, Training and Consulting in 1985, as a career, starting his company CSC Knowledge Systems being pioneers in the field of Computer Education & Training. He has authored several books on Computing / Systems / IT / Management / Business Administration. He is a Professor, Mentor, Project Guide, Examiner for Diploma / Under Graduate / Graduate / Post Graduate programs in Engineering, Computer

Science, Computer Applications, IT, Bio Informatics, Management, Business Administration. His Technology Practicing, Research and Consulting profile includes Distributed Ledger Technology with Ethereum, Cloud Computing, Business Analysis, Business Intelligence, Competitive Intelligence, Business and Big Data Analytics, Application Development, RDBMSs, Software Engineering, Core Programming Languages. He has been Panellist, Chair, Speaker, Editorial Member, and Tutorial Presenter at various forums. He is an Academician and Industry Expert with over three decades of experience, 50+ book publications and a large number of students mentored by him.

About the Reviewers Gaurav Aroraa is a tech enthusiast and technical consultant with more than 23 years of experience in the industry. He has a Doctorate in Computer Science. Gaurav is a Microsoft MVP award recipient. He is a lifetime member of the Computer Society of India (CSI), an advisory member and senior mentor at IndiaMentor, certified as a Scrum trainer and coach, ITIL- F certified, and PRINCE-F and PRINCE-P certified. Gaurav is an open- source developer and a contributor to the Microsoft TechNet community. He has authored books across the technologies, including Microservices by Examples Using .NET Core (BPB Publications). Blog links: https://gaurav-arora.com/blog/ LinkedIn Profile: https://www.linkedin.com/in/aroragaurav/ Chitra Lele is a young software engineer, software solution architect, record-setting author, award-winning poet, and research scholar. Chitra runs her software startup firm, Chitra Lele & Associates, which designs software solutions based on Ethically-Aligned Design principles. She is also the founder of a social transformation initiative called Chitra Cares (chitracares.com) and it is dedicated to community building projects. Through her software projects, peace work, academic books, and social transformation initiatives, Chitra strives to contribute to the greater good of the world. Priyanka Dive is a Devops Architect Experienced Engineer with a demonstrated history of working in the information technology and services industry for more than seven years. She has a strong engineering professional skilled in Linux System Administration, Docker, Apache Kafka, Jenkins, Kubernetes, and Shell Scripting. She has worked as DevOps engineer implementing CI-CD using Jenkins on Kubernetes cluster. Hands-on experience with cloud services like AWS and Azure. She is co-author of the book "Devops for Salesforce" & currently working with ChordX as Devops Architect.

Acknowledgement First and Foremost, we are grateful to all Researchers and Industrial Developers world-wide for their contributions to various concepts and technologies concerning xOps practices – DevOps, DataOps, MLOps. The genesis of a book of this sort is to be found, in activities far earlier than the publisher’s invitation to write a book on it. It was our Academic – Industry interaction, the Innovation Council(s) at universities and the delivery of courses / workshops /seminars in xOps – DevOps, DataOps, MLOps to the Engineering / Technology /Management / Business Administration/ Computer Applications Students/Faculty Members (FDPs) for their skill development and industry readiness, which interested us in the field and with the work / research done by us over a period of time looking at the businesses world over, made us decide on writing a book on Combining DataOps, MLOps and DevOps. In the coming years, new technologies, global economics, and many other factors will present innumerable changes for business and society to navigate. Starting now, leaders need to be more flexible, responsive, and decisive than ever before. In order to survive, flourish, succeed, and win, businesses need to look ahead. It is this endeavor and objective with which we would like to serve our readers through this book. Thanks to all our Students / Faculty members of Engineering, Technology, Management, Business Administration, Computer Applications for providing us an equally innovative teaching - learning experience. We would like to thank our publishers, the BPB Publications team for the valuable support provided throughout the entire process. The ceaseless cooperation by the team is greatly acknowledged. They are wonderful to work with! The final tribute and appreciation, however is reserved for our families without whose support, co-operation, and understanding, this would not have been possible. Their inspiration and love has been instrumental in providing us the necessary support all through our professional careers.

We sincerely hope that this book on Combining DataOps, MLOPs and DevOps will meet the desired demands of all our valued readers.

Preface Successful projects require input, effort, insight, foresight, hindsight, and collaboration from people across the organization. This book on Combining DataOps, MLOps and DevOps is aimed primarily at managers and individual contributors in leadership roles who see friction within their organization and are looking for concrete, actionable steps they can take toward implementing or improving a xOps culture in their work environment. Individual contributors at all levels looking for practical suggestions for easing some of the pain points will find actionable takeaways. Our readers have a mix of professional roles, as xOps is a professional and cultural movement that stresses the iterative efforts to breakdown information silos, monitor relationships, and repair misunderstandings that arise between teams within an organization. The book covers a wide range of xOps practices, skills and theory, including an introduction to the foundational ideas and concepts. After reading this book, you will have a solid understanding of what having a xOps culture means practically for your organization, how to encourage effective collaboration such that individual contributors from different backgrounds and teams with different goals and working styles can work together productively, how to help teams collaborate to maximize their value while increasing employee satisfaction and balancing conflicting organizational goals, and how to choose tools and workflows that complement your organization. DevOps is a set of practices that combines software development (Dev) and Information Technology Operations (Ops) with the aim to shorten the system development life cycle and provide continuous delivery with high software quality. The intent is to combine agile software development practices with continuous deployment in order to have a constant flow of new functionality and resultant value delivery to customers. Also referred to as continuous deployment, new functionality can be rolled out whenever it

is ready, the effects measured and the feedback used to inform the next (rapid) cycle of development. DataOps is an automated, process-oriented methodology, used by analytic and data teams to improve the quality and reduce the cycle time of data analytics. Although this sounds very different from DevOps, in most product companies it’s tightly inter-connected with the products deployed in the field. Consequently, the data analytics is primarily focused on R & D teams that need to know if the intended outcomes of their development efforts are indeed accomplished as part of the continuous deployment pipeline. MLOps is a practice for collaboration and communication between Data Scientists and Operations professionals to help manage the production machine learning life cycle. Whereas traditionally, Data Scientists would develop a model based on a data set and then move on with their lives, currently in many systems, ML/DL models are constantly evolving due to changes to the data or new algorithmic insights and need to be deployed frequently as well. Once deployed, they need to be monitored to ensure that models that perform better in training also perform better during operations. “Ops” for all these stands for “operations” and the key is to remember that for any system, product or solution, there is only one operation function taking care of it. So, Dev, Data, and ML all have to integrate with the same Ops. Concluding, whatever “Ops” you are working on, it all has to come together in the same operation and consequently, you will need to work in cross-functional teams to ensure that you are reaching the desired outcomes. Over the EIGHT Chapters in this book, you will learn the following: Chapter 1 introduces you to Container: Containerization is the New Virtualization. To quickly and reliably run your application from one computing environment to another you require a software unit called Container that packages code and all its dependencies in order that your application runs. Chapter 2 discusses Docker with Containers for Developing and Deploying Software. Docker is an open platform for developing, shipping, and running applications. By taking the advantage of Docker’s methodologies for shipping, testing, and deploying code quickly, you can significantly reduce the delay between writing code and running it in production.

Chapter 3 describes DevOps to Build at Scale a Culture of Collaboration, Affinity, and Tooling. It is a software engineering culture that unifies the development and operations team, under an umbrella of tools to automate every stage of software development. It automates the service management for the support of operational objectives and improves understanding of the layers in the production environment stack for the development objectives. Chapter 4 introduces Docker Containers for Microservices Architecture Design. A microservice is an isolated, loosely-coupled unit of development that works on a single concern. The idea is that applications become easier to build and maintain when they are broken down into smaller, composable pieces which work together. Each component is continuously developed and separately maintained, and the application is then the sum of its constituent components. Chapter 5 introduces Kubernetes – The Cluster Manager for Container developed in Google Labs to manage containerized applications in different kind of environments such as physical, virtual, and cloud infrastructure. It is an open source system which helps in creating and managing containerization of application. Chapter 6 describes Data Engineering with DataOps, an automated, process-oriented methodology, used by analytic and data teams, to improve the quality and reduce the cycle time of data analytics. It focuses on continuous delivery by leveraging on-demand IT resources and by automating test and deployment of analytics. DataOps is the alignment of people, process, and technology to enable the rapid, automated, and secure management of data. Chapter 7 introduces you to MLOps: Engineering Machine Learning Operations. MLOps is the communication between Data Scientists and the Operations team. Deeply collaborative in nature, it is designed to eliminate waste, automate as much as possible, and produce richer, more consistent insights with machine learning, becoming a game changer for a business. Solving the combined concerns of the development, deployment, and operational phases of a machine learning application is MLOps. Chapter 8 discusses the xOps best practices. The DevOps best practices are described wherein the Developers and the Operations teams use DevOps as an approach to agile software development to build, test, deploy, and monitor applications with speed, quality, and control. DataOps best

practices are described where DataOps serves as the key that enables business enterprises to extract the maximum value from their data. MLOps best practices are described where MLOps is the process of operationalising data science and machine learning solutions using code and best practices that promote efficiency, speed, and robustness. The book caters to the need of application developers – front-end, back-end, end-to end (full stack) and students who want to build good proficiency in the entire process of application development.

Coloured Images Please follow the link to download the Coloured Images of the book: https://rebrand.ly/0f5a9c We have code bundles from our rich catalogue of books and videos available at https://github.com/bpbpublications. Check them out! Errata We take immense pride in our work at BPB Publications and follow best practices to ensure the accuracy of our content to provide with an indulging reading experience to our subscribers. Our readers are our mirrors, and we use their inputs to reflect and improve upon human errors, if any, that may have occurred during the publishing processes involved. To let us maintain the quality and help us reach out to any readers who might be having difficulties due to any unforeseen errors, please write to us at : errata@bpbonline.com Your support, suggestions and feedbacks are highly appreciated by the BPB Publications’ Family. Did you know that BPB offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.bpbonline.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at: business@bpbonline.com for more details. At www.bpbonline.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive

exclusive discounts and offers on BPB books and eBooks.

Piracy If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at business@bpbonline.com with a link to the material. If you are interested in becoming an author If there is a topic that you have expertise in, and you are interested in either writing or contributing to a book, please visit www.bpbonline.com. We have worked with thousands of developers and tech professionals, just like you, to help them share their insights with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea. Reviews Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions. We at BPB can understand what you think about our products, and our authors can see your feedback on their book. Thank you! For more information about BPB, please visit www.bpbonline.com.

Table of Contents 1. Container – Containerization is the New Virtualization Introduction Structure Objectives Introducing containers Working of containers Virtual machines and containers Hypervisor Hosted hypervisor Bare metal hypervisor Virtual machine Container OCI container Container runtime Container-as-a-Service Container host Container in J2EE Applications based on container Comparing container and virtual machine Benefits of container Docker and the rise of MicroServices Monolithic architecture Drawbacks “Microservices” come to the rescue! Technological evolution Docker Docker advantages Designing a microservices architecture with Docker containers Microservices: what are they? Building a microservice architecture – the challenges Microservices rescued by Docker Patterns to enable your architecture Moving to microservices

cgroups and namespaces in Linux Containerization and its benefits Conclusion Key terms Questions 2. Docker with Containers for Developing and Deploying Software Introduction Structure Objectives Docker and its use Is Docker a VM? The standards of a container and its industry leadership rancher Swarm Amazon ECS Build tool Engine Client Daemon File Image Container and container image—the differences The Union File System Volumes Containers in Docker Container implementation Namespaces Explaining Control Groups Docker application programming interface Docker’s purpose Client in Docker Remote API in Docker DevOps Docker Benefits of Docker How do these benefits benefit the users? What has Docker done in helping create this foundation?

Formation of OCI and evolution of Docker’s role Installation of Docker on Linux Docker version Docker support for Windows The ToolBox for Docker Docker support for Windows The ToolBox Setup for Docker Working with the Toolbox for Docker Hub Images Displaying the images Images downloading in Docker Removal of images The Docker inspect command Containers in Docker Running a container Container listing Working with containers in Docker Container lifecycle in Docker The architecture of Docker Conclusion Key terms Questions 3. DevOps to Build at Scale a Culture of Collaboration, Affinity, and Tooling Introduction Structure Objectives DevOps essentials Benefits of DevOps Introducing DevOps Development and operations, agile and DevOps, and waterfall model Dev and Ops Agile for DevOps Software development with waterfall model The challenges of the waterfall model