Node.js web development server-side development with Node 10 made easy (David Herron)（Z-Library）

Page 1

(This page has no text content)

Page 2

Node.js Web Development Fourth Edition David Herron BIRMINGHAM - MUMBAI

Page 3

Node.js Web Development Fourth Edition Copyright 2018 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Commissioning Editor: Amarabha Banerjee Acquisition Editor: Larissa Pinto Content Development Editor: Gauri Pradhan Technical Editor: Leena Patil Copy Editor: Safis Editing Project Coordinator: Sheejal Shah Proofreader: Safis Editing Indexer: Mariammal Chettiyar Graphics: Jason Monteiro Production Coordinator: Shraddha Falebhai First published: August 2011 Second edition: July 2013 Third edition: June 2016 Fourth edition: May 2018 Production reference: 1240518 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78862-685-9

Page 4

To my mother, Herron, and to the memory of my father, James, since I would not exist without them To my partner Maggie for being my loving partner throughout our joint life-journey

Page 5

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website. Why subscribe? Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals Improve your learning with Skill Plans built especially for you Get a free eBook or video every month Mapt is fully searchable Copy and paste, print, and bookmark content PacktPub.com Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details. At , you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

Page 6

Contributors About the author David Herron is a software engineer in Silicon Valley, working on projects from an X.400 e- mail server to assist launching the OpenJDK project, to Yahoo's Node.js application-hosting platform, and a solar array performance monitoring service. David writes about electric vehicles, green technology on The Long Tail Pipe website, and about other topics, including Node.js, on TechSparx website. Using Node.js, he developed the AkashaCMS static website generator. I wish to thank my mother, Evelyn, for everything; my father, Jim; my sister, Patti; my brother, Ken; my partner Maggie for being there and encouraging me, and the many years we expect to have with each other. I wish to thank Dr. Kubota of the University of Kentucky for believing in me, giving me my first computing job, and overseeing 6 years of learning the art of computer system maintenance. I am grateful to Ryan Dahl, the creator of Node.js, and the current Node.js core team members. Some platforms are just plain hard to work with, but not Node.js.

Page 7

About the reviewer Nicholas Duffy has had a wide-ranging career, holding positions from analyst to business intelligence architect, to software engineer, and even golf professional. He has a passion for all things data and software engineering, specializing in cloud architecture, Python, and Node.js. He is a frequent contributor to open source projects and is also a lifelong New York Mets fan. I'd like to thank my wife, Anne, and our boys, Jack and Chuck, for their never ending- support in whatever endeavor I pursue. Packt is searching for authors like you If you're interested in becoming an author for Packt, please visit and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

Page 8

Table of Contents Preface 1 Chapter 1: About Node.js 7 The capabilities of Node.js 9 Server-side JavaScript 10 Why should you use Node.js? 11 Popularity 11 JavaScript at all levels of the stack 12 Leveraging Google's investment in V8 12 Leaner, asynchronous, event-driven model 13 Microservice architecture 13 Node.js is stronger for having survived a major schism and hostile fork 13 Threaded versus event-driven architecture 14 Performance and utilization 17 Is Node.js a cancerous scalability disaster? 19 Server utilization, the business bottom line, and green web hosting 21 Embracing advances in the JavaScript language 22 Deploying ES2015/2016/2017/2018 JavaScript code 24 Node.js, the microservice architecture, and easily testable systems 25 Node.js and the Twelve-Factor app model 26 Summary 26 Chapter 2: Setting up Node.js 27 System requirements 27 Installing Node.js using package managers 28 Installing on macOS with MacPorts 28 Installing on macOS with Homebrew 29 Installing on Linux, *BSD, or Windows from package management systems 30 Installing Node.js in the Windows Subsystem for Linux (WSL) 30 Opening an administrator-privileged PowerShell on Windows 31 Installing the Node.js distribution from nodejs.org 31 Installing from source on POSIX-like systems 33 Installing prerequisites 33 Installing developer tools on macOS 34 Installing from source for all POSIX-like systems 34 Installing from source on Windows 36 Installing multiple Node.js instances with nvm 37 Installing nvm on Windows 39 Native code modules and node-gyp 39 Node.js versions policy and what to use 41

Page 9

Table of Contents [ ii ] Editors and debuggers 42 Running and testing commands 42 Node.js's command-line tools 43 Running a simple script with Node.js 44 Conversion to async functions and the Promise paradigm 47 Launching a server with Node.js 48 NPM – the Node.js package manager 49 Node.js, ECMAScript 2015/2016/2017, and beyond 51 Using Babel to use experimental JavaScript features 54 Summary 58 Chapter 3: Node.js Modules 59 Defining a module 59 CommonJS and ES2015 module formats 61 CommonJS/Node.js module format 62 ES6 module format 63 JSON modules 67 Supporting ES6 modules on older Node.js versions 68 Demonstrating module-level encapsulation 69 Finding and loading CommonJS and JSON modules using require 71 File modules 71 Modules baked into Node.js binary 71 Directories as modules 72 Module identifiers and pathnames 72 An example of application directory structure 75 Finding and loading ES6 modules using import 77 Hybrid CommonJS/Node.js/ES6 module scenarios 78 Dynamic imports with import() 78 The import.meta feature 79 npm - the Node.js package management system 80 The npm package format 80 Finding npm packages 82 Other npm commands 84 Installing an npm package 84 Installing a package by version number 84 Global package installs 85 Avoiding global module installation 86 Maintaining package dependencies with npm 86 Automatically updating package.json dependencies 87 Fixing bugs by updating package dependencies 87 Packages that install commands 88 Configuring the PATH variable to handle commands installed by modules 89 Configuring the PATH variable on Windows 90 Avoiding modifications to the PATH variable 90 Updating outdated packages you've installed 91 Installing packages from outside the npm repository 91

Page 10

Table of Contents [ iii ] Initializing a new npm package 92 Declaring Node.js version compatibility 92 Publishing an npm package 92 Explicitly specifying package dependency version numbers 93 The Yarn package management system 94 Summary 95 Chapter 4: HTTP Servers and Clients 96 Sending and receiving events with EventEmitters 96 JavaScript classes and class inheritance 97 The EventEmitter Class 98 The EventEmitter theory 100 HTTP server applications 102 ES2015 multiline and template strings 105 HTTP Sniffer – listening to the HTTP conversation 107 Web application frameworks 109 Getting started with Express 110 Setting environment variables in Windows cmd.exe command line 114 Walking through the default Express application 115 The Express middleware 119 Middleware and request paths 120 Error handling 121 Calculating the Fibonacci sequence with an Express application 122 Computationally intensive code and the Node.js event loop 127 Algorithmic refactoring 129 Making HTTP Client requests 133 Calling a REST backend service from an Express application 135 Implementing a simple REST server with Express 136 Refactoring the Fibonacci application for REST 139 Some RESTful modules and frameworks 142 Summary 143 Chapter 5: Your First Express Application 144 Promises, async functions, and Express router functions 144 Promises and error handling 147 Flattening our asynchronous code 148 Promises and generators birthed async functions 148 Express and the MVC paradigm 152 Creating the Notes application 153 Your first Notes model 154 Understanding ES-2015 class definitions 155 Filling out the in-memory Notes model 157 The Notes home page 158 Adding a new note – create 161 Viewing notes – read 165

Page 11

Table of Contents [ iv ] Editing an existing note – update 167 Deleting notes – destroy 168 Theming your Express application 170 Scaling up – running multiple Notes instances 172 Summary 175 Chapter 6: Implementing the Mobile-First Paradigm 176 Problem – the Notes app isn't mobile friendly 177 Mobile-first paradigm 178 Using Twitter Bootstrap on the Notes application 180 Setting it up 180 Adding Bootstrap to application templates 182 Alternative layout frameworks 184 Flexbox and CSS Grids 184 Mobile-first design for the Notes application 185 Laying the Bootstrap grid foundation 185 Responsive page structure for the Notes application 187 Using icon libraries and improving visual appeal 187 Responsive page header navigation bar 188 Improving the Notes list on the front page 190 Cleaning up the Note viewing experience 192 Cleaning up the add/edit note form 193 Cleaning up the delete-note window 196 Building a customized Bootstrap 197 Pre-built custom Bootstrap themes 201 Summary 203 Chapter 7: Data Storage and Retrieval 205 Data storage and asynchronous code 205 Logging 206 Request logging with Morgan 207 Debugging messages 209 Capturing stdout and stderr 209 Uncaught exceptions 210 Unhandled Promise rejections 211 Using the ES6 module format 211 Rewriting app.js as an ES6 module 211 Rewriting bin/www as an ES6 module 214 Rewriting models code as ES6 modules 214 Rewriting router modules as ES6 modules 216 Storing notes in the filesystem 217 Dynamic import of ES6 modules 220 Running the Notes application with filesystem storage 222 Storing notes with the LevelUP data store 223 Storing notes in SQL with SQLite3 227

Page 12

Table of Contents [ v ] SQLite3 database schema 227 SQLite3 model code 228 Running Notes with SQLite3 232 Storing notes the ORM way with Sequelize 233 Sequelize model for the Notes application 234 Configuring a Sequelize database connection 238 Running the Notes application with Sequelize 239 Storing notes in MongoDB 240 MongoDB model for the Notes application 242 Running the Notes application with MongoDB 246 Summary 247 Chapter 8: Multiuser Authentication the Microservice Way 248 Creating a user information microservice 249 User information model 251 A REST server for user information 256 Scripts to test and administer the user authentication server 263 Login support for the Notes application 266 Accessing the user authentication REST API 266 Login and logout routing functions 270 Login/logout changes to app.js 274 Login/logout changes in routes/index.mjs 275 Login/logout changes required in routes/notes.mjs 276 View template changes supporting login/logout 278 Running the Notes application with user authentication 281 Twitter login support for the Notes application 284 Registering an application with Twitter 284 Implementing TwitterStrategy 286 Securely keeping secrets and passwords 293 The Notes application stack 293 Summary 295 Chapter 9: Dynamic Client/Server Interaction with Socket.IO 296 Introducing Socket.IO 298 Initializing Socket.IO with Express 298 Real-time updates on the Notes homepage 303 The Notes model as an EventEmitter class 303 Real-time changes in the Notes home page 305 Changing the homepage and layout templates 306 Running Notes with real-time homepage updates 308 Real-time action while viewing notes 309 Changing the note view template for real-time action 310 Running Notes with real-time updates while viewing a note 312 Inter-user chat and commenting for Notes 313 Data model for storing messages 313 Adding messages to the Notes router 316

Page 13

Table of Contents [ vi ] Changing the note view template for messages 317 Using a Modal window to compose messages 318 Sending, displaying, and deleting messages 320 Running Notes and passing messages 323 Other applications of Modal windows 325 Summary 325 Chapter 10: Deploying Node.js Applications 327 Notes application architecture and deployment considerations 328 Traditional Linux Node.js service deployment 329 Prerequisite – provisioning the databases 330 Installing Node.js on Ubuntu 331 Setting up Notes and user authentication on the server 331 Adjusting Twitter authentication to work on the server 335 Setting up PM2 to manage Node.js processes 335 Node.js microservice deployment with Docker 341 Installing Docker on your laptop 342 Starting Docker with Docker for Windows/macOS 343 Kicking the tires of Docker 344 Creating the AuthNet for the user authentication service 345 MySQL container for Docker 346 Initializing AuthNet 348 Script execution on Windows 349 Linking Docker containers 349 The db-userauth container 350 Dockerfile for the authentication service 353 Configuring the authentication service for Docker 355 Building and running the authentication service Docker container 356 Exploring Authnet 357 Creating FrontNet for the Notes application 360 MySQL container for the Notes application 360 Dockerizing the Notes application 361 Controlling the location of MySQL data volumes 368 Docker deployment of background services 369 Deploying to the cloud with Docker compose 369 Docker compose files 371 Running the Notes application with Docker compose 375 Summary 385 Chapter 11: Unit Testing and Functional Testing 386 Assert – the basis of testing methodologies 387 Testing a Notes model 388 Mocha and Chai – the chosen test tools 389 Notes model test suite 389 Configuring and running tests 393 More tests for the Notes model 394 Testing database models 397 Using Docker to manage test infrastructure 401

Page 14

Table of Contents [ vii ] Docker Compose to orchestrate test infrastructure 402 Executing tests under Docker Compose 406 MongoDB setup under Docker and testing Notes against MongoDB 407 Testing REST backend services 409 Automating test results reporting 413 Frontend headless browser testing with Puppeteer 414 Setting up Puppeteer 414 Improving testability in the Notes UI 415 Puppeteer test script for Notes 417 Running the login scenario 420 The Add Note scenario 420 Mitigating/preventing spurious test errors in Puppeteer scripts 422 Configuring timeouts 423 Tracing events on the Page and the Puppeteer instance 423 Inserting pauses 424 Avoiding WebSockets conflicts 425 Taking screenshots 427 Summary 427 Chapter 12: Security 429 HTTPS/TLS/SSL using Let's Encrypt 430 Associating a domain name with Docker-based cloud hosting 431 A Docker container to manage Let's Encrypt SSL certificates 435 Cross-container mounting of Let's Encrypt directories to the notes container 437 Adding HTTPS support to Notes 441 Put on your Helmet for across-the-board security 443 Using Helmet to set the Content-Security-Policy header 444 Using Helmet to set the X-DNS-Prefetch-Control header 446 Using Helmet to set the X-Frame-Options header 446 Using Helmet to remove the X-Powered-By header 446 Improving HTTPS with Strict Transport Security 447 Mitigating XSS attacks with Helmet 447 Addressing Cross-Site Request Forgery (CSRF) attacks 448 Denying SQL injection attacks 449 Sequelize deprecation warning regarding operator injection attack 449 Scanning for known vulnerabilities 450 Using good cookie practices 453 Summary 454 Other Books You May Enjoy 456 Index 459

Page 15

Preface Node.js is a server-side JavaScript platform using an event-driven, non-blocking I/O model, allowing users to build fast and scalable transaction-intensive applications running in real time. It plays a significant role in the software development world and liberates JavaScript from the web browser. With Node.js, we can reuse our JavaScript skills for general software development on a large range of systems. It runs atop the ultra-fast JavaScript engine at the heart of Google's Chrome browser, V8, and adds a fast and robust library of asynchronous network I/O modules. The primary focus of Node.js is developing high performance, highly scalable web applications, and it also sees a widespread use in other areas. Electron, the Node.js-based wrapper around the Chrome engine, is the basis for popular desktop applications, such as Atom and Visual Studio Code editors, GitKraken, Postman, Etcher, and the desktop Slack client. Node.js is popular for developing Internet of Things devices and sees a tremendous adoption in microservice development and for building tools for frontend web developers and more. Node.js, as a lightweight high-performance platform, fits microservice development like a glove. The Node.js platform uses an asynchronous single-thread system, with asynchronously invoked callback functions (also known as event handlers) and an event loop, as opposed to a traditional thread-based architecture. The theory is that threaded systems are notoriously difficult to develop, and that threads themselves impose an architectural burden on app servers. Node.js's goal is to provide an easy way to build scalable network servers. The whole Node.js runtime is designed around asynchronous execution. JavaScript was chosen as the language because anonymous functions and other language elements provide an excellent base for implementing asynchronous computation. Who this book is for We assume that you have some knowledge of JavaScript and possibly have experience with server-side code development, and that you are looking for a different way of developing server-side code.

Page 16

Preface [ 2 ] Server-side engineers may find the concepts behind Node.js refreshing. It offers a new perspective on web application development and a different take on server architectures from the monoliths we deal with in other programming languages. JavaScript is a powerful language and Node.js's asynchronous nature plays to its strengths. Having JavaScript on both the frontend and the backend gives a whole new meaning. Developers experienced with browser-side JavaScript will find it productive to bring that knowledge to a new territory. Although our focus is on web application development, Node.js knowledge can be applied in other areas as well. As said earlier, Node.js is widely used to develop many types of applications. What this book covers , About Node.js, introduces you to the Node.js platform. It covers its uses, the technological architecture choices in Node.js, its history, the history of server-side JavaScript, why JavaScript should be liberated from the browser, and important recent advances in the JavaScript scene. , Setting up Node.js, goes over setting up a Node.js developer environment. This includes installing Node.js on Windows, macOS, and Linux. Important tools are covered, including the npm and yarn package management systems and Babel, which is used for transpiling modern JavaScript into a form that's runnable on older JavaScript implementations. , Node.js Modules, explores the module as the unit of modularity in Node.js applications. We dive deep into understanding and developing Node.js modules and using npm to maintain dependencies. We learn about the new module format, ES6 Modules, that should supplant the CommonJS module format currently used in Node.js, and are natively supported in Node.js 10.x. , HTTP Servers and Clients, starts exploring web development with Node.js. We develop several small webserver and client applications in Node.js. We use the Fibonacci algorithm to explore the effects of heavy-weight, long-running computations on a Node.js application. We also learn several mitigation strategies, and have our first experience with developing REST services. , Your First Express Application, begins the section on developing a note-taking application. The first step is getting a basic application running.

Page 17

Preface [ 3 ] , Implementing the Mobile-First Paradigm, uses Bootstrap V4 to implement responsive web design. We take a look at integrating a popular icon set so that we can have pictorial buttons, and go over compiling a custom Bootstrap theme. , Data Storage and Retrieval, ensures that we don't lose our notes when we restart the application. We explore several database engines and a method to enable easily switching between them at will. , Multiuser Authentication the Microservice Way, adds user authentication to the note-taking application. Both logged-in and anonymous users can access the application, with varying capabilities based on role. Authentication is supported both for locally stored credentials and for using OAuth against Twitter. , Dynamic Client/Server Interaction with Socket.IO, lets our users talk with each other in real time. JavaScript code will run in both the browser and the server, with Socket.IO providing the plumbing needed for real-time event exchange. Users will see notes change as they're edited by other users and can leave messages/comments for others. , Deploying Node.js Applications, helps us understand Node.js application deployment. We look at both traditional Linux service deployment using an script and using Docker for both local development and deployment on cloud hosting services. , Unit Testing and Functional Testing, takes a look at three test development models: unit testing, REST testing, and functional testing. We'll use the popular Mocha and Chai frameworks for the first two, and Puppeteer for the third. Puppeteer uses a headless version of Chrome to support running tests. Docker is used to facilitate setting up and tearing down test environments. , Security, explores techniques and tools required to mitigate the risk of security intrusions. Intelligently using Docker is a great first step if only because it can easily limit the attack surface of your application. The Node.js community has developed a suite of tools that integrate with Express to implement several critical security technologies. To get the most out of this book The basic requirement is to install Node.js and have a programmer-oriented text editor. The editor need not be anything fancy, vi/vim will even do in a pinch. We will show you how to install everything that's needed. It's all open source software that can be easily downloaded from websites. The most important tool is the one between your ears.

Page 18

Preface [ 4 ] Some chapters require database engines, such as MySQL and MongoDB. Although Node.js is a cross-platform software development platform, some third-party modules are written in C/C++ and must be compiled during installation. To do so, native- code development tools such as C/C++ compilers are required, and Python is required to run the tool-chain. The details are covered in , Setting up Node.js. Microsoft is involved with the Node.js project and to ensure developer productivity with Node.js on Windows. Download the example code files You can download the example code files for this book from your account at . If you purchased this book elsewhere, you can visit and register to have the files emailed directly to you. You can download the code files by following these steps: Log in or register at .1. Select the SUPPORT tab.2. Click on Code Downloads & Errata.3. Enter the name of the book in the Search box and follow the onscreen4. instructions. Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of: WinRAR/7-Zip for Windows Zipeg/iZip/UnRarX for Mac 7-Zip/PeaZip for Linux The code bundle for the book is also hosted on GitHub at . We also have other code bundles from our rich catalog of books and videos available at . Check them out!

Page 19

Preface [ 5 ] Conventions used There are a number of text conventions used throughout this book. : Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "The object encapsulates the HTTP protocol, and its method creates a whole web server, listening on the port specified in the method." A block of code is set as follows: When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold: fibonacciAsync(urlP.query['n'], fibo => { res.end('Fibonacci '+ urlP.query['n'] +'='+ fibo); }); Any command-line input or output is written as follows: $ node --version v8.9.1 Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "In the Start menu, enter PowerShell in the applications search box." Warnings or important notes appear like this.

Page 20

Preface [ 6 ] Tips and tricks appear like this. Get in touch Feedback from our readers is always welcome. General feedback: Email and mention the book title in the subject of your message. If you have questions about any aspect of this book, please email us at . Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit , selecting your book, clicking on the Errata Submission Form link, and entering the details. Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at copyright@packtpub.com with a link to the material. If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit . Reviews Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you! For more information about Packt, please visit .