Node.

JS

Yoni Goldberg
About Me – Yoni Goldberg

Backend. Only
Node. Only
Top 15 Node.JS projects on GitHub
Some Formalities Before We Take Off

Course flow
Prerequisites
Environment setup
Supporting materials
Questions (www.clarify.live)
Post course meeting
Intro

What is Node.JS?
 It All
Began 8
Years
Ago…
2018 - 8 Years Later

8M 1st 2nd
Servers worldwide Most popular Most loved
framework technology
(StackOverflow) (StackOverlow)
Platforms Packages
Compared

200,000
Maven Java – 220,000
150,000
100,000 Nugget .NET - ~100,000

PyPi Python - ~200,000

page
07
Platforms Packages
Compared

720,000
NPM Node.JS – 596,000

Maven Java – 220,000

Nugget .NET - ~100,000

220,000
150,000 180,000
PyPi Python - ~200,000

page
08
What Is Node Secret Sauce?

“A language that doesn't affect the way you

think about programming, is not worth
knowing”

john doe
1. Node Secret Sauce - House Of Many Ideas

OOP vs Functional

Dynamic typing vs Strict typing

Adhoc and lightweight vs Robust architecture

2. Node Secret Sauce - Code once, deploy anywhere

Desktop IOT Smart Devices

Browser Web Server

3. Node Secret Sauce - Vibrant Community

726,000 packages

Re-inventing the wheel every year

Open source obsessive

4. Node Secret Sauce - Freedom

“We’re JavaScript developers.

Where we are going there are no
roads”
Agenda - The World Of Node.JS

Your Code + Community

Code
+ Node
Libraries

+ Service
Architecture + Web
Framework

+ Testing

+ Production Setup

+ Node Engine
Section

Development Basics
The Basic Pillars - JS
The Basic Pillars - Modules
Creating Modules

Module = file

Choose what to export

Exporting code -> module.exports = someFunction

Importing code -> require(‘./someModule.js’)

NPM – Node Package Manager

Solution = 10% your code, 90% others code

Describes properties and dependencies

Documented within package.json

Marketplace of components (720K!)

Hello-World Exercise

The Packing Advisor

Goal: Combine JS, NPM and modules (require)

Time: 20 minutes
Requirements:
Get input – City name
Output – whether we need to pack a coat or not
Logic – If the temperature will drop below 15 degree in the next 5 days - output that a coat is needed
Tech: use the NPM package "weather-js", encapsulate login in a Class
Starter file: ~/basics/the-packing-advisor/start.js
Section

Asynchronous Programming
Async Programming - Context

Node is a-synchronous by nature

Asynchronous programming is sometimes challenging

Mastering a-synchronous patterns is a must have skill for

Node developers
Synchronous Programming Flow
= Waiting & doing nothing
category one - Lorem
1. const options = {dbName: orders}
ipsum dlor sit amet //0.1ms
= Executing code
category four - =Lorem
2. const orders DB.getOrders(options)
ipsum dolor sit amet //8ms

3. logger.log(‘orders fetched’) //4ms

1% Executing
Synchronous Programming – Simple And Wasteful

Program is executed line by line

Blocks the thread while waiting

Requests are handled using multiple threads

A-Synchronous Programming Flow
= Waiting doing nothing
category one - Lorem
1. const options = {dbName: orders}
ipsum dlor sit amet //0.1ms
= Executing code
category four - =Lorem
2. const orders Don’t Wait
ipsum ->sitDB.getOrders(options)
dolor amet //0.1ms

3. Don’t Wait -> logger.log(‘orders fetched’) //0.1ms

We didn’t wait. But who will handle the result once it arrives?

100% Executing
A-Synchronous Programming – Efficient And Complex

Program execution’s flow can’t be predicted

Thread is never blocked

Node uses one thread which can handle ~10,000 requests per
second

Node.JS is single threaded, non-blocking

 Async Techniques
Compared
Callback Promise Async Await

Nesting

Try-Catch

Debugging

Stacktrace

Synchronous loop

Asynchronous loop

Bottom Line
Option 1: Callback

The eldest paradigm (v0.1)

Main idea - Tell an async code/function which function to

invoke when it finishes

//Example: call some async function

searchOrders(‘USA’ , function(error, result){
console.log(‘This are the orders’ + result)
})
Async Demo Requirements

Get products from DB

Return products that a user ordered

Get input JSON with configuration – return in English or with local translation
Get current user id from DB
Fetch user orders
Based on orders, get all user products (translated or in English)
Return products. That’s it
Callback Exercise

The travelling recommender

Goal: Understand callbacks flow

Time: 25 minutes
Requirements:
Get input - traveling city, country & date
Output – whether it’s recommended to travel to the given city on the given date
Logic – If the given date falls on a vacation, print ‘No. The streets will be crowded!’
If the temperature might drop below 12 Celsius degree, print ‘Oh no, might be too cold’. Other wise print ‘Sure, go for it’
Bonus: get dates range, vacation start and end date and check for the entire period
Tech: use callbacks only, use the NPM packages: ‘weather-js’ & ‘date-holidays’
 Async Techniques
Compared

Callback Promise Async Await

Nesting High

Try-Catch No

Debugging Tolerable

Stacktrace Named functions

Synchronous loop No

Asynchronous loop No

Bottom Line
Option 2: Promises

Grown by the community, now built-in

Main idea - the async code/function immediately provide

the caller with an object that will yield event when the
async operation completed

//Example: call some async function

let ordersPromise = searchOrders(‘USA’);
ordersPromise.then(function (result) {
console.log(‘This are the orders’ + result)
})
Promises Exercise

The travelling recommender

Goal: Understand promises flow

Time: 25 minutes
Requirements:
Get input - traveling city, country & date
Output – whether it’s recommended to travel to the given city on the given date
Logic – If the given date falls on a vacation, print ‘No. The streets will be crowded!’
If the temperature might drop below 12 Celsius degree, print ‘Oh no, might be too cold’. Other wise print ‘Sure, go for it’
Bonus: get dates range, vacation start and end date and check for the entire period
Tech: use promises only, use the NPM packages: ‘weather-js’ & ‘date-holidays’
Promises Exercise

The travelling recommender – quick version

Goal: Understand promises flow
Time: 25 minutes
Requirements:
Get input - traveling city, country
Output – whether it’s recommended to travel to the given city on the given date. If the given date falls on a vacation OR the
temperature might drop below 12 Celsius degree, print ‘No. The streets will be crowded!’ Other wise print ‘Sure, go for it’
Bonus: (a) Get multiple locations and check for each (b) Get dates range and check for the entire period
How:
Use promises only and return a promise. Get some help from the functions isWeatherWarmEnough (returns callback,
promisify it) and checkHolidays (return a promise)
Starter file: ~/async/travel-recommend-promises/start-with-helpers.js
 Async Techniques
Compared

Callback Promise Async Await

Nesting High

Try-Catch No

Debugging Tolerable

Stacktrace Named functions

Synchronous loop No

Asynchronous loop No

Bottom Line
Option 3: Async/await

New, since v8.9

Main idea – synchronous programming alike!

//Example: call some async function

let result = await searchOrders(‘USA’);
console.log(‘This are the orders’ + result)
From Promises To
Async/await

function do(){ async function getOrders(){

new Promise((resolve, reject) =>{ const user = await logIn(“user”,
logIn("username", “p") “p”);
.then((user) => { const orders = await getOrders(user);
getOrders(user.name)})
.then((orders)=> { return orders
resolve(orders) }
}
})

}
Async/await Exercise

The travelling recommender

Goal: Understand async/await flow

Time: 25 minutes
Requirements:
Get input - traveling city, country & date
Output – whether it’s recommended to travel to the given city on the given date
Logic – If the given date falls on a vacation, print ‘No. The streets will be crowded!’
If the temperature might drop below 12 Celsius degree, print ‘Oh no, might be too cold’. Other wise print ‘Sure, go for it’
Bonus: get dates range, vacation start and end date and check for the entire period
Tech: use async/await only, use the NPM packages: ‘weather-js’ & ‘date-holidays’
A-Synchronous Programming Heaven

Async/await is your best friend – use it whenever possible

To run multiple functions simultaneously - use Promise.all

To avoid callbacks – Apply Util.Promisify over old packages

Section

Building REST API

The World Of Node.JS

Your Code + Community

Code
+ Node
Libraries

+ Node Engine Let’s

discuss
+ Web Application this now

+ Testing

+ Production Setup

= Great Solution
Anatomy Of Any Web App
Express Coverage

#must #quick-start
Express
Node.JS Framework
Popularity

18,679,0 Express – 18M downloads/day

60

KOA – 0.4M downloads/day

HAPI – 0.5M downloads/day

407,720 500,657 358,583 Restify – 0.35M downloads/day

page
047
What Express Brings To The Table

Simplification over Node’s built-in web server

Web request objects filled with properties and methods

Routing – invoke custom code per URL

Huge echo-system of utilities

 Express
Middleware
What Are
Express Middleware
Middleware
Syntax

//1. define middleware func //2. Register

function(req, res, next){ app.use(middlewarefunc)
//custom code here
}
Middlewares & API Exercise

The rate limiter

Goal: Practice middleware programming

Time: 25 minutes
Requirements:
Get API input – write an API that receives a city name GET: /api/travel and returns whether it’s recommended for a vacation (based on the
exercise ‘The travel advisor’)
Logic for all incoming requests (all routes) – a specific user/ip can only request 4 requests per minute. If a visitor exceeds this limit - HTTP
status 429 should be returned and the request should not be processed
Logic for GET: /api/travel – if some parameter is missing, return HTTP status 400. Otherwise return HTTP status 200
Tech: Implement using ExpressJS, use middleware to limit the amount of requests
Starter file: ~/examples/the-rate-limiter/start.js
Express Exercise

The travel recommender API

Goal: Practice basic Express programming

Time: 25 minutes
Requirements:
Get API input – write an API that receives a city & country name GET: /api/travel and returns whether it’s recommended for a vacation (based
on the exercise ‘The travel advisor’).
Logic for all incoming requests (all routes) – if the request body is larger than 1000 characters length – return 400 http status
Bonus: (A) If some parameter is missing, return HTTP status 400. Otherwise return HTTP status 200. (B) A specific IP can not
perform more than 5 requests/sec
Tech: Implement using ExpressJS, use middleware to limit the amount of requests
Starter file: ~/examples/express/travel-recommender /start.js
Section

Project Structure
Inspiration

Warning: highly subjective topic

Based on the most popular paradigms:

• Domain Driven Design

• Microservices
• Clean Architecture
• Hexagonal Architecture
• Onion Architecture
Break Your App Into Small
Autonomous Components

Benefits:
Small is simple
Deploy with confidence
Resiliency
Tech agnostic
Granularly scalable
Moving From Spaghetti
To Ravioli
 Microservices guidelines

#must #quick-start #nice-to-have #complex

1 Small and simple components 1

Deployed independently

2 Share no logic 2 Own a server or a container

3 Interaction via message queue (better) 3 Separate DB

or REST

4 Separate DB tables 4 Scale independently

Domain Driven Design (DDD)

#must #quick-start

Interest Over Time

2004 2018
DDD Core Principle -
Isolate The Domain
#must #quick-start

This is
where the
business
features live
DDD Layers In Node.JS

#must #quick-start

Express, etc

Custom code

NPM
Clean Architecture

#must #quick-start
“Heavy architecture” (Martin Fowler)

#must #quick-start
Practical Project Structure Guidelines

Project = single small microservice (+self contained)

Isolate the domain

Express != microservice

Express === Single entry point to your microservice

Layers = API, Domain: services, entities, data-access

Anatomy Of a Microservice

#must #quick-start
Anatomy Of a Microservice

#must #quick-start
Project Structure Demo
Section

NPM
NPM – Overview

Mashup architecture – 10% your code, 90% community

Has a competitor – Yarn (by Facebook)

Maintained by npm inc. company

640K packages available

NPM Typical Flow
NPM – Publishing a Package

Run mkdir ‘your-package-name’

npm init

Code it!

Add tests + readme.md + version

npm adduser

npm publish
Publishing NPM Package Exercise

Publish the ‘travelling recommender’

Goal: Understand package wrapping and life-cycle

Time: 20 minutes
Requirements:
Convert the ‘Travelling recommender’ exercise into NPM package and publish. Ensure to include some basic readme.m
Tech: Run ‘npm init’ to add package.json, ‘npm adduser’ & ‘npm publish’
Starter file: ~/examples/the-traveler/start.js
Lifecycle Of a Package

Publishers update packages constantly

When code is deployed in production newer dependency

version might be used

Lock dependencies to prevent drifting

Use ‘semver’ to understand package changes

Semver PATCH version when
you make backwards-
compatible bug fixes.

1.7.3
MAJOR version
when you make
incompatible API
changes

MINOR version when

you add functionality in
a backwards-
compatible manner
NPM – Popular Commands

npm init – craft an empty solution

npm install – add new local/online package
npm install <pkg> --save-dev – add package for dev purposes
npm uninstall <pkg> --save – remove a package
npm start – run the application
npm test – test the application
npm publish [tag] – share your app/package with the world
Section

Error Handling
The Ultimate Goals Of Error Handling

01 02 03
Visibility through Optimized API results Maximum uptime
operational dashboard
Error Handling Typical Flow
Throwing Errors
const user = await logIn("username",
"password");

if(!user){
throw new appError('userDoesntExist', 400,
'Couldnt find the user when trying to get
products', true);
}
 Catchers

Local vs Global

Promise vs Async/await

Express caught exceptions

Uncaught exceptions
Error Handler

If(!err.isOperational){
process.exit(1)
}
 Operational vs Developer Errors

Operational Non-trusted, Developer

Definition Run-time errors that the program can predict or reason Errors that the program can’t reason about their impact
about

Examples Invalid input, query timeout, 3rd party server not Compilation error, startup error, unknown error, errors
responding originated from ‘stateful’ objects

Typical scope Request Compilation, system-wide

Bottom Line Log and continue Log and exit

Error Handline Exercise

Protecting the ‘travelling recommender’

Goal: Practice Express error handling

Time: 20 minutes
Requirements:
Based on the ‘Travelling recommender’ exercise. If the given city doesn’t exist – return HTTP status 404.
Bonus: if there’s no internet connection (turn it off), the program won’t be able to access the weather service. In that case, return HTTP
status 500
Tech: Extend the build-in Error object with few additional properties. Use Express error-middleware.
Section

Authentication & Authorization

Challenges with
traditional auth flows
Auth flow
with JWT token
Middlewares & API Exercise

JWT Based Authentication

Goal: Understand how stateless authentication works

Time: 20 minutes
Requirements:
Login endpoint – receive username & password, if they are correct then return a signed JWT token with the user id and name. Otherwise, if
the credentials are wrong, return 404
Travel recommender endpoint – based on the existing API from previous exercises, prevent accessing to these API without valid token
Tech: Use the NPM package ‘jsonwebtoken’ to sign and verify tokens. Use a middleware to authenticate requests
Starter file: ~/examples/express-jwt/start.js
Section

Node Mechanics
Why Should I Care About
How Node Works?

01 02
When future code will run? When does my code become CPU-
Differences between various API:
setTimeout vs setImmediate vs
intensive?
callback, etc How sensitive is Node to synchronous
code?

03
Should I cluster?
Use one Node process or multiple?
The first thing
About Node.JS mechanics

Single Threaded
Non-Blocking
The first thing
About Node.JS mechanics

Single Threaded
Non-Blocking
Node.JS
Building Blocks

App Code Libuv Kernel

JS IO
01
Thread
s
V8

JS Engine
02 03 04
Node.JS
Building Blocks

App Code Libuv

“a magical place filled with unicorns and
Kernel
rainbows, and is the reason Node can
essentially be "single threaded" while
JS still allowing an arbitrary number of
IO
01 operations to be handled in the
background”
john doe
Thread
s
V8

JS Engine
02 03 04
Node.JS Flags:
Building Blocks --POOL_SIZE

App Code Libuv Kernel

JS IO
01 Flags:
--inspect
--use_strict
Thread
--many_other s
V8

JS Engine
02 03 04
4 Facts About
V8

01 02
Fast Single Threaded
Transform JS into machine code Runs using one thread only

03 04
Replaceable Sometimes not fast enough
Other vendors aim to provide their own GC, dynamic typing, etc – all come with a
implementation price

page
095
Typical Program Flow

Demonstrating
Node internals with
a real code
 Code V8 Stack Async API

console.log(‘starting’)

validate = () => {console.log(‘validate’)};

saveUser = () => {console.log(‘save’)};

validate()
saveUser()

console.log(‘end’)
Main

Console Task Queue

Timer Network Immediate
 Code V8 Stack Async API

console.log(‘starting’)

validate = () => {console.log(‘validate’)};

saveUser = () => {console.log(‘save’)};

validate() Console.log
saveUser()

console.log(‘end’)
Main

Console Task Queue

starting
Timer Network Immediate
 Code V8 Stack Async API

console.log(‘starting’)

validate = () => {console.log(‘validate’)};

saveUser = () => {console.log(‘save’)};

validate()
saveUser()

console.log(‘end’)
Main

Console Task Queue

starting
Timer Network Immediate
 Code V8 Stack Async API

console.log(‘starting’)

validate = () => {console.log(‘validate’)};

saveUser = () => {console.log(‘save’)};

validate() Validate()
saveUser()

console.log(‘end’)
Main

Console Task Queue

starting
validate Timer Network Immediate
 Code V8 Stack Async API

console.log(‘starting’)

validate = () => {console.log(‘validate’)};

saveUser = () => {console.log(‘save’)};

validate()
saveUser()

console.log(‘end’)
Main

Console Task Queue

starting
validate Timer Network Immediate
 Code V8 Stack Async API

console.log(‘starting’)

validate = () => {console.log(‘validate’)};

saveUser = () => {console.log(‘save’)};

validate() saveUser()
saveUser()

console.log(‘end’)
Main

Console Task Queue

starting
validate Timer Network Immediate
save
 Code V8 Stack Async API

console.log(‘starting’)

validate = () => {console.log(‘validate’)};

saveUser = () => {console.log(‘save’)};

validate()
saveUser()

console.log(‘end’)
Main

Console Task Queue

starting
validate Timer Network Immediate
save
 Code V8 Stack Async API

console.log(‘starting’)

validate = () => {console.log(‘validate’)};

saveUser = () => {console.log(‘save’)};

validate() Console.log()
saveUser()

console.log(‘end’)
Main

Console Task Queue

starting
validate Timer Network Immediate
save
end
 Code V8 Stack Async API

console.log(‘starting’)

validate = () => {console.log(‘validate’)};

saveUser = () => {console.log(‘save’)};

validate()
saveUser()

console.log(‘end’)
Main

Console Task Queue

starting
validate Timer Network Immediate
save
end
 Code V8 Stack Async API

console.log(‘starting’)

validate = () => {console.log(‘validate’)};

saveUser = () => {console.log(‘save’)};

validate()
saveUser()

console.log(‘end’)

Console Task Queue

starting
validate Timer Network Immediate
save
end
 Code Stack Async API
DB.save()
console.log(‘starting’)

validate = () =>
{console.log(‘validate’);saveUser()}; DB.save()

saveUser = () => {console.log(‘save’); saveUser()

await DB.save(user, callback);} Another Request
Console.log
Validate()
Callback
validate()

console.log(‘end’) Main

Console Task Queue

starting
Timer IO Immediate
validate
save Callback
end
Take
Away

Avoid 2. Monitor The 3. Replicate

1. Event Lop Processes
Starvation
Minimize CPU Measure average Diagnose whether
intensive tasks; avoid queue time; multiple processes
process.nextTick; use yields better
async API performance

page
0108
Part 6

Continuous Quality
Testing & Inspecting
Continuous Integration Inspection Trends

Old school

Modern
Demo

Live demo of running Jenkins

declarative pipeline in Jenkins,
getting mini build dashboard in
Slack, complexity reports,
coverage reports, live code
analysis with Wallaby and more
 CI Market – Flexible vs Easy

CircleCI GitLab

CodeShip Travis
B
Easy

AWS CodeBuild

A
Jenkins

Popularity
Flexible
Take Away

01 02
Simple CI? Easy! Everything Docker
Getting test, coverage and linting CI tools are built around dockers - plan
automation is now a breeze your application around Docker

03 04
Plenty Of Tools Developer Experience
The market is not short of rich analysis CI is merely a bot that should buy
tools, don’t miss your free lunch developers time to handle strategic issues

page
0113
Part 7

DevOps
Production High-level
Topology
thank you.
Appendix

Node.JS
Best Practices
Appendix

Part 1 – Project Structure

Practices
1.1 Structure your solution by components

TL;DR: The worst large applications pitfall is maintaining a huge code

base with hundreds of dependencies - such a monolith slows down
developers as they try to incorporate new features. Instead, partition your
code into components, each gets its own folder or a dedicated codebase,
and ensure that each unit is kept small and simple. Visit 'Read More'
below to see examples of correct project structure

Otherwise: When developers who code new features struggle to realize

the impact of their change and fear to break other dependant
components - deployments become slower and more risky. It's also
considered harder to scale-out when all the business units are not
separated
1.1 Structure your solution by components
One Paragraph Explainer
For medium sized apps and above, monoliths are really bad - having one big software with many
dependencies is just hard to reason about and often leads to spaghetti code. Even smart architects
— those who are skilled enough to tame the beast and 'modularize' it — spend great mental effort
on design, and each change requires carefully evaluating the impact on other dependent objects.
The ultimate solution is to develop small software: divide the whole stack into self-contained
components that don't share files with others, each constitutes very few files (e.g. API, service, data
access, test, etc.) so that it's very easy to reason about it. Some may call this 'microservices'
architecture — it's important to understand that microservices are not a spec which you must
follow, but rather a set of principles. You may adopt many principles into a full-blown microservices
architecture or adopt only a few. Both are good as long as you keep the software complexity low.
The very least you should do is create basic borders between components, assign a folder in your
project root for each business component and make it self-contained - other components are
allowed to consume its functionality only through its public interface or API. This is the foundation
for keeping your components simple, avoid dependency hell and pave the way to full-blown
microservices in the future once your app grows.
 "Scaling requires scaling of the entire application"

“Monolithic applications can be successful, but increasingly people are feeling

frustrations with them - especially as more applications are being deployed to
the cloud. Change cycles are tied together - a change made to a small part of the
application requires the entire monolith to be rebuilt and deployed. Over time it's
often hard to keep a good modular structure, making it harder to keep changes
that ought to only affect one module within that module. Scaling requires scaling
of the entire application rather than parts of it that require greater resource.”

From the blog,

MartinFowler.com
"So what does the architecture of your application scream?"

“...if you were looking at the architecture of a library, you’d likely see a
grand entrance, an area for check-in-out clerks, reading areas, small
conference rooms, and gallery after gallery capable of holding
bookshelves for all the books in the library. That architecture would
scream: Library.”

From the blog, 

uncle-bob
1.1 Structure your solution by components

Good:
Structure your
solution by
self-contained
components
1.1 Structure your solution by components

Bad: Group
your files by
technical
role
1.2 Layer your components, keep Express within its
boundaries

TL;DR: Each component should contain 'layers' - a dedicated object for the

web, logic and data access code. This not only draws a clean separation of
concerns but also significantly eases mocking and testing the system.
Though this is a very common pattern, API developers tend to mix layers by
passing the web layer objects (Express req, res) to business logic and data
layers - this makes your application dependant on and accessible by
Express only

Otherwise: App that mixes web objects with other layers can not be
accessed by testing code, CRON jobs and other non-Express callers
1.2 Layer your components, keep Express within its
boundaries

Separate
component
code into
layers: web,
services, and
DAL
1.3 Wrap common utilities as NPM packages

TL;DR: In a large app that constitutes a large code base, cross-

cutting-concern utilities like logger, encryption and alike, should be
wrapped by your own code and exposed as private NPM packages.
This allows sharing them among multiple code bases and projects

Otherwise: You'll have to invent your own deployment and

dependency wheel
1.3 Wrap common utilities as NPM packages
One Paragraph Explainer

Once you start growing and have different components on different servers
which consumes similar utilities, you should start managing the
dependencies - how can you keep 1 copy of your utility code and let
multiple consumer components use and deploy it? well, there is a tool for
that, it's called npm... Start by wrapping 3rd party utility packages with your
own code to make it easily replaceable in the future and publish your own
code as private npm package. Now, all your code base can import that code
and benefit free dependency management tool. It's possible to publish npm
packages for your own private use without sharing it publicly using 
private modules, private registry or local npm packages
1.3 Wrap common utilities as NPM packages

Sharing your
own common
utilities across
environments
and
components
1.4 Separate Express 'app' and 'server'

TL;DR: Avoid the nasty habit of defining the entire Express app in a single huge

file - separate your 'Express' definition to at least two files: the API declaration
(app.js) and the networking concerns (WWW). For even better structure, locate
your API declaration within components

Otherwise: Your API will be accessible for testing via HTTP calls only (slower
and much harder to generate coverage reports). It probably won't be a big
pleasure to maintain hundreds of lines of code in a single file
1.4 Separate Express 'app' and 'server'

One Paragraph Explainer

The latest Express generator comes with a great practice that is worth to
keep - the API declaration is separated from the network related
configuration (port, protocol, etc). This allows testing the API in-process,
without performing network calls, with all the benefits that it brings to the
table: fast testing execution and getting coverage metrics of the code. It also
allows deploying the same API under flexible and different network
conditions. Bonus: better separation of concerns and cleaner code
Code example: API declaration, should reside in app.js

var app = express();

app.use(bodyParser.json());
app.use("/api/events", events.API);
app.use("/api/forms", forms);
Code example: Server network declaration, should reside in
/bin/www

var app = require('../app');

var http = require('http');

/**
* Get port from environment and store in Express.
*/

var port = normalizePort(process.env.PORT || '3000');

app.set('port', port);

/**
* Create HTTP server.
*/

var server = http.createServer(app);

Example: test your API in-process using supertest (popular
testing package)

const app = express();

app.get('/user', function(req, res) {

res.status(200).json({ name: 'tobi' });
});

request(app)
.get('/user')
.expect('Content-Type', /json/)
.expect('Content-Length', '15')
.expect(200)
.end(function(err, res) {
if (err) throw err;
});
1.5 Use environment aware, secure and hierarchical config

TL;DR: A perfect and flawless configuration setup should

ensure (a) keys can be read from file AND from
environment variable (b) secrets are kept outside
committed code (c) config is hierarchical for easier
findability. There are a few packages that can help tick
most of those boxes like rc, nconf and config
Otherwise: Failing to satisfy any of the config
requirements will simply bog down the development or
devops team. Probably both
1.5 Use environment aware, secure and hierarchical config

One Paragraph Explainer

When dealing with configuration data, many things can just annoy and slow down:
1.setting all the keys using process environment variables becomes very tedious when in need to inject 100 keys
(instead of just committing those in a config file), however when dealing with files only the DevOps admins cannot
alter the behavior without changing the code. A reliable config solution must combine both configuration files +
overrides from the process variables
2.when specifying all keys in a flat JSON, it becomes frustrating to find and modify entries when the list grows bigger.
A hierarchical JSON file that is grouped into sections can overcome this issue + few config libraries allow to store the
configuration in multiple files and take care to union all at runtime. See example below
3.storing sensitive information like DB password is obviously not recommended but no quick and handy solution
exists for this challenge. Some configuration libraries allow to encrypt files, others encrypt those entries during GIT
commits or simply don't store real values for those entries and specify the actual value during deployment via
environment variables.
4.some advanced configuration scenarios demand to inject configuration values via command line (vargs) or sync
configuration info via a centralized cache like Redis so multiple servers will use the same configuration data.
Some configuration libraries can provide most of these features for free, have a look at NPM libraries like rc, nconf
 and configwhich tick many of these requirements.
Code Example – hierarchical config helps to find entries and
maintain huge config files
{
// Customer module configs
"Customer": {
"dbConfig": {
"host": "localhost",
"port": 5984,
"dbName": "customers"
},
"credit": {
"initialLimit": 100,
// Set low for development
"initialDays": 1
}
}
}
Part 2 – Error Handling
Practices
2.1 Use Async-Await or promises for async error handling

TL;DR: Handling async errors in callback style is probably the fastest

way to hell (a.k.a the pyramid of doom). The best gift you can give to
your code is using a reputable promise library or async-await instead
which enables a much more compact and familiar code syntax like try-
catch

Otherwise: Node.js callback style, function(err, response), is a

promising way to un-maintainable code due to the mix of error handling
with casual code, excessive nesting and awkward coding patterns
2.1 Use Async-Await or promises for async error handling

One Paragraph Explainer

Callbacks don’t scale well since most programmers are not familiar with them. They
force to check errors all over, deal with nasty code nesting and make it difficult to
reason about the code flow. Promise libraries like BlueBird, async, and Q pack a
standard code style using RETURN and THROW to control the program flow.
Specifically, they support the favorite try-catch error handling style which allows
freeing the main code path from dealing with errors in every function
Code Example – using promises to catch errors

doWork()
.then(doWork)
.then(doOtherWork)
.then((result) => doWork)
.catch((error) => {throw error;})
.then(verify);
Anti pattern code example – callback style error handling

getData(someParameter, function(err, result) {

if(err !== null) {
// do something like calling the given callback function and pass the error
getMoreData(a, function(err, result) {
if(err !== null) {
// do something like calling the given callback function and pass the error
getMoreData(b, function(c) {
getMoreData(d, function(e) {
if(err !== null ) {
// you get the idea? 
}
})
});
}
});
}
});
"We have a problem with promises"

“……And in fact, callbacks do something even more sinister: they deprive us of the stack,
which is something we usually take for granted in programming languages. Writing code
without a stack is a lot like driving a car without a brake pedal: you don’t realize how badly
you need it until you reach for it and it’s not there. The whole point of promises is to give
us back the language fundamentals we lost when we went async: return, throw, and the
stack. But you have to know how to use promises correctly in order to take advantage of
them.”

From the blog

pouchdb.com
"The promises method is much more compact"

“………The promises method is much more compact, clearer and quicker to

write. If an error or exception occurs within any of the ops it is handled by
the single .catch() handler. Having this single place to handle all errors
means you don’t need to write error checking for each stage of the work.”

From the blog

gosquared.com
"Promises are native ES6, can be used with generators"

“….Callbacks have a lousy error-handling story. Promises are better. Marry

the built-in error handling in Express with promises and significantly lower
the chances of an uncaught exception. Promises are native ES6, can be
used with generators, and ES7 proposals like async/await through
compilers like Babel”

From the blog

StrongLoop
"All those regular flow control constructs you are used to are
completely broken"

“……One of the best things about asynchronous, callback-based programming is

that basically all those regular flow control constructs you are used to are
completely broken. However, the one I find most broken is the handling of
exceptions. Javascript provides a fairly familiar try…catch construct for dealing
with exceptions. The problem with exceptions is that they provide a great way of
short-cutting errors up a call stack, but end up being completely useless of the
error happens on a different stack…”
From the blog
Benno’s
2.2 Use only the built-in Error object

TL;DR: Many throws errors as a string or as some custom type – this

complicates the error handling logic and the interoperability between modules.
Whether you reject a promise, throw an exception or an emit error – using only
the built-in Error object will increase uniformity and prevent loss of information

Otherwise: When invoking some component, being uncertain which type of errors

come in return – it makes proper error handling much harder. Even worse, using
custom types to describe errors might lead to loss of critical error information
like the stack trace!
2.2 Use only the built-in Error object
One Paragraph Explainer
The permissive nature of JS along with its variety code-flow options (e.g.
EventEmitter, Callbacks, Promises, etc) pushes to great variance in how
developers raise errors – some use strings, other define their own custom
types. Using Node.js built-in Error object helps to keep uniformity within your
code and with 3rd party libraries, it also preserves significant information
like the StackTrace. When raising the exception, it’s usually a good practice
to fill it with additional contextual properties like the error name and the
associated HTTP error code. To achieve this uniformity and practices,
consider extending the Error object with additional properties, see code
example below
Code Example – doing it right

// throwing an Error from typical function, whether sync or async

if(!productToAdd)
throw new Error("How can I add new product when no value provided?");

// 'throwing' an Error from EventEmitter

const myEmitter = new MyEmitter();
myEmitter.emit('error', new Error('whoops!'));

// 'throwing' an Error from a Promise

return new Promise(function (resolve, reject) {
return DAL.getProduct(productToAdd.id).then((existingProduct) => {
if(existingProduct != null)
reject(new Error("Why fooling us and trying to add an existing product?"));
});
});
Code example – Anti Pattern

// throwing a string lacks any stack trace information and other important data properties
if(!productToAdd)
throw ("How can I add new product when no value provided?");
Code example – doing it even better

// centralized error object that derives from Node’s Error

function appError(name, httpCode, description, isOperational) {
Error.call(this);
Error.captureStackTrace(this);
this.name = name;
//...other properties assigned here
};

appError.prototype.__proto__ = Error.prototype;

module.exports.appError = appError;

// client throwing an exception

if(user == null)
throw new appError(commonErrors.resourceNotFound, commonHTTPErrors.notFound, "further explanation", true)
"I don’t see the value in having lots of different types"

“…Personally, I don’t see the value in having lots of different types of error
objects – JavaScript, as a language, doesn’t seem to cater to Constructor-
based error-catching. As such, differentiating on an object property seems far
easier than differentiating on a Constructor type…”

From the blog,

Ben Nadel ranked 5 for the keywords “Node.js error object”
"A string is not an error"

“…passing a string instead of an error results in reduced interoperability between

modules. It breaks contracts with APIs that might be performing instanceof Error checks,
or that want to know more about the error. Error objects, as we’ll see, have very
interesting properties in modern JavaScript engines besides holding the message passed
to the constructor…”

From the blog,

devthought.com ranked 6 for the keywords “Node.js error object”
"Inheriting from Error doesn’t add too much value"

“…One problem that I have with the Error class is that is not so simple to extend. Of course,
you can inherit the class and create your own Error classes like HttpError, DbError, etc.
However, that takes time and doesn’t add too much value unless you are doing something
with types. Sometimes, you just want to add a message and keep the inner error, and
sometimes you might want to extend the error with parameters, and such…”

From the blog

machadogj
"All JavaScript and System errors raised by Node.js inherit from
Error"

“…All JavaScript and System errors raised by Node.js inherit from, or are instances of, the standard JavaScript
Error class and are guaranteed to provide at least the properties available on that class. A generic JavaScript
Error object that does not denote any specific circumstance of why the error occurred. Error objects capture a
“stack trace” detailing the point in the code at which the Error was instantiated, and may provide a text
description of the error. All errors generated by Node.js, including all System and JavaScript errors, will either be
instances of or inherit from, the Error class…”

From
Node.js official documentation
2.3 Distinguish operational vs programmer errors

TL;DR: Operational errors (e.g. API received an invalid input) refer to known

cases where the error impact is fully understood and can be handled
thoughtfully. On the other hand, programmer error (e.g. trying to read undefined
variable) refers to unknown code failures that dictate to gracefully restart the
application

Otherwise: You may always restart the application when an error appears, but
why let ~5000 online users down because of a minor, predicted, operational
error? the opposite is also not ideal – keeping the application up when an
unknown issue (programmer error) occurred might lead to an unpredicted
behavior. Differentiating the two allows acting tactfully and applying a balanced
approach based on the given context
2.3 Distinguish operational vs programmer errors

One Paragraph Explainer

Distinguishing the following two error types will minimize your app downtime
and helps avoid crazy bugs: Operational errors refer to situations where you
understand what happened and the impact of it – for example, a query to
some HTTP service failed due to connection problem. On the other hand,
programmer errors refer to cases where you have no idea why and
sometimes where an error came from – it might be some code that tried to
read an undefined value or DB connection pool that leaks memory.
Operational errors are relatively easy to handle – usually logging the error is
enough. Things become hairy when a programmer error pops up, the
application might be in an inconsistent state and there’s nothing better you
can do than to restart gracefully
Code Example – marking an error as operational (trusted)

// marking an error object as operational

var myError = new Error("How can I add new product when no value provided?");
myError.isOperational = true;

// or if you're using some centralized error factory (see other examples at the bullet "Use only the built-in Error object")
function appError(commonType, description, isOperational) {
Error.call(this);
Error.captureStackTrace(this);
this.commonType = commonType;
this.description = description;
this.isOperational = isOperational;
};

throw new appError(errorManagement.commonErrors.InvalidInput, "Describe here what happened", true);

"Programmer errors are bugs in the program"

“…The best way to recover from programmer errors is to crash immediately. You should run your
programs using a restarter that will automatically restart the program in the event of a crash. With a
restarter in place, crashing is the fastest way to restore reliable service in the face of a transient
programmer error…”

From the blog,

Joyent ranked 1 for the keywords “Node.js error handling”
"No safe way to leave without creating some undefined
brittle state"

“…By the very nature of how throw works in JavaScript, there is almost never any way to safely “pick
up where you left off”, without leaking references, or creating some other sort of undefined brittle
state. The safest way to respond to a thrown error is to shut down the process. Of course, in a
normal web server, you might have many connections open, and it is not reasonable to abruptly shut
those down because an error was triggered by someone else. The better approach is to send an
error response to the request that triggered the error while letting the others finish in their normal
time, and stop listening for new requests in that worker.”

From
Node.js official documentation
"Otherwise you risk the state of your application"

“…So, unless you really know what you are doing, you should perform a graceful
restart of your service after receiving an “uncaughtException” exception event.
Otherwise, you risk the state of your application, or that of 3rd party libraries to
become inconsistent, leading to all kinds of crazy bugs…”

From the blog,

debugable.com ranked 3 for the keywords “Node.js uncaught exception”
“There are three schools of thoughts on error handling”

…There are primarily three schools of thoughts on error handling:

1. Let the application crash and restart it.

2. Handle all possible errors and never crash.
3. A balanced approach between the two

From the blog:

JS Recipes
2.4 Handle errors centrally, not within an Express middleware

TL;DR: Error handling logic such as mail to admin and logging

should be encapsulated in a dedicated and centralized object
that all endpoints (e.g. Express middleware, cron jobs, unit-
testing) call when an error comes in

Otherwise: Not handling errors within a single place will lead to

code duplication and probably to improperly handled errors
 2.4 Handle errors centrally, not within an Express
middleware

One Paragraph Explainer

Without one dedicated object for error handling, greater are the chances of
important errors hiding under the radar due to improper handling. The error
handler object is responsible for making the error visible, for example by writing
to a well-formatted logger, sending events to some monitoring product or to an
admin directly via email. A typical error handling flow might be: Some module
throws an error -> API router catches the error -> it propagates the error to the
middleware (e.g. Express, KOA) who is responsible for catching errors -> a
centralized error handler is called -> the middleware is being told whether this
error is an untrusted error (not operational) so it can restart the app gracefully.
Note that it’s a common, yet wrong, practice to handle errors within Express
middleware – doing so will not cover errors that are thrown in non-web interfaces
Code Example – a typical error flow

// DAL layer, we don't handle errors here

DB.addDocument(newCustomer, (error, result) => {
if (error)
throw new Error("Great error explanation comes here", other useful parameters)
});

// API route code, we catch both sync and async errors and forward to the middleware
try {
customerService.addNew(req.body).then((result) => {
res.status(200).json(result);
}).catch((error) => {
next(error)
});
}
catch (error) {
next(error);
}

// Error handling middleware, we delegate the handling to the centralized error handler
app.use((err, req, res, next) => {
errorHandler.handleError(err).then((isOperationalError) => {
if (!isOperationalError)
next(err);
});
});
Code example – handling errors within a dedicated object

module.exports.handler = new errorHandler();

function errorHandler(){
this.handleError = function (error) {
return
logger.logError(err).then(sendMailToAdminIfCritical).then(saveInOpsQueueIfCritical).then(determineIfOperationalError)
;
}
}
Code Example – Anti Pattern: handling errors within the
middleware

// middleware handling the error directly, who will handle Cron jobs and testing errors?
app.use((err, req, res, next) => {
logger.logError(err);
if(err.severity == errors.high)
mailer.sendMail(configuration.adminMail, "Critical error occured", err);
if(!err.isOperational)
next(err);
});
"Sometimes lower levels can’t do anything useful except
propagate the error to their caller"

“…You may end up handling the same error at several levels of the stack. This happens when lower levels can’t do
anything useful except propagate the error to their caller, which propagates the error to its caller, and so on. Often,
only the top-level caller knows what the appropriate response is, whether that’s to retry the operation, report an
error to the user, or something else. But that doesn’t mean you should try to report all errors to a single top-level
callback, because that callback itself can’t know in what context the error occurred…”

From the blog

Joyent, ranked 1 for the keywords “Node.js error handling”
"Handling each err individually would result in tremendous
duplication"

“……In Hackathon Starter api.js controller alone, there are over 79

occurrences of error objects. Handling each err individually would result
in a tremendous amount of code duplication. The next best thing you
can do is to delegate all error handling logic to an Express
middleware…”

From the blog

JS Recipes ranked 17 for the keywords “Node.js error handling”
 "HTTP errors have no place in your database code"

“……You should set useful properties in error objects, but use such properties
consistently. And, don’t cross the streams: HTTP errors have no place in your
database code. Or for browser developers, Ajax errors have a place in the code
that talks to the server, but not code that processes Mustache templates…”

From the blog

Daily JS ranked 14 for the keywords “Node.js error handling”
2.5 Document API errors using Swagger

TL;DR: Let your API callers know which errors might come in return
so they can handle these thoughtfully without crashing. This is
usually done with REST API documentation frameworks like
Swagger

Otherwise: An API client might decide to crash and restart only

because he received back an error he couldn’t understand. Note:
the caller of your API might be you (very typical in a microservice
environment)
2.5 Document API errors using Swagger

One Paragraph Explainer

REST APIs return results using HTTP status codes, it’s absolutely
required for the API user to be aware not only about the API schema
but also about potential errors – the caller may then catch an error
and tactfully handle it. For example, your API documentation might
state in advance that HTTP status 409 is returned when the customer
name already exists (assuming the API register new users) so the
caller can correspondingly render the best UX for the given situation.
Swagger is a standard that defines the schema of API documentation
offering an eco-system of tools that allow creating documentation
easily online, see print screens below
"You have to tell your callers what errors can happen"

“We’ve talked about how to handle errors, but when you’re writing a
new function, how do you deliver errors to the code that called your
function? …If you don’t know what errors can happen or don’t know
what they mean, then your program cannot be correct except by
accident. So if you’re writing a new function, you have to tell your
callers what errors can happen and what they mean…”

From the blog

Joyent, ranked 1 for the keywords “Node.js logging”
2.6 Shut the process gracefully when a stranger comes to
town

TL;DR: When an unknown error occurs (a developer error, see best

practice number #3)- there is uncertainty about the application
healthiness. A common practice suggests restarting the process
carefully using a ‘restarter’ tool like Forever and PM2

Otherwise: When an unfamiliar exception is caught, some object

might be in a faulty state (e.g an event emitter which is used globally
and not firing events anymore due to some internal failure) and all
future requests might fail or behave crazily
2.6 Shut the process gracefully when a stranger comes to
town
One Paragraph Explainer
Somewhere within your code, an error handler object is responsible for
deciding how to proceed when an error is thrown – if the error is trusted (i.e.
operational error, see further explanation within best practice #3) then writing
to log file might be enough. Things get hairy if the error is not familiar – this
means that some component might be in a faulty state and all future
requests are subject to failure. For example, assuming a singleton, stateful
token issuer service that threw an exception and lost its state – from now it
might behave unexpectedly and cause all requests to fail. Under this
scenario, kill the process and use a ‘Restarter tool’ (like Forever, PM2, etc) to
start over with a clean slate.
Code example: deciding whether to crash

// Assuming developers mark known operational errors with error.isOperational=true, read best practice #3
process.on('uncaughtException', function(error) {
errorManagement.handler.handleError(error);
if(!errorManagement.handler.isTrustedError(error))
process.exit(1)
});

// centralized error handler encapsulates error-handling related logic

function errorHandler() {
this.handleError = function (error) {
return logger.logError(err)
.then(sendMailToAdminIfCritical)
.then(saveInOpsQueueIfCritical)
.then(determineIfOperationalError);
}

this.isTrustedError = function (error) {

return error.isOperational;
}
}
"The best way is to crash"

…The best way to recover from programmer errors is to crash immediately.

You should run your programs using a restarter that will automatically
restart the program in the event of a crash. With a restarter in place, crashing
is the fastest way to restore reliable service in the face of a transient
programmer error…

From the blog

Joyent
"There are three schools of thoughts on error handling"

…There are primarily three schools of thoughts on error handling:

1. Let the application crash and restart it.

2. Handle all possible errors and never crash.
3. A balanced approach between the two

From the blog:

JS Recipes
"No safe way to leave without creating some undefined brittle
state"

“…By the very nature of how throw works in JavaScript, there is almost never any way to safely “pick
up where you left off”, without leaking references, or creating some other sort of undefined brittle
state. The safest way to respond to a thrown error is to shut down the process. Of course, in a
normal web server, you might have many connections open, and it is not reasonable to abruptly shut
those down because an error was triggered by someone else. The better approach is to send an
error response to the request that triggered the error while letting the others finish in their normal
time, and stop listening for new requests in that worker.”

From
Node.js official documentation
 2.7 Use a mature logger to increase error visibility

TL;DR: A set of mature logging tools like Winston, Bunyan or Log4J,

will speed-up error discovery and understanding. So forget about
console.log

Otherwise: Skimming through console.logs or manually through messy

text file without querying tools or a decent log viewer might keep you
busy at work until late
2.7 Use a mature logger to increase error visibility

One Paragraph Explainer

We all love console.log but obviously, a reputable and persistent logger like 
Winston, Bunyan (highly popular) or Pino (the new kid in town which is focused
on performance) is mandatory for serious projects. A set of practices and tools
will help to reason about errors much quicker – (1) log frequently using
different levels (debug, info, error), (2) when logging, provide contextual
information as JSON objects, see example below. (3) watch and filter logs using
a log querying API (built-in in most loggers) or a log viewer software (4) Expose
and curate log statement for the operation team using operational intelligence
tools like Splunk
Code Example – Winston Logger in action

// your centralized logger object

var logger = new winston.Logger({
level: 'info',
transports: [
new (winston.transports.Console)(),
new (winston.transports.File)({ filename: 'somefile.log' })
]
});

// custom code somewhere using the logger

logger.log('info', 'Test Log Message with some parameter %s', 'some parameter', { anything: 'This is metadata' });
Code Example – Querying the log folder (searching for entries)

var options = {
from: new Date - 24 * 60 * 60 * 1000,
until: new Date,
limit: 10,
start: 0,
order: 'desc',
fields: ['message']
};

// Find items logged between today and yesterday.

winston.query(options, function (err, results) {
// execute callback with results
});
"Logger Requirements"

Lets identify a few requirements (for a logger):

1. Timestamp each log line. This one is pretty self-explanatory – you should be able to tell when
each log entry occurred.
2. Logging format should be easily digestible by humans as well as machines.
3. Allows for multiple configurable destination streams. For example, you might be writing trace
logs to one file but when an error is encountered, write to the same file, then into error file and
send an email at the same time…

From the blog

Strong Loop
2.8 Test error flows using your favorite test framework
TL;DR: Whether professional automated QA or plain manual
developer testing – Ensure that your code not only satisfies positive
scenario but also handle and return the right errors. Testing
frameworks like Mocha & Chai can handle this easily (see code
examples within the "Gist popup")

Otherwise: Without testing, whether automatically or manually, you

can’t rely on our code to return the right errors. Without meaningful
errors – there’s no error handling
2.8 Test error flows using your favorite test framework

One Paragraph Explainer

Testing ‘happy’ paths is no better than testing failures. Good testing
code coverage demands to test exceptional paths. Otherwise, there
is no trust that exceptions are indeed handled correctly. Every unit
testing framework, like Mocha & Chai, supports exception testing
(code examples below). If you find it tedious to test every inner
function and exception you may settle with testing only REST API
HTTP errors.
Code example: ensuring the right exception is thrown using
Mocha & Chai

describe("Facebook chat", () => {

it("Notifies on new chat message", () => {
var chatService = new chatService();
chatService.participants = getDisconnectedParticipants();
expect(chatService.sendMessage.bind({ message: "Hi" })).to.throw(ConnectionError);
});
});
Code example: ensuring API returns the right HTTP error code

it("Creates new Facebook group", function (done) {

var invalidGroupInfo = {};
httpRequest({
method: 'POST',
uri: "facebook.com/api/groups",
resolveWithFullResponse: true,
body: invalidGroupInfo,
json: true
}).then((response) => {
// if we were to execute the code in this block, no error was thrown in the operation above
}).catch(function (response) {
expect(400).to.equal(response.statusCode);
done();
});
});
2.9 Discover errors and downtime using APM products

TL;DR: Monitoring and performance products (a.k.a APM)

proactively gauge your codebase or API so they can
automagically highlight errors, crashes and slow parts that you
were missing

Otherwise: You might spend great effort on measuring API

performance and downtimes, probably you’ll never be aware
which are your slowest code parts under real-world scenario and
how these affect the UX
2.9 Discover errors and downtime using APM products

One Paragraph Explainer

Exception != Error. Traditional error handling assumes the existence of Exception
but application errors might come in the form of slow code paths, API downtime,
lack of computational resources and more. This is where APM products come in
handy as they allow to detect a wide variety of ‘burried’ issues proactively with a
minimal setup. Among the common features of APM products are for example
alerting when the HTTP API returns errors, detect when the API response time
drops below some threshold, detection of ‘code smells’, features to monitor
server resources, operational intelligence dashboard with IT metrics and many
other useful features. Most vendors offer a free plan.
 2.9 Discover errors and downtime using APM products

Wikipedia about APM

In the fields of information technology and systems management, Application

Performance Management (APM) is the monitoring and management of
performance and availability of software applications. APM strives to detect and
diagnose complex application performance problems to maintain an expected
level of service. APM is “the translation of IT metrics into business meaning ([i.e.]
value) Major products and segments
2.9 Discover errors and downtime using APM products
Understanding the APM marketplace
APM products constitute 3 major segments:
1.Website or API monitoring – external services that constantly monitor uptime and performance via HTTP requests.
Can be set up in few minutes. Following are few selected contenders: Pingdom, Uptime Robot, and New Relic
2.Code instrumentation – product family which requires embedding an agent within the application to use features like
slow code detection, exception statistics, performance monitoring and many more. Following are few selected
contenders: New Relic, App Dynamics
3.Operational intelligence dashboard – this line of products is focused on facilitating the ops team with metrics and
curated content that helps to easily stay on top of application performance. This usually involves aggregating multiple
sources of information (application logs, DB logs, servers log, etc) and upfront dashboard design work. Following are
few selected contenders: Datadog, Splunk, Zabbix
2.9 Discover errors and downtime using APM products

Example:
UpTimeRobot.Com –
Website monitoring
dashboard
2.9 Discover errors and downtime using APM products

Example:
AppDynamics.Com –
end to end monitoring
combined with code
instrumentation
 2.10 Catch unhandled promise rejections

TL;DR: Any exception thrown within a promise will get swallowed

and discarded unless a developer didn’t forget to explicitly handle.
Even if your code is subscribed to process.uncaughtException!
Overcome this by registering to the event
process.unhandledRejection

Otherwise: Your errors will get swallowed and leave no trace.

Nothing to worry about
 2.10 Catch unhandled promise rejections
One Paragraph Explainer
Typically, most of modern Node.js/Express application code runs within promises – whether
within the .then handler, a function callback or in a catch block. Surprisingly, unless a developer
remembered to add a .catch clause, errors thrown at these places are not handled by the
uncaughtException event-handler and disappear. Recent versions of Node added a warning
message when an unhandled rejection pops, though this might help to notice when things go
wrong but it's obviously not a proper error handling method. The straightforward solution is to
never forget adding .catch clauses within each promise chain call and redirect to a centralized
error handler. However, building your error handling strategy only on developer’s discipline is
somewhat fragile. Consequently, it’s highly recommended using a graceful fallback and
subscribe to process.on(‘unhandledRejection’, callback) – this will ensure that any promise error,
if not handled locally, will get its treatment.
Code example: these errors will not get caught by any error
handler (except unhandledRejection)

DAL.getUserById(1).then((johnSnow) => {
// this error will just vanish
if(johnSnow.isAlive == false)
throw new Error('ahhhh');
});
Code example: Catching unresolved and rejected promises

process.on('unhandledRejection', (reason, p) => {

// I just caught an unhandled promise rejection, since we already have fallback handler for unhandled errors (see below), let throw and let him handle that
throw reason;
});
process.on('uncaughtException', (error) => {
// I just received an error that was never handled, time to handle it and then decide whether a restart is needed
errorManagement.handler.handleError(error);
if (!errorManagement.handler.isTrustedError(error))
process.exit(1);
});
"If you can make a mistake, at some point you will"

Let’s test your understanding. Which of the following would

you expect to print an error to the console?

From the blog

James Nelson
"If you can make a mistake, at some point you will"

Promise.resolve(‘promised value’).then(() => {

throw new Error(‘error’);
});

Promise.reject(‘error value’).catch(() => {

throw new Error(‘error’);
});

new Promise((resolve, reject) => {

throw new Error(‘error’);
});
"If you can make a mistake, at some point you will"

I don’t know about you, but my answer is that I’d expect all of them to print an error.
However, the reality is that a number of modern JavaScript environments won’t
print errors for any of them.The problem with being human is that if you can make
a mistake, at some point you will. Keeping this in mind, it seems obvious that we
should design things in such a way that mistakes hurt as little as possible, and that
means handling errors by default, not discarding them.

From the blog

James Nelson
2.11 Fail fast, validate arguments using a dedicated library

TL;DR: This should be part of your Express best practices –

Assert API input to avoid nasty bugs that are much harder to track
later. The validation code is usually tedious unless you are using a
very cool helper library like Joi

Otherwise: Consider this – your function expects a numeric

argument “Discount” which the caller forgets to pass, later on,
your code checks if Discount!=0 (amount of allowed discount is
greater than zero), then it will allow the user to enjoy a discount.
OMG, what a nasty bug. Can you see it?
2.11 Fail fast, validate arguments using a
dedicated library
One Paragraph Explainer
We all know how checking arguments and failing fast is important to avoid
hidden bugs (see anti-pattern code example below). If not, read about explicit
programming and defensive programming. In reality, we tend to avoid it due to
the annoyance of coding it (e.g. think of validating hierarchical JSON object with
fields like email and dates) – libraries like Joi and Validator turn this tedious task
into a breeze.
2.11 Fail fast, validate arguments using a dedicated library

Wikipedia: Defensive Programming

Defensive programming is an approach to improve software and source code, in
terms of General quality – reducing the number of software bugs and problems.
Making the source code comprehensible – the source code should be readable
and understandable so it is approved in a code audit. Making the software
behave in a predictable manner despite unexpected inputs or user actions.
Code example: validating complex JSON input
using ‘Joi’

var memberSchema = Joi.object().keys({

password: Joi.string().regex(/^[a-zA-Z0-9]{3,30}$/),
birthyear: Joi.number().integer().min(1900).max(2013),
email: Joi.string().email()
});

function addNewMember(newMember)
{
// assertions come first
Joi.assert(newMember, memberSchema); //throws if validation fails
// other logic here
}
Anti-pattern: no validation yields nasty bugs

// if the discount is positive let's then redirect the user to pring his discount coupons
function redirectToPrintDiscount(httpResponse, member, discount)
{
if(discount != 0)
httpResponse.redirect(`/discountPrintView/${member.id}`);
}

redirectToPrintDiscount(httpResponse, someMember);
// forgot to pass the parameter discount, why the heck was the user redirected to the discount screen?
"You should throw these errors immediately"

“A degenerate case is where someone calls an asynchronous function but doesn’t

pass a callback. You should throw these errors immediately since the program is
broken and the best chance of debugging it involves getting at least a stack trace
and ideally a core file at the point of the error. To do this, we recommend
validating the types of all arguments at the start of the function.”

From the blog:

Joyent
Appendix

Part 3 – Code Style Practices

3.1 Use ESLint

TL;DR: ESLint is the de-facto standard for checking possible code errors and
fixing code style, not only to identify nitty-gritty spacing issues but also to
detect serious code anti-patterns like developers throwing errors without
classification. Though ESLint can automatically fix code styles, other tools like 
prettier and beautify are more powerful in formatting the fix and work in
conjunction with ESLint

Otherwise: Developers will focus on tedious spacing and line-width concerns

and time might be wasted overthinking about the project's code style
3.2 Node.js Specific Plugins

TL;DR: On top of ESLint standard rules that cover vanilla JS only, add
Node-specific plugins like eslint-plugin-node, eslint-plugin-mocha and 
eslint-plugin-node-security

Otherwise: Many faulty Node.js code patterns might escape under the

radar. For example, developers might require(variableAsPath) files with a
variable given as path which allows attackers to execute any JS script.
Node.js linters can detect such patterns and complain early
3.3 Start a Codeblock's Curly Braces on the Same Line

TL;DR: The opening curly braces of a code block

should be in the same line of the opening
statement
Code Example

// Do
function someFunction() {
// code block
}

// Avoid
function someFunction()
{
// code block
}
3.4 Don't Forget the Semicolon

TL;DR: While not unanimously agreed upon, it is still recommended

to put a semicolon at the end of each statement. This will make your
code more readable and explicit to other developers who read it

Otherwise: As seen in the previous section, JavaScript's interpreter

automatically adds a semicolon at the end of a statement if there
isn't one which might lead to some undesired results
3.5 Name Your Functions

TL;DR: Name all functions, including closures and callbacks. Avoid

anonymous functions. This is especially useful when profiling a
node app. Naming all functions will allow you to easily understand
what you're looking at when checking a memory snapshot

Otherwise: Debugging production issues using a core dump

(memory snapshot) might become challenging as you notice
significant memory consumption from anonymous functions
3.6 Naming conventions for variables, constants, functions and
classes

TL;DR: Use lowerCamelCase when naming constants, variables and functions

and UpperCamelCase (capital first letter as well) when naming classes. This will
help you to easily distinguish between plain variables/functions, and classes that
require instantiation. Use descriptive names, but try to keep them short

Otherwise: Javascript is the only language in the world which allows invoking a

constructor ("Class") directly without instantiating it first. Consequently, Classes
and function-constructors are differentiated by starting with UpperCamelCase
Code Example

// for class name we use UpperCamelCase

class SomeClassExample {}

// for const names we use the const keyword and lowerCamelCase

const config = {
key: 'value'
};

// for variables and functions names we use lowerCamelCase

let someVariableExample = 'value';
function doSomething() {}
3.7 Prefer const over let. Ditch the var

TL;DR: Using const means that once a variable is assigned, it cannot be

reassigned. Preferring const will help you to not be tempted to use the same
variable for different uses, and make your code clearer. If a variable needs to be
reassigned, in a for loop, for example, use let to declare it. Another important
aspect of let is that a variable declared using it is only available in the block scope
in which it was defined. var is function scoped, not block scoped, and shouldn't be
used in ES6 now that you have const and let at your disposal

Otherwise: Debugging becomes way more cumbersome when following a variable

that frequently changes
3.8 Requires come first, and not inside functions

TL;DR: Require modules at the beginning of each file, before and outside of any
functions. This simple best practice will not only help you easily and quickly tell
the dependencies of a file right at the top but also avoids a couple of potential
problems

Otherwise: Requires are run synchronously by Node.js. If they are called from

within a function, it may block other requests from being handled at a more
critical time. Also, if a required module or any of its own dependencies throw an
error and crash the server, it is best to find out about it as soon as possible, which
might not be the case if that module is required from within a function
3.9 Do Require on the folders, not directly on the files

TL;DR: When developing a module/library in a folder, place an index.js

file that exposes the module's internals so every consumer will pass
through it. This serves as an 'interface' to your module and eases
future changes without breaking the contract

Otherwise: Changing the internal structure of files or the signature may

break the interface with clients
Code example

// Do
module.exports.SMSProvider = require('./SMSProvider');
module.exports.SMSNumberResolver = require('./SMSNumberResolver');

// Avoid
module.exports.SMSProvider = require('./SMSProvider/SMSProvider.js');
module.exports.SMSNumberResolver = require('./SMSNumberResolver/SMSNumberResolver.js');
 3.10 Use the === operator

TL;DR: Prefer the strict equality operator === over the

weaker abstract equality operator ==. == will compare two
variables after converting them to a common type. There is
no type conversion in ===, and both variables must be of
the same type to be equal

Otherwise: Unequal variables might return true when

compared with the == operator
Code example

'' == '0' // false

0 == '' // true
0 == '0' // true

false == 'false' // false

false == '0' // true

false == undefined // false

false == null // false
null == undefined // true

' \t\r\n ' == 0 // true

3.11 Use Async Await, avoid callbacks

TL;DR: Node 8 LTS now has full support for Async-await. This is a new way of
dealing with asynchronous code which supersedes callbacks and promises.
Async-await is non-blocking, and it makes asynchronous code look synchronous.
The best gift you can give to your code is using async-await which provides a
much more compact and familiar code syntax like try-catch

Otherwise: Handling async errors in callback style is probably the fastest way to

hell - this style forces to check errors all over, deal with awkward code nesting and
make it difficult to reason about the code flow
3.12 Use Fat (=>) Arrow Functions

TL;DR: Though it's recommended to use async-await and avoid

function parameters when dealing with older API that accept promises
or callbacks - arrow functions make the code structure more compact
and keep the lexical context of the root function (i.e. 'this')

Otherwise: Longer code (in ES5 functions) is more prone to bugs and

cumbersome to read
Appendix

Part 4 – Testing And Overall

Quality Practices
4.1 At the very least, write API (component) testing
TL;DR: Most projects just don't have any automated testing due to
short timetables or often the 'testing project' run out of control and
being abandoned. For that reason, prioritize and start with API testing
which is the easiest to write and provide more coverage than unit
testing (you may even craft API tests without code using tools like 
Postman. Afterward, should you have more resources and time,
continue with advanced test types like unit testing, DB testing,
performance testing, etc

Otherwise: You may spend long days on writing unit tests to find out
that you got only 20% system coverage
4.2 Detect code issues with a linter
TL;DR: Use a code linter to check basic quality and detect anti-
patterns early. Run it before any test and add it as a pre-commit git-
hook to minimize the time needed to review and correct any issue.
Also check Section 3 on Code Style Practices

Otherwise: You may let pass some anti-pattern and possible

vulnerable code to your production environment.
4.3 Carefully choose your CI platform (Jenkins vs CircleCI vs
Travis vs Rest of the world)
TL;DR: Your continuous integration platform (CICD) will host all the quality tools
(e.g test, lint) so it should come with a vibrant ecosystem of plugins. Jenkins
 used to be the default for many projects as it has the biggest community along
with a very powerful platform at the price of complex setup that demands a
steep learning curve. Nowadays, it became much easier to set up a CI solution
using SaaS tools like CircleCI and others. These tools allow crafting a flexible CI
pipeline without the burden of managing the whole infrastructure. Eventually, it's
a trade-off between robustness and speed - choose your side carefully

Otherwise: Choosing some niche vendor might get you blocked once you need
some advanced customization. On the other hand, going with Jenkins might
burn precious time on infrastructure setup
4.3 Carefully choose your CI platform (Jenkins vs CircleCI vs
Travis vs Rest of the world)
One Paragraph Explainer
The CI world used to be the flexibility of Jenkins vs the simplicity of SaaS vendors.
The game is now changing as SaaS providers like CircleCI and Travis offer robust
solutions including Docker containers with minimum setup time while Jenkins tries
to compete on 'simplicity' segment as well. Though one can setup rich CI solution in
the cloud, should it required to control the finest details Jenkins is still the platform
of choice. The choice eventually boils down to which extent the CI process should be
customized: free and setup free cloud vendors allow to run custom shell commands,
custom docker images, adjust the workflow, run matrix builds and other rich
features. However, if controlling the infrastructure or programming the CI logic using
a formal programming language like Java is desired - Jenkins might still be the
choice. Otherwise, consider opting for the simple and setup free cloud option
Code Example – a typical cloud CI configuration. Single .yml file
and that's it
version: 2
jobs:
build:
docker:
- image: circleci/node:4.8.2
- image: mongo:3.4.4
steps:
- checkout
- run:
name: Install npm wee
command: npm install
test:
docker:
- image: circleci/node:4.8.2
- image: mongo:3.4.4
steps:
- checkout
- run:
name: Test
command: npm test
- run:
name: Generate code coverage
command: './node_modules/.bin/nyc report --reporter=text-lcov'
- store_artifacts:
path: coverage
prefix: coverage
4.3 Carefully choose your CI platform (Jenkins vs CircleCI vs
Travis vs Rest of the world)

Circle CI -
almost zero
setup cloud
CI
4.3 Carefully choose your CI platform (Jenkins vs CircleCI vs
Travis vs Rest of the world)

Jenkins -
sophisticated
and robust
CI
4.4 Constantly inspect for vulnerable dependencies
TL;DR: Even the most reputable dependencies such as
Express have known vulnerabilities. This can get easily
tamed using community and commercial tools such as 🔗 
nsp that can be invoked from your CI on every build

Otherwise: Keeping your code clean from vulnerabilities

without dedicated tools will require to constantly follow
online publications about new threats. Quite tedious
4.5 Tag your tests

TL;DR: Different tests must run on different scenarios: quick smoke, IO-less,

tests should run when a developer saves or commits a file, full end-to-end tests
usually run when a new pull request is submitted, etc. This can be achieved by
tagging tests with keywords like #cold #api #sanity so you can grep with your
testing harness and invoke the desired subset. For example, this is how you
would invoke only the sanity test group with Mocha: mocha --grep 'sanity‘

Otherwise: Running all the tests, including tests that perform dozens of DB

queries, any time a developer makes a small change can be extremely slow and
keeps developers away from running tests
4.6 Check your test coverage, it helps to identify wrong test
patterns

TL;DR: Code coverage tools like Istanbul/NYC are great for 3

reasons: it comes for free (no effort is required to benefit this
reports), it helps to identify a decrease in testing coverage, and last
but not least it highlights testing mismatches: by looking at colored
code coverage reports you may notice, for example, code areas that
are never tested like catch clauses (meaning that tests only invoke
the happy paths and not how the app behaves on errors). Set it to fail
builds if the coverage falls under a certain threshold

Otherwise: There won't be any automated metric telling you when a

large portion of your code is not covered by testing
 4.7 Inspect for outdated packages
TL;DR: Use your preferred tool (e.g. 'npm outdated' or npm
-check-updates to detect installed packages which are outdated, inject
this check into your CI pipeline and even make a build fail in a severe
scenario. For example, a severe scenario might be when an installed
package is 5 patch commits behind (e.g. local version is 1.3.1 and
repository version is 1.3.8) or it is tagged as deprecated by its author -
kill the build and prevent deploying this version

Otherwise: Your production will run packages that have been explicitly

tagged by their author as risky
4.8 Use docker-compose for e2e testing

TL;DR: End to end (e2e) testing which includes live data used to be the weakest
link of the CI process as it depends on multiple heavy services like DB. Docker-
compose turns this problem into a breeze by crafting production-like environment
using a simple text file and easy commands. It allows crafting all the dependent
services, DB and isolated network for e2e testing. Last but not least, it can keep a
stateless environment that is invoked before each test suite and dies right after

Otherwise: Without docker-compose teams must maintain a testing DB for each

testing environment including developers machines, keep all those DBs in sync
so test results won't vary across environments
Appendix

Part 5 – Going to Production

Practices
5.1. Monitoring!

TL;DR: Monitoring is a game of finding out issues before customers

do – obviously this should be assigned unprecedented importance.
The market is overwhelmed with offers thus consider starting with
defining the basic metrics you must follow (my suggestions inside),
then go over additional fancy features and choose the solution that
ticks all boxes. Click ‘The Gist’ below for an overview of the solutions

Otherwise: Failure === disappointed customers. Simple

5.1. Monitoring!
One Paragraph Explainer
At the very basic level, monitoring means you can easily identify when bad things happen at production. For
example, by getting notified by email or Slack. The challenge is to choose the right set of tools that will satisfy
your requirements without breaking your bank. May I suggest, start with defining the core set of metrics that
must be watched to ensure a healthy state – CPU, server RAM, Node process RAM (less than 1.4GB), the
number of errors in the last minute, number of process restarts, average response time. Then go over some
advanced features you might fancy and add to your wish list. Some examples of a luxury monitoring feature:
DB profiling, cross-service measuring (i.e. measure business transaction), front-end integration, expose raw
data to custom BI clients, Slack notifications and many others.
Achieving the advanced features demands lengthy setup or buying a commercial product such as Datadog,
NewRelic and alike. Unfortunately, achieving even the basics is not a walk in the park as some metrics are
hardware-related (CPU) and others live within the node process (internal errors) thus all the straightforward
tools require some additional setup. For example, cloud vendor monitoring solutions (e.g. AWS CloudWatch, 
Google StackDriver) will tell you immediately about the hardware metrics but not about the internal app
behavior. On the other end, Log-based solutions such as ElasticSearch lack the hardware view by default. The
solution is to augment your choice with missing metrics, for example, a popular choice is sending application
logs to Elastic stack and configure some additional agent (e.g. Beat) to share hardware-related information to
get the full picture.
5.1. Monitoring!

Monitoring
example: AWS
cloudwatch default
dashboard. Hard to
extract in-app
metrics
5.1. Monitoring!

Monitoring
example:
StackDriver
default
dashboard. Hard
to extract in-app
metrics
5.1. Monitoring!

Monitoring
example:
Grafana as the
UI layer that
visualizes raw
data
“What Other Bloggers Say”

“…We recommend you to watch these signals for all of your services:
Error Rate: Because errors are user facing and immediately affect
your customers. Response time: Because the latency directly affects
your customers and business. Throughput: The traffic helps you to
understand the context of increased error rates and the latency too.
Saturation: It tells how “full” your service is. If the CPU usage is 90%,
can your system handle more traffic? …”

From the blog 

Rising Stack
5.2. Increase transparency using smart logging

TL;DR: Logs can be a dumb warehouse of debug statements or the

enabler of a beautiful dashboard that tells the story of your app. Plan
your logging platform from day 1: how logs are collected, stored and
analyzed to ensure that the desired information (e.g. error rate,
following an entire transaction through services and servers, etc) can
really be extracted

Otherwise: You end-up with a black box that is hard to reason about,

then you start re-writing all logging statements to add additional
information
5.2. Increase transparency using smart logging
One Paragraph Explainer
Since you print out log statements anyway and you're obviously in a need of some interface that wraps up
production information where you can trace errors and core metrics (e.g. how many errors happen every hour
and which is your slowest API end-point) why not invest some moderate effort in a robust logging framework
that will tick all boxes? Achieving that requires a thoughtful decision on three steps:
1. smart logging – at the bare minimum you need to use a reputable logging library like Winston, Bunyan and
write meaningful information at each transaction start and end. Consider to also format log statements as
JSON and provide all the contextual properties (e.g. user id, operation type, etc) so that the operations team can
act on those fields. Include also a unique transaction ID at each log line, for more information refer to the bullet
below “Write transaction-id to log”. One last point to consider is also including an agent that logs the system
resource like memory and CPU like Elastic Beat.
2. smart aggregation – once you have comprehensive information on your servers file system, it’s time to
periodically push these to a system that aggregates, facilities and visualizes this data. The Elastic stack, for
example, is a popular and free choice that offers all the components to aggregate and visualize data. Many
commercial products provide similar functionality only they greatly cut down the setup time and require no
hosting.
3. smart visualization – now the information is aggregated and searchable, one can be satisfied only with the
power of easily searching the logs but this can go much further without coding or spending much effort. We can
now show important operational metrics like error rate, average CPU throughout the day, how many new users
opted-in in the last hour and any other metric that helps to govern and improve our app
5.2. Increase transparency using smart logging

Visualization
Example: Kibana
(part of the Elastic
stack) facilitates
advanced
searching on log
content
5.2. Increase transparency using smart logging

Visualization
Example: Kibana
(part of the
Elastic stack)
visualizes data
based on logs
“Logger Requirements”

Lets identify a few requirements (for a logger):

1. Timestamp each log line. This one is pretty self-explanatory – you should be able
to tell when each log entry occurred.
2. Logging format should be easily digestible by humans as well as machines.
3. Allows for multiple configurable destination streams. For example, you might be
writing trace logs to one file but when an error is encountered, write to the same file,
then into error file and send an email at the same time…

From the blog 

Strong Loop
5.3. Delegate anything possible (e.g. gzip, SSL) to a
reverse proxy

TL;DR: Node is awfully bad at doing CPU intensive tasks like

gzipping, SSL termination, etc. You should use ‘real’ middleware
services like nginx, HAproxy or cloud vendor services instead

Otherwise: Your poor single thread will stay busy doing

infrastructural tasks instead of dealing with your application core
and performance will degrade accordingly
5.3. Delegate anything possible (e.g. gzip, SSL) to a reverse
proxy
One Paragraph Explainer

It’s very tempting to cargo-cult Express and use its rich middleware offering for
networking related tasks like serving static files, gzip encoding, throttling
requests, SSL termination, etc. This is a performance kill due to its single
threaded model which will keep the CPU busy for long periods (Remember,
Node’s execution model is optimized for short tasks or async IO related tasks).
A better approach is to use a tool that expertise in networking tasks – the most
popular are nginx and HAproxy which are also used by the biggest cloud
vendors to lighten the incoming load on node.js processes.
Nginx Config Example – Using nginx to compress server
responses
# configure gzip compression
gzip on;
gzip_comp_level 6;
gzip_vary on;

# configure upstream
upstream myApplication {
server 127.0.0.1:3000;
server 127.0.0.1:3001;
keepalive 64;
}

#defining web server

server {
# configure server with ssl and error pages
listen 80;
listen 443 ssl;
ssl_certificate /some/location/sillyfacesociety.com.bundle.crt;
error_page 502 /errors/502.html;

# handling static content

location ~ ^/(images/|img/|javascript/|js/|css/|stylesheets/|flash/|media/|static/|robots.txt|humans.txt|favicon.ico)
{
root /usr/local/silly_face_society/node/public;
access_log off;
expires max;
}
“What Other Bloggers Say”

“…It’s very easy to fall into this trap – You see a package like Express and think “Awesome! Let’s get started”
– you code away and you’ve got an application that does what you want. This is excellent and, to be honest,
you’ve won a lot of the battle. However, you will lose the war if you upload your app to a server and have it
listen on your HTTP port because you’ve forgotten a very crucial thing: Node is not a web server. As soon as
any volume of traffic starts to hit your application, you’ll notice that things start to go wrong: connections
are dropped, assets stop being served or, at the very worst, your server crashes. What you’re doing is
attempting to have Node deal with all of the complicated things that a proven web server does really well.
Why reinvent the wheel? This is just for one request, for one image and bearing in mind this is the memory
that your application could be used for important stuff like reading a database or handling complicated
logic; why would you cripple your application for the sake of convenience?”

From the blog 

Mubaloo
“What Other Bloggers Say”

“Although express.js has built-in static file handling through some connect
middleware, you should never use it. Nginx can do a much better job of handling
static files and can prevent requests for non-dynamic content from clogging our
node processes…”

From the blog 

Argteam
5.4. Lock dependencies
TL;DR: Your code must be identical across all environments, but amazingly
NPM lets dependencies drift across environments by default – when you install
packages at various environments it tries to fetch packages’ latest patch
version. Overcome this by using NPM config files, .npmrc, that tell each
environment to save the exact (not the latest) version of each package.
Alternatively, for finer grain control use NPM” shrinkwrap”. *Update: as of NPM5,
dependencies are locked by default. The new package manager in town, Yarn,
also got us covered by default

Otherwise: QA will thoroughly test the code and approve a version that will
behave differently at production. Even worse, different servers at the same
production cluster might run different code
 5.4. Lock dependencies
One Paragraph Explainer

Your code depends on many external packages, let’s say it ‘requires’ and use momentjs-2.1.4, then
by default when you deploy to production NPM might fetch momentjs 2.1.5 which unfortunately
brings some new bugs to the table. Using NPM config files and the argument –save-exact=true
instructs NPM to refer to the exact same version that was installed so the next time you run npm
install (in production or within a Docker container you plan to ship forward for testing) the same
dependent version will be fetched. An alternative and popular approach is using a .shrinkwrap file
(easily generated using NPM) that states exactly which packages and versions should be installed
so no environment can get tempted to fetch newer versions than expected.

• Update: as of NPM 5, dependencies are locked automatically using .shrinkwrap. Yarn, an

emerging package manager, also locks down dependencies by default.
Code example: .npmrc file that instructs NPM to use exact
versions

// save this as .npmrc file on the project directory

save-exact:true
Code example: shrinkwrap.json file that distills the exact
dependency tree

{
"name": "A",
"dependencies": {
"B": {
"version": "0.0.1",
"dependencies": {
"C": {
"version": "0.1.0"
}
}
}
}
}
Code example: NPM 5 dependencies lock file – package.json

{
"name": "package-name",
"version": "1.0.0",
"lockfileVersion": 1,
"dependencies": {
"cacache": {
"version": "9.2.6",
"resolved": "https://registry.npmjs.org/cacache/-/cacache-9.2.6.tgz",
"integrity": "sha512-YK0Z5Np5t755edPL6gfdCeGxtU0rcW/DBhYhYVDckT+7AFkCCtedf2zru5NRbBLFk6e7Agi/RaqTOAfiaipUfg=="
},
"duplexify": {
"version": "3.5.0",
"resolved": "https://registry.npmjs.org/duplexify/-/duplexify-3.5.0.tgz",
"integrity": "sha1-GqdzAC4VeEV+nZ1KULDMquvL1gQ=",
"dependencies": {
"end-of-stream": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.0.0.tgz",
"integrity": "sha1-1FlucCc0qT5A6a+GQxnqvZn/Lw4="
}
}
}
}
}
5.5. Guard process uptime using the right tool
TL;DR: The process must go on and get restarted upon
failures. For simple scenarios, ‘restarter’ tools like PM2 might
be enough but in today ‘dockerized’ world – a cluster
management tools should be considered as well

Otherwise: Running dozens of instances without a clear

strategy and too many tools together (cluster management,
docker, PM2) might lead to a DevOps chaos
5.5. Guard process uptime using the right tool

One Paragraph Explainer

At the base level, Node processes must be guarded and restarted upon failures. Simply put, for
small apps and those who don’t use containers – tools like PM2 are perfect as they bring
simplicity, restarting capabilities and also rich integration with Node. Others with strong Linux
skills might use systemd and run Node as a service. Things get more interesting for apps that
use Docker or any container technology since those are usually accompanied by cluster
management and orchestration tools (e.g. AWS ECS, Kubernetes, etc) that deploy, monitor and
heal containers. Having all those rich cluster management features including container restart,
why mess up with other tools like PM2? There’s no bulletproof answer. There are good reasons
to keep PM2 within containers (mostly its containers specific version pm2-docker) as the first
guarding tier – it’s much faster to restart a process and provide Node-specific features like
flagging to the code when the hosting container asks to gracefully restart. Other might choose
to avoid unnecessary layers. To conclude this write-up, no solution suits them all and getting to
know the options is the important thing
 “What Other Bloggers Say”

“... In development, you started your app simply from the command line with node
server.js or something similar. But doing this in production is a recipe for disaster. If
the app crashes, it will be offline until you restart it. To ensure your app restarts if it
crashes, use a process manager. A process manager is a “container” for
applications that facilitate deployment, provides high availability, and enables you to
manage the application at runtime.”

From the 
Express Production Best Practices
“What Other Bloggers Say”

“... Understanding NodeJS Clustering in Docker-Land “Docker containers are

streamlined, lightweight virtual environments, designed to simplify processes to
their bare minimum. Processes that manage and coordinate their own resources
are no longer as valuable. Instead, management stacks like Kubernetes, Mesos,
and Cattle have popularized the concept that these resources should be
managed infrastructure-wide. CPU and memory resources are allocated by
“schedulers”, and network resources are managed by stack-provided load
balancers.”

From the Medium blog post 

Understanding Node Clustering

5.6. Utilize all CPU cores

TL;DR: At its basic form, a Node app runs on a single CPU core while all other are
left idling. It’s your duty to replicate the Node process and utilize all CPUs – For
small-medium apps you may use Node Cluster or PM2. For a larger app consider
replicating the process using some Docker cluster (e.g. K8S, ECS) or deployment
scripts that are based on Linux init system (e.g. systemd)

Otherwise: Your app will likely utilize only 25% of its available resources(!) or
even less. Note that a typical server has 4 CPU cores or more, naive deployment
of Node.js utilizes only 1 (even using PaaS services like AWS beanstalk!)
5.6. Utilize all CPU cores

One Paragraph Explainer

It might not come as a surprise that in its basic form, Node runs over a single
thread=single process=single CPU. Paying for beefy hardware with 4 or 8 CPU
and utilizing only one sounds crazy, right? The quickest solution which fits
medium sized apps is using Node’s Cluster module which in 10 lines of code
spawns a process for each logical core and route requests between the
processes in a round-robin style. Even better, use PM2 which sugarcoats the
clustering module with a simple interface and cool monitoring UI. While this
solution works well for traditional applications, it might fall short for applications
that require top-notch performance and robust DevOps flow. For those advanced
use cases, consider replicating the NODE process using custom deployment
script and balancing using a specialized tool such as nginx or use a container
engine such as AWS ECS or Kubernetees that have advanced features for
deployment and replication of processes.
5.6. Utilize all CPU cores

Comparison:
Balancing using
Node’s cluster vs
nginx
“What Other Bloggers Say”

“... The second approach, Node clusters, should, in theory, give the
best performance. In practice, however, distribution tends to be very
unbalanced due to operating system scheduler vagaries. Loads have
been observed where over 70% of all connections ended up in just two
processes, out of a total of eight ...”

From the 
Node.js documentation
“What Other Bloggers Say”

“... Clustering is made possible with Node’s cluster module. This enables a master
process to spawn worker processes and distribute incoming connections among
the workers. However, rather than using this module directly, it’s far better to use
one of the many tools out there that do it for you automatically; for example node-
pm or cluster-service ...”

From the blog 

StrongLoop
“What Other Bloggers Say”

“... Node cluster is simple to implement and configure, things are kept
inside Node’s realm without depending on other software. Just
remember your master process will work almost as much as your
worker processes and with a little less request rate than the other
solutions ...”

From the Medium post 

Node.js process load balance performance: comparing
cluster module, iptables, and Nginx
  5.7. Create a ‘maintenance endpoint’

TL;DR: Expose a set of system-related information, like

memory usage and REPL, etc in a secured API. Although it’s
highly recommended to rely on standard and battle-tests
tools, some valuable information and operations are easier
done using code

Otherwise: You’ll find that you’re performing many “diagnostic

deploys” – shipping code to production only to extract some
information for diagnostic purposes
  5.7. Create a ‘maintenance endpoint’

One Paragraph Explainer

A maintenance endpoint is a highly secure HTTP API that is part of the app code and its purpose is to be
used by the ops/production team to monitor and expose maintenance functionality. For example, it can
return a heap dump (memory snapshot) of the process, report whether there are some memory leaks and
even allow to execute REPL commands directly. This endpoint is needed where the conventional DevOps
tools (monitoring products, logs, etc) fail to gather some specific type of information or you choose not to
buy/install such tools. The golden rule is using professional and external tools for monitoring and
maintaining the production, these are usually more robust and accurate. That said, there are likely to be
cases where the generic tools will fail to extract information that is specific to Node or to your app – for
example, should you wish to generate a memory snapshot at the moment GC completed a cycle – few npm
libraries will be glad to perform this for you but popular monitoring tools will likely miss this functionality. It is
important to keep this endpoint private and accessibly only by admins because it can become a target of a
DDOS attack.
Code example: generating a heap dump via code

const heapdump = require('heapdump');

// Check if request is authorized

function isAuthorized(req) {
// ...
}

router.get('/ops/heapdump', (req, res, next) => {

if (!isAuthorized(req)) {
return res.status(403).send('You are not authorized!');
}

logger.info('About to generate heapdump');

heapdump.writeSnapshot((err, filename) => {

console.log('heapdump file is ready to be sent to the caller', filename);
fs.readFile(filename, "utf-8", (err, data) => {
res.end(data);
});
});
});
5.8. Discover errors and downtime using APM products

TL;DR: Monitoring and performance products (a.k.a APM) proactively

gauge codebase and API so they can auto-magically go beyond
traditional monitoring and measure the overall user-experience
across services and tiers. For example, some APM products can
highlight a transaction that loads too slow on the end-users side
while suggesting the root cause

Otherwise: You might spend great effort on measuring API

performance and downtimes, probably you’ll never be aware which is
your slowest code parts under real-world scenario and how these
affects the UX
5.8. Discover errors and downtime using APM products

One Paragraph Explainer

APM (application performance monitoring) refers to a family of products that aims
to monitor application performance from end to end, also from the customer
perspective. While traditional monitoring solutions focus on Exceptions and
standalone technical metrics (e.g. error tracking, slow server endpoints, etc), in the
real world our app might create disappointed users without any code exceptions,
for example, if some middleware service performed real slow. APM products
measure the user experience from end to end, for example, given a system that
encompasses frontend UI and multiple distributed services – some APM products
can tell how fast a transaction that spans multiple tiers last. It can tell whether the
user experience is solid and point to the problem. This attractive offering comes
with a relatively high price tag hence it’s recommended for large-scale and complex
products that require going beyond straightforward monitoring.
5.8. Discover errors and downtime using APM products

APM example – a
commercial product
that visualizes cross-
service app
performance
5.8. Discover errors and downtime using APM products

APM example – a
commercial product
that emphasizes the
user experience
score
5.8. Discover errors and downtime using APM products

APM example – a
commercial
product that
highlights slow
code paths
5.9. Make your code production-ready

TL;DR: Code with the end in mind, plan for production from

day 1. This sounds a bit vague so I’ve compiled a few
development tips that are closely related to production
maintenance (click Gist below)

Otherwise: A world champion IT/DevOps guy won’t save a

system that is badly written
5.9. Make your code production-ready

One Paragraph Explainer

Following is a list of development tips that greatly affect the production maintenance and stability:
•The twelve-factor guide – Get familiar with the Twelve factors guide
•Be stateless – Save no data locally on a specific web server (see separate bullet – ‘Be Stateless’)
•Cache – Utilize cache heavily, yet never fail because of cache mismatch
•Test memory – gauge memory usage and leaks as part your development flow, tools such as ‘memwatch’ can greatly
facilitate this task
•Name functions – Minimize the usage of anonymous functions (i.e. inline callback) as a typical memory profiler will
provide memory usage per method name
•Use CI tools – Use CI tool to detect failures before sending to production. For example, use ESLint to detect reference
errors and undefined variables. Use –trace-sync-io to identify code that uses synchronous APIs (instead of the async
version)
•Log wisely – Include in each log statement contextual information, hopefully in JSON format so log aggregators tools such
as Elastic can search upon those properties (see separate bullet – ‘Increase visibility using smart logs’). Also, include
transaction-id that identifies each request and allows to correlate lines that describe the same transaction (see separate
bullet – ‘Include Transaction-ID’)
•Error management – Error handling is the Achilles’ heel of Node.js production sites – many Node processes are crashing
because of minor errors while others hang on alive in a faulty state instead of crashing. Setting your error handling strategy
is absolutely critical, read here my error handling best practices
5.10. Measure and guard the memory usage

TL;DR: Node.js has controversial relationships with memory: the v8

engine has soft limits on memory usage (1.4GB) and there are known
paths to leaks memory in Node’s code – thus watching Node’s
process memory is a must. In small apps, you may gauge memory
periodically using shell commands but in medium-large app consider
baking your memory watch into a robust monitoring system

Otherwise: Your process memory might leak a hundred megabytes a

day like how it happened at Walmart
 5.10. Measure and guard the memory usage

One Paragraph Explainer

In a perfect world, a web developer shouldn’t deal with memory leaks. In reality,
memory issues are a known Node’s gotcha one must be aware of. Above all,
memory usage must be monitored constantly. In the development and small
production sites, you may gauge manually using Linux commands or NPM tools
and libraries like node-inspector and memwatch. The main drawback of this
manual activities is that they require a human being actively monitoring – for
serious production sites, it’s absolutely vital to use robust monitoring tools e.g.
(AWS CloudWatch, DataDog or any similar proactive system) that alerts when a
leak happens. There are also few development guidelines to prevent leaks: avoid
storing data on the global level, use streams for data with dynamic size, limit
variables scope using let and const.
“What Other Bloggers Say”

... ”As we already learned, in Node.js JavaScript is compiled to native code by V8.
The resulting native data structures don’t have much to do with their original
representation and are solely managed by V8. This means that we cannot actively
allocate or deallocate memory in JavaScript. V8 uses a well-known mechanism
called garbage collection to address this problem.”

From the blog 

Dyntrace
“What Other Bloggers Say”

... “Although this example leads to obvious results the process is

always the same: Create heap dumps with some time and a fair
amount of memory allocation in between Compare a few dumps to find
out what’s growing”

From the blog 

Dyntrace
“What Other Bloggers Say”

... “fault, Node.js will try to use about 1.5GBs of memory, which has to be capped when running on
systems with less memory. This is the expected behavior as garbage collection is a very costly
operation. The solution for it was adding an extra parameter to the Node.js process: node –
max_old_space_size=400 server.js –production ” “Why is garbage collection expensive? The V8
JavaScript engine employs a stop-the-world garbage collector mechanism. In practice, it means
that the program stops execution while garbage collection is in progress.”

From the blog 

Dyntrace
5.11. Get your frontend assets out of Node
TL;DR: Serve frontend content using dedicated middleware
(nginx, S3, CDN) because Node performance really gets hurt
when dealing with many static files due to its single threaded
model

Otherwise: Your single Node thread will be busy streaming

hundreds of html/images/angular/react files instead of
allocating all its resources for the task it was born for –
serving dynamic content
 5.11. Get your frontend assets out of Node

One Paragraph Explainer

In a classic web app the backend serves the frontend/graphics to the browser, a very common approach in the
Node’s world is to use Express static middleware for streamlining static files to the client. BUT – Node is not a
typical webapp as it utilizes a single thread that is not optimized to serve many files at once. Instead, consider
using a reverse proxy (e.g. nginx, HAProxy), cloud storage or CDN (e.g. AWS S3, Azure Blob Storage, etc) that
utilizes many optimizations for this task and gain much better throughput. For example, specialized middleware
like nginx embodies direct hooks between the file system and the network card and uses a multi-threaded
approach to minimize intervention among multiple requests.
Your optimal solution might wear one of the following forms:
1.Using a reverse proxy – your static files will be located right next to your Node application, only requests to
the static files folder will be served by a proxy that sits in front of your Node app such as nginx. Using this
approach, your Node app is responsible deploying the static files but not to serve them. Your frontend’s
colleague will love this approach as it prevents cross-origin-requests from the frontend.
2.Cloud storage – your static files will NOT be part of your Node app content, they will be uploaded to services
like AWS S3, Azure BlobStorage, or other similar services that were born for this mission. Using this approach,
your Node app is not responsible deploying the static files neither to serve them, hence a complete decoupling
is drawn between Node and the Frontend which is anyway handled by different teams.
Configuration example: typical nginx configuration for serving
static files

# configure gzip compression

gzip on;
keepalive 64;

# defining web server

server {
listen 80;
listen 443 ssl;

# handle static content

location ~ ^/(images/|img/|javascript/|js/|css/|stylesheets/|flash/|media/|static/|robots.txt|humans.txt|
favicon.ico) {
root /usr/local/silly_face_society/node/public;
access_log off;
expires max;
}
5.11. Get your frontend assets out of Node

“…In development, you can use res.sendFile() to serve static files. But don’t do
this in production, because this function has to read from the file system for
every file request, so it will encounter significant latency and affect the overall
performance of the app. Note that res.sendFile() is not implemented with the
sendfile system call, which would make it far more efficient. Instead, use serve-
static middleware (or something equivalent), that is optimized for serving files
for Express apps. An even better option is to use a reverse proxy to serve static
files; see Use a reverse proxy for more information…”

From the blog 

StrongLoop
5.13. Use tools that automatically detect vulnerabilities

TL;DR: Even the most reputable dependencies such as Express have

known vulnerabilities (from time to time) that can put a system at
risk. This can get easily tamed using community and commercial
tools that constantly check for vulnerabilities and warn (locally or at
GitHub), some can even patch them immediately

Otherwise: Otherwise: Keeping your code clean from vulnerabilities

without dedicated tools will require to constantly follow online
publications about new threats. Quite tedious
5.13. Use tools that automatically detect vulnerabilities

One Paragraph Explainer

Modern Node applications have tens and sometimes hundreds of
dependencies. If any of the dependencies you use has a known
security vulnerability your app is vulnerable as well. The following
tools automatically check for known security vulnerabilities in
your dependencies:
•nsp - Node Security Project
•snyk - Continuously find & fix vulnerabilities in your dependencies
“What Other Bloggers Say”

“...Using to manage your application’s dependencies is powerful and convenient. But the
packages that you use may contain critical security vulnerabilities that could also affect your
application. The security of your app is only as strong as the “weakest link” in your dependencies.
Fortunately, there are two helpful tools you can use to ensure the third-party packages you use:
and requireSafe. These two tools do largely the same thing, so using both might be overkill, but
“better safe than sorry” are words to live by when it comes to security...”

From the blog

StrongLoop 
 5.14. Assign ‘TransactionId’ to each log statement

TL;DR: Assign the same identifier, transaction-id: {some value}, to each

log entry within a single request. Then when inspecting errors in logs,
easily conclude what happened before and after. Unfortunately, this is
not easy to achieve in Node due to its async nature, see code
examples inside

Otherwise: Looking at a production error log without the context –

what happened before – makes it much harder and slower to reason
about the issue
 5.14. Assign ‘TransactionId’ to each log statement

One Paragraph Explainer

A typical log is a warehouse of entries from all components and requests. Upon detection of some suspicious
line or error, it becomes hairy to match other lines that belong to the same specific flow (e.g. the user “John”
tried to buy something). This becomes even more critical and challenging in a microservice environment when
a request/transaction might span across multiple computers. Address this by assigning a unique transaction
identifier value to all the entries from the same request so when detecting one line one can copy the id and
search for every line that has similar transaction Id. However, achieving this In Node is not straightforward as
a single thread is used to serve all requests –consider using a library that that can group data on the request
level – see code example on the next slide. When calling other microservice, pass the transaction Id using an
HTTP header like “x-transaction-id” to keep the same context.
Code example: typical Express configuration

// when receiving a new request, start a new isolated context and set a transaction Id. The following example is using the NPM library
continuation-local-storage to isolate requests

const { createNamespace } = require('continuation-local-storage');

var session = createNamespace('my session');

router.get('/:id', (req, res, next) => {

session.set('transactionId', 'some unique GUID');
someService.getById(req.params.id);
logger.info('Starting now to get something by Id');
});

// Now any other service or components can have access to the contextual, per-request, data
class someService {
getById(id) {
logger.info(“Starting to get something by Id”);
// other logic comes here
}
}

// The logger can now append the transaction-id to each entry so that entries from the same request will have the same value
class logger {
info (message)
{console.log(`${message} ${session.get('transactionId')}`);}
}
5.15. Set NODE_ENV=production

TL;DR: Set the environment variable NODE_ENV to ‘production’ or

‘development’ to flag whether production optimizations should get
activated – many NPM packages determining the current
environment and optimize their code for production

Otherwise: Omitting this simple property might greatly degrade

performance. For example, when using Express for server-side
rendering omitting NODE_ENV makes the slower by a factor of three!
5.15. Set NODE_ENV=production

One Paragraph Explainer

Process environment variables is a set of key-value pairs made available to any
running program, usually for configuration purposes. Though any variables can be
used, Node encourages the convention of using a variable called NODE_ENV to flag
whether we’re in production right now. This determination allows components to
provide better diagnostics during development, for example by disabling caching or
emitting verbose log statements. Any modern deployment tool – Chef, Puppet,
CloudFormation, others – support setting environment variables during
deployment
Code example: Setting and reading the NODE_ENV environment
variable

// Setting environment variables in bash before starting the node

process
$ NODE_ENV=development
$ node

// Reading the environment variable using code

if (process.env.NODE_ENV === “production”)
useCaching = true;
“What Other Bloggers Say”

“...In Node.js there is a convention to use a variable called NODE_ENV to set the
current mode. We see that it, in fact, reads NODE_ENV and defaults to
‘development’ if it isn’t set. We clearly see that by setting NODE_ENV to production
the number of requests Node.js can handle jumps by around two-thirds while the
CPU usage even drops slightly. Let me emphasize this: Setting NODE_ENV to
production makes your application 3 times faster!”

From the blog 

dynatrace
5.16. Design automated, atomic and zero-downtime
deployments

TL;DR: Researches show that teams who perform many deployments – lowers

the probability of severe production issues. Fast and automated deployments that
don’t require risky manual steps and service downtime significantly improves the
deployment process. You should probably achieve that using Docker combined
with CI tools as they became the industry standard for streamlined deployment

Otherwise: Long deployments -> production down time & human-related error ->
team unconfident and in making deployment -> less deployments and features
5.17. Use an LTS release of Node.js

TL;DR: Ensure you are using an LTS version of Node.js to receive

critical bug fixes, security updates and performance improvements

Otherwise: Newly discovered bugs or vulnerabilities could be used to

exploit an application running in production, and your application may
become unsupported by various modules and harder to maintain
5.17. Use an LTS release of Node.js

One Paragraph Explainer

Ensure you are using an LTS(Long Term Support) version of Node.js in
production to receive critical bug fixes, security updates and performance
improvements.
LTS versions of Node.js are supported for at least 18 months and are indicated
by even version numbers (e.g. 4, 6, 8). They're best for production since the LTS
release line is focussed on stability and security, whereas the 'Current' release
line has a shorter lifespan and more frequent updates to the code. Changes to
LTS versions are limited to bug fixes for stability, security updates, possible
npm updates, documentation updates and certain performance improvements
that can be demonstrated to not break existing applications.
Appendix

Part 6 – Security Best Practices

6.1. Embrace linter security rules

TL;DR: Make use of security linter plugins such as eslint

-plugin-security to enforce a policy for secure code (e.g. no use of
eval, require with variables, etc).

Otherwise: Developers in the project may not follow consistent

code security practices, leading to vulnerabilities being introduced,
or sensitive secrets committed into remote repositories.
6.1. Embrace linter security rules

One Paragraph Explainer

Security plugins for ESLint such as eslint-plugin-security offer code
security checks based on a number of known vulnerabilities, such as
unsafe regex, unsafe use of eval(), and non literal filenames being used
when accessing the file system within an application. The use of git
hooks such as pre-git allows to further enforce any rules on source
control before they get distributed to remotes, one of which can be to
check that no secrets were added to source control.
“What Other Bloggers Say”

“Linting doesn’t have to be just a tool to enforce pedantic rules about whitespace,
semicolons or eval statements. ESLint provides a powerful framework for
eliminating a wide variety of potentially dangerous patterns in your code (regular
expressions, input validation, and so on). I think it provides a powerful new tool
that’s worthy of consideration by security-conscious JavaScript developers.”

From the blog 

Adam Baldwin
6.2. Limit concurrent requests using a balancer or a
middleware

TL;DR: Implement rate limiting using an external service such

as NGINX, or if that is not possible use a rate limiting
middleware within the application

Otherwise: An application could be subject to an attack

resulting in a denial of service where real users receive
degraded service, or an unavailable application.
6.2. Limit concurrent requests using a balancer or a
middleware

One Paragraph Explainer

Rate limiting should be implemented in your application to protect a Node.js

application from being overwhelmed by too many requests at the same time.
Rate limiting is a task best performed with a service designed for this task,
such as nginx, however it is also possible with application middleware such
as express-rate-limiter.
Code example: Express rate limiting middleware for certain
routes

var RateLimit = require('express-rate-limit');

// important if behind a proxy to ensure client IP is passed to req.ip
app.enable('trust proxy');

var apiLimiter = new RateLimit({

windowMs: 15*60*1000, // 15 minutes
max: 100,
delayMs: 0 // disabled
});

// only apply to requests that begin with /user/

app.use('/user/', apiLimiter);
“What Other Bloggers Say”

“Rate limiting can be used for security purposes, for example to slow down
brute‑force password‑guessing attacks. It can help protect against DDoS attacks
by limiting the incoming request rate to a value typical for real users, and (with
logging) identify the targeted URLs. More generally, it is used to protect upstream
application servers from being overwhelmed by too many user requests at the
same time.”

From the

NGINX blog
6.3 Extract secrets from config files or use NPM package that
encrypts them

TL;DR: Never store plain-text secrets in configuration files or source code.

Instead, make use of secrets management systems like Vault, Docker Secrets,
or inject secrets as environment variables accessible to an application. As a last
result, storing secrets in source control must be encrypted, and managed
(rolling keys, expiring, auditing, etc). Make use of pre-commit/push hooks to
check for accidental commit of secrets.

Otherwise: Source control for even private repositories, can mistakenly be made

public, at which point all secret has been exposed outside. Access to source
control for an external party will inadvertently provide access to related systems
(database, apis, etc).
6.4. Prevent SQL/noSQL injection with ORM/ODM or any other
DAL packages
TL;DR: To prevent SQL/noSQL injection attacks always make use of
an ORM/ODM or a database library that escapes or supports named
or indexed parameter binding, and takes care of validating user input
for expected types. Never use JS template strings or string
concatenation to inject values into queries.

Otherwise: Un-validated user input could lead to operator injection

when working with MongoDB for noSQL, and unescaped use of
proper ORM/ODM will allow easy SQL injection attacks.
6.5. Collection of common generic security best practices (15
items)

TL;DR: There are many built-in and external ways to secure

your Node.js applications starting from simple steps like
running your server with ssl/tls enabled to more advanced
features.
Common Node.js security best practices

The common security guidelines section contains best

practices that are standardized in many frameworks and
conventions, running an application with ssl/tls for example
should be a common guideline and convention followed in
every setup to achieve great security benefits.
Use SSL/TLS to encrypt the client-server connection

TL;DR: In the times of free SSL/TLS certificates and easy configuration of

those, you do no longer have to weigh advantages and disadvantages of using
a secure server because the advantages such as security, support of modern
technology and trust clearly outweigh the disadvantages like minimal overhead
compared to pure http.

Otherwise: Attackers could perform man-in-the-middle attacks, spy on your

users' behaviour and perform even more malicious actions when the
connection is unencrypted
Using HTTPS to encrypt the client-server connection

One Paragraph Explainer

Using services like the Let'sEncrypt certificate authority providing free

ssl/tls certificates, you can easily obtain a certificate to secure your
application. Node.js frameworks like Express (based on the core https
module) support ssl/tls based servers easily, so the configuration can
be done in a few lines of additional code.

You can also configure ssl/tls on your reverse proxy pointing to your
application for example using nginx or HAProxy.
Code Example – Enabling SSL/TLS using the Express
framework

const express = require('express');

const https = require('https');
const app = express();
const options = {
// The path should be changed accordingly to your setup
cert: fs.readFileSync('./sslcert/fullchain.pem'),
key: fs.readFileSync('./sslcert/privkey.pem')
};
https.createServer(options, app).listen(443);
Comparing secret values and hashes securely

TL;DR: When comparing secret values or hashes like HMAC digests, you should
use the crypto.timingSafeEqual(a, b) function Node provides out of the box since
Node.js v6.6.0. This method compares two given objects and keeps comparing
even if data does not match. The default equality comparison methods would
simply return after a character mismatch, allowing timing attacks based on the
operation length.

Otherwise: Using default equality comparison operators you might expose critical
information based on the time taken to compare two objects
Generating random strings using Node.js

TL;DR: Using a custom-built function generating pseudo-random strings for

tokens and other security-sensitive use cases might actually not be as random as
you think, rendering your application vulnerable to cryptographic attacks. When
you have to generate secure random strings, use the crypto.RandomBytes(size,
[callback]) function using available entropy provided by the system.

Otherwise: When generating pseudo-random strings without cryptographically

secure methods, attackers might predict and reproduce the generated results,
rendering your application insecure
6.6. Adjust the response HTTP headers for enhanced security

TL;DR: Your application should be using secure headers to prevent

attackers from using common attacks like cross-site scripting (XSS),
clickjacking and other malicious attacks. These can be configured
easily using modules like helmet.

Otherwise: Attackers could perform attacks on your application's

users, leading to insecurity
6.6. Adjust the response HTTP headers for enhanced security

One Paragraph Explainer

There are security-related headers used to secure your application

further. The most important headers are listed below. You can also
visit the sites linked at the bottom of this page to get more
information on this topic. You can easily set these headers using the 
Helmet module for express (Helmet for koa).
6.7. Constantly and automatically inspect for vulnerable
dependencies

TL;DR: With the npm ecosystem it is common to have many

dependencies for a project. Dependencies should always be kept in
check as new vulnerabilities are found. Use tools like nsp or snyk to
track, monitor and patch vulnerable dependencies. Integrate these
tools with your CI setup so you catch a vulnerable dependency before
it makes it to production.

Otherwise: An attacker could detect your web framework and attack

with all it's known vulnerabilities.
6.7. Constantly and automatically inspect for vulnerable
dependencies
One Paragraph Explainer
The majority of Node.js applications rely heavily on a large number of third
party modules from NPM or Yarn package managers due to ease and speed of
development. However, the downside to this benefit is the security risks of
including unknown vulnerabilities into your application, which is a risk
recognised by it's place in the OWASP top critical web application security
risks list.
There are a number of tools available to help identify third party packages in
Node.js applications which have been identified as vulnerable by the
community to mitigate the risk of introducing them into your project. These
can be used periodically from CLI tools or included as part of your
application's build process.
6.8. Avoid using the Node.js Crypto library for passwords, use
Bcrypt

TL;DR: Passwords or secrets (API keys) should be stored using a

secure hash function like bcrypt, that should be a preferred choice over
its JavaScript implementation due to performance and security
reasons.

Otherwise: Passwords or secrets that are persisted without using a

secure hash function are vulnerable to brute forcing and dictionary
attacks that will lead to their disclosure eventually.
6.8. Avoid using the Node.js Crypto library for passwords, use
Bcrypt
One Paragraph Explainer
When storing user passwords, using an adaptive hashing algorithm such as
bcrypt, offered by the bcrypt npm module is recommended as opposed to using
the native Node.js Crypto module. Math.random() should also never be used for
as part of any password or token generation due to it's predictability.

The bcrypt module or similar should be used as opposed to the JavaScript

implementation, as when using bcrypt, a number of 'rounds' can be specified in
order to provide a secure hash. This sets the work factor or the number of
'rounds' the data is processed for, and more hashing rounds leads to more secure
hash (although this at the cost of CPU time). The introduction of hashing rounds
means that the brute force factor is significantly reduced, as password crackers
are slowed down increasing the time required to generate one attempt.
Code Example – using promises to catch
errors

// asynchronously generate a secure password using 10 hashing rounds

bcrypt.hash('myPassword', 10, function(err, hash) {
// Store secure hash in user record
});

// compare a provided password input with saved hash

bcrypt.compare('somePassword', hash, function(err, match) {
if(match) {
// Passwords match
} else {
// Passwords don't match
}
});
“What Other Bloggers Say”

“... it’s not just using the right hashing algorithm. I’ve talked extensively about how
the right tool also includes the necessary ingredient of “time” as part of the
password hashing algorithm and what it means for the attacker who’s trying to
crack passwords through brute-force.”

From the blog by

Max McCarty
 6.9. Use middleware that sanitizes input and output

TL;DR: Here we will write about sanitizing/escaping/encoding both the input and

output. The top frontend libraries handle output encoding well, but nevertheless
you should always make sure to output encode your user generated data in the
correct context, and can make use of libraries such as node-esapi or escape-html.

Otherwise: Failure to encode user generated when outputting it can result in XSS,

Log Injection or other vulnerabilities. Input validation should always be performed
to confirm one is working with expected types and data properties (length, range,
etc).
6.10. Validate the incoming JSON schemas

TL;DR: Validate the incoming requests` body payload and ensure it qualifies

the expectations, fail fast if it doesn't. To avoid tedious validation coding
within each route you may use lightweight JSON-based validation schemas
such as jsonschema or JOI

Otherwise: Your generosity and permissive approach greatly increases the

attack surface and encourage the attacker to try out many inputs until it finds
some combination that crashes the application
6.10. Validate the incoming JSON schemas

One Paragraph Explainer

Validation is about being very explicit on what payload our app is willing to accept and failing fast
should the input deviates from the expectations. This minimizes an attackers surface who can no
longer try out payloads with a different structure, values and length. Practically it prevents attacks
like DDOS (code is unlikely to fail when the input is well defined) and Insecure Deserialization
(JSON contain no surprises ). Though validation can be coded or rely upon classes and types (TS,
ES6 classes) the community seems to increasingly like JSON-based schemas as these allow
declaring complex rules without coding and share the expectations with the frontend. JSON-
schema is an emerging standard that is supported by many NPM libraries and tools (e.g. 
jsonschema, POSTMAN), JOI is also highly popular with sweet syntax. Typically JSON syntax can't
cover all validation scenario and custom code or pre-baked validation frameworks like validator.js
 come in handy. Regardless of the chosen syntax, ensure to run the validation as early as possible -
For example, by using Express middleware that validates the request body before the request is
passed to the route handler
Example - JSON-Schema validation rules

{
"$schema": "http://json-schema.org/draft-06/schema#",
"title": "Product",
"description": "A product from Acme's catalog",
"type": "object",
"properties": {
"name": {
"description": "Name of the product",
"type": "string"
},
"price": {
"type": "number",
"exclusiveMinimum": 0
}
},
"required": ["id", "name", "price"]
}
Example - Validating an entity using JSON-Schema

const JSONValidator = require("jsonschema").Validator;

class Product {

validate() {
var v = new JSONValidator();

return v.validate(this, schema);

}

static get schema() {

//define JSON-Schema, see example above
}
}
Example - Usage of middleware validator

// The validator is a generic middleware that gets the entity it should validate and takes care to return
// HTTP status 400 (Bad Request) should the body payload validation fail
router.post("/" , **validator(Product.validate)**, async (req, res, next) => {
// route handling code goes here
});
“What Other Bloggers Say”

“Validating user input is one of the most important things to do when it comes to
the security of your application. Failing to do it correctly can open up your
application and users to a wide range of attacks, including command injection,
SQL injection or stored cross-site scripting.”

From the blog

Nemeth Gergley
6.11. Support blacklisting JWT tokens

TL;DR: When using JWT(for example, with Passport.js), a

custom implementation for blacklisting untrusted or invalid
tokens is required to revoke the token's access, due to the
completely stateless approach of JWT.

Otherwise: Expired, or misplaced tokens could be used

maliciously by a third party to access an application
impersonating the owner of the token.
6.13. Run Node.js as non-root user

TL;DR: There are common scenario where nodejs runs as a root user with
unlimited permissions. For example this is the default behaviour in Docker
containers. It's recommended to create a non-root user and always run the
process on this user behalf by invoking the container with the flag "-u username“

Otherwise: An attacker who manages to run a script on the server gets unlimited
power over the local machine (e.g. change iptable and re-route traffic to his
server)
6.13. Run Node.js as non-root user

One Paragraph Explainer

According to the 'Principle of least privilege' a user/process must be able to
access only the necessary information and resources. Granting root access to
an attacker opens a whole new world of malicious ideas like routing traffic to
other servers. In practice, most Node.js apps don't need root access and don't
run with such privileges. However, there are two common scenarios that might
push to root usage: (1) to gain access to privilege port (e.g. port 80) Node.js
must run as root (2) Docker containers by default run as root(!). It's
recommended for Node.js webservers to listen on non-privilege ports and rely on
a reverse-proxy like nginx to redirect incoming traffic from port 80 to your
Node.js application. When building a Docker image, highly secured apps should
run the container with an alternate non-root user. Most Docker clusters (e.g.
Swarm, Kubernetes) allow setting the security context declaratively
Code example - Building a Docker image as non-root

FROM node:latest
COPY package.json .
RUN npm install
COPY . .
EXPOSE 3000
USER node
CMD ["node", "server.js"]
 “What Other Bloggers Say”

“By default, Docker runs container as root which inside of the container can pose
as a security issue. You would want to run the container as an unprivileged user
wherever possible. The node images provide the node user for such purpose. The
Docker Image can then be run with the node user in the following way:”

From the Repository docker-node by 

eyalzek
6.14. Limit payload size using a reverse-proxy or a middleware

TL;DR: DOS attacks of many varieties can be mitigated by limiting

the body size of incoming requests to your web application.

Otherwise: your application will have to deal with large requests,

unable to process the other important work it has to accomplish,
leading to performance implications and vulnerability towards DOS
attacks
6.14. Limit payload size using a reverse-proxy or a middleware

One Paragraph Explainer

Parsing request bodies, for example JSON-encoded payloads, is a performance-

heavy operation, especially with larger requests. When handling incoming requests
in your web application, you should limit the size of their respective payloads.
Incoming requests with unlimited body/payload sizes can lead to your application
performing badly or crashing due to a denial-of-service outage or other unwanted
side-effects. Many popular middleware-solutions for parsing request bodies, such
as the already-included body-parser package for express, expose options to limit
the sizes of request payloads, making it easy for developers to implement this
functionality. You can also integrate a request body size limit in your reverse-
proxy/web server software if supported. Below are examples for limiting request
sizes using express and/or nginx.
Example code for express

const express = require('express');

const app = express();

app.use(express.json({ limit: '300kb' })); // body-parser defaults to a body size limit of 100kb

// Request with json body

app.post('/json', (req, res) => {

// Check if request payload content-type matches json, because body-parser does not check for content types
if (!req.is('json')) {
return res.sendStatus(415); // -> Unsupported media type if request doesn't have JSON body
}

res.send('Hooray, it worked!');
});

app.listen(3000, () => console.log('Example app listening on port 3000!'));

Example configuration for nginx

http {
...
# Limit the body size for ALL incoming requests to 1 MB
client_max_body_size 1m;
}

server {
...
# Limit the body size for incoming requests to this specific server block to 1 MB
client_max_body_size 1m;
}

location /upload {
...
# Limit the body size for incoming requests to this route to 1 MB
client_max_body_size 1m;
}
 6.15. Avoid JS eval statements

TL;DR: eval may be used to evaluate javascript code during run-time, but it is
not just a performance concern but also an important security concern due to
malicious javascript code that may be sourced from user input. Another
language feature that should be avoided is new Function constructor.
setTimeout and setInterval should never be passed dynamic javascript code
either.

Otherwise: Malicious javascript code finds a way into a text passed into eval or
other real-time evaluating javascript language functions, it will gain complete
access to javascript permissions on the page, often manifesting as an XSS
attack.
 6.15. Avoid JS eval statements

One Paragraph Explainer

eval(), setTimeout() and setInterval() are global functions often used in Node.js
which accept a string parameter representing a JavaScript expression, statement,
or sequence of statements. The security concern of using these functions are the
possibility that untrusted user input that might find it's way into code execution
which can lead to server compromise, as evaluating user inputted code essentially
allows an attacker to perform any actions that you can. It is suggested to refactor
code to not rely on the use of these functions where user input could be passed to
the function and executed.
Code example

// example of malicious code which an attacker was able to input

userInput = "require('child_process').spawn('rm', ['-rf', '/'])";
// malicious code executed
eval(userInput);
 “What Other Bloggers Say”

“The eval() function is perhaps of the most frowned upon JavaScript pieces from a
security perspective. It parses a JavaScript string as text, and executes it as if it
were a JavaScript code. Mixing that with untrusted user input that might find it’s
way to eval() is a recipe for disaster that can end up with server compromise.”

From the Essential Node.js Security book by

Liran Tal
6.16. Prevent malicious RegEx from overloading your single
thread execution

TL;DR: Regular Expressions, while being handy, pose a real threat to

JavaScript applications at large, and the Node.js platform in
particular due to the fact that they require CPU cycles to compute a
pattern test. Use the aforementioned validator.js package to validate
data instead of writing your own, or make use of safe-regex to
detect vulnerable regex patterns.

Otherwise: Poorly written regexes could be susceptible to Regular

Expressions DoS attacks that will block the event loop completely.
For example, the popular moment package was found vulnerable in
Nov 2017.
6.16. Prevent malicious RegEx from overloading your single
thread execution
One Paragraph Explainer

The risk that is inherent with the use of Regular Expressions is the
computational resources that require to parse text and match a given
pattern. For the Node.js platform, where a single-thread event-loop is
dominant, a CPU-bound operation like resolving a regular expression
pattern will render the application unresponsive. Avoid regex when
possible or defer the task to a dedicated library like validator.js, or 
safe-regex to check if the regex pattern is safe.
Some OWASP examples for vulnerable regex patterns:
•(a|aa)+
•([a-zA-Z]+)*
Code Example – using promises to catch
errors

var saferegex = require('safe-regex');

var emailRegex = /^([a-zA-Z0-9])(([\-.]|[_]+)?([a-zA-Z0-9]+))*(@){1}[a-z0-9]+[.]{1}(([a-z]
{2,3})|([a-z]{2,3}[.]{1}[a-z]{2,3}))$/;

// should output false because the emailRegex is vulnerable to redos attacks

console.log(saferegex(emailRegex));

// instead of the regex pattern, use validator:

var validator = require('validator');
console.log(validator.isEmail('[email protected]'));
 “What Other Bloggers Say”

“Often, programmers will use RegEx to validate that an input received from a user
conforms to an expected condition. A vulnerable Regular Expression is known as
one which applies repetition to a repeating capturing group, and where the string
to match is composed of a suffix of a valid matching pattern plus characters that
aren't matching the capturing group.”

From the Essential Node.js Security book by

Liran Tal
6.17. Avoid module loading require(someVariable) using a
variable
TL;DR: Avoid requiring/importing another file with a path that was given as
parameter due to the concern that it could have originated from user input.
This rule can be extended for accessing files in general (i.e. fs.readFile()) or
other sensitive resource access with dynamic variables originating from user
input.

Otherwise: Malicious user input could find its way to a parameter that is used
to require tampered files, for example a previously uploaded file on the
filesystem, or access already existing system files.
Code example

// insecure, as helperPath variable may have been modified by user input

const uploadHelpers = require(helperPath);

// secure
const uploadHelpers = require('./helpers/upload');
6.18. Run unsafe code in a sandbox

TL;DR: Here we will write about the need to run

untrusted code, for example third-party components.
In that case, we'd like to isolate our code from
weakness of the 3rd party code. We can achieve that
by running our code in a sandbox
 6.19. Take extra care when working with child processes

TL;DR: Avoid using child_process.exec when possible or

validate and sanitize input to mitigate shell injection attacks.
Prefer using child_process.execFile which by definition will
only execute a single command with a set of attributes and
will not allow shell parameter expansion.

Otherwise: Naive use of child_process.exec could result in

remote command execution or shell injection attacks due to
malicious user input passed to an unsanitized system
command.
6.20. Hide error details from client (e.g. default Express
behaviour)

TL;DR: Use the default Express error handler, or ensure that custom

error handlers avoid exposing error details to the client when running
an application outside of development.

Otherwise: Sensitive application details such as server filepaths,

third party modules in use, and other internal workings of the
application which could be exploited by an attacker from
information found in a stack trace.
6.20. Hide error details from client (e.g. default Express
behaviour)
One Paragraph Explainer

Exposing application error details to the client in production should be avoided due to risk
of exposing sensitive application details such as server filepaths, third party modules in
use, and other internal workings of the application which could be exploited by an attacker.
Express comes with a built-in error handler, which takes care of any errors that might be
encountered in the app. This default error-handling middleware function is added at the
end of the middleware function stack. If you pass an error to next() and you do not handle
it in a custom error handler, it will be handled by the built-in Express error handler; the error
will be written to the client with the stack trace. This behaviour will be true when
NODE_ENV is set to development, however when NODE_ENV is set to production, the stack
trace is not written, only the HTTP response code.
Code example

// production error handler

// no stacktraces leaked to user
app.use(function(err, req, res, next) {
res.status(err.status || 500);
res.render('error', {
message: err.message,
error: {}
});
});
6.21. Modify session middleware settings
TL;DR: Using the default settings for session middleware can be
expose your app to module and framework specific hijacking
attacks in a similar way to the X-Powered-By header.

Otherwise: Cookies could be sent over insecure connections, and

an attacker can use session identification to identify the
underlying framework of the web application, as well as module-
specific vulnerabilities.
6.21. Modify session middleware settings

One Paragraph Explainer

Many popular session middlewares do not apply best practice/secure cookie settings out of the box.
Tweaking these settings from the defaults offers more protection for both the user and the application, by
reducing the threat of attacks such as session hijacking and session identification.

The most common setting left to default is the session name - in express-session this is connect.sid. An
attacker can use this information to identify the underlying framework of the web application as well as
module specific vulnerabilities. Changing this value to something other than the default will make it harder
to determine what session mechanism is being used.

Also in express-session, the option cookie.secure is set to false as the default value. Changing this to true
will restrict transport of the cookie to https only which provides safety from man in the middle type attacks
Code example: Setting secure cookie settings

// using the express session middleware

app.use(session({
secret: 'youruniquesecret', // secret string used in the signing of the session ID that is stored in the cookie
name: 'youruniquename', // set a unique name to remove the default connect.sid
cookie: {
httpOnly: true, // minimize risk of XSS attacks by restricting the client from reading the cookie
secure: true, // only send cookie over https
maxAge: 60000*60*24 // set cookie expiry length in ms
}
}));
 “What Other Bloggers Say”

“...Express has default cookie settings that aren’t highly secure. They can be
manually tightened to enhance security - for both an application and its user.*”
From the NodeSource blog:

From the

NodeSource blog
6.21. Modify session middleware
settings

One Paragraph Explainer

Avoid requiring/importing another file with a path that was

given as parameter due to the concern that it could have
originated from user input. This rule can be extended for
accessing files in general (i.e. fs.readFile()) or other sensitive
resource access with dynamic variables originating from user
input.
6.21. Modify session middleware settings

One Paragraph Explainer

Avoid requiring/importing another file with a path that was

given as parameter due to the concern that it could have
originated from user input. This rule can be extended for
accessing files in general (i.e. fs.readFile()) or other sensitive
resource access with dynamic variables originating from user
input.
End