Node - Js Design Patterns Sample Chapter

Fr
ee
Sa
pl
In this package, you will find:
The author biography

A preview chapter from the book, Chapter 1 "Node.js Design Fundamentals"
A synopsis of the books content
More information on Node.js Design Patterns
About the Author

Mario Casciaro is a software engineer and technical lead with a passion for open source.
He began programming with a Commodore 64 when he was 12, and grew up with Pascal
and Visual Basic. His programming skills evolved by experimenting with x86 assembly
language, C, C++, PHP, and Java. His relentless work on side projects led him to
discover JavaScript and Node.js, which quickly became his new passion.
In his professional career, he worked with IBM for several yearsfirst in Rome and then
in the Dublin Software Lab. At IBM, Mario worked on products for brands such as
Tivoli, Cognos, and Collaboration Solutions, using a variety of technologies from C to
PHP and Java. He then plunged into the adventurous world of start ups to work full time
on Node.js projects. He ended up working in a lighthouse, at D4H Technologies, where
he led the development of a real-time platform to manage emergency operations.
Acknowledgments
This book is the result of an amazing amount of work, knowledge, and perseverance from
many people. A big thanks goes to the entire team at Packt who made this book a reality,
from the editors to the project coordinator; in particular, I would like to thank Rebecca
You and Sriram Neelakantan for their guidance and patience during the toughest parts of
the writing process. Kudos to Alan Shaw, Joel Purra, and Afshin Mehrabani who
dedicated their time and expertise to reviewing the technical content of the book; every
comment and advice was really invaluable in bringing this work up to production quality.
This book would not exist without the efforts of so many people who made Node.js a
realityfrom the big players, who continuously inspired us, to the contributor of the
smallest module.
In these months, I also learned that a book is only possible with the support and
encouragement of all the people around you. My gratitude goes to all my friends who
heard the phrase "Today I can't, I have to work on the book" too many times; thanks to
Christophe Guillou, Zbigniew Mrowinski, Ryan Gallagher, Natalia Lopez, Ruizhi Wang,
and Davide Lionello for still talking to me. Thanks to the D4H crew, for their inspiration
and understanding, and for giving me the chance to work on a first-class product.
Thanks to all the friends back in Italy, to the legendary company of Taverna and
Centrale, to the lads of Lido Marini for always giving me a great time, laughing and
having fun. I'm sorry for not being present in the past few months.
Thanks to my Mom and Dad, and to my brother and sister, for their unconditional love
and support.
At last, you have to know that there is another person who wrote this book along with
me, that's Miriam, my girlfriend, who walked throughout this long journey with me and
supported me night and day, regardless of how difficult it was. There's nothing more one
could wish for. I send all my love and gratitude to her. Many adventures await us.
Node.js Design Patterns

Node.js is considered by many as a game-changerthe biggest shift of the decade in web
development. It is loved not just for its technical capabilities, but also for the change of
paradigm that it introduced in web development.
First, Node.js applications are written in JavaScript, the language of the web, the only
programming language supported natively by a majority of web browsers. This aspect
only enables scenarios such as single-language application stacks and sharing of code
between the server and the client. Node.js itself is contributing to the rise and evolution
of the JavaScript language. People realize that using JavaScript on the server is not as bad
as it is in the browser, and they will soon start to love it for its pragmatism and for its
hybrid nature, half way between object-oriented and functional programming.
The second revolutionizing factor is its single-threaded, asynchronous architecture.
Besides obvious advantages from a performance and scalability point of view, this
characteristic changed the way developers approach concurrency and parallelism.
Mutexes are replaced by queues, threads by callbacks and events, and synchronization
by causality.
The last and most important aspect of Node.js lies in its ecosystem: the npm package
manager, its constantly growing database of modules, its enthusiastic and helpful
community, and most importantly, its very own culture based on simplicity, pragmatism,
and extreme modularity.
However, because of these peculiarities, Node.js development gives you a very different
feel compared to the other server-side platforms, and any developer new to this paradigm
will often feel unsure about how to tackle even the most common design and coding
problem effectively. Common questions include: "How do I organize my code?", "What's
the best way to design this?", "How can I make my application more modular?", "How do
I handle a set of asynchronous calls effectively?", "How can I make sure that my
application will not collapse while it grows?", or more simply "What's the right way of
doing this?" Fortunately, Node.js has become a mature-enough platform and most of
these questions can now be easily answered with a design pattern, a proven coding
technique, or a recommended practice. The aim of this book is to guide you through this
emerging world of patterns, techniques, and practices, showing you what the proven
solutions to the common problems are and teaching you how to use them as the starting
point to building the solution to your particular problem.
By reading this book, you will learn the following:
The "Node way". How to use the right point of view when approaching a Node.js
design problem. You will learn, for example, how different traditional design
patterns look in Node.js, or how to design modules that do only one thing.
A set of patterns to solve common Node.js design and coding problems.

You will be presented with a "Swiss army knife" of patterns, ready-touse in order to efficiently solve your everyday development and design
problems.
How to write modular and efficient Node.js applications. You will gain
an understanding of the basic building blocks and principles of writing
large and well-organized Node.js applications and you will be able to
apply these principles to novel problems that don't fall within the scope
of existing patterns.
Throughout the book, you will be presented with several real-life libraries and
technologies, such as LevelDb, Redis, RabbitMQ, ZMQ, Express, and many others. They
will be used to demonstrate a pattern or technique, and besides making the example more
useful, these will also give you great exposure to the Node.js ecosystem and its set
of solutions.
Whether you use or plan to use Node.js for your work, your side project, or for an open
source project, recognizing and using well-known patterns and techniques will allow you
to use a common language when sharing your code and design, and on top of that, it will
help you get a better understanding about the future of Node.js and how to make your
own contributions a part of it.
What This Book Covers

Chapter 1, Node.js Design Fundamentals, serves as an introduction to the world of
Node.js application design by showing the patterns at the core of the platform itself. It
covers the reactor pattern, the callback pattern, the module pattern, and the
observer pattern.
Chapter 2, Asynchronous Control Flow Patterns, introduces a set of patterns and
techniques for efficiently handling asynchronous control flow in Node.js. This chapter
teaches you how to mitigate the "callback hell" problem using plain JavaScript, the async
library, Promises, and Generators.
Chapter 3, Coding with Streams, dives deeply into one of the most important patterns in
Node.js: Streams. It shows you how to process data with transform streams and how to
combine them into different layouts.
Chapter 4, Design Patterns, deals with a controversial topic: traditional design patterns in
Node.js. It covers the most popular conventional design patterns and shows you how
unconventional they might look in Node.js.
Chapter 5, Wiring Modules, analyzes the different solutions for linking the modules of an
application together. In this chapter, you will learn design patterns such as Dependency
Injection and Service locator.
Chapter 6, Recipes, takes a problem-solution approach to show you how some common
coding and design challenges can be solved with ready-to-use solutions.
Chapter 7, Scalability and Architectural Patterns, teaches you the basic techniques and
patterns for scaling a Node.js application.
Chapter 8, Messaging and Integration Patterns, presents the most important messaging
patterns, teaching you how to build and integrate complex distributed systems using
ZMQ and AMQP.
Node.js Design
Fundamentals
Some principles and design patterns literally define the Node.js platform and its
ecosystem; the most peculiar ones are probably its asynchronous nature and its
programming style that makes heavy use of callbacks. However, there are other
fundamental components that characterize the platform; for example, its module
system, which allows multiple versions of the same dependency to coexist in an
application, and the observer pattern, implemented by the EventEmitter class,
which perfectly complements callbacks when dealing with asynchronous code.
It's therefore important that we first dive into these fundamental principles and
patterns, not only for writing correct code, but also to be able to take effective
design decisions when it comes to solving bigger and more complex problems.
Another aspect that characterizes Node.js is its philosophy. Approaching Node.js
is in fact way more than simply learning a new technology; it's also embracing a
culture and a community. We will see how this greatly influences the way we
design our applications and components, and the way they interact with those
created by the community.
In this chapter, we will learn the following topics:
The Node.js philosophy, the "Node way"
The reactor pattern: the mechanism at the heart of the Node.js

asynchronous architecture
The Node.js callback pattern and its set of conventions
The module system and its patterns: the fundamental mechanisms for
organizing code in Node.js
The observer pattern and its Node.js incarnation: the EventEmitter class
Node.js Design Fundamentals
The Node.js philosophy

Every platform has its own philosophya set of principles and guidelines that are
generally accepted by the community, an ideology of doing things that influences
the evolution of a platform, and how applications are developed and designed.
Some of these principles arise from the technology itself, some of them are enabled
by its ecosystem, some are just trends in the community, and others are evolutions
of different ideologies. In Node.js, some of these principles come directly from its
creator, Ryan Dahl, from all the people who contributed to the core, from charismatic
figures in the community, and some of the principles are inherited from the
JavaScript culture or are influenced by the Unix philosophy.
None of these rules are imposed and they should always be applied with common
sense; however, they can prove to be tremendously useful when we are looking for a
source of inspiration while designing our programs.
You can find an extensive list of software development philosophies
in Wikipedia at http://en.wikipedia.org/wiki/List_of_
software_development_philosophies.
Small core
The Node.js core itself has its foundations built on a few principles; one of
these is, having the smallest set of functionality, leaving the rest to the so-called
userland (or userspace), the ecosystem of modules living outside the core. This
principle has an enormous impact on the Node.js culture, as it gives freedom to the
community to experiment and iterate fast on a broader set of solutions within the
scope of the userland modules, instead of being imposed with one slowly evolving
solution that is built into the more tightly controlled and stable core. Keeping the
core set of functionality to the bare minimum then, not only becomes convenient
in terms of maintainability, but also in terms of the positive cultural impact that it
brings on the evolution of the entire ecosystem.
Small modules
Node.js uses the concept of module as a fundamental mean to structure the code of a
program. It is the brick for creating applications and reusable libraries called packages
(a package is also frequently referred to as just module; since, usually it has one
single module as an entry point). In Node.js, one of the most evangelized principles
is to design small modules, not only in terms of code size, but most importantly in
terms of scope.
[8]
Chapter 1
This principle has its roots in the Unix philosophy, particularly in two of its
precepts, which are as follows:
"Small is beautiful."
"Make each program do one thing well."
Node.js brought these concepts to a whole new level. Along with the help of npm,
the official package manager, Node.js helps solving the dependency hell problem
by making sure that each installed package will have its own separate set of
dependencies, thus enabling a program to depend on a lot of packages without
incurring in conflicts. The Node way, in fact, involves extreme levels of reusability,
whereby applications are composed of a high number of small, well-focused
dependencies. While this can be considered unpractical or even totally unfeasible
in other platforms, in Node.js this practice is encouraged. As a consequence, it is
not rare to find npm packages containing less than 100 lines of code or exposing
only one single function.
Besides the clear advantage in terms of reusability, a small module is also considered
to be the following:
Easier to understand and use

Simpler to test and maintain
Perfect to share with the browser
Having smaller and more focused modules empowers everyone to share or reuse
even the smallest piece of code; it's the Don't Repeat Yourself (DRY) principle
applied at a whole new level.
Small surface area

In addition to being small in size and scope, Node.js modules usually also have the
characteristic of exposing only a minimal set of functionality. The main advantage
here is an increased usability of the API, which means that the API becomes clearer
to use and is less exposed to erroneous usage. Most of the time, in fact, the user of a
component is interested only in a very limited and focused set of features, without
the need to extend its functionality or tap into more advanced aspects.
In Node.js, a very common pattern for defining modules is to expose only one piece
of functionality, such as a function or a constructor, while letting more advanced
aspects or secondary features become properties of the exported function or
constructor. This helps the user to identify what is important and what is secondary.
It is not rare to find modules that expose only one function and nothing else, for the
simple fact that it provides a single, unmistakably clear entry point.
[9]
Another characteristic of many Node.js modules is the fact that they are created to
be used rather than extended. Locking down the internals of a module by forbidding
any possibility of an extension might sound inflexible, but it actually has the
advantage of reducing the use cases, simplifying its implementation, facilitating its
maintenance, and increasing its usability.
Simplicity and pragmatism

Have you ever heard of the Keep It Simple, Stupid (KISS) principle? Or the
famous quote:
"Simplicity is the ultimate sophistication."
Leonardo da Vinci
Richard P. Gabriel, a prominent computer scientist coined the term worse is better to
describe the model, whereby less and simpler functionality is a good design choice
for software. In his essay, The rise of worse is better, he says:
"The design must be simple, both in implementation and interface. It is more
important for the implementation to be simple than the interface. Simplicity is
the most important consideration in a design."
Designing a simple, as opposed to a perfect, feature-full software, is a good
practice for several reasons: it takes less effort to implement, allows faster shipping
with less resources, is easier to adapt, and is easier to maintain and understand.
These factors foster the community contributions and allow the software itself to
grow and improve.
In Node.js, this principle is also enabled by JavaScript, which is a very pragmatic
language. It's not rare, in fact, to see simple functions, closures, and object literals
replacing complex class hierarchies. Pure object-oriented designs often try to
replicate the real world using the mathematical terms of a computer system without
considering the imperfection and the complexity of the real world itself. The truth is
that our software is always an approximation of the reality and we would probably
have more success in trying to get something working sooner and with reasonable
complexity, instead of trying to create a near-perfect software with a huge effort and
tons of code to maintain.
Throughout this book, we will see this principle in action many times. For example,
a considerable number of traditional design patterns, such as Singleton or Decorator
can have a trivial, even if sometimes not foolproof implementation and we will see
how an uncomplicated, practical approach most of the time is preferred to a pure,
flawless design.
[ 10 ]
Chapter 1
The reactor pattern

In this section, we will analyze the reactor pattern, which is the heart of the Node.js
asynchronous nature. We will go through the main concepts behind the pattern,
such as the single-threaded architecture and the non-blocking I/O, and we will
see how this creates the foundation for the entire Node.js platform.
I/O is slow
I/O is definitely the slowest among the fundamental operations of a computer.
Accessing the RAM is in the order of nanoseconds (10e-9 seconds), while accessing
data on the disk or the network is in the order of milliseconds (10e-3 seconds). For
the bandwidth, it is the same story; RAM has a transfer rate consistently in the order
of GB/s, while disk and network varies from MB/s to, optimistically, GB/s. I/O
is usually not expensive in terms of CPU, but it adds a delay between the moment
the request is sent and the moment the operation completes. On top of that, we also
have to consider the human factor; often, the input of an application comes from a
real person, for example, the click of a button or a message sent in a real-time chat
application, so the speed and frequency of I/O don't depend only on technical
aspects, and they can be many orders of magnitude slower than the disk or network.
Blocking I/O
In traditional blocking I/O programming, the function call corresponding to an
I/O request will block the execution of the thread until the operation completes.
This can go from a few milliseconds, in case of a disk access, to minutes or even
more, in case the data is generated from user actions, such as pressing a key. The
following pseudocode shows a typical blocking read performed against a socket:
//blocks the thread until the data is available
data = socket.read();
//data is available
print(data);
It is trivial to notice that a web server that is implemented using blocking I/O will
not be able to handle multiple connections in the same thread; each I/O operation
on a socket will block the processing of any other connection. For this reason, the
traditional approach to handle concurrency in web servers is to kick off a thread
or a process (or to reuse one taken from a pool) for each concurrent connection
that needs to be handled. This way, when a thread blocks for an I/O operation
it will not impact the availability of the other requests, because they are handled
in separate threads.
[ 11 ]
The following image illustrates this scenario:

Idle time
handle data
from A
Connection A
Connection B
Server
handle data
from A
Thread
handle data from B
Thread
Connection C
handle data from C
Thread
The preceding image lays emphasis on the amount of time each thread is idle,
waiting for new data to be received from the associated connection. Now, if we
also consider that any type of I/O can possibly block a request, for example, while
interacting with databases or with the filesystem, we soon realize how many times a
thread has to block in order to wait for the result of an I/O operation. Unfortunately,
a thread is not cheap in terms of system resources, it consumes memory and causes
context switches, so having a long running thread for each connection and not using
it for most of the time, is not the best compromise in terms of efficiency.
Non-blocking I/O
In addition to blocking I/O, most modern operating systems support another
mechanism to access resources, called non-blocking I/O. In this operating mode,
the system call always returns immediately without waiting for the data to be read
or written. If no results are available at the moment of the call, the function will
simply return a predefined constant, indicating that there is no data available to
return at that moment.
For example, in Unix operating systems, the fcntl() function is used to
manipulate an existing file descriptor to change its operating mode to non-blocking
(with the O_NONBLOCK flag). Once the resource is in non-blocking mode, any read
operation will fail with a return code, EAGAIN, in case the resource doesn't have any
data ready to be read.
[ 12 ]
Chapter 1
The most basic pattern for accessing this kind of non-blocking I/O is to actively
poll the resource within a loop until some actual data is returned; this is called
busy-waiting. The following pseudocode shows you how it's possible to read
from multiple resources using non-blocking I/O and a polling loop:
resources = [socketA, socketB, pipeA];
while(!resources.isEmpty()) {
for(i = 0; i < resources.length; i++)
resource = resources[i];
//try to read
var data = resource.read();
if(data === NO_DATA_AVAILABLE)
//there is no data to read at the
continue;
if(data === RESOURCE_CLOSED)
//the resource was closed, remove
resources.remove(i);
else
//some data was received, process
consumeData(data);
}
}
moment
it from the list
it
You can see that, with this simple technique, it is already possible to handle
different resources in the same thread, but it's still not efficient. In fact, in the
preceding example, the loop will consume precious CPU only for iterating over
resources that are unavailable most of the time. Polling algorithms usually result
in a huge amount of wasted CPU time.
Event demultiplexing
Busy-waiting is definitely not an ideal technique for processing non-blocking
resources, but luckily, most modern operating systems provide a native mechanism
to handle concurrent, non-blocking resources in an efficient way; this mechanism
is called synchronous event demultiplexer or event notification interface. This
component collects and queues I/O events that come from a set of watched
resources, and block until new events are available to process. The following is the
pseudocode of an algorithm that uses a generic synchronous event demultiplexer to
read from two different resources:
socketA, pipeB;
watchedList.add(socketA, FOR_READ);
//[1]
watchedList.add(pipeB, FOR_READ);
while(events = demultiplexer.watch(watchedList)) {
//event loop
[ 13 ]
//[2]

foreach(event in events) {
//[3]
//This read will never block and will always return data
data = event.resource.read();
if(data === RESOURCE_CLOSED)
//the resource was closed, remove it from the watched list
demultiplexer.unwatch(event.resource);
else
//some actual data was received, process it
consumeData(data);
}
}
These are the important steps of the preceding pseudocode:

1. The resources are added to a data structure, associating each one of them
with a specific operation, in our example a read.
2. The event notifier is set up with the group of resources to be watched.
This call is synchronous and blocks until any of the watched resources is
ready for a read. When this occurs, the event demultiplexer returns from
the call and a new set of events is available to be processed.
3. Each event returned by the event demultiplexer is processed. At this point,
the resource associated with each event is guaranteed to be ready to read
and to not block during the operation. When all the events are processed, the
flow will block again on the event demultiplexer until new events are again
available to be processed. This is called the event loop.
It's interesting to see that with this pattern, we can now handle several I/O
operations inside a single thread, without using a busy-waiting technique. The
following image shows us how a web server would be able to handle multiple
connections using a synchronous event demultiplexer and a single thread:
Idle time
Connection A
Connection B
Server
handle data
from A
Connection C
[ 14 ]
handle data
from C
handle data
from B
Thread
Chapter 1
The previous image helps us understand how concurrency works in a single-threaded

application using a synchronous event demultiplexer and non-blocking I/O. We can
see that using only one thread does not impair our ability to run multiple I/O bound
tasks concurrently. The tasks are spread over time, instead of being spread across
multiple threads. This has the clear advantage of minimizing the total idle time of the
thread, as clearly shown in the image. This is not the only reason for choosing this
model. To have only a single thread, in fact, also has a beneficial impact on the way
programmers approach concurrency in general. Throughout the book, we will see how
the absence of in-process race conditions and multiple threads to synchronize, allows
us to use much simpler concurrency strategies.
In the next chapter, we will have the opportunity to talk more about the concurrency
model of Node.js.
The reactor pattern

We can now introduce the reactor pattern, which is a specialization of the algorithm
presented in the previous section. The main idea behind it is to have a handler
(which in Node.js is represented by a callback function) associated with each I/O
operation, which will be invoked as soon as an event is produced and processed by
the event loop. The structure of the reactor pattern is shown in the following image:
Application
Request
I/O
5b
Execute
Handler
1
5a
Event Queue
Event Demultiplexer
Resource Operation
3
Resource Operation
Event
Handler
Event
Handler
Event
Handler
Handler
Handler
Event Loop
6
[ 15 ]
This is what happens in an application using the reactor pattern:

1. The application generates a new I/O operation by submitting a request to
the Event Demultiplexer. The application also specifies a handler, which will
be invoked when the operation completes. Submitting a new request to the
Event Demultiplexer is a non-blocking call and it immediately returns the
control back to the application.
2. When a set of I/O operations completes, the Event Demultiplexer pushes the
new events into the Event Queue.
3. At this point, the Event Loop iterates over the items of the Event Queue.
4. For each event, the associated handler is invoked.
5. The handler, which is part of the application code, will give back the
control to the Event Loop when its execution completes (5a). However,
new asynchronous operations might be requested during the execution
of the handler (5b), causing new operations to be inserted in the Event
Demultiplexer (1), before the control is given back to the Event Loop.
6. When all the items in the Event Queue are processed, the loop will block
again on the Event Demultiplexer which will then trigger another cycle.
The asynchronous behavior is now clear: the application expresses the interest to
access a resource at one point in time (without blocking) and provides a handler,
which will then be invoked at another point in time when the operation completes.
A Node.js application will exit automatically when there are no more
pending operations in the Event Demultiplexer, and no more events
to be processed inside the Event Queue.
We can now define the pattern at the heart of Node.js.

Pattern (reactor): handles I/O by blocking until new events are
available from a set of observed resources, and then reacting by
dispatching each event to an associated handler.
[ 16 ]
Chapter 1
The non-blocking I/O engine of Node.js libuv

Each operating system has its own interface for the Event Demultiplexer:
epoll on Linux, kqueue on Mac OS X, and I/O Completion Port API (IOCP) on
Windows. Besides that, each I/O operation can behave quite differently depending
on the type of the resource, even within the same OS. For example, in Unix, regular
filesystem files do not support non-blocking operations, so, in order to simulate a
non-blocking behavior, it is necessary to use a separate thread outside the Event
Loop. All these inconsistencies across and within the different operating systems
required a higher-level abstraction to be built for the Event Demultiplexer. This is
exactly why the Node.js core team created a C library called libuv, with the
objective to make Node.js compatible with all the major platforms and normalize
the non-blocking behavior of the different types of resource; libuv today represents
the low-level I/O engine of Node.js.
Besides abstracting the underlying system calls, libuv also implements the reactor
pattern, thus providing an API for creating event loops, managing the event queue,
running asynchronous I/O operations, and queuing other types of tasks.
A great resource to learn more about libuv is the free
online book created by Nikhil Marathe, which is available
at http://nikhilm.github.io/uvbook/.
The recipe for Node.js

The reactor pattern and libuv are the basic building blocks of Node.js, but we need
the following three other components to build the full platform:
A set of bindings responsible for wrapping and exposing libuv and other
low-level functionality to JavaScript.
V8, the JavaScript engine originally developed by Google for the Chrome
browser. This is one of the reasons why Node.js is so fast and efficient.
V8 is acclaimed for its revolutionary design, its speed, and for its efficient
memory management.
A core JavaScript library (called node-core) that implements the high-level

Node.js API.
[ 17 ]
Finally, this is the recipe of Node.js, and the following image represents its
final architecture:
Userland modules and applications
Node.js
Core Javascript API (node-core)
Bindings
libuv
V8
The callback pattern

Callbacks are the materialization of the handlers of the reactor pattern and they are
literally one of those imprints that give Node.js its distinctive programming style.
Callbacks are functions that are invoked to propagate the result of an operation
and this is exactly what we need when dealing with asynchronous operations.
They practically replace the use of the return instruction that, as we know, always
executes synchronously. JavaScript is a great language to represent callbacks,
because as we know, functions are first class objects and can be easily assigned to
variables, passed as arguments, returned from another function invocation, or stored
into data structures. Also, closures are an ideal construct for implementing callbacks.
With closures, we can in fact reference the environment in which a function was
created, practically, we can always maintain the context in which the asynchronous
operation was requested, no matter when or where its callback is invoked.
If you need to refresh your knowledge about closures, you can refer to
the article on the Mozilla Developer Network at https://developer.
mozilla.org/en-US/docs/Web/JavaScript/Guide/Closures.
In this section, we will analyze this particular style of programming made of

callbacks instead of the return instructions.
[ 18 ]
Chapter 1
The continuation-passing style

In JavaScript, a callback is a function that is passed as an argument to another
function and is invoked with the result when the operation completes. In functional
programming, this way of propagating the result is called continuation-passing
style, for brevity, CPS. It is a general concept, and it is not always associated with
asynchronous operations. In fact, it simply indicates that a result is propagated by
passing it to another function (the callback), instead of directly returning it to the caller.
Synchronous continuation-passing style

To clarify the concept, let's take a look at a simple synchronous function:
function add(a, b) {
return a + b;
}
There is nothing special here; the result is passed back to the caller using the
return instruction; this is also called direct style, and it represents the most
common way of returning a result in synchronous programming. The equivalent
continuation-passing style of the preceding function would be as follows:
function add(a, b, callback) {
callback(a + b);
}
The add() function is a synchronous CPS function, which means that it will
return a value only when the callback completes its execution. The following
code demonstrates this statement:
console.log('before');
add(1, 2, function(result) {
console.log('Result: ' + result);
});
console.log('after');
Since add() is synchronous, the previous code will trivially print the following:
before
Result: 3
after
[ 19 ]
Asynchronous continuation-passing style

Now, let's consider the case where the add() function is asynchronous, which is
as follows:
function addAsync(a, b, callback) {
setTimeout(function() {
callback(a + b);
}, 100);
}
In the previous code, we simply use setTimeout() to simulate an asynchronous

invocation of the callback. Now, let's try to use this function and see how the order
of the operations changes:
console.log('before');
addAsync(1, 2, function(result) {
console.log('Result: ' + result);
});
console.log('after');
The preceding code will print the following:

before
after
Result: 3
Since setTimeout() triggers an asynchronous operation, it will not wait anymore

for the callback to be executed, but instead, it returns immediately giving the control
back to addAsync(), and then back to its caller. This property in Node.js is crucial,
as it allows the stack to unwind, and the control to be given back to the event loop as
soon as an asynchronous request is sent, thus allowing a new event from the queue
to be processed.
[ 20 ]
Chapter 1
The following image shows how this works:

Function invocation
console.log(before)
2
Transfer of control
addAysnc(...)
3
Event Loop
setTimeout(...)
6
console.log(after)
When the async
operation
completes
7
callback(a + b)
10
8
9
console.log(Result:
+result)
When the asynchronous operation completes, the execution is then resumed

starting from the callback provided to the asynchronous function that caused the
unwinding. The execution will start from the Event Loop, so it will have a fresh stack.
This is where JavaScript comes in really handy, in fact, thanks to closures it is trivial to
maintain the context of the caller of the asynchronous function, even if the callback is
invoked at a different point in time and from a different location.
A synchronous function blocks until it completes its operations.
An asynchronous function returns immediately and the result is
passed to a handler (in our case, a callback) at a later cycle of the
event loop.
[ 21 ]
Non continuation-passing style callbacks

There are several circumstances in which the presence of a callback argument might
make you think that a function is asynchronous or is using a continuation-passing
style; that's not always true, let's take, for example, the map() method of the
Array object:
var result = [1, 5, 7].map(function(element) {
return element 1;
});
Clearly, the callback is just used to iterate over the elements of the array, and not to
pass the result of the operation. In fact, the result is returned synchronously using a
direct style. The intent of a callback is usually clearly stated in the documentation of
the API.
Synchronous or asynchronous?
We have seen how the order of the instructions changes radically depending
on the nature of a function - synchronous or asynchronous. This has strong
repercussions on the flow of the entire application, both in correctness and efficiency.
The following is an analysis of these two paradigms and their pitfalls. In general,
what must be avoided, is creating inconsistency and confusion around the nature
of an API, as doing so can lead to a set of problems which might be very hard to
detect and reproduce. To drive our analysis, we will take as example the case of an
inconsistently asynchronous function.
An unpredictable function
One of the most dangerous situations is to have an API that behaves synchronously
under certain conditions and asynchronously under others. Let's take the following
code as an example:
var fs = require('fs');
var cache = {};
function inconsistentRead(filename, callback) {
if(cache[filename]) {
//invoked synchronously
callback(cache[filename]);
} else {
//asynchronous function
[ 22 ]
Chapter 1
fs.readFile(filename, 'utf8', function(err, data) {
cache[filename] = data;
callback(data);
});
}
}
The preceding function uses the cache variable to store the results of different
file read operations. Please bear in mind that this is just an example, it does not
have error management, and the caching logic itself is suboptimal. Besides this,
the preceding function is dangerous because it behaves asynchronously until the
cache is not setwhich is until the fs.readFile() function returns its resultsbut
it will also be synchronous for all the subsequent requests for a file already in the
cachetriggering an immediate invocation of the callback.
Unleashing Zalgo
Now, let's see how the use of an unpredictable function, such as the one that we
defined previously, can easily break an application. Consider the following code:
function createFileReader(filename) {
var listeners = [];
inconsistentRead(filename, function(value) {
listeners.forEach(function(listener) {
listener(value);
});
});
return {
onDataReady: function(listener) {
listeners.push(listener);
}
};
}
When the preceding function is invoked, it creates a new object that acts as a notifier,
allowing to set multiple listeners for a file read operation. All the listeners will be
invoked at once when the read operation completes and the data is available. The
preceding function uses our inconsistentRead() function to implement this
functionality. Let's now try to use the createFileReader() function:
var reader1 = createFileReader('data.txt');
reader1.onDataReady(function(data) {
console.log('First call data: ' + data);
[ 23 ]

//...sometime later we try to read again from
//the same file
var reader2 = createFileReader('data.txt');
reader2.onDataReady(function(data) {
console.log('Second call data: ' + data);
});
});
The preceding code will print the following output:

First call data: some data
As you can see, the callback of the second operation is never invoked. Let's see why:
During the creation of reader1, our inconsistentRead() function behaves

asynchronously, because there is no cached result available. Therefore, we
have all the time to register our listener, as it will be invoked later in another
cycle of the event loop, when the read operation completes.
Then, reader2 is created in a cycle of the event loop in which the

cache for the requested file already exists. In this case, the inner call to
inconsistentRead() will be synchronous. So, its callback will be invoked
immediately, which means that also all the listeners of reader2 will be
invoked synchronously. However, we are registering the listeners after the
creation of reader2, so they will never be invoked.
The callback behavior of our inconsistentRead() function is really unpredictable,

as it depends on many factors, such as the frequency of its invocation, the filename
passed as argument, and the amount of time taken to load the file.
The bug that we've just seen might be extremely complicated to identify and
reproduce in a real application. Imagine to use a similar function in a web server,
where there can be multiple concurrent requests; imagine seeing some of those
requests hanging, without any apparent reason and without any error being logged.
This definitely falls under the category of nasty defects.
Isaac Z. Schlueter, creator of npm and former Node.js project lead, in one of his blog
posts compared the use of this type of unpredictable functions to unleashing Zalgo.
If you're not familiar with Zalgo, you are invited to find out what it is.
[ 24 ]
Chapter 1
You can find the original Isaac Z. Schlueter's post at http://blog.izs.

me/post/59142742143/designing-apis-for-asynchrony.
Using synchronous APIs

The lesson to learn from the unleashing Zalgo example is that it is imperative for an
API to clearly define its nature, either synchronous or asynchronous.
One suitable fix for our inconsistentRead() function, is to make it totally
synchronous. This is possible because Node.js provides a set of synchronous
direct style APIs for most of the basic I/O operations. For example, we can use the
fs.readFileSync() function in place of its asynchronous counterpart. The code
would now be as follows:
var cache = {};
function consistentReadSync(filename) {
return cache[filename];
} else {
cache[filename] = fs.readFileSync(filename, 'utf8');
return cache[filename];
}
}
We can see that the entire function was also converted to a direct style. There is no
reason for the function to have a continuation-passing style if it is synchronous.
In fact, we can state that it is always a good practice to implement a synchronous
API using a direct style; this will eliminate any confusion around its nature and
will also be more efficient from a performance perspective.
Pattern: prefer the direct style for purely synchronous functions.
[ 25 ]
Please bear in mind that changing an API from CPS to a direct style, or from
asynchronous to synchronous, or vice versa might also require a change to the
style of all the code using it. For example, in our case, we will have to totally
change the interface of our createFileReader() API and adapt it to work
always synchronously.
Also, using a synchronous API instead of an asynchronous one has some caveats:
A synchronous API might not be always available for the needed

functionality.
A synchronous API will block the event loop and put the concurrent requests
on hold. It practically breaks the Node.js concurrency, slowing down the
whole application. We will see later in the book what this really means for
our applications.
In our consistentReadSync() function, the risk of blocking the event loop is

partially mitigated, because the synchronous I/O API is invoked only once per each
filename, while the cached value will be used for all the subsequent invocations. If
we have a limited number of static files, then using consistentReadSync() won't
have a big effect on our event loop. Things can change quickly if we have to read
many files and only once. Using synchronous I/O in Node.js is strongly discouraged
in many circumstances; however, in some situations, this might be the easiest and
most efficient solution. Always evaluate your specific use case in order to choose the
right alternative.
Use blocking API only when they don't affect the ability of the
application to serve concurrent requests.
Deferred execution
Another alternative for fixing our inconsistentRead() function is to make it purely
asynchronous. The trick here is to schedule the synchronous callback invocation
to be executed "in the future" instead of being run immediately in the same event
loop cycle. In Node.js, this is possible using process.nextTick(), which defers
the execution of a function until the next pass of the event loop. Its functioning is
very simple; it takes a callback as an argument and pushes it on the top of the event
queue, in front of any pending I/O event, and returns immediately. The callback will
then be invoked as soon as the event loop runs again.
[ 26 ]
Chapter 1
Let's apply this technique to fix our inconsistentRead() function as follows:

var cache = {};
function consistentReadAsync(filename, callback) {
process.nextTick(function() {
callback(cache[filename]);
});
} else {
//asynchronous function
cache[filename] = data;
callback(data);
});
}
}
Now, our function is guaranteed to invoke its callback asynchronously,

under any circumstances.
Another API for deferring the execution of code is setImmediate(), whichdespite
the namemight actually be slower than process.nextTick(). While their purpose
is very similar, their semantic is quite different. Callbacks deferred with process.
nextTick() run before any other I/O event is fired, while with setImmediate(),
the execution is queued behind any I/O event that is already in the queue. Since
process.nextTick() runs before any already scheduled I/O, it might cause I/O
starvation under certain circumstances, for example, a recursive invocation; this
can never happen with setImmediate(). We will learn to appreciate the difference
between these two APIs when we analyze the use of deferred invocation for running
synchronous CPU-bound tasks later in the book.
Pattern: we guarantee that a callback is invoked asynchronously by
deferring its execution using process.nextTick().
[ 27 ]
Node.js callback conventions

In Node.js, continuation-passing style APIs and callbacks follow a set of specific
conventions. These conventions apply to the Node.js core API but they are also
followed virtually by every userland module and application. So, it's very important
that we understand them and make sure that we comply whenever we need to
design an asynchronous API.
Callbacks come last

In Node.js, if a function accepts in input a callback, this has to be passed as the last
argument. Let's take the following Node.js core API as an example:
fs.readFile(filename, [options], callback)
As you can see from the signature of the preceding function, the callback is always
put in last position, even in the presence of optional arguments. The motivation
for this convention is that the function call is more readable in case the callback is
defined in place.
Error comes first

In CPS, errors are propagated as any other type of result, which means using the
callback. In Node.js, any error produced by a CPS function is always passed as
the first argument of the callback, and any actual result is passed starting from the
second argument. If the operation succeeds without errors, the first argument
will be null or undefined. The following code shows you how to define a callback
complying with this convention:
fs.readFile('foo.txt', 'utf8', function(err, data) {
if(err)
handleError(err);
else
processData(data);
});
It is a good practice to always check for the presence of an error, as not doing so will
make it harder for us to debug our code and discover the possible points of failures.
Another important convention to take into account is that the error must always be
of type Error. This means that simple strings or numbers should never be passed as
error objects.
[ 28 ]
Chapter 1
Propagating errors
Propagating errors in synchronous, direct style functions is done with the
well-known throw command, which causes the error to jump up in the call
stack until it's caught.
In asynchronous CPS however, proper error propagation is done by simply passing
the error to the next callback in the CPS chain. The typical pattern looks as follows:
function readJSON(filename, callback) {
var parsed;
if(err)
//propagate the error and exit the current function
return callback(err);
try {
//parse the file contents
parsed = JSON.parse(data);
} catch(err) {
//catch parsing errors
}
//no errors, propagate just the data
callback(null, parsed);
});
};
The detail to notice in the previous code is how the callback is invoked when we
want to pass a valid result and when we want to propagate an error.
Uncaught exceptions
You might have seen from the readJSON() function defined previously that in
order to avoid any exception to be thrown into the fs.readFile() callback, we
put a try-catch block around JSON.parse(). Throwing inside an asynchronous
callback, in fact, will cause the exception to jump up to the event loop and never be
propagated to the next callback.
[ 29 ]
In Node.js, this is an unrecoverable state and the application will simply shut down
printing the error to the stderr interface. To demonstrate this, let's try to remove the
try-catch block from the readJSON() function defined previously:
function readJSONThrows(filename, callback) {
if(err)
//no errors, propagate just the data
callback(null, JSON.parse(data));
});
};
Now, in the function we just defined, there is no way of catching an eventual

exception coming from JSON.parse(). Let's try, for example, to parse an invalid
JSON file with the following code:
readJSONThrows('nonJSON.txt', function(err) {
console.log(err);
});
This would result in the application being abruptly terminated and the following
exception being printed on the console:
SyntaxError: Unexpected token d
at Object.parse (native)
at [...]/06_uncaught_exceptions/uncaught.js:7:25
at fs.js:266:14
at Object.oncomplete (fs.js:107:15)
Now, if we look at the preceding stack trace, we will see that it starts somewhere
from the fs.js module, practically from the point at which the native API has
completed reading and returned its result back to the fs.readFile() function, via
the event loop. This clearly shows us that the exception traveled from our callback
into the stack that we saw, and then straight into the event loop, where it's finally
caught and thrown in the console.
[ 30 ]
Chapter 1
This also means that wrapping the invocation of readJSONThrows() with a

try-catch block will not work, because the stack in which the block operates
is different from the one in which our callback is invoked. The following code
shows the anti-pattern that we just described:
try {
readJSONThrows('nonJSON.txt', function(err, result) {
[...]
});
} catch(err) {
console.log('This will not catch the JSON parsing exception');
}
The preceding catch statement will never receive the JSON parsing exception,
as it will travel back to the stack in which the exception was thrown, and we just
saw that the stack ends up in the event loop and not with the function that triggers
the asynchronous operation.
We already said that the application is aborted the moment an exception reaches
the event loop; however, we still have a last chance to perform some cleanup
or logging before the application terminates. In fact, when this happens, Node.js
emits a special event called uncaughtException just before exiting the process.
The following code shows a sample use case:
process.on('uncaughtException', function(err){
console.error('This will catch at last the ' +
'JSON parsing exception: ' + err.message);
//without this, the application would continue
process.exit(1);
});
It's important to understand that an uncaught exception leaves the application in a

state that is not guaranteed to be consistent, which can lead to unforeseeable problems.
For example, there might still have incomplete I/O requests running, or closures might
have become inconsistent. That's why it is always advised, especially in production, to
exit anyway from the application after an uncaught exception is received.
[ 31 ]
The module system and its patterns

Modules are the bricks for structuring non-trivial applications, but also the main
mechanism to enforce information hiding by keeping private all the functions and
variables that are not explicitly marked to be exported. In this section, we will
introduce the Node.js module system and its most common usage patterns.
The revealing module pattern

One of the major problems with JavaScript is the absence of namespacing.
Programs run in the global scope polluting it with data that comes from both
internal application code and dependencies. A popular technique to solve this
problem is called revealing module pattern and it looks like the following:
var module = (function() {
var privateFoo = function() {...};
var privateVar = [];
var export = {
publicFoo: function() {...},
publicBar: function() {...}
}
return export;
})();
This pattern leverages a self-invoking function to create a private scope, exporting

only the parts that are meant to be public. In the preceding code, the module variable
contains only the exported API, while the rest of the module content is practically
inaccessible from outside. As we will see in a moment, the idea behind this pattern is
used as a base for the Node.js module system.
Node.js modules explained

CommonJS is a group with the aim to standardize the JavaScript ecosystem, and one
of their most popular proposals is called CommonJS modules. Node.js built its module
system on top of this specification, with the addition of some custom extensions. To
describe how it works, we can make an analogy with the revealing module pattern,
where each module runs in a private scope, so that every variable that is defined
locally does not pollute the global namespace.
[ 32 ]
Chapter 1
A homemade module loader

To explain how this works, let's build a similar system from scratch. The code that
follows creates a function that mimics a subset of the functionality of the original
require() function of Node.js.
Let's start by creating a function that loads the content of a module, wraps it into a
private scope, and evaluates it:
function loadModule(filename, module, require) {
var wrappedSrc =
'(function(module, exports, require) {' +
fs.readFileSync(filename, 'utf8') +
'})(module, module.exports, require);';
eval(wrappedSrc);
}
The source code of a module is essentially wrapped into a function, as it was for
the revealing module pattern. The difference here is that we pass a list of variables
to the module, in particular: module, exports, and require. Make a note of how
the exports argument of the wrapping function is initialized with the contents of
module.exports, as we will talk about this later.
Please bear in mind that this is only an example and you will rarely
need to evaluate some source code in a real application. Features such
as eval() or the functions of the vm module (http://nodejs.org/
api/vm.html) can be easily used in the wrong way or with the wrong
input, thus opening a system to code injection attacks. They should
always be used with extreme care or avoided altogether.
Let's now see what these variables contain by implementing our require() function:
var require = function(moduleName) {
console.log('Require invoked for module: ' + moduleName);
var id = require.resolve(moduleName);
//[1]
if(require.cache[id]) {
//[2]
return require.cache[id].exports;
}
//module metadata
var module = {
exports: {},
id: id
};
//[3]
[ 33 ]

//Update the cache
require.cache[id] = module;
//[4]
//load the module

loadModule(id, module, require);
//[5]
//return exported variables

return module.exports;
//[6]
};
require.cache = {};
require.resolve = function(moduleName) {
/* resolve a full module id from the moduleName */
}
The preceding function simulates the behavior of the original require() function
of Node.js, which is used to load a module. Of course, this is just for educative
purposes and it does not accurately or completely reflect the internal behavior of
the real require() function, but it's great to understand the internals of the Node.js
module system, how a module is defined, and loaded. What our homemade module
system does is explained as follows:
1. A module name is accepted as input and the very first thing that we do is
resolve the full path of the module, which we call id. This task is delegated
to require.resolve(), which implements a specific resolving algorithm
(we will talk about it later).
2. If the module was already loaded in the past, it should be available in the
cache. In this case, we just return it immediately.
3. If the module was not yet loaded, we set up the environment for the first
load. In particular, we create a module object that contains an exports
property initialized with an empty object literal. This property will be used
by the code of the module to export any public API.
4. The module object is cached.
5. The module source code is read from its file and the code is evaluated, as we
have seen before. We provide to the module, the module object that we just
created, and a reference to the require() function. The module exports its
public API by manipulating or replacing the module.exports object.
6. Finally, the content of module.exports, which represents the public API of
the module, is returned to the caller.
As we see, there is nothing magical behind the workings of the Node.js module
system; the trick is all in the wrapper we create around a module's source code
and the artificial environment in which we run it.
[ 34 ]
Chapter 1
Defining a module
By looking at how our homemade require() function works, we should now know
how to define a module. The following code gives us an example:
//load another dependency
var dependency = require('./anotherModule');
//a private function
function log() {
console.log('Well done ' + dependency.username);
}
//the API to be exported for public use
module.exports.run = function() {
log();
};
The essential concept to remember is that everything inside a module is private

unless it's assigned to the module.exports variable. The contents of this variable
are then cached and returned when the module is loaded using require().
Defining globals
Even if all the variables and functions that are declared in a module are defined
in its local scope, it is still possible to define a global variable. In fact, the module
system exposes a special variable called global, which can be used for this
purpose. Everything that is assigned to this variable will end up automatically
in the global scope.
Please note that polluting the global scope is considered a bad practice
and nullifies the advantage of having a module system. So, use it only
if you really know what you are doing.
module.exports vs exports
For many developers who are not yet familiar with Node.js, a common source of
confusion is the difference between using exports and module.exports to expose
a public API. The code of our homemade require function should again clear
any doubt. The variable exports is just a reference to the initial value of module.
exports; we have seen that such a value is essentially a simple object literal created
before the module is loaded.
[ 35 ]
This means that we can only attach new properties to the object referenced by the
exports variable, as shown in the following code:
exports.hello = function() {
console.log('Hello');
}
Reassigning the exports variable doesn't have any effect, because it doesn't
change the contents of module.exports, it will only reassign the variable itself.
The following code is therefore wrong:
exports = function() {
}
If we want to export something other than an object literal, as for example a function,
an instance, or even a string, we have to reassign module.exports as follows:
module.exports = function() {
}
require is synchronous
Another important detail that we should take into account is that our homemade
require function is synchronous. In fact, it returns the module contents using a
simple direct style, and no callback is required. This is true for the original Node.js
require() function too. As a consequence, any assignment to module.export must
be synchronous as well. For example, the following code is incorrect:
module.exports = function() {...};
}, 100);
This property has important repercussions in the way we define modules, as it

limits us to mostly using synchronous code during the definition of a module.
This is actually one of the most important reasons why the core Node.js libraries
offer synchronous APIs as an alternative to most of the asynchronous ones.
[ 36 ]
Chapter 1
If we need some asynchronous initialization steps for a module, we can always

define and export an uninitialized module that is initialized asynchronously at
a later time. The problem with this approach though, is that loading such a module
using require does not guarantee that it's ready to be used. In Chapter 6, Recipes,
we will analyze this problem in detail and we will present some patterns to solve
this issue elegantly.
For the sake of curiosity, you might want to know that in its early days,
Node.js used to have an asynchronous version of require(), but it
was soon removed because it was overcomplicating a functionality that
was actually meant to be used only at initialization time, and where
asynchronous I/O brings more complexities than advantages.
The resolving algorithm

The term dependency hell, describes a situation whereby the dependencies of a
software, in turn depend on a shared dependency, but require different incompatible
versions. Node.js solves this problem elegantly by loading a different version of a
module depending on where the module is loaded from. All the merits of this feature
go to npm and also to the resolving algorithm used in the require function.
Let's now give a quick overview of this algorithm. As we saw, the resolve() function
takes a module name (which we will call here, moduleName) as input and it returns
the full path of the module. This path is then used to load its code and also to identify
the module uniquely. The resolving algorithm can be divided into the following three
major branches:
File modules: If moduleName starts with "/" it's considered already an

absolute path to the module and it's returned as it is. If it starts with "./",
then moduleName is considered a relative path, which is calculated starting
from the requiring module.
Core modules: If moduleName is not prefixed with "/" or "./", the algorithm
will first try to search within the core Node.js modules.
Package modules: If no core module is found matching moduleName,

then the search continues by looking for a matching module into the first
node_modules directory that is found navigating up in the directory
structure starting from the requiring module. The algorithm continues
to search for a match by looking into the next node_modules directory
up in the directory tree, until it reaches the root of the filesystem.
[ 37 ]
For file and package modules, both the individual files and directories can match
moduleName. In particular, the algorithm will try to match the following:
<moduleName>.js
<moduleName>/index.js
The directory/file specified in the main property of <moduleName>/

package.json
The complete, formal documentation of the resolving algorithm can
be found at http://nodejs.org/api/modules.html#modules_
all_together.
The node_modules directory is actually where npm installs the dependencies

of each package. This means that, based on the algorithm we just described,
each package can have its own private dependencies. For example, consider
the following directory structure:
myApp
foo.js
node_modules
depA
index.js
depB
bar.js
node_modules
depA
index.js
depC
foobar.js
node_modules
depA
index.js
[ 38 ]
Chapter 1
In the preceding example, myApp, depB, and depC all depend on depA; however,
they all have their own private version of the dependency! Following the rules of the
resolving algorithm, using require('depA') will load a different file depending on
the module that requires it, for example:
Calling require('depA') from /myApp/foo.js will load /myApp/node_

modules/depA/index.js
Calling require('depA') from /myApp/node_modules/depB/bar.js will

load /myApp/node_modules/depB/node_modules/depA/index.js
Calling require('depA') from /myApp/node_modules/depC/foobar.js

will load /myApp/node_modules/depC/node_modules/depA/index.js
The resolving algorithm is the magic behind the robustness of the Node.js
dependency management, and is what makes it possible to have hundreds or even
thousands of packages in an application without having collisions or problems of
version compatibility.
The resolving algorithm is applied transparently for us when we
invoke require(); however, if needed, it can still be used directly
by any module by simply invoking require.resolve().
The module cache

Each module is loaded and evaluated only the first time it is required, since any
subsequent call of require() will simply return the cached version. This should result
clear by looking at the code of our homemade require function. Caching is crucial
for performances, but it also has some important functional implications:
It makes it possible to have cycles within module dependencies
It guarantees, to some extent, that always the same instance is returned

when requiring the same module from within a given package
The module cache is exposed in the require.cache variable, so it is possible to

directly access it if needed. A common use case is to invalidate any cached module
by deleting the relative key in the require.cache variable, a practice very useful
during testing but very dangerous if applied in normal circumstances.
[ 39 ]
Cycles
Many consider circular dependencies as an intrinsic design issue, but it is something
which might actually happen in a real project, so it's useful for us to know at least how
this works in Node.js. If we look again at our homemade require() function, we
immediately get a glimpse of how this might work and what are its caveats.
Suppose we have two modules defined as follows:
Module a.js:
exports.loaded = false;
var b = require('./b');
module.exports = {
bWasLoaded: b.loaded,
loaded: true
};
Module b.js:
exports.loaded = false;
var a = require('./a');
module.exports = {
aWasLoaded: a.loaded,
loaded: true
};
Now, let's try to load these from another module, main.js, as follows:
var a = require('./a');
var b = require('./b');
console.log(a);
console.log(b);
The preceding code will print the following output:

{ bWasLoaded: true, loaded: true }
{ aWasLoaded: false, loaded: true }
This result reveals the caveats of circular dependencies. While both the modules
are completely initialized the moment they are required from the main module,
the a.js module will be incomplete when it is loaded from b.js. In particular,
its state will be the one that it reached the moment it required b.js. This behavior
should ring another bell, which will be confirmed if we swap the order in which
the two modules are required in main.js.
[ 40 ]
Chapter 1
If you try it, you will see that this time it will be the module a.js that will receive an
incomplete version of b.js. We understand now that this can become quite a fuzzy
business if we lose control of which module is loaded first, which can happen quite
easily if the project is big enough.
Module definition patterns

The module system, besides being a mechanism for loading dependencies, is also
a tool for defining APIs. As for any other problem related to API design, the main
factor to consider is the balance between private and public functionality. The aim
is to maximize information hiding and API usability, while balancing these with other
software qualities like extensibility and code reuse.
In this section, we will analyze some of the most popular patterns for defining
modules in Node.js; each one has its own balance of information hiding,
extensibility, and code reuse.
Named exports
The most basic method for exposing a public API is using named exports, which
consists in assigning all the values we want to make public to properties of the object
referenced by exports (or module.exports). In this way, the resulting exported
object becomes a container or namespace for a set of related functionality.
The following code shows a module implementing this pattern:
//file logger.js
exports.info = function(message) {
console.log('info: ' + message);
};
exports.verbose = function(message) {
console.log('verbose: ' + message);
};
The exported functions are then available as properties of the loaded module,
as shown in the following code:
//file main.js
var logger = require('./logger');
logger.info('This is an informational message');
logger.verbose('This is a verbose message');
Most of the Node.js core modules use this pattern.
[ 41 ]
The CommonJS specification only allows the use of the exports

variable to expose public members. Therefore, the named exports
pattern is the only one that is really compatible with the CommonJS
specification. The use of module.exports is an extension provided
by Node.js to support a broader range of module definition patterns,
as those we are going to see next.
Exporting a function
One of the most popular module definition patterns consists in reassigning the
whole module.exports variable to a function. Its main strength it's the fact that
it exposes only a single functionality, which provides a clear entry point for the
module, and makes it simple to understand and use; it also honors the principle
of small surface area very well. This way of defining modules is also known in the
community as substack pattern, after one of its most prolific adopters, James Halliday
(nickname substack). The following code is an example of this pattern:
//file logger.js
module.exports = function(message) {
console.log('info: ' + message);
};
A possible extension of this pattern is using the exported function as namespace

for other public APIs. This is a very powerful combination, because it still gives the
module the clarity of a single entry point (the main exported function), but it also
allows us to expose other functionalities that have secondary or more advanced
use cases. The following code shows you how to extend the module we defined
previously by using the exported function as a namespace:
module.exports.verbose = function(message) {
console.log('verbose: ' + message);
};
The following code demonstrates how to use the module that we just defined:
//file main.js
logger('This is an informational message');
logger.verbose('This is a verbose message');
[ 42 ]
Chapter 1
Even though exporting just a function might seem a limitation, in reality, it's a
perfect way to put the emphasis on a single functionalitythe most important for
the modulewhile giving less visibility to secondary aspects, which are instead
exposed as properties of the exported function itself.
Pattern (substack): expose the main functionality of a module by
exporting only one function. Use the exported function as namespace
to expose any auxiliary functionality.
Exporting a constructor
A module that exports a constructor is a specialization of a module that exports a
function. The difference is that with this new pattern, we allow the user to create
new instances using the constructor, but we also give them the ability to extend its
prototype and forge new classes. The following is an example of this pattern:
//file logger.js
function Logger(name) {
this.name = name;
};
Logger.prototype.log = function(message) {
console.log('[' + this.name + '] ' + message);
};
Logger.prototype.info = function(message) {
this.log('info: ' + message);
};
Logger.prototype.verbose = function(message) {
this.log('verbose: ' + message);
};
module.exports = Logger;
And, we can use the preceding module as follows:

//file logger.js
var Logger = require('./logger');
var dbLogger = new Logger('DB');
dbLogger.info('This is an informational message');
var accessLogger = new Logger('ACCESS');
accessLogger.verbose('This is a verbose message');
Exporting a constructor still provides a single entry point for the module, but
compared to the substack pattern, it exposes a lot more of the module internals;
however on the other side it allows much more power when it comes to extending
its functionality.
[ 43 ]
A variation of this pattern consists in applying a guard against invocations that don't
use the new instruction. This little trick allows us to use our module as a factory.
The following code shows you how this works:
if(!(this instanceof Logger)) {
return new Logger(name);
}
this.name = name;
};
The trick is simple; we check whether this exists and is an instance of Logger. If any
of these conditions is false, it means that the Logger() function was invoked without
using new, so we proceed with creating the new instance properly and returning it to
the caller. This technique allows us to use the module also as a factory, as shown in
the following code:
//file logger.js
var Logger = require('./logger');
var dbLogger = Logger('DB');
accessLogger.verbose('This is a verbose message');
Exporting an instance
We can leverage the caching mechanism of require() to easily define stateful
instancesobjects with a state created from a constructor or a factory, which can be
shared across different modules. The following code shows an example of this pattern:
//file logger.js
this.count = 0;
this.name = name;
};
Logger.prototype.log = function(message) {
this.count++;
console.log('[' + this.name + '] ' + message);
};
module.exports = new Logger('DEFAULT');
This newly defined module can then be used as follows:

//file main.js
logger.log('This is an informational message');
[ 44 ]
Chapter 1
Because the module is cached, every module that requires the logger module
will actually always retrieve the same instance of the object, thus sharing its state.
This pattern is very much like creating a Singleton, however, it does not guarantee
the uniqueness of the instance across the entire application, as it happens in the
traditional Singleton pattern. When analyzing the resolving algorithm, we have seen
in fact, that a module might be installed multiple times inside the dependency tree
of an application. This results with multiple instances of the same logical module, all
running in the context of the same Node.js application. In Chapter 5, Wiring Modules,
we will analyze the consequences of exporting stateful instances and some of the
patterns we can use as alternatives.
An extension to the pattern we just described, consists in exposing the constructor
used to create the instance, in addition to the instance itself. This allows the user to
create new instances of the same object, or even to extend it if necessary. To enable
this, we just need to assign a new property to the instance, as shown in the following
line of code:
module.exports.Logger = Logger;
Then, we can use the exported constructor to create other instances of the class,
as follows:
var customLogger = new logger.Logger('CUSTOM');
customLogger.log('This is an informational message');
From a usability perspective, this is similar to using an exported function as

namespace; the module exports the default instance of an objectthe piece of
functionality we might want to use most of the timewhile more advanced
features, such as the ability to create new instances or extend the object, are still
made available through less exposed properties.
Modifying other modules or the global scope

A module can even export nothing. This can look a bit out of place, however,
we should not forget that a module can modify the global scope and any object
in it, including other modules in the cache. Please note that these are in general
considered bad practices, but since this pattern can be useful and safe under some
circumstances (for example, for testing) and is sometimes used in the wild, it is
worth to know and understand it. So, we said a module can modify other modules
or objects in the global scope. Well, this is called monkey patching, which generally
refers to the practice of modifying the existing objects at runtime to change or extend
their behavior or to apply temporary fixes.
[ 45 ]
The following example shows you how we can add a new function to
another module:
//file patcher.js
// ./logger is another module
require('./logger').customMessage = function() {
console.log('This is a new functionality');
};
Using our new patcher module would be as easy as writing the following code:
//file main.js
require('./patcher');
logger.customMessage();
In the preceding code, patcher must be required before using the logger module
for the first time in order to allow the patch to be applied.
The techniques described here are all dangerous ones to apply. The main concern
is that, to have a module that modifies the global namespace or other modules is an
operation with side effects. In other words, it affects the state of entities outside their
scope, which can have consequences that are not always predictable, especially when
multiple modules interact with the same entities. Imagine to have two different
modules trying to set the same global variable, or modifying the same property
of the same module; the effects might be unpredictable (which module wins?),
but most importantly it would have repercussions on the entire application.
The observer pattern

Another important and fundamental pattern used in Node.js is the observer
pattern. Together with reactor, callbacks, and modules, this is one of the pillars
of the platform and an absolute prerequisite for using many node-core and
userland modules.
Observer is an ideal solution for modeling the reactive nature of Node.js, and a
perfect complement for callbacks. Let's give a formal definition as follows:
Pattern (observer): defines an object (called subject), which can notify a
set of observers (or listeners), when a change in its state happens.
[ 46 ]
Chapter 1
The main difference from the callback pattern is that the subject can actually notify
multiple observers, while a traditional continuation-passing style callback will
usually propagate its result to only one listener, the callback.
The EventEmitter
In traditional object-oriented programming, the observer pattern requires interfaces,
concrete classes, and a hierarchy; in Node.js, all becomes much simpler. The observer
pattern is already built into the core and is available through the EventEmitter class.
The EventEmitter class allows us to register one or more functions as listeners,
which will be invoked when a particular event type is fired. The following image
visually explains the concept:
Listener
Event A
EventEmitter
Listener
Event B
Listener
The EventEmitter is a prototype, and it is exported from the events core module.
The following code shows how we can obtain a reference to it:
var EventEmitter = require('events').EventEmitter;
var eeInstance = new EventEmitter();
The essential methods of the EventEmitter are given as follows:
on(event, listener): This method allows you to register a new listener
once(event, listener): This method registers a new listener, which is
emit(event, [arg1], []): This method produces a new event and

provides additional arguments to be passed to the listeners
removeListener(event, listener): This method removes a listener for
(a function) for the given event type (a string)
then removed after the event is emitted for the first time
the specified event type
[ 47 ]
All the preceding methods will return the EventEmitter instance to allow chaining.
The listener function has the signature, function([arg1], []), so it simply
accepts the arguments provided the moment the event is emitted. Inside the listener,
this refers to the instance of the EventEmitter that produces the event.
We can already see that there is a big difference between a listener and a traditional
Node.js callback; in particular, the first argument is not an error, but it can be any
data passed to emit() at the moment of its invocation.
Create and use an EventEmitter

Let's see how we can use an EventEmitter in practice. The simplest way is to create
a new instance and use it directly. The following code shows a function, which uses
an EventEmitter to notify its subscribers in real time when a particular pattern is
found in a list of files:
function findPattern(files, regex) {
var emitter = new EventEmitter();
files.forEach(function(file) {
fs.readFile(file, 'utf8', function(err, content) {
if(err)
return emitter.emit('error', err);
emitter.emit('fileread', file);
var match = null;
if(match = content.match(regex))
match.forEach(function(elem) {
emitter.emit('found', file, elem);
});
});
});
return emitter;
}
The EventEmitter created by the preceding function will produce the following
three events:
fileread: This event occurs when a file is read
found: This event occurs when a match has been found
error: This event occurs when an error has occurred during the reading
of the file
[ 48 ]
Chapter 1
Let's see now how our findPattern() function can be used:

findPattern(
['fileA.txt', 'fileB.json'],
/hello \w+/g
)
.on('fileread', function(file) {
console.log(file + ' was read');
})
.on('found', function(file, match) {
console.log('Matched "' + match + '" in file ' + file);
})
.on('error', function(err) {
console.log('Error emitted: ' + err.message);
});
In the preceding example, we registered a listener for each of the three event types
produced by the EventEmitter which was created by our findPattern() function.
Propagating errors
The EventEmitter - as it happens for callbacks - cannot just throw exceptions
when an error condition occurs, as they would be lost in the event loop if the
event is emitted asynchronously. Instead, the convention is to emit a special event,
called error, and to pass an Error object as an argument. That's exactly what we
are doing in the findPattern() function that we defined earlier.
It is always a good practice to register a listener for the
error event, as Node.js will treat it in a special way and will
automatically throw an exception and exit from the program if
no associated listener is found.
Make any object observable

Sometimes, creating a new observable object directly from the EventEmitter
class is not enough, as this makes it impractical to provide functionality that
goes beyond the mere production of new events. It is more common, in fact, to
have the need to make a generic object observable; this is possible by extending
the EventEmitter class.
[ 49 ]
To demonstrate this pattern, let's try to implement the functionality of the

findPattern() function in an object as follows:
var util = require('util');
function FindPattern(regex) {
EventEmitter.call(this);
this.regex = regex;
this.files = [];
}
util.inherits(FindPattern, EventEmitter);
FindPattern.prototype.addFile = function(file) {
this.files.push(file);
return this;
};
FindPattern.prototype.find = function() {
var self = this;
self.files.forEach(function(file) {
fs.readFile(file, 'utf8', function(err, content) {
if(err)
return self.emit('error', err);
self.emit('fileread', file);
var match = null;
if(match = content.match(self.regex))
match.forEach(function(elem) {
self.emit('found', file, elem);
});
});
});
return this;
};
[ 50 ]
Chapter 1
The FindPattern prototype that we defined extends the EventEmitter using the
inherits() function provided by the core module util. This way, it becomes a
full-fledged observable class. The following is an example of its usage:
var findPatternObject = new FindPattern(/hello \w+/);
findPatternObject
.addFile('fileA.txt')
.addFile('fileB.json')
.find()
.on('found', function(file, match) {
console.log('Matched "' + match + '" in file ' + file);
})
.on('error', function(err) {
console.log('Error emitted ' + err.message);
});
We can now see how the FindPattern object has a full set of methods, in addition to
being observable by inheriting the functionality of the EventEmitter.
This is a pretty common pattern in the Node.js ecosystem, for example, the
Server object of the core http module defines methods such as listen(), close(),
setTimeout(), and internally it also inherits from the EventEmitter function,
thus allowing it to produce events, such as request, when a new request is received,
or connection, when a new connection is established, or closed, when the
server is closed.
Other notable examples of objects extending the EventEmitter are Node.js streams.
We will analyze streams in more detail in Chapter 3, Coding with Streams.
Synchronous and asynchronous events

As with callbacks, events can be emitted synchronously or asynchronously, and it is
crucial that we never mix the two approaches in the same EventEmitter, but even
more importantly, when emitting the same event type, to avoid to produce the same
problems that we described in the Unleashing Zalgo section.
[ 51 ]
The main difference between emitting synchronous or asynchronous events lies in

the way listeners can be registered. When the events are emitted asynchronously,
the user has all the time to register new listeners even after the EventEmitter is
initialized, because the events are guaranteed not to be fired until the next cycle
of the event loop. That's exactly what is happening in the findPattern() function.
We defined this function previously and it represents a common approach that is
used in most Node.js modules.
On the contrary, emitting events synchronously requires that all the listeners are
registered before the EventEmitter function starts to emit any event. Let's look
at an example:
function SyncEmit() {
this.emit('ready');
}
util.inherits(SyncEmit, EventEmitter);
var syncEmit = new SyncEmit();
syncEmit.on('ready', function() {
console.log('Object is ready to be used');
});
If the ready event was emitted asynchronously, then the previous code would
work perfectly; however, the event is produced synchronously and the listener is
registered after the event was already sent, so the result is that the listener is never
invoked; the code will print nothing to the console.
Contrarily to callbacks, there are situations where using an EventEmitter in a
synchronous fashion makes sense, given its different purpose. For this reason,
it's very important to clearly highlight the behavior of our EventEmitter in its
documentation to avoid confusion, and potentially a wrong usage.
EventEmitter vs Callbacks
A common dilemma when defining an asynchronous API is to check whether
to use an EventEmitter or simply accept a callback. The general differentiating
rule is semantic: callbacks should be used when a result must be returned in
an asynchronous way; events should instead be used when there is a need to
communicate that something has just happened.
[ 52 ]
Chapter 1
But besides this simple principle, a lot of confusion is generated from the fact that the
two paradigms are most of the time equivalent and allow you to achieve the same
results. Consider the following code for an example:
function helloEvents() {
var eventEmitter = new EventEmitter();
eventEmitter.emit('hello', 'world');
}, 100);
return eventEmitter;
}
function helloCallback(callback) {
callback('hello', 'world');
}, 100);
}
The two functions helloEvents() and helloCallback() can be considered

equivalent in terms of functionality; the first communicates the completion of the
timeout using an event, the second uses a callback to notify the caller instead,
passing the event type as an argument. But what really differentiates them is the
readability, the semantic, and the amount of code that is required to be implemented
or used. While we cannot give a deterministic set of rules to choose between one or
the other style, we can certainly provide some hints to help take the decision.
As a first observation, we can say that callbacks have some limitations when it comes
to supporting different types of events. In fact, we can still differentiate between
multiple events by passing the type as an argument of the callback, or by accepting
several callbacks, one for each supported event. However, this cannot exactly be
considered an elegant API. In this situation, an EventEmitter can give a better
interface and leaner code.
Another case where the EventEmitter might be preferable is when the same event
can occur multiple times, or not occur at all. A callback, in fact, is expected to be
invoked exactly once, whether the operation is successful or not. The fact that we have
a possibly repeating circumstance should let us think again about the semantic nature
of the occurrence, which is more similar to an event that has to be communicated
rather than a result; in this case an EventEmitter is the preferred choice.
Lastly, an API using callbacks can notify only that particular callback, while
using an EventEmitter function it's possible for multiple listeners to receive
the same notification.
[ 53 ]
Combine callbacks and EventEmitter

There are also some circumstances where an EventEmitter can be used in
conjunction with a callback. This pattern is extremely useful when we want to
implement the principle of small surface area by exporting a traditional asynchronous
function as the main functionality, while still providing richer features, and more
control by returning an EventEmitter. One example of this pattern is offered by the
node-glob module (https://npmjs.org/package/glob), a library that performs
glob-style file searches. The main entry point of the module is the function it exports,
which has the following signature:
glob(pattern, [options], callback)
The function takes pattern as the first argument, a set of options, and a callback
function which is invoked with the list of all the files matching the provided pattern.
At the same time, the function returns an EventEmitter that provides a more
fine-grained report over the state of the process. For example, it is possible to be
notified in real-time when a match occurs by listening to the match event, to obtain
the list of all the matched files with the end event, or to know whether the process
was manually aborted by listening to the abort event. The following code shows
how this looks:
var glob = require('glob');
glob('data/*.txt', function(error, files) {
console.log('All files found: ' + JSON.stringify(files));
}).on('match', function(match) {
console.log('Match found: ' + match);
});
As we can see, the practice of exposing a simple, clean, and minimal entry point
while still providing more advanced or less important features with secondary
means is quite common in Node.js, and combining EventEmitter with traditional
callbacks is one of the ways to achieve that.
Pattern: create a function that accepts a callback and returns an
EventEmitter, thus providing a simple and clear entry point for
the main functionality, while emitting more fine-grained events
using the EventEmitter.
[ 54 ]
Chapter 1
Summary
In this chapter, we have seen how the Node.js platform is based on a few important
principles that provide the foundation to build efficient and reusable code. The
philosophy and the design choices behind the platform have, in fact, a strong
influence on the structure and behavior of every application and module we create.
Often, for a developer moving from another technology, these principles might
seem unfamiliar and the usual instinctive reaction is to fight the change by trying
to find more familiar patterns inside a world which in reality requires a real shift in
the mindset. On one hand, the asynchronous nature of the reactor pattern requires
a different programming style made of callbacks and things that happen at a later
time, without worrying too much about threads and race conditions. On the other
hand, the module pattern and its principles of simplicity and minimalism creates
interesting new scenarios in terms of reusability, maintenance, and usability.
Finally, besides the obvious technical advantages of being fast, efficient, and based
on JavaScript, Node.js is attracting so much interest because of the principles
we have just discovered. For many, grasping the essence of this world feels like
returning to the origins, to a more humane way of programming for both size
and complexity and that's why developers end up falling in love with Node.js.
In the next chapter, we will focus our attention on the mechanisms to handle
asynchronous code, we will see how callbacks can easily become our enemy, and
we will learn how to fix that by using some simple principles, patterns, or even
constructs that do not require a continuation-passing style programming.
[ 55 ]
Get more information Node.js Design Patterns
Where to buy this book

You can buy Node.js Design Patterns from the Packt Publishing website.
Alternatively, you can buy the book from Amazon, BN.com, Computer Manuals and most internet
book retailers.
Click here for ordering and shipping details.
www.PacktPub.com
Stay Connected:

Node - Js Design Patterns Sample Chapter

Uploaded by

Copyright:

Available Formats

Node - Js Design Patterns Sample Chapter

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Node - Js Design Patterns Sample Chapter

Uploaded by

Copyright:

Available Formats

What is the book about?

What is the book about?

Who is the author?

Who is the author?

Fr

In this package, you will find:

The author biography

About the Author

Node.js Design Patterns

A set of patterns to solve common Node.js design and coding problems.

What This Book Covers

The Node.js philosophy, the "Node way"

The reactor pattern: the mechanism at the heart of the Node.js

The Node.js callback pattern and its set of conventions

Node.js Design Fundamentals

The Node.js philosophy

Easier to understand and use

Small surface area

Node.js Design Fundamentals

Simplicity and pragmatism

The reactor pattern

Node.js Design Fundamentals

The following image illustrates this scenario:

handle data from B

it from the list

Node.js Design Fundamentals

These are the important steps of the preceding pseudocode:

The previous image helps us understand how concurrency works in a single-threaded

The reactor pattern

Node.js Design Fundamentals

This is what happens in an application using the reactor pattern:

We can now define the pattern at the heart of Node.js.

The non-blocking I/O engine of Node.js libuv

The recipe for Node.js

A core JavaScript library (called node-core) that implements the high-level

Node.js Design Fundamentals

Userland modules and applications

The callback pattern

In this section, we will analyze this particular style of programming made of

The continuation-passing style

Synchronous continuation-passing style

Node.js Design Fundamentals

Asynchronous continuation-passing style

In the previous code, we simply use setTimeout() to simulate an asynchronous

The preceding code will print the following:

Since setTimeout() triggers an asynchronous operation, it will not wait anymore

The following image shows how this works:

When the asynchronous operation completes, the execution is then resumed

Node.js Design Fundamentals

Non continuation-passing style callbacks

Node.js Design Fundamentals

The preceding code will print the following output:

During the creation of reader1, our inconsistentRead() function behaves

Then, reader2 is created in a cycle of the event loop in which the

The callback behavior of our inconsistentRead() function is really unpredictable,

You can find the original Isaac Z. Schlueter's post at http://blog.izs.

Using synchronous APIs

Pattern: prefer the direct style for purely synchronous functions.

Node.js Design Fundamentals