BROUGHT TO YOU IN PARTNERSHIP WITH

JavaScript
CONTENTS

•  JavaScript Test Automation

Fundamentals

Test Automation
−  A General Approach to Testing
−  Testing JavaScript Applications

•  Essentials of Test Automation

Frameworks
Frameworks
−  WebDriverIO
−  Nightwatch
−  Puppeteer
Six Essential Frameworks for Creating −  Playwright

Automated Tests −  TestCafe

−  Cypress

JUSTIN ALBANO •  Conclusion

SOFTWARE ENGINEER, IBM

Web development has changed dramatically over the last three 2. Command – The logic of the test case that is intended
decades. At the advent of the Web, nearly all applications consisted to verify some behavior
of static content rendered in one or two browsers. Since that time,
3. Postconditions – Assertions that must be true once
the Web has evolved into a colossal collection of dynamic web pages
the test is complete
and applications rendered in half-a-dozen or more browsers on
behalf of billions of users. Written formally, using Hoare Logic, a test case can be expressed as:

In addition to dramatic changes in web applications, JavaScript {P} C {Q}

test frameworks have also morphed to keep pace with this ever-
where:
evolving ecosystem. In this Refcard, we will delve into the conceptual
•  {P} are the preconditions of the test case
underpinnings of modern JavaScript test automation frameworks
and explore six of the most popular frameworks available today. •  {C} is the command to be executed
Along this journey, we will compare the different design and •  {Q} are the postconditions of the test case that
architecture decisions of each framework and how well these various we expect to be to be true
trade-offs are suited for different requirements and contexts.

JAVASCRIPT TEST AUTOMATION

FUNDAMENTALS
To understand JavaScript automated testing frameworks, we must
first have a foundational understanding of how to approach testing in
the general sense, including the purpose and structure of test cases
and the mindset we use to create effective test cases.

A GENERAL APPROACH TO TESTING

A test framework is primarily responsible for running a test suite or
a collection of test cases. Each test case is designed to exercise a
single, logical unit of behavior in an application and ensure that the
targeted behavior operates as expected. Generally, test cases are
structured in three parts:

1. Preconditions – Assertions that must be true before the

test can execute

1
 NEW

NEW

NEW

NEW
 REFCARD | JAVASCRIPT TEST AUTOMATION FR AMEWORKS

We can write tests in natural language — called Behavior-Driven This tight relationship between JavaScript and the browser allows
Development (BDD) — using the following clauses: us to programmatically execute basic actions we take for granted,
such as clicking a button or hovering over a specific Hypertext
1. Given – The environment and context of the test case
Markup Language (HTML) element. Due to JavaScript's capability to
2. When – The logic that is under test
interact with the browser and execute these actions, JavaScript test
3. Then – The assertions about the results of the test
frameworks must also be able to inspect the results of these actions,

For example: as seen in the following diagram:

Given I have addends of 3 and 5

When I sum them
Then the result is 8

Lastly, we can translate our test cases into code, as seen in the
pseudocode example below of our previous BDD test case:

// Given
a = 3
b = 5

// When
For example, if we click a button, we may want to assert that a different
result = a + b
HTML element changes to a specific color, a dialog box is displayed,

// Then or a Representational State Transfer (REST) request is made to some

assert result == 8 back-end service. Additionally, we may want to take screenshots

or videos of a test case's results — or the entire execution of a test
Since our test case is now expressed as code, we can execute it the case — to ensure that the results match our aesthetic specifications
same way we execute our application code. In practice, this means (e.g., graphic designs).
executing all, or a specific subset, of our test cases each time we
Since users can choose from a host of different browsers (e.g., Chrome,
make changes to our application. This ensures that regressions —
Firefox, Safari, Edge), we may also want to ensure that our application
bugs that cause previously working components to fail — are not
behaves and appears as expected in multiple browsers. This adds an
introduced in our application.
additional requirement to our test framework: interoperability with
Reporting is another important responsibility of a test framework. different browsers.
Apart from simply denoting whether each test passed or failed,
Conceptually, each browser has its own application programming
frameworks are also responsible for describing why a test case failed.
interface (API) that allows an application to inspect the results of
For example, if the value of result in our previous test case was 7,
an interaction with the browser. Over the years, as more and more
we expect that our report would include a description of the failure,
browsers with vastly different APIs became popular, standards were
informing us that the actual result of the test, 7, does not match our
devised that abstract the details of each browser and allow test
expected result, 8.
frameworks to interact with browsers in an agnostic fashion.
TESTING JAVASCRIPT APPLICATIONS
These advancements have led to three different types of JavaScript
The generalized approach to testing and test frameworks above
test frameworks:
holds true for nearly all programming languages and ecosystems.
While JavaScript can be executed outside of a browser using 1. Standardized – Uses a standard API, which ensures

Node.Js (Node), a vast majority of the JavaScript applications we see interoperability with all browsers that support the

today are browser-based. This is unsurprising, as JavaScript has a standard, but it can be inefficient and lack support for

significant advantage over nearly all other programming languages browser-specific features

in this realm: All modern browsers include a JavaScript engine that 2. Non-Standardized – Uses browser-specific protocols that
allows JavaScript code to be executed natively within the browser, have not been standardized, which may be more efficient
which brings its own set of advantages and challenges. and feature-rich, but it may not support all browsers

3 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | JAVASCRIPT TEST AUTOMATION FR AMEWORKS

3. Proprietary – Uses a customized mechanism — such as Note that the -y flag — “yes” — accepts all prompts during the
proxying or code injection — to interact with browsers, which initialization process, removing the need for manual interaction.
can be efficient and feature-rich, but it is more complex and Next, we must install the WebDriverIO Command Line Interface
may introduce quirky limitations (CLI) tool through NPM and configure the project using the
wdio config command:
Each of these categories has its advantages and disadvantages, and
the frameworks that use them excel and deteriorate in different ways. npm i --save-dev @wdio/cli

In the following section, we will look at six of the more prominent npx wdio config -y

frameworks, exploring how they utilize one of the three approaches

The configuration command creates a wdio.conf.js file that
above to allow us to create automated tests in JavaScript.
contains the WebDriverIO configuration for our project. Within this
ESSENTIALS OF TEST AUTOMATION configuration file, we can see the following field:
FRAMEWORKS
specs: [
As JavaScript automated testing frameworks have matured over './test/specs/**/*.js'
the last two decades, six have risen as the most popular and can be ]
categorized by their protocols:

1. World Wide Web Consortium (W3C) WebDriver – WebDriverIO This specs field specifies the location of our test case specifications.

and Nightwatch By default, a test file, example.e2e.js, will be created under the
./test/specs directory. We can run this test using the following
2. Chrome DevTools Protocol (CDP) – Puppeteer and Playwright
command:
3. Proprietary – TestCafe and Cypress
npx wdio run wdio.conf.js
In this section, we will delve into how we can install and run each
framework, walk through an exploratory example of a simple test
The example.e2e.js test will open a browser window and attempt
case, and compare each framework to understand which is the best
to authenticate using a login form. Once the test is complete, we can
fit for our use cases. Note: The examples that follow assume Node
see the successful results of our test case in the WebDriverIO output:
and the Node Package Manager (NPM) are already installed.
Spec Files: 1 passed, 1 total (100% completed) in
WEBDRIVERIO 00:00:08
WebDriverIO is an open-source project that utilizes the W3C
WebDriver standard to interact with browsers in a truly cross- For more information, see the following WebDriverIO resources:
compatible manner. All modern web browsers, including Chrome, •  What is WebDriver.IO?
Firefox, Safari, and Edge, support the WebDriver standard, which •  Automation Protocol
means that WebDriverIO supports nearly all browsers out of the box. •  Getting Started
Unfortunately, the Web has changed significantly since the WebDriver •  Boilerplate Projects
standard was devised, and it lags in its support for modern web •  NPM
development and testing.
NIGHTWATCH
To supplement deficiencies of the WebDriver standard, WebDriverIO
Nightwatch is an open-source project similar to WebDriverIO — both
also supports CDP, which enables the framework to use the
use the WebDriver standard to interact with the browser — but
standardized WebDriver interactions, while also allowing us to
Nightwatch focuses more on usability. This focus on ease-of-use
perform additional, more modern interactions with the browser.
results in cleaner syntax, selection through JavaScript and CSS or
This CDP interaction occurs over WebSockets to the browser, which
XPath, and support for Page Object Models (POMs).
means WebDriverIO can interact with the browser without the need
The usability of Nightwatch is also reflected in how quickly a test case
for a patched browser or external service.
can be created. Similar to WebDriverIO, we must first create a new
To run a WebDriverIO test, we must first create a new Node project: Node project:

mkdir webdriverio-examples mkdir nightwatch-examples

cd webdriverio-examples/ cd nightwatch-examples/
npm init -y npm init -y

4 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | JAVASCRIPT TEST AUTOMATION FR AMEWORKS

To install Nightwatch, we can run the following command: WebDriverIO, on the other hand, exposes many of these details,
which allows us to fine-tune and tweak our test configuration as
npm i --save-dev nightwatch geckodriver \
needed. In general, Nightwatch should be preferred when simplicity
chromedriver
and ease-of-use are a priority, while WebDriverIO should be used
when a greater level of control is required.
We can then run an example script, ecosia.js — contained in
node_modules/nightwatch/examples/tests/ — by running the
For more information, see the following Nightwatch resources:
following command:
•  Getting Started
npx nightwatch \ •  Developer Guide
node_modules/nightwatch/examples/tests/ecosia.js •  GitHub
•  NPM
The ecosia.js script performs the following operations:
PUPPETEER
1. Opens the Ecosia Search Engine — https://www.ecosia.org/
Puppeteer is an open-source project maintained by the Google
2. Asserts that the title is Ecosia
Chrome DevTools team that provides an abstracted API that
3. Asserts that the search box is visible interacts with Chrome and Chromium browsers over CDP. By default,
4. Enters nightwatch into the search box Puppeteer runs headlessly, but it can be configured to run using
5. Asserts that the Search button is visible Chrome or Chromium — in the same manner as WebDriverIO and
6. Clicks the Search button Nightwatch — as well.

7. Asserts that the results contain Nightwatch.js

To get started with Puppeteer, we must create a new Node project:

After we run this test case, we can see that all four of our assertions
mkdir puppeteer-examples
have passed: cd puppeteer-examples/
npm init -y
[Ecosia.org Demo] Test Suite
============================
ℹ Connected to localhost on port 4444 (9932ms).
Next, we need to install the Puppeteer packages:

Using: firefox (87.0) on linux 5.4.0-71-generic

npm i --save-dev puppeteer
platform.

Note that the puppeteer package installs Puppeteer and an

✓ Running Demo test ecosia.org:
accompanying version of the Chromium browser.

✓ Element <body> was visible after 466

If we want to forgo the installation of Chromium, we can install the
milliseconds.
puppeteer-core package instead. Regardless of our selection,
✓ Testing if the page title contains 'Ecosia'
(11ms) we can create a test script, example.js — from the official
✓ Testing if element <input[type=search]> is Puppeteer Usage page — that opens https://example.com/ and
visible (72ms) takes a screenshot of the browser window:
✓ Testing if element <button[type=submit]> is
const puppeteer = require('puppeteer');
visible (56ms)
✓ Testing if element <.mainline-results> contains
(async () => {
text 'Nightwatch.js' (3446ms)
const browser = await puppeteer.launch();
const page = await browser.newPage();
OK. 5 assertions passed. (7.359s)
await page.goto('https://example.com');
await page.screenshot({ path: 'example.png'
Compared to WebDriverIO, Nightwatch is quick to start and test
});
cases are simple to implement. This begs the question: Why use
WebDriverIO, then? What Nightwatch gains in simplicity and agility, it await browser.close();
sacrifices in control. To reduce the burden on developers, Nightwatch })();
abstracts a large portion of the details about the interaction of the
framework with the browser.

5 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | JAVASCRIPT TEST AUTOMATION FR AMEWORKS

We can then run this script using the following command: This command ensures that the browsers we require — Chromium,
Firefox, and WebKit — are also installed and ready for use. We can
node example.js
install these packages using the following commands:

Once the command completes, we can see a screenshot of https:// npm i --save-dev playwright
example.com/ in the current directory: sudo npx playwright install-deps

We can now create a new test case, example.js — provided by

Playwright — with the following content:

const playwright = require('playwright');

(async () => {
for (const browserType of ['chromium',
'firefox', 'webkit']) {
const browser = await playwright[browserType].
This is only a small fraction of the capability that Puppeteer contains. launch();
Puppeteer also allows us to programmatically perform nearly all the const context = await browser.newContext();
actions that we can perform manually in a browser, including: const page = await context.newPage();
await page.goto('http://whatsmyuseragent.
•  Providing input
org/');
•  Interacting with forms await page.screenshot({ path: `example-
•  Navigating Single-Page Applications (SPAs) ${browserType}.png` });

•  Obtaining performance timing information await browser.close();

}
•  Testing Chrome extensions
})();

For more information, see the following Puppeteer resources:

This test case opens http://whatsmyuseragent.org/ in Chrome,
•  GitHub
Firefox, and Safari, and it takes a screenshot of the web page in each
•  Google Web Developer page
browser. We can run this test case using the following command:
•  NPM

node example.js
PLAYWRIGHT
Playwright is the spiritual successor to Puppeteer. It was created
Once execution completes, we can see the following three
by Microsoft in 2020 (by the same team that created the original
screenshots (note the contents of each page differs based on
Puppeteer at Google) with the goal of bringing the same rich
the browser in which it was loaded):
functionality supported by Puppeteer to all mainstream browsers.
Instead of utilizing standard APIs for Firefox and WebKit browser Screenshot 1: Chromium
engines, Playwright patches these engines to enable support for
its APIs. This allows Playwright to support Chromium, Firefox, and
WebKit on Windows, Linux, and MacOS at the expense of requiring
that test cases be executed in a different browser than the one a user
will experience (i.e., a patched browser).

To create a new test case, we must create a new Node project:

mkdir playwright-examples
cd playwright-examples/
npm init -y

Once we have created the new project, we can install the playwright
package. We must also install the Playwright dependencies using the SEE SCREENSHOTS 2 AND 3 ON NEXT PAGE

npx playwright install-deps command.

6 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | JAVASCRIPT TEST AUTOMATION FR AMEWORKS

Screenshot 2: Firefox TestCafe is a powerful tool, and its architecture differs greatly from
the previous frameworks we have seen. This proxy architecture
ensures that tests are executed quickly and a vast array of browsers
are supported. The catch is that events — such as a button click —
are not executed natively in a browser, which differs from the user
experience and can lead to disparity in testing.

For example, a button may not be clickable when rendered by a

browser, but it is possible that a TestCafe test case can click the
button anyway.

To run a TestCafe test case, we must first create a new Node project:

mkdir testcafe-examples
Screenshot 3: WebKit cd testcafe-examples/
npm init -y

We can then install the testcafe package using the following

command:

npm i --save-dev testcafe

Next, we can create a simple test case, example.js — provided

by TestCafe — that enters text into an input field and clicks a
submit button:

import { Selector } from 'testcafe';

Although Playwright extends the features of Puppeteer to a wider fixture `Getting Started`
array of browsers, it does not replace Puppeteer. While it does offer a .page `http://devexpress.github.io/testcafe/
greater degree of interoperability, Playwright requires that patched example`;

browser releases be used — which introduces a disparity between

the browsers used by our applications' customers and the browsers test('My first test', async t => {

we use for testing. await t

.typeText('#developer-name', 'John Smith')
For more information, see the following Playwright resources: .click('#submit-button');
•  GitHub });

•  NPM
Finally, we can execute our test case using the following command:
TESTCAFE
npx testcafe chrome example.js
TestCafe is an open-source project that acts as a proxy server and
injects the code used for testing. In practice, TestCafe executes a
The testcafe command accepts two arguments: (1) the alias of the
test case that exercises a developer-specified URL, which is changed
browser to use and (2) the test script to execute. In our case, we want
by TestCafe to the URL of the proxy server, injecting the HTML and
to execute our test case in Chrome, so we supply an alias of chrome.
JavaScript necessary to execute the test. This page is then delivered
to the browser and the test case is executed. When more advanced A complete list of browser aliases can be found on the Test Cafe
interaction is needed, developers can use Client Scripts and Client Browsers page.
Functions. TestCafe also automatically waits for page objects to load,
which reduces the burden on developers when accessing certain
elements on a web page.

7 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | JAVASCRIPT TEST AUTOMATION FR AMEWORKS

Once the test successfully completes, the testcafe tool outputs the Once we create a new project, we can install the cypress package:
following, denoting that our test case, My first test, has passed:
npm i --save-dev cypress
Running tests in:
- Chrome 93.0.4577.63 / Linux 0.0
With the cypress package installed, we can create a new test in the

Getting Started
file, cypress/integration/example_spec.js:
✓ My first test
mkdir -p cypress/integration/
touch cypress/integration/example_spec.js
1 passed (8s)

For more information, see the following TestCafe resources: Once we create the example_spec.js file, we can add the following
content — provided by Cypress:
•  Getting Started
•  Guides
describe('My First Test', () => {
•  GitHub
it('clicking "type" navigates to a new url', ()
•  NPM
=> {
cy.visit('https://example.cypress.io')
CYPRESS
Cypress is an open-source project that runs test cases directly
cy.contains('type').click()
within the browser. Unlike other test frameworks that make remote
requests to the API of the browser or a standardized API, Cypress
cy.url().should('include', '/commands/
runs as a Node server process, which means that it runs in the same
actions')
event-loop as our application under test. This has many distinct
})
advantages over alternative approaches, including:
})
•  Faster test execution

•  Access to the network layer and web traffic Next, we can open the Cypress App using the following command:

•  Ability to execute operating system tasks (e.g., taking

npx cypress open
screenshots or video)

•  Access to useful debugging information With the dashboard open, we can click on our example_spec.js
file to open it:
Like TestCafe, Cypress also supports automatic waiting, which
removes the need for developers to add artificial waits for page
elements to load. This architecture does have some drawbacks,
though, including:

•  Supports only Chrome-based and Firefox browsers

•  Struggles with certain web security issues (although

workarounds are available)
Opening our example_spec.js file (by clicking on it) will open a
•  Constrains each test to a single origin
browser window and execute our test case:
•  Lacks multi-tab support

Although Cypress brings a lot to the table in terms of its continuity

with the application under test, this ability comes with some
sacrifices. Cypress can be a great automated test framework, but its
limitations should be well understood.

To run a Cypress test, we must first create a new Node project:

mkdir cypress-examples
cd cypress-examples/
npm init -y

8 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | JAVASCRIPT TEST AUTOMATION FR AMEWORKS

Apart from running our tests visually, we can also execute them
using the command line, which removes the need to open our test WRITTEN BY JUSTIN ALBANO,
SOFTWARE ENGINEER, IBM
case in a browser.
Justin Albano is a Software Engineer at IBM
For more information, see the following Cypress resources: responsible for building software-storage and
backup/recovery solutions for some of the largest
•  Getting Started Guide
worldwide companies, focusing on Spring-based REST API and
•  GitHub MongoDB development. When not working or writing, he can be
found practicing Brazilian Jiu-Jitsu, playing or watching hockey,
•  NPM
drawing, or reading.

CONCLUSION
The Web has changed dramatically over the last 30 years, and the
JavaScript automated test ecosystem has evolved alongside it to
better suit the needs of modern web application development.

DZone, a Devada Media Property, is the resource software developers,

Currently, three general approaches have solidified: engineers, and architects turn to time and again to learn new skills, solve
software development problems, and share their expertise. Every day,
1. Using standards such as WebDriver hundreds of thousands of developers come to DZone to read about the latest
technologies, methodologies, and best practices. That makes DZone the ideal
2. Using non-standard APIs for each browser, such as CDP place for developer marketers to build product and brand awareness and drive
sales. DZone clients include some of the most innovative technology and tech-
enabled companies in the world including Red Hat, Cloud Elements, Sensu, and
3. Using proprietary approaches such as proxy servers and pure Sauce Labs.
Node code
Devada, Inc.
600 Park Offices Drive
Regardless of the technique used, each JavaScript automated test Suite 150
Research Triangle Park, NC 27709
framework has its advantages and sacrifices. As we have seen in this
888.678.0399 | 919.678.0300
Refcard, the better we understand where each framework excels and
Copyright © 2021 Devada, Inc. All rights reserved. No part of this publication
falls short, the more equipped we are to select the test framework may be reproduced, stored in a retrieval system, or transmitted, in any form or
by means of electronic, mechanical, photocopying, or otherwise, without prior
that best fits our environment and context. written permission of the publisher.

9 BROUGHT TO YOU IN PARTNERSHIP WITH