SCHOOL OF INFORMATION TECHNOLOGY & ENGINEERING

WINTER SEMESTER 2022-23

Subject Code & Name: ITE2004 & Software Testing

Program: B.Tech - IT

Digital Assignment - 1

Garv Tandon (20BIT0130)

Tanishq Tyagi (20BIT0192)
Priyanka Chowdhury (20BIT0197)
Manmeet Singh Tuteja (20BIT0383)

Evaluating Software Testing Frameworks for a Node JS Application

Introduction

Unit Testing
Unit testing is a software testing technique that involves testing small, independent
units of code in isolation. These units of code are usually the most minor logical
parts of a system. It's important that unit tests are isolated from external
dependencies.

Benefits of Unit Testing

1. Unit testing detects code defects. Each code component should have suitable
test cases. A test failure on code changes signifies a bug. Moreover,
pinpointing an error is easier.
 2. Unit tests provide self-documentation. A new member can learn about the
codebase by examining them to understand component interactions and
behavior expectations.
3. Unit tests simplify debugging, highlighting recent changes when tests fail,
and aiding in pinpointing and resolving errors.
4. Unit tests facilitate code refactoring by allowing changes to be tested,
ensuring that the targeted unit still performs as expected.
5. Lower expenses in resolving errors and system failures.

Factors for evaluating the Node JS framework:

1. Ease of Setup
2. Well-supported
3. A wide array of feature sets
4. Speed
5. Ease of reporting
6. Ease of integration

Mocha
Mocha has been around for a long; it was first launched in November of 2011. It
does, however, rely on third-party assertions, mocking, and spying tools like Sinon
and Chai, unlike other frameworks such as Jest and Jasmine. It is extremely
extensible, with numerous plugins, extensions, and libraries built to run on top of
it.

Installation Command: npm i –global mocha

Pros:
1. Extensible, including support for many assertions and mocking libraries.
2. Simple asynchronous testing
3. It is relatively simple to add generator support to test suites. All you have to
do to utilize generators with the co-mocha package is need it in your tests.
4. Some CI servers and plugins for others support it.

Cons:
 1. The usage of additional libraries can increase configuration complexity and
maintenance labour.

Jest
Facebook created and maintains the Jest JavaScript testing framework. Its
popularity has slowly climbed since 2016, when only 6% of respondents to that
year's "State of JS" survey claimed they have used and would use Jest again. This
percentage increased to a quarter of respondents in 2017, then to 40% in 2018.
According to the most recent version, 73 percent of JavaScript developers had tried
Jest and intended to use it again.

Installation Command: npm install --save-dev jest

Pros:
1. Comprehensive documentation contains extensive directions for setting up
testing, writing different sorts of tests, and utilizing its numerous features, as
well as excellent examples.
2. Simple installation with flexible and simple configuration and less
boilerplate code than competing frameworks.
3. Parallel test execution is enabled.
4. To maximize performance, tests are parallelized by running them in their
own processes.
5. Snapshots, coverage, and test monitoring are all useful aspects.

Cons:
1. Multiple messages are displayed for the same issue.
2. It may necessitate more dependencies during the first setup (e.g., Babel)

Jasmine
Jasmine, was created by Pivotal Labs and released in 2010. It is designed to run on
any JavaScript-enabled platform and is highly adaptable and compatible with a
wide range of other testing frameworks and libraries, including Sinon and Chai.
Because of its longevity, it has a sizable community and plenty of support in the
form of libraries, blog articles, and tutorials.
Installation Command: npm install -g jasmine

Pros:
1. Setup is simple — Jasmine has a CLI tool that generates a spec file and a
JSON configuration file, so one can start testing your code with just one
command.
2. It has been thoroughly tested, and documented, and is backed up by
numerous tutorials on how to use it.
3. Development based on behavior and descriptive syntax
4. Many CI servers support it, and plugins are available for those that don't
have out-of-the-box support.

Cons:
1. Complicated error logs
2. A specific suffix is required for test files (e.g., spec.js)
3. The assertion library is not as extensive as Chai's.

AVA
AVA's emphasis is on minimalism. It has a simple API but supports advanced
features. It achieves its lightning-fast performance by running tests in parallel as
separate Node procedures. It does not create test globals, unlike other testing
frameworks including Jest and Jasmine.

Installation Command: npm i ava

Pros:
1. Simple to use.
2. To install and configure AVA, simply run npm init ava.
3. Native ES6/ES7 support is being tested in parallel.
4. Async functions are built-in.
5. If a promise is returned, you do not need to terminate the test; it will
terminate when the promise resolves.

Cons:
 1. AVA is a relatively new technology.
2. The community is still developing, and there isn't as much documentation or
tutorials as there are for other testing frameworks.
3. On GitHub, AVA has a large number of open issues.

Comparison between the Node JS Testing Frameworks:

Feature Mocha Jest Jasmine AVA

Language JavaScript JavaScript JavaScript JavaScript
Jest's own
assertions or
Assertion any assertion
Library Any library Any Any
Requires
Parallel Test external Built-in Requires
Running library support external library Built-in support
Requires Requires
manual manual
Isolated isolation of Automatic isolation of Automatic test
Tests tests test isolation tests isolation
Requires
DOM external Built-in Requires No built-in
Testing library support external library support
Requires
external
Spies/Stubs/ libraries like Built-in Built-in No built-in
Mocks Sinon support support support
Requires
external Requires
Test library like Built-in external library
Coverage Istanbul support like Istanbul Built-in support
 Fast,
Slower than comparable to Slower than
Performance Jest and AVA AVA Jest and AVA Fastest
Requires more
setup and Easy to set up Easy to set up Easy to set up
Ease of Use configuration and use and use and use
Large and Large and Large and Smaller and less
Community active active active active

Methodology
We created a basic Javascript module called Math.js and utilized different
Javascript testing frameworks to showcase their functionalities by writing test
cases for the module.
We wrote tests in the following frameworks.

1. Ava
2. Jasmine
3. Jest
4. Mocha
System under test
The Math.js module we will use as a sample is very straightforward. It consists of
four functions, all of which take two arguments and return a result.
● Math.add()
● Math.subtract()
● Math.multiply()
● Math.divide()
Testing with Ava:
Result:

Testing with Jasmine :

math.spec.js
Result:

In contrast to Ava, Jasmine includes globals for every function, so you don't need
to import anything. (This is one of the reasons we preferred Ava over Jasmine or
Mocha; both of the latter frameworks create globals by default which increases the
load of memory.)
We also need to create a spec(configuration file) for Jasmine which was not needed
for Ava.
Testing with Jest:
Result:

Jest code is very similar to Jasmine code. The reason for that is that jest is just a
wrapper around Jasmine. The main difference is that to use our implementation of
Math we had to call jest.unmock() which was not needed in Jasmine.
Testing with Mocha:
Result

Jasmine and Mocha are comparable, but Mocha is much more customizable. In
actuality, Mocha lacks both built-in assertions and mocking while Jasmine does.
Instead, a mocking library and an assertion library must be included explicitly. We
have used the inbuilt node js assertions.

Summary:
1. Assertion libraries: Mocha and Jasmine require external assertion libraries
such as Chai or should.js, while Jest has its built-in assertions. AVA supports
both built-in and external assertion libraries.

2. Mocking and Spying: Mocha and Jasmine require external mocking and
spying libraries like Sinon, while Jest has built-in mocking and spying
functionality. AVA also requires external libraries for mocking and spying.

3. Extensibility: Mocha is highly extensible and supports various assertion and

mocking libraries, while Jest has fewer extension points but comes with
useful features like snapshots, coverage, and test watching. Jasmine is also
extensible and compatible with a variety of testing frameworks and libraries.
AVA has a simple API, supporting advanced features and easy to use.
4. Documentation and Community Support: Jest has comprehensive
documentation, examples, and a growing community, while Mocha,
Jasmine, and AVA also have good documentation but with less community
support.

5. Test Parallelization: Mocha runs tests serially, while Jest and AVA run tests
in parallel, which can speed up the test execution process.

6. Performance: Jest and AVA are optimized for performance, and both run
tests in separate processes, while Mocha and Jasmine do not.

7. Syntax: Mocha and Jasmine use BDD-style syntax, while Jest and AVA use
a combination of BDD and TDD-style syntax.

8. Configuration and Setup: Jest has an easy setup with flexible and
straightforward configuration, while Mocha and Jasmine require more
configuration and set up. AVA is easy to install and set up.

9. Testing Approach: Mocha and Jasmine support both unit and integration
testing, while Jest emphasizes snapshot testing and functional testing. AVA
also supports both unit and integration testing.

10. Error Handling and Logging: Jest provides useful error messages and
stack traces, while Mocha and Jasmine's error messages can be difficult to
understand. AVA has its error messages, which are concise and easy to
understand.
FRAMEWORK MOCHA JEST JASMINE AVA
Open-source YES YES YES YES
In-built coverage
reporting NO NO YES NO

Parallel test running NO YES YES NO

Snapshots NO YES YES NO

In-built spies YES NO YES NO

In-built mocking YES NO YES NO

In-built assertions YES YES YES NO

ES2017 support NO YES YES NO

Assertions NO YES YES YES

Code coverage NO YES NO NO

Snapshot testing NO YES NO NO

Extensibility YES YES YES YES

Ready-To-Go NO YES YES YES

Mocks NO YES YES YES