Advanced Programming

Techniques
TESTING
Introduction
Testing and Quality Automation:
◦ Testing
◦ Selecting test cases
◦ Writing unit tests
◦ Checking for test coverage
◦ Exercise
◦ Testing with Pytest; with fake; doctest; unittest; mock
◦ References:
Expert Python Programming, 4th ed, Chapter 10, Code
◦ Expert Python Programming, 3rd ed, Test-Driven Development
◦ https://python-course.eu/advanced-python/tests-doctest-unittest.php
◦ https://python-course.eu/advanced-python/pytest.php
◦ https://python-textbok.readthedocs.io/en/1.0/Packaging_and_Testing.html

2
Testing and Quality Automation
 Complete software products are often composed of many
layers and relies on many external or interchangeable
components, such as operating systems, libraries, databases,
caches, web APIs, or clients used to interact with the code.
 The verification of complex modern software correctness often
requires you to go beyond your code. You need to consider the
environment in which your code runs, variations of
components that can be replaced, and the ways your code can
be interacted with.
 Developers of high-quality software often employ special
testing techniques that allow them to quickly and reliably verify
that the code they write meets desired acceptance criteria.

3
Testing and Quality Automation
 Another concern of complex software is its maintainability.
This can be understood as how easy it is to sustain the ongoing
development of a piece of software.
 Maintainable software is software that requires little effort to
introduce new changes and where there is a low risk of
introducing new defects upon change.
 Automated testing helps in reducing the risk of change by
enforcing that known use cases are properly covered by
existing and future code. But it is not enough to ensure that
future changes will be easy to implement.
 That is why modern testing methodologies also rely on
automated code quality measurement and testing to enforce
specific coding conventions, highlight potentially erroneous
code fragments, or scan for security vulnerabilities.

4
Testing and Quality Automation
 We review the most popular testing and quality automation
techniques that are often employed by professional Python
developers.
 We cover the following topics:
 The principles of test-driven development
 Writing tests with pytest
 Quality automation
 Mutation testing
 Useful testing utilities
 Packages from PyPI: (in terminal, PowerShell, etc)
pytest; redis; coverage; mypy; mutmut; faker; freezegun
$ py -m pip --version
$ py -m ensurepip --default-pip (if not installed)
$ pip install <package-name>

5
Errors and Tests
Usually, programmers and program developers spend a great deal of their
time with debugging and testing.
It is hard to give exact percentages, because it highly depends on other
factors like the individual programming style, the problems to be solved
and of course on the qualification of a programmer.
Without doubt, the programming language is another important factor.
Even though it is hardly possible to completely eliminate all errors in a
software product, we should always work ambitiously to this end, i.e., to
keep the number of errors minimal.
There are various kinds of errors.
 Syntactical errors
 Semantic errors
 Errors caused by lack of understanding of a language construct.
 Errors due to logically incorrect code conversion.

6
Automated Testing
For our code to be suitable for automated testing, we need to organize it in
logical subunits which are easy to import and use independently from outside
the program.
We should already be doing this by using functions and classes and avoiding
reliance on global variables. If a function relies only on its input parameters to
produce result, we can easily import this function into a separate testing
module, and check that various examples of input produce the expected
results. Each matching set of input and expected output is called a test case.
 Tests which are applied to individual components in our code are known as
unit tests – they verify that each of the components is working correctly.
 Testing the interaction between different components in a system is known
as integration testing.
 A test can be called a functional test if it tests a particular feature, or
function of the code – this is usually a relatively high-level specification of a
requirement, not an actual single function.

7
Unit Tests
 Unit Tests are used for testing units or components of the code,
typically, classes or functions. The underlying concept is to simplify
the testing of large programming systems by testing "small" units.
 Unit testing is a method whereby individual units of source code are
tested to determine if they meet the requirements, i.e. return the
expected output for all possible - or defined - input data.
 A unit can be seen as the smallest testable part of a program, which
are often functions or methods from classes.
 Testing one unit should be independent from the other units as a
unit is "quite" small, i.e. manageable to ensure complete
correctness.
 Usually, this is not possible for large scale systems like large software
programs or operating systems.

8
Module Tests with name
 Every module has a name, which is defined in the built-in attribute
__name__.
 Let's assume that we have written a module "xyz" which we have
saved as "xyz.py".
 If we import this module with "import xyz", the string "xyz" will be
assigned to __name__.
 If we call the file xyz.py as a standalone program, i.e. in the following
way,

$ python3 xyz.py
the value of __name__ will be the string '__main__'.
It is possible to create a simple module test inside of a module file, - in
our case the file "xyz.py", - by using an if statement and checking the
value of __name__.

9
doctest Module
 The doctest module is often considered easier to use than the
unittest, though the latter is more suitable for more complex tests.
 doctest is a test framework that comes prepackaged with Python.
 The doctest module searches for pieces of text that look like
interactive Python sessions inside the documentation parts of a
module, and then executes (or reexecutes) the commands of those
sessions to verify that they work exactly as shown, i.e., that the
same results can be achieved.
 In other words: The help text of the module is parsed, for example,
python sessions. These examples are run and the results are
compared to the expected value.
 Usage of doctest: "doctest" has to be imported. The part of an
interactive Python sessions with the examples and the output has to
be copied inside the docstring the corresponding function.

10
unittest Module
 The Python module unittest is a unit testing framework, which is
based on Erich Gamma's JUnit and Kent Beck's Smalltalk testing
framework.
 The module contains the core framework classes that form the basis
of the test cases and suites (TestCase, TestSuite and so on), and also
a text-based utility class for running the tests and reporting the
results (TextTestRunner).
 The most obvious difference to the module "doctest" lies in the fact
that the test cases of the module "unittest" are not defined inside
the module, which has to be tested.
 The major advantage is clear: program documentation and test
descriptions are separate from each other. The price you have to pay
on the other hand, is an increase of work to create the test cases.

11
The Principles of Test-Driven
Development (TDD)
 Testing is one of the most important elements of the software
development process. It is so important that there is even a
software development methodology called Test-Driven
Development (TDD).
 TDD advocates writing software requirements as tests as the
first and foremost step in developing code.
 The principle is simple: you focus on the tests first.
 Use tests to describe the behavior of the software, verify it,
and check for potential errors.
 Only when those tests are complete should you proceed with
the actual implementation to satisfy the tests.

12
The Principles of Test-Driven
Development (TDD)
 TDD, in its simplest form, is an iterative process that consists of
the following steps:
1. Writing tests: Tests should reflect the specification of a
functionality or improvement that has not been implemented yet.
2. Run tests: At this stage, all new tests should fail as the feature or
improvement is not yet implemented.
3. Write a minimal valid implementation: The main focus at this
stage should be satisfying all the tests written in step 1.
4. Run tests: At this stage, all tests should pass. If any of them fail, the
code should be revised until it satisfies the requirements.
5. Hone and polish: When all tests are satisfied, the code can be
progressively refactored until it meets desired quality standards.
After each change, all tests should be rerun to ensure that no
functionality was broken.

13
The Principles of Test-Driven
Development (TDD)
 It is important to follow some basic principles:
 Keep the size of the tested unit small:
 A unit of code is a simple autonomous piece of software that (preferably)
should do only one thing.
 A single unit test should exercise one function or method with a single set
of arguments.
 Keep tests small and focused: Every test should verify only one
aspect/requirement of the intended functionality. Small tests
pinpoint problems better and are just easier to read.
 Keep tests isolated and independent: If a test relies on a specific
state of the execution environment, the test itself should ensure that
all preconditions are satisfied. Similarly, any side effects of the test
should be cleaned up after the execution. Those preparation and
cleanup phases of every test are also known as setup and teardown.

14
The Principles of Test-Driven
Development (TDD)
 The Python standard library comes with two built-in modules created
exactly for the purpose of automated tests:
 doctest : https://docs.python.org/3/library/doctest.html
A testing module for testing interactive code examples found in
docstrings. It is a convenient way of merging documentation with
tests. It is theoretically capable of handling unit tests but it is more
often used to ensure that snippets of the code found in docstrings
reflect the correct usage examples.
 unittest : https://docs.python.org/3/library/unittest.html
A full-fledged testing framework inspired by JUnit (a popular Java
testing framework). It comes with a built-in test runner that is able to
discover test modules across the whole codebase and execute
specific test selections.
These two modules together can satisfy most of the testing needs of
even the most demanding developers.
Unfortunately, doctest concentrates on a very specific use case for
tests (the testing of code examples) and unittest requires a rather
large amount of boilerplate due to class-oriented test organization

15
Pytest
 Many professional programmers prefer using one of the
third-party frameworks available on PyPI. One such
framework is pytest.
 unittest and doctest are still great and useful but pytest
is almost always a better and more practical choice.
 Writing tests with pytest
Code from Expert Python Programming, 4th ed.,
Chapter 10, 01 - Writing tests with pytest
- batch_1st_iteration.py
- batch.py
- test_batch.py

16
Testing with Pytest
Many tests follow a very common structure:
 Setup: This is the step where the test data and all other
prerequisites are prepared. In our case the setup consists of
preparing iterable and batch_size arguments.
 Execution: This is when the actual tested unit of code is put into
use and the results are saved for later inspection. In our case it is a
call to the batches() function.
 Validation: In this step we verify that the specific requirement is
met by inspecting the results of unit execution. In our case these
are all the assert statements used to verify the saved output.
 Cleanup: This is the step where all resources that could affect other
tests are released or returned back to the state they were in before
the setup step. We didn't acquire any such resources, so in our case
this step can be skipped.

17
Testing with Pytest
Ensure pytest is installed:
$ pip install pytest
$ pytest –v
collected 3 items
test_batch.py::test_batch_on_lists PASSED [ 33%]
test_batch.py::test_batch_order PASSED [ 66%]
test_batch.py::test_batch_sizes PASSED [100%]
3 passed in 0.03s

If you did not implement the functions properly, the tests will fail.

18
Test Parametrization
 Using a direct comparison of the received and expected function output is a
common method for writing short unit tests.
 test_batch_on_lists() slightly breaks the keep tests small and focused
principle.
 test_batch_with_loop() improves the test structure by moving the
preparation of all the samples to the separate setup part of the test and
then iterating over the samples in the main execution part.
 pytest comes with native support for test parameterization in the form of
the @pytest.mark.parametrize decorator, which requires at least two
positional arguments:
 argnames: list of argument names that pytest will use to provide test
parameters to the test function as arguments. It can be a comma-separated
string or a list/tuple of strings.
 argvalues: an iterable of parameter sets for each individual test run. Usually, it
is a list of lists or a tuple of tuples.
 There four separate instances of the test_batch_parameterized() test run.

19
Test Parametrization
 Test parameterization effectively puts a part of classic test
responsibility—the setup of the test context—outside of
the test function.
 This allows for greater reusability of the test code and gives
more focus on what really matters: unit execution and the
verification of the execution outcome.

20
Pytest’s fixtures
 Another way of extracting the setup responsibility from the
test body is through the use of reusable test fixtures.
 pytest already has great native support for reusable fixtures
that is truly magical.
 The term "fixture" comes from mechanical and electronic
engineering. It is a physical device that can take the form of
a clamp or grip that holds the tested hardware in a fixed
position and configuration (hence the name "fixture") to
allow it to be consistently tested in a specific environment.
 Software testing fixtures serve a similar purpose. They
simulate a fixed environment configuration that tries to
mimic the real usage of the tested software component.
21
Pytest’s fixtures
 Using fixtures to provide connectivity for external services like
Redis. The installation of Redis is simple and does not require
any custom configuration to use it for testing purposes.
 Fixtures can be anything from specific objects used as input
arguments, through environment variable configurations, to sets
of data stored in a remote database that are used during the
testing procedure.
 In pytest, a fixture is a reusable piece of setup and/or teardown
code that can be provided as a dependency to the test functions.
 pytest has a built-in dependency injection mechanism that
allows for writing modular and scalable test suites.

22
Pytest’s fixtures
 To create a pytest fixture you need to define a named
function and decorate it with the @pytest.fixture
decorator as in the following example:
import pytest
@pytest.fixture
def dependency():
return "fixture value"

 pytest runs fixture functions before test execution.

 The return value of the fixture function (here "fixture
value") will be provided to the test function as an input
argument.

23
Pytest’s fixtures
 It is also possible to provide both setup and cleanup code in the
same fixture function by using the following generator syntax:
@pytest.fixture
def dependency_as_generator():
# setup code
yield "fixture value"
# teardown code

 When generator syntax is used, pytest will obtain the yielded

value of the fixture function and keep it suspended until the test
finishes its execution.
 After the test finishes, pytest will resume execution of all used
fixture functions just after the yield statement regardless of the
test result (failure or success).
 This allows for the convenient and reliable cleanup of the test
environment.

24
Pytest’s fixtures
 To use a fixture within a test you need to use its name as an
input argument of the test function:
def test_fixture(dependency):
pass

 When starting a pytest runner, pytest will collect all fixture

uses by inspecting the test function signatures and
matching the names with available fixture functions.
 By default, there are a few ways that pytest will discover
fixtures and perform their name resolution:
 Local fixtures: from the same module that they are defined in; take
precedence over shared fixtures
 Shared fixtures: in the conftest module stored in the same
directory as the test module or any of its parent directories.
 Plugin fixtures: provide their own fixtures; matched last.

25
Pytest’s fixtures
 Last but not least, fixtures can be associated with specific scopes that
decide the lifetime of fixture values. These scopes are extremely
important for fixtures implemented as generators because they
determine when cleanup code is executed.
 There are five scopes available:
 "function" scope: This is the default scope. A fixture function with the
"function" scope will be executed once for every individual test run and
will be destroyed afterward.
 "class" scope: This scope can be used for test methods written in the
xUnit style (based on the unittest module). Fixtures with this scope
are destroyed after the last test in a test class.
 "module" scope: destroyed after the last test in the test module.
 "package" scope: destroyed after the last test in the given test package
(collection of test modules).
 "session" scope: This is kind of a global scope. Fixtures with this scope
live though the entire runner execution; destroyed after the last test.

26
Markers in Pytest
 Test functions can be marked or tagged by decorating them with
'pytest.mark.’.
 Such a marker can be used to select or deselect test functions.
 You can see the markers which exist for your test suite by typing
$ pytest --markers
 Registering markers
 skipif markers
 Parametrization with markers
 Prints in functions
 Command Line Options / Fixtures
Read from: https://python-course.eu/advanced-python/pytest.php

27
Mocks and the unittest.mock
module
Mock objects are generic fake objects that can be used to isolate the
tested code. They automate the building process of the fake object's
input and output. There is a greater level of use of mock objects in
statically typed languages, where monkey patching is harder, but they
are still useful in Python to shorten the code that mimics external APIs.
There are a lot of mock libraries available in Python, but the most
recognized one is unittest.mock, which is provided in the standard
library.
See example
Run
$ py.test -v --tb line

28
Quality automation
We can measure various metrics of the software that are known
to be highly correlated with the quality of code.
The following are a few:
 The percentage of code covered by tests
 The number of code style violations
 The amount of documentation
 Complexity metrics, such as McCabe's cyclomatic complexity
 The number of static code analysis warnings
Many projects use code quality testing in their continuous
integration workflows. A good and popular approach is to test at
least the basic metrics (test coverage, static code analysis, and
code style violations) and not allow the merging of any code to
the main branch that scores poorly on these metrics.

29
Test coverage
Test coverage, also known as code coverage, is a very useful metric that
provides objective information on how well some given source code is
tested.
It is simply a measurement of how many, and which lines of code are
executed during the test run. It is often expressed as a percentage, and
100% coverage means that every line of code was executed during tests.
The most popular code coverage tool for measuring Python code is the
coverage package, and it is freely available on PyPI.
Its usage is very simple and consists only of two steps:
1. Running the test suite using the coverage tool
2. Reporting the coverage report in the desired format

You need Redis installed to be able to test everything.

30
Mutation testing
 Countless projects with high coverage discover new bugs in parts of
the code that are already covered by tests.
 Requirements may not be clear, and tests do not cover what they were
supposed to cover.
 Tests may include errors.
 Tests are just code and like any other code are susceptible to bugs.

 Sometimes bad tests are just empty shells - they execute some units
of code and compare some results but do not actually care about
really verifying software correctness.
 One of the ways to verify the quality of tests is to deliberately modify
the code in a way that we know would definitely break the software
and see if tests can discover the issue.

31
Mutation testing
 If at least one test fails, we are sure that they are good enough to
capture that error.
 If none of the tests fails, we may need to consider revisiting the test
suite.
 As possibilities for errors are countless, it is hard to perform this
procedure often and repeatedly without the aid of tools and specific
methodologies. One such methodology is mutation testing.
 Mutation testing works on the hypothesis that most faults of software are
introduced by small errors like off-by-one errors, flipping comparison
operators, wrong ranges, and so on.
 There is also the assumption that these small mistakes cascade into larger
faults that should be recognizable by tests.

32
Mutation testing
 Mutation testing uses well-defined modification operators known as
mutations that simulate small and typical programmer mistakes.
 Examples of typical programming mistakes:
 Replacing the == operator with the is operator
 Replacing a 0 literal with 1
 Switching the operands of the < operator
 Adding a suffix to a string literal
 Replacing a break statement with continue
 In each round of mutation testing, the original program is modified
slightly to produce a so-called mutant.
 If the mutant can pass all the tests, we say that it survived the test.
 If at least one of the tests failed, we say that it was killed during the test.
 The purpose of mutation testing is to strengthen the test suite so that it
does not allow new mutants to survive.

33
Mutation testing
 Example: test an is_prime() function that is supposed to verify
whether an integer number is a prime number or not.
 A prime number is a natural number greater than 1 that is divisible
only by itself and 1.
 There is no easy way to test the is_prime() function other than
providing some sample data.
 We will start with the following simple test (test_primes.py):
from primes import is_prime
def test_primes_true():
assert is_prime(5)
assert is_prime(7)
def test_primes_false():
assert not is_prime(4)
assert not is_prime(8)

34
Mutation testing
 A very naïve implementation and easy to understand implementation
of the is_prime() function:
def is_prime(number):
if not isinstance(number, int) or number < 0:
return False
if number in (0, 1):
return False
for element in range(2, number):
if number % element == 0:
return False
return True

35
Mutation testing
 There are a few mutation testing tools available on PyPI. One that
seems the simplest to use is mutmut, and we will use it in our
mutation testing session.
 mutmut requires you to define a minor configuration that tells it how
tests are run and how to mutate your code. It uses its own [mutmut]
section in the common setup.cfg file.
 Our configuration will be the following:
[mutmut]
paths_to_mutate=primes.py
runner=python -m pytest –x
 The paths_to_mutate variable specifies paths of the source files
that mutmut is able to mutate.
 The runner variable specifies the command that is used to run tests.

36
Mutation testing
 Mutation testing in large projects can take a substantial amount of
time so it is crucial to guide mutmut on what it is supposed to mutate,
just to save time.
 Here we use pytest with the -x flag, which tells pytest to abort testing
on the first failure.
 The mutation testing session involves mutmut tool's usage. The work
starts with the run subcommand (after installing mutmut):
$ mutmut run

37
Mutation testing
- Mutation testing starting -
These are the steps:
1. A full test suite run will be made to make sure we
can run the tests successfully and we know how long
it takes (to detect infinite loops for example)
2. Mutants will be generated and checked
Results are stored in .mutmut-cache.
Print found mutants with `mutmut results`.
Legend for output:
🎉🎉 Killed mutants. The goal is for everything to end up in this bucket.
⏰ Timeout. Test suite took 10 times as long as the baseline so were killed.
🤔🤔 Suspicious. Tests took a long time, but not long enough to be fatal.
🙁🙁 Survived. This means your tests need to be expanded.
🔇🔇 Skipped. Skipped.
mutmut cache is out of date, clearing it...
1. Running tests without mutations
⠧ Running...Done
2. Checking mutants
⠇ 15/15 🎉🎉 8 ⏰ 0 🤔🤔 0 🙁🙁 7 🔇🔇 0

38
Mutation testing
 We can get a detailed view of the results by running the mutmut
results command:
$ mutmut results

To apply a mutant on disk:

mutmut apply <id>

To show a mutant:
mutmut show <id>

Survived 🙁🙁 (7)
---- primes.py (7) ----
2-5, 7-9

39
Mutation testing
 We can see that 7 mutants survived, and their identifiers are in the 2-5
and 7-9 ranges.
 The output also shows useful information on how to review mutants
using the mutmut show <id> command.
 We can also review mutants in bulk using the source file name as the
<id> value.

$ mutmut show 2
$ mutmut show 9
$ mutmut show primes.py

40
Mutation testing
 If we extend the test suite to cover more corner cases and invalid
values, it will probably make it more robust.
 The following is a revised set of tests:
from primes import is_prime
def test_primes_true():
assert is_prime(2)
assert is_prime(5)
assert is_prime(7)
def test_primes_false():
assert not is_prime(-200)
assert not is_prime(3.1)
assert not is_prime(0)
assert not is_prime(1)
assert not is_prime(4)
assert not is_prime(8)

41
Mutation testing
 Mutation testing is an interesting technique that can strengthen the
quality of tests. One problem is that it does not scale well.
 On larger projects, the number of potential mutants will be big and in
order to validate them, you must run the whole test suite.
 It will take a lot of time to execute a single mutation session if you
have many long-running tests. Hence, mutation testing works well
with simple unit tests but is very limited when it comes to integration
testing. Still, it is a great tool for poking holes in those perfect
coverage test suites.
 So far, we have been focusing on systematic tools and approaches for
writing tests and quality automation. These systematic approaches
create a good foundation for your testing operations but do not
guarantee that you will be efficient in writing tests or that testing will
be easy. Testing sometimes can be tedious and boring.
 What makes it more fun is the large collection of utilities available on
PyPI that allows you to reduce the boring parts.
42
Useful testing utilities
 Faking realistic data values (with faker package and pytest)
 Names of people
 Addresses
 Telephone numbers
 Email addresses
 Identification numbers like tax or social security identifiers
 Faking time values (with breakpoint() or freezegun package)
 It may happen that for some reason you would like to change the way
your application experiences the passage of time.
 This could be useful in testing time-sensitive processing like work
scheduling or inspecting the automatically assigned creation timestamps
of specific objects.
 Freezing time should of course be done really carefully. If your application
actively checks the current time with time.time() and waits until a
certain time passes, you could easily lock it in indefinite sleep.

43
Another Example
See the Examples and Exercise 1 and its answer at:
https://python-textbok.readthedocs.io/en/1.0/Packaging_and_Testing.html#testing

44
Summary
1. The most important thing in developing software with TDD is
always starting with tests. That is the only way to ensure that
code units are easily testable.
2. Caring about software correctness and maintainability does not
end with testing and quality automation.
• They allow us to verify the requirements we know about and fix
bugs we have discovered.
• We can of course deepen the testing suite, and we have learned
that mutation testing is an effective technique to discover
potential testing blindspots, but this approach has its limits.
3. What follows next is usually the constant monitoring of the
application and listening to bug reports submitted by the users.
4. Before your users will be able to get their hands on your
application, you need to package and ship it.
45