Lecture 8: Unit Testing

and Test Automation

Gregory Gay
DIT635 - February 12, 2021
Today’s Goals
• Unit Testing
• Testing of individual classes
• Writing and executing test cases
• How to write unit tests in JUnit.
• Executing tests as part of a build script.

2018-08-27 Chalmers University of Technology 2

Testing Stages API GUI CLI

• We interact with systems API

through interfaces.
• APIs, GUIs, CLIs
• Systems built from subsystems.
API
• With their own interfaces.
• Subsystems built from units.
• Communication via method calls.
• Set of methods is an interface.
3
Unit Testing
• Testing the smallest “unit” that can be tested.
• Often, a class and its methods.
• Tested in isolation from all other units.
• Mock the results from other classes.
• Test input = method calls.
• Test oracle = assertions on output/class variables.

4
Testing Percentages
• Unit tests verify behavior
of a single class.
• 70% of your tests.
• System tests verify class
interactions.
• 20% of your tests.
• GUI tests verify
end-to-end journeys.
• 10% of your tests.

5
Unit Testing Account

• For a unit, tests should: - name

- personnummer
• Test all “jobs” associated with the unit. - balance
• Individual methods belonging to a class. Account (name,
• Sequences of methods that can interact. personnummer, Balance)

• Set and check class variables. withdraw (double amount)

• Examine how variables change after deposit (double amount)
changeName(String name)
method calls.
getName()
• Put the variables into all possible states getPersonnummer()
(types of values). getBalance()

6
Unit Testing - Account
Account
Unit tests should cover:
- name ● Set and check class variables.
- personnummer ○ Can any methods change name,
- balance
personnummer, balance?
Account (name, ○ Does changing those create problems?
personnummer, Balance)
● Each “job” performed by the class.
withdraw (double amount)
deposit (double amount) ○ Single methods or method sequences.
changeName(String name) ■ Vary the order methods are called.
getName()
getPersonnummer() ○ Each outcome of each “job” (error
getBalance() handling, return conditions).
7
Unit Testing - Account
Account
Some tests we might want to write:
- name
• Execute constructor, verify fields.
- personnummer
- balance
• Check the name, change the name,
make sure changed name is in place.
Account (name,
personnummer, Balance) • Check that personnummer is correct.
withdraw (double amount) • Check the balance, withdraw money,
deposit (double amount) verify that new balance is correct.
changeName(String name)
getName() • Check the balance, deposit money,
getPersonnummer()
getBalance() verify that new balance is correct.
8
Unit Testing - Account
Account
Some potential error cases:
• Withdraw more than is in balance.
- name
- personnummer • Withdraw a negative amount.
- balance
• Deposit a negative amount.
withdraw(name,
Account (double amount)
deposit (double amount)
personnummer, Balance) • Withdraw/Deposit a small amount
changeName(String name) (potential rounding error)
getName()
withdraw (double amount)
getPersonnummer()
deposit (double amount) • Change name to a null reference.
getBalance()
changeName(String name)
getName() • Can we set an “malformed” name?
getPersonnummer()
getBalance() • (i.e., are there any rules on a valid name?)

9
Unit Testing and Test Automation

10
Executing Tests
• How do you run test cases on the program?
• System level: could run code and check results by hand.
• Please don’t do this.
• Humans are slow, expensive, and error-prone.
• Exception - exploratory and acceptance testing.
• Test design requires effort and creativity.
• Test execution should not.

11
Test Automation
• Development of software to separate repetitive
tasks from creative aspects of testing.
• Control over how and when tests are executed.
• Control environment and preconditions/setup.
• Automatic comparison of predicted and actual output.
• Automatic hands-free re-execution of tests.

12
Testing Requires Writing Code
• The component to be tested must be isolated and
driven using method or interface calls.
• Untested dependencies must be mocked with
reliable substitutions.
• The deployment environment must be simulated by
a controllable harness.

13
Test Scaffolding
• Test scaffolding is a set of programs written to
support test automation.
• Not part of the product, often temporary
• Allows for:
• Testing before all components complete.
• Testing independent components.
• Control over testing environment.

14
Test Scaffolding
• A driver substitutes for a main or calling program.
• Test cases are drivers.
• A harness substitutes for part of the deployment
environment.
• A stub (or mock object) substitutes for system
functionality that has not been tested.
• Support for recording and managing test execution.

15
 Test Scaffolding
● Simulates the execution environment.
● Can control network conditions,
environmental factors,
operating systems.

● Templates that provide functionality

● Initializes objects and allow testing in isolation
● Initializes parameter variables
● Performs the test
● Performs any necessary
cleanup steps.
● Checks the correspondence between the produced and
expected output and renders a test verdict.
16
Writing an Executable Test Case
• Test Input
• Any required input data.
• Expected Output (Test Oracle)
• What should happen, i.e., values or exceptions.
• Initialization
• Any steps that must be taken before test execution.
• Test Steps
• Interactions (e.g., method calls), and output comparisons.
• Tear Down
• Steps that must be taken after execution to prepare for the next test.
17
Writing a Unit Test
JUnit is a Java-based toolkit
for writing executable tests.
• Choose a target from the
code base.
• Write a “testing class”
containing a series of unit
tests centered around
testing that target.

18
JUnit Test Skeleton
@Test annotation defines a single test:
Type of scenario, and expectation on outcome.
I.e., or

19
Writing JUnit Tests Convention - name the test class
after the class it is testing.

Each test is denoted with keyword

@test.

Initialization
Test Steps Input
Oracle

20
Test Fixtures - Shared Initialization
@BeforeEach annotation defines a common test
initialization method:

21
Test Fixtures - Teardown Method
@AfterEach annotation defines a common test tear
down method:

22
More Test Fixtures
• @BeforeAll defines
initialization to take
place before any
tests are run.
• @AfterAll defines
tear down after all
tests are done.

23
Assertions
Assertions are a "language" of testing - constraints that
you place on the output.

24
assertEquals
● Compares two items for
equality.
● For user-defined classes,
relies on method.
○ Compare field-by-field
○

rather than

● assertArrayEquals
compares arrays of items.
25
assertFalse, assertTrue
● Take in a string and a
boolean expression.
● Evaluates the expression
and issues pass/fail based
on outcome.
● Used to check
conformance of solution to
expected properties.
26
assertSame, assertNotSame

● Checks whether two

objects are clones.
● Are these variables aliases
for the same object?
○ assertEquals uses
.equals().
○ assertSame uses ==

27
assertNull, assertNotNull

● Take in an object and

checks whether it is
null/not null.
● Can be used to help
diagnose and void null
pointer exceptions.

28
Grouping Assertions

● Grouped assertions are

executed.
○ Failures are reported
together.
○ Preferred way to
compare fields of two
data structures.

29
assertThat not(allOf(...))
anyOf
items- -ata least
haseveryItem
both
allOf -- -two
either
all-contains
-list ifitems
properties
allpass
listedif one
one
all of
ofinthese
the
of must
properties
an listed
list
these beproperties
indicated
must
must
match
met. subset
properties
a of items,
be trueis true.
butproperties
are
cantrue,
alsothe
property. must
testbe
contain should
true items.
other fail.

30
Testing Exceptions
● When testing error
handling, we expect
exceptions to be thrown.
○ assertThrows checks
whether the code block
throws the expected
exception.
○ assertEquals can be
used to check the
contents of the stack
trace.
31
Testing Performance

● assertTimeout can be
used to impose a time
limit on an action.
○ Time limit stated using ofMilis(..),
ofSeconds(..), ofMinutes(..)
○ Result of action can be captured as
well, allowing checking of result
correctness.

32
Unit Testing - Account
Account
• Withdraw money, verify balance.
- name
- personnummer
- balance

Account (name,
personnummer, Balance)

withdraw (double amount)

deposit (double amount)
changeName(String name)
getName()
getPersonnummer()
getBalance()
33
Unit Testing - Account
Account
• Withdraw more than is in balance.
• (should throw an exception with
- name appropriate error message)
- personnummer
- balance

Account (name,
personnummer, Balance)

withdraw (double amount)

deposit (double amount)
changeName(String name)
getName()
getPersonnummer()
getBalance()
34
Unit Testing - Account
Account
• Withdraw a negative amount.
• (should throw an exception with
- name appropriate error message)
- personnummer
- balance

Account (name,
personnummer, Balance)

withdraw (double amount)

deposit (double amount)
changeName(String name)
getName()
getPersonnummer()
getBalance()
35
Let’s take a break.

36
Best Practices
• Use assertions instead of print statements

• The first will always pass (no assertions)

37
Best Practices
• If code is non-deterministic, tests should give deterministic results.

• Tests for this method should not specify exact time, but properties
of a “good” execution.
• The time should be positive, not negative or 0.
• A range on the allowed times.

38
Best Practices
• Test negative scenarios and boundary cases, in
addition to positive scenarios.
• Can the system handle invalid data?
• Method expects a string of length 8, with A-Z,a-z,0-9.
• Try non-alphanumeric characters. Try a blank value. Try strings
with length < 8, > 8

• Boundary cases test extreme values.

• If method expects numeric value 1 to 100, try 1 and 100.
• Also, 0, negative, 100+ (negative scenarios).

39
Best Practices
• Test only one unit at a time.
• Each scenario in a separate test case.
• Helps in isolating and fixing faults.
• Don’t use unnecessary assertions.
• Specify how code should work, not a list of observations.
• Generally, each unit test performs one assertion
• Or all assertions are related.

40
Best Practices
• Make each test independent of all others.
• Use @BeforeEach and @AfterEach to set up state and clear state
before the next test case.

• Create unit tests to target exceptions.

• If an exception should be thrown based on certain input, make
sure the exception is thrown.

41
Scaffolding
• Mock objects and drivers are written as
replacements for other parts of the system.
• May be required if pieces of the system do not exist.
• Scaffolding allows control over test execution and
greater observability to judge test results.
• Simulate dependencies and test components in isolation.
• Ability to set up specialized testing scenarios.
• Ability to replace part of the program with a version more
suited to testing.
42
Unit Testing - Object Mocking
WeatherData
Unit may depend on unfinished (or
temperature
untested) components. Can mock windSpeed
windDirection
those components. pressure Thermometer
lastReadingTime
• Same interface as real component, ther_identifier
temperature
but hand-created simulation. collect()
summarize(time) get()
• Can be used to simulate abnormal shutdown()
operation or rare events. Mock_Thermometer restart()

• Ex. Place exact data in database ther_identifier

temperature
needed to hit special outcome.
get()
shutdown() get(){
restart() return 98;
}
43
Mocking Example
• Declare a mock object:

• Specify method behavior:

• Returns “first”:
• Returns null:
• Because behavior for “99” is not specified.

• both return
“element”, as all input are specified.
44
Mocking Within a Test

45
Build Systems

46
Build Systems
• Building, running tests, packaging and distributing
are very common, effort-intensive tasks.
• Building and deploying should be as easy as possible.
• Build systems ease process by automating as
much as possible.
• Repetitive tasks can be automated and run at-will.

47
Build Systems
• Allow control over code compilation, test execution,
executable packaging, and deployment.
• Script defines actions that can be automatically
invoked at any time.
• Many frameworks for build scripting.
• Most popular for Java include Ant, Maven, Gradle.
• Gradle is very common for Android projects.

48
 Build Lifecycle
Validate Compile Test Package Verify Install Deploy

• Validate the project is correct and all necessary information

is available
• Compile the source code of the project.
• Test the source code using a suitable unit testing framework.
• Run unit tests against classes and subsystem
integration tests against groups of classes.
• Take the compiled code and package it in its distributable
format, such as a JAR.
49
 Build Lifecycle
Validate Compile Test Package Verify Install Deploy

• Verify - run system tests for quality/correctness.

• System tests require a packaged executable.
• This is also when tests of non-functional criteria like
performance are executed.
• Install the package for use as a dependency in
other projects locally.
• Deploy the package to the installation environment.

50
Apache Ant
• Build system for Java.
• Build scripts define targets that can be executed on
command.
• Correspond to lifecycle phases or other automated tasks.
• Targets can trigger other targets.
• Build scripts written in XML.
• Platform neutral, But can invoke platform-specific commands.
• Human and machine readable.
• Created automatically by many IDEs (Eclipse).
51
A Basic Build Script

• File typically named build.xml, and placed in the base

directory of the project.
• Build script requires project element and at least one target.
• Project defines a name and a default target.
• This target prints project information.
• Echo prints information to the terminal.
52
Targets

• A target is a collection of tasks you want to run in a

single unit.
• Targets can depend on other targets.
• deploy command will call package target, which will call
clean and compile first.
• Dependencies denoted using the depends attribute.

53
Targets

• Target attributes:
• name defines the name of the target (required)
• depends lists dependencies of the target.
• description is used to describe the target.
• if and unless allow execution of the target to depend on
a conditional attribute.
• Execute target if attribute is true, or execute unless true.
54
Executing targets

• In the command line, invoke:

• ant <target name>
• If no target is supplied, the default will be executed.
• In this case, ant and ant info give same result because
info is default target.
55
Properties
• XML does not natively allow variable declaration.
• Instead, create property elements, which can be referred
to by name.

56
Properties

• Properties have a name and a value.

• Property value is referred to as ${property name}.
• Ant pre-defines ant.version, ant.file (location of the build
file), ant.project.name, ant.project.default-target, and
other properties.
57
Property Files
• Separate file can define static properties.
• Allows reuse of build file in different environments
(development, testing, production).
• Allows easy lookup of property values.
• Called build.properties and stored in the same
directory as build script.
• Lists one property per line:
Comments can be added using

58
Property Files
• build.xml

• build.properties

59
 • Properties whose value determined by and
Conditions and or expressions.
• And requires that each property is true.
• Both foo.txt and bar.txt must exist.
• (available is an Ant command that
checks for file existence)
• Or requires that 1+ properties true.
• Calling myTarget.check creates
property (myTarget.run) that is true if
both files are present.
• When myTarget is called, it will run only
if myTarget.run is true.

60
Ant Utilities
• Fileset generates list of files matching criteria for
inclusion or exclusion.
• ** means that the file can be in any subdirectory.
• * allows partial file name matches.

61
Ant Utilities

• Path is used to represent a classpath.

• pathelement is used to add items or other paths to the path.

62
Building a Project

• Properties src.dir and build.dir define where the source files

are stored and where the built classes are deployed.
• Path master-classpath includes all JAR files in the lib folder
and all files in the build.dir folder.
63
Building a Project

• The clean target is used to prepare for the build process by

cleaning up any remnants of previous builds.
• In this case, it deletes all compiled files (.class)
• May also remove JAR files or other temporary artifacts that will be
regenerated by the build.
64
Building a Project

• The build target will create the build directory, compile the
source code (using javac), and place the class files in the
build directory.
• Can specify which java version to target (1.8).
• Must reference the classpath to use during compilation.
65
Creating a JAR File
• The jar command creates executable from compiled classes.

• destfile is the location to place the JAR file.

• basedir is the base directory of included files.
• includes defines the files to include in the JAR.
• excludes prevents certain files from being added.
• The manifest declares metadata about the JAR.
• Attribute Main-Class makes the JAR executable.
66
Running Unit Tests
• JUnit tests run using the junit command.

• test entries list the test classes to execute.

• haltonfailure will stop test execution if any tests fail, haltonerror if
errors occur.
• printsummary displays test statistics (number of tests run, number of
failures/errors, time elapsed).
• timeout will stop a test and issue an error if the specified time limit is
exceeded.
67
We Have Learned
• Test automation can lower cost and improve the
quality of testing.
• Automation involves creating drivers, harnesses,
stubs, and oracles.
• Test cases are often written in unit testing
frameworks as executable code.
• Assertions allow examination of output for failures.

68
We Have Learned
• Testing is not all that can be automated.
• Project compilation, installation, deployment, etc.
• Project build automation:
• Automating the entire compilation, testing, and
deployment process.
• Ant is an XML-based tool for automating build process.

69
Next Time
• Exercise Session: Unit Testing Practice
• Next Wednesday: Structural Testing
• Pezze and Young, Ch. 5.3 and 12
• Assignment 1 due Sunday.
• Assignment 2 out.
• (Based on Lectures 8-10)

70