Lecture 13: Model-Based Testing

Gregory Gay
DIT635 - March 2, 2022
Models and Software Analysis
• Before and while building products, engineers
analyze models to address design questions.
• Software is no different.
• Software models capture different ways that the
software behaves during execution.

2
Behavior Modeling
• Abstraction - simplify problem by identifying and
focusing only on important aspects.
• Solve a simpler problem, then apply to the big problem.
• A model is a simplified representation of an artifact.
• Ignores all irrelevant elements of that artifact.

3
Software Models
• Model is an abstraction of system being developed.
• By abstracting unnecessary details, powerful analyses
can be performed.
• Can be extracted from specifications and design (or
from code)
• Illustrates intended behavior of the system.
• Often state machines.
• Events cause the system to react, changing its internal state.

4
Model-Driven Development
• Models often created during requirements analysis.
• Allows refinement of requirements.
• Prove that properties hold over model.
• Finite State Verification (next class) - used to analyze
requirements, plan development, create test cases.

• Can generate code from models.

• Can create tests using model.

5
Model-Based Testing
• Models describe what happens input applied to
certain functionality.
• Model structure can be exploited:
• Coverage criteria used to identify important paths.
• Steps taken to perform functionality in different ways or to
get different outcomes.

6
Finite State Machines

7
State Machines
• Program execution as sequence of states
transformed by actions.
• “Behavior” is state -> action -> state transitions.
• Set of all possible real behaviors is often infinite.
• Called the “state space” of the program.
• Models simplify a functionality or class state space into a
finite set of states.

8
Finite State Machines
• Nodes represent states
• Abstract description of the current
value of an entity’s attributes.
• Edges represent transitions
• Events cause state to change.
• Labeled event [guard] / activity
• event: The event that triggered the transition.
• guard: Conditions that must be true to transition.
• activity: Output behavior when this transition is taken.

9
Terminology
• Event - An input that occurs at a defined time.
• The user presses a self-test button.
• The alarm goes off.
• Condition - Internal or external property describing
a change over time.
• The fuel level has risen over a threshold.
• The alarm has been on for ten seconds.

10
Terminology
• State - Abstract description of the current value of
the entity’s attributes.
• (ex: Not “X=5; Y=10”, but “Normal Operating Mode”)
• The controller is in the “self-test” state after the self-test
button has been pressed, and leaves it when the reset
button has been pressed.
• The tank is in the “too-low” state when the fuel level is
below the set threshold for N seconds.

11
States, Transitions, and Guards
• States change in response to events (transition).
• When multiple transitions are possible, the choice is
guided by the current conditions.
• Also called the guards on a transition.
• We take the transition that satisfies all guards.

12
State Transitions
Transitions are labeled as:
event [guard] / activity
• event: The event that triggered the transition.
• guard: Conditions required to take this transition.
• activity: Output when this transition is taken.

13
State Transitions
Transitions are labeled in the form:
event [guard] / activity
• All three are optional.
• Missing Activity: No output from this transition.
• Missing Guard: Always take transition following event.
• Missing Event: Take this transition immediately after
entering preceding state (if guards met).

14
State Transition Examples
Transitions are labeled in the form:
event [guard] / activity
• Controller enters “self-test” mode after test button is
pressed, leaves when reset button is pressed.
• User pressing self-test, reset buttons are events.
• The tank enters “too-low” state when fuel level <
threshold for N seconds.
• Fuel level < threshold for N seconds is a guard.

15
Example: Gumball Machine

[gumballs > 0] Waiting for

Quarter

user ejects quarter user inserts quarter

Quarter
Inserted user turns crank
[gumballs -1 > 0]
Out of Gumball / dispense
Gumballs Sold gumball
[gumballs -1 = 0] / dispense
gumball

16
 More on Transitions
Guards must be mutually exclusive
If event occurs and no
Able to transition is valid, then
Purchase
last bill ejected last bill ejected event is ignored.
[balance = 0] [balance > 0 &&
balance < needed]
Waiting for More Money last bill ejected [balance >
Money Needed
0 && balance >= needed]

17
Internal Activities
Can react to events and ● Special events: entry
conditions without and exit.
transitioning using internal ● Other activities occur
activities. each “time step”, until a
Typing transition occurs.
entry / highlight all ○ Entry and exit not
exit / update field re-triggered.
character entered / add to field
help requested [verbose] / open help page
help requested [minimal] / update status bar

18
Example: Maintenance Tracking
• Customers send products for maintenance.
• Maintenance tracking function notes current
stage of maintenance process for each customer.
• Transition path determined by a set of rules.
• States = stages of process.
• Transitions shows paths through process.
• Model only what software tracks and controls!

19
Example: Maintenance Tracking
If the product is covered by warranty or maintenance contract,
maintenance can be requested through the web site or by bringing the
item to a designated maintenance station. No Maintenance
Waiting for Pick Up
If the maintenance is requested by web and the customer is a US
resident, the item is picked up from the customer. Otherwise, the
customer will ship the item. Request - No Warranty
If the product is not covered by warranty or the warranty number is not
valid, the item must be brought to a maintenance station. The station
informs the customer of the estimated cost. Maintenance starts when
the customer accepts the estimate. If the customer does not accept, the
item is returned. Wait for Acceptance Wait for Returning
20
 Example: Maintenance Tracking
Repair at Station
If the maintenance station cannot solve the problem, the
product is sent to the regional headquarters (if in the US) or
the main headquarters (otherwise). If the regional
headquarters cannot solve the problem, the product is sent
to main headquarters. Repair at Regional HQ Repair at Main HQ

Maintenance is suspended if some components are not

available. Wait for Component
Once repaired, the product is returned to the customer.
Repaired

21
Example:
Maintenance Tracking

22
 Model

Example - Computer Model ModelID

Slots

• Many classes have stateful behavior. selectModel(modelID)

deselectModel
• States = class variables addComponent(slot,
component)
• Transitions = method calls removeComponent(slot)
isLegalConfiguration()
• Derive model from class and create tests.
Slot
• Ex: We sell computers on our website. Model
Model class represents a model of Component
Required
computer. incorporate(model)
bind(component)
unbind()
isBound()

23
Slot Specification
Slot represents a configuration choice in all instances of a particular model of
computer. A given model may have zero or more slots, each of which is marked
as required or optional. If a slot is marked as required, it must be bound to a
suitable component in all legal configurations. Slot offers the following methods:
• Incorporate: Make a slot part of a model, and mark it as either required or
optional. All instances of a model incorporate the same slots.
• Bind: Associate a compatible component with a slot.
• Unbind: The unbind operation breaks the binding of a component to a slot,
reversing the effect of a previous bind operation.
• IsBound: Returns true if a component is currently bound to a slot, or false
if the slot is currently empty.

24
Slot State Machine
isBound()

unbind()
No Component Component isBound()
No Model Bound Bound
incorporate
bind
(model) (component)
unbind()

• Do not derive too many states.

• Map variables to abstract values, not a state for each
possible combination of values.
• Model how a method affects a class.
• States only need to capture interactions between
methods and the class state.
25
Example - Model
Model represents the current configuration of a model of computer.
• A given model may have zero or more slots, each of which is
marked as required or optional.
• Each slot may contain a single component.
• To be a legal model, the model ID must exist in the ModelDB, each
slot marked as required must be filled, the configuration must
match that of the ModelDB entry for the model ID, and the optional
components must match those allowed for that model in the
ModelDB.

26
Example - Model
● selectModel(modelID): Sets the model ID to the value passed in, as long as the
model ID is set to “no model selected”. A model ID must be set before any other
services are requested.
● deselectModel(): Sets the model ID to “no model selected”. If the configuration was
previously judged to be legal, it is no longer legal.
● addComponent(slot, component): Adds the selected component to the selected
slot. If the configuration was previously judged to be legal, it is no longer legal.
● removeComponent(slot): Removes the selected component to the selected slot. If
the configuration was previously judged to be legal, it is no longer legal.
● isLegalConfiguration(): Compares the current configuration to the entry in
ModelDB. If the configuration is valid, the Model’s isLegal field is set to “true”.

27
Choosing States
No Model Legal
Configuring
Selected Configuration

• What does the class represent?

• e.g., a computer model.
• What causes method results to differ?
• e.g., whether the model is legal or illegal.
• Can the class be in any other states?
• e.g., we may not have set the model yet, we could still be
making decisions and have not determined legality.

28
Choosing Transitions and Initial State
No Model
Selected
deselectModel()
selectModel(model)
deselectModel() addComponent
Configuring (slot,component)

remove isLegalConfiguration()
Component(slot) [legalConfig=false]
isLegalConfiguration() remove
[legalConfig=true] addComponent Component(slot)
Legal (slot,component)

Configuration

29
Activity - Secret Panel Controller
You must design a state machine for the controller of a
secret panel in Dracula’s castle.
Dracula wants to keep his valuables in a safe that’s hard to find. So, to
reveal the lock to the safe, Dracula must remove a strategic candle
from its holder. This will reveal the lock only if the door is closed. Once
Dracula can see the lock, he can insert his key to open the safe. For
extra safety, the safe can only be opened if he replaces the candle first.
If someone attempts to open the safe without replacing the candle, a
monster is unleashed.
Take a short break if needed. https://bit.ly/33OZyjc (Problem 1)
30
Activity Solution
Open
safe closed / close
panel
key turned [candle in] /
open safe
Lock
Wait
Revealed
candle removed [door closed] /
reveal lock key turned [candle out] /
release monster
Monster
Unleashed

31
Model Coverage Criteria

32
Test Creation
• Tests created from models can be applied to the
real program.
• Events translated into method/API calls.
• Program output (abstracted) should match model output.
• Model coverage maps to requirements coverage.
• Tests should be effective for verification.
• Exercises stateful behavior thoroughly.
• Coverage criteria based on states, transitions, paths.

33
State Coverage
• Each state must be reached by test cases.
• Unless model has been placed in each state, faults
cannot be revealed.
• Num. of Covered States / Number of States
• Easy to understand and obtain, but low
fault-revealing power.
• Software takes action during transitions
• Most states can be reached through multiple transitions.

34
Transition Coverage
• A transition specifies a pre/post-condition.
• “If system is in state S and sees event I, then after
reacting to it, the system will be in state T.”
• Faulty system could violate (pre, post-condition) pairs.
• Every transition must be covered by test cases.
• Num. Covered Transitions / Number of Transitions

35
Example: Maintenance ● If no “final” states, we
could achieve transition
coverage with one large
test case.
○ Smarter to target
sections in different
test cases.

Example Paths:
T1: request w/ no warranty (0->2) - estimate costs
(2->4) - reject (4->1) - pick up (1->0)
T2: 0->5->2->4->5->6->0
T3: 0->3->5->9->6->0
T4: 0->3->5->7->5->8->7->8->9->7->9->6->0
T5: 0->5->8->6->0
36
Example - Slot
isBound()

unbind()
No Component Component isBound()
No Model Bound Bound
incorporate
bind
(model) (component)
unbind()

• Tests are series of method calls.

• incorporate(model), isBound(), unbind()
• incorporate(model), bind(component), isBound()
• incorporate(model), bind(component), unbind(), isBound()

37
 Example - Model
No Model
TC1: Selected
selectModel(M1) [M1, 1 slots = C1]
deselectModel()
deselectModel() selectModel(model)
selectModel(M1)
addComponent(S1,C1) deselectModel() addComponent
isLegalConfiguration() //true
remove Configuring (slot,component)
deselectModel()
Component(slot) isLegalConfi isLegalConfi
guration() guration()
TC2:
[legalConfig [legalConfig
selectModel(M1) [M1, 1 slot = C1]
Valid
=true] =false] addComponent
addComponent(S1,C1)
(slot,component)
isLegalConfiguration() //true Configuration
addComponent(S2,C2)
remove
isLegalConfiguration() // false
Component(slot)
removeComponent(S2)
isLegalConfiguration() // true
removeComponent(S1)

38
Path Coverage Criteria
• Transition coverage based on assumption that
transitions are independent.
• Many machines exhibit “history sensitivity”.
• Transitions available depend on path taken.
• “wait for component” in Maintenance Tracking example.

• Path-based metrics can cope with sensitivity.

39
Path Coverage Metrics
• Single State Path Coverage
• Requires that each subpath that traverses states at most
once to be included in a path that is exercised.
• Single Transition Path Coverage
• Requires that each subpath that traverses a transition at
most once to be included in a path that is exercised.
• Boundary Interior Loop Coverage
• Each distinct loop must be exercised minimum, an
intermediate, and a large number of times.
40
Single State (Transition) Path Coverage
Single State (or
Transition) Path
Coverage
● Each subpath
that traverses a state (or
transition) at most once
must be exercised.

41
Boundary Interior Loop Coverage
Boundary Interior
Loop Coverage
● Each loop must be
exercised 1, 2, N
times.
● (N = some higher
number)

42
Activity https://bit.ly/33OZyjc (Problem 2)

For this model, derive test suites that achieve state and
transition coverage.

43
Activity - State Coverage
[true,1], [false,2], [false, 65]

44
Activity - Transition Coverage
1. [true,1], [false,2], [false, 65], [true, 66], [false, 77], [true,
78], [false, 79], [false, 140], [false, 141]
2. [false, 1]

45
We Have Learned
• Models can be used to systematically create tests.
• Exercises stateful behavior of a class or functionality.
• Maps well to requirements.
• State machines model expected behavior.
• Cover states, transitions, non-looping paths, loops.
• Can also verify properties over models as part of
verification (next class).

46
Next Time
• Finite State Verification
• Optional Reading - Pezze and Young, Chapter 8
• Assignment 3
• Due Sunday, March 13
• Questions?

2018-08-27 Chalmers University of Technology 47