CPS3230

FUNDAMENTALS OF SOFTWARE
TESTING
–
MODEL-BASED TESTING
& RUNTIME VERIFICATION
CHRISTIAN COLOMBO
UNIVERSITY OF MALTA

NOVEMBER 2022
OUTLINE

• High-Level Introduction
• A Specification Language – Finite State Automata
• An Overview of Runtime Verification
• Model-Based Testing
• System Verification in a Nutshell
PART I

HIGH-LEVEL
INTRODUCTION
SPECIFICATION

• What is a system specification?

SPECIFICATION

• What is a system specification?

Acceptable behaviour
Spec

All
behaviours
IMPLEMENTATION?

• Where does the

implementation fit in?

Spec

All
behaviours
IMPLEMENTATION

• The behaviours allowed by

the implementation are a subset
An implementation
of those allowed by the Spec which satisfies the
specification
specification
All
behaviours
IMPLEMENTATION

There should not be

Spec more than 5 successive
bad logins in 1 minute
All
behaviours
 IMPLEMENTATION

• The behaviours allowed by the implementation are a subset of those allowed by the
specification

After 3 successive bad

logins, the user is
blocked for 2 minutes Spec
from making another
attempt All
behaviours
WHAT ARE WE TRYING TO DO?

Specificatio
n
Impleme
nt-ation

Does the
implementation really
follow the specification?
THE TESTING PERSPECTIVE

• Consider Unit Testing

Test Oracle
Impleme
nt-ation
THE RUNTIME VERIFICATION
PERSPECTIVE

User Monitor
Impleme
nt-ation
LIGHTING SYSTEM EXAMPLE

• There are three levels of light intensity:

dim, normal, and bright which the user
can switch between.

• If the lights are bright, increasing the

intensity should not affect the intensity.
The case is similar for dim light and
decreasing the intensity.
EXERCISE

• How would you implement unit tests for this?

• You write a test with an input and you verify on the output
• What about a monitor?
• You monitor the brightness change method and, based on the
input parameters, the output is verified.
CONCLUSION

• Monitors require generalisation

• Generalisation makes things more challenging
• Hence the need for a specification language (other than
the programming language itself)
PART II

A SPECIFICATION LANGUAGE
FINITE STATE AUTOMATA
REMEMBER THIS?
ALGORITHM

1. Start off from a pre-determined starting state and from the beginning of

the tape.
2. Read in the next value off the tape.
3. Depending on the current state and the value just read determine the
next state (via some internal table).
4. If the current state is a terminal one (from an internal list of terminal
states) the machine may stop.
5. If the state is not a terminal one, advance the tape by one position.
6. Jump back to instruction 2.
FSA
LIGHTING SYSTEM EXAMPLE

• There are three levels of light intensity:

dim, normal, and bright which the user
can switch between.

• If the lights are bright, increasing the

intensity should not affect the intensity.
The case is similar for dim light and
decreasing the intensity.
SOLUTION

dim brighten brighten brighten

dim normal bright

dim dim
 LIGHTING SYSTEM SIMULATOR

• The lights can be turned on and off using one input.

• The intensity of the light can be adjusted using two inputs for lowering
and increasing the intensity.
• There are three levels of light intensity: dim, normal, and bright.
• If the lights are bright, increasing the intensity should not affect the
intensity. The case is similar for dim light and decreasing the intensity.
• The simulator starts with the lights off.
• Finally, when the lights are turned on, the intensity is normal by
default, regardless of the intensity of the light when it was last turned
EXERCISE

• Come up with an FSA which represents the behaviour of

the system described in the previous slides
FINITE STATE AUTOMATON
 OLD TELEPHONE EXAMPLE
• When the phone is “on hook”
• If it is ringing and someone lifts the receiver, connect the call
• If it is not ringing and someone lifts the receiver, sound the dial
tone and let the user input the numbers. Once the number is
inputted, setup the call.

• When the is “off hook”

• If the phone is placed back on hook,
Terminate the dial tone or the call.
 OLD TELEPHONE EXAMPLE
• Telephone input: “lift the receiver”
• When is this input applicable?
• What behaviours are expected?
 HINTS FOR FSA MODELLING
• The general procedure:
• Make a list of all the possible states
• Make a list of the inputs/actions
• For each input, note the situations in which the input can be applied
and when it can’t
• For each situation, note further situations when the input causes
different behaviours
• What variables do we need to model all possible situations
 MODELLING EXAMPLE
• The input – lift receiver – is only applicable if handset is in
place
• The input can generate two behaviours:
• If a call is incoming then connect to the other side
• Otherwise sound the dial tone
• What variables do we need to encode this system?
 MODELLING EXAMPLE
• Phone Status = {on hook, off hook}
• Incoming Call (Ringing) = {yes, no}
 MODELLING EXAMPLE
• What other inputs does a telephone system have?
 MODELLING EXAMPLE
• Hang up
• Dial number

• ... When are these applicable?

 MODELLING EXAMPLE
• When the telephone is off the hook
• When there is the dial tone

• ... What are the possible behaviours?

 MODELLING EXAMPLE
• Hang up
• Stop dial tone
• Stop call
• Dial
• Connect to person (assume call is successful)

• ... What are the variables required?

 MODELLING EXAMPLE
• Phone Status = {on hook, off hook}
• Incoming Call (Ringing) = {yes, no}
• Oncall={yes, no}
• Dial tone={yes, no}
 EXERCISE 2
• Draw the FSA for the old telephone example
 EXERCISE 2
Lift receiver

On hook, Off hook,

Not ringing, Hangup receiver Not ringing,
No dial tone, Dial tone,
Not on call Not on call
Press number
sequence
ring Stop ringing Hangup receiver

On hook, Off hook,

Ringing, Not ringing,
No dial tone, No dial tone,
Not on call On call

Lift receiver
PART III

AN OVERVIEW OF
RUNTIME VERIFICATION
RUNTIME VERIFICATION

System

Specification
RUNTIME VERIFICATION

System

RV
tool

Specification
RUNTIME VERIFICATION

System
System

RV
tool
Monitor
Specification
Verifier
RV ARCHITECTURE

Monitor

System Verifier
RV ARCHITECTURE

Monitor
events

System Verifier

ok/error
 RV ARCHITECTURE

• Monitoring checks whether the system advances beyond failure.

• Upon failure one can attempt to fix the problem since the
monitor stops immediately after the failure.
• The cost in overheads can be prohibitive if the volume of
monitored events is large.
• The system can break real-time properties because of monitor-
caused delays.
BUILDING THE ARCHITECTURE
SEPARATION OF CONCERNS

• Separating specification from the system requires

additional work to weave the two together.

System

Specification
SEPARATION OF CONCERNS

• Separating specification from the system requires

additional work to weave the two together.

System

RV
Specification
tool
SEPARATION OF CONCERNS

• Separating specification from the system requires

additional work to weave the two together.

System

RV Weaving
Specification
tool instructions
SEPARATION OF CONCERNS

• Separating specification from the system requires

additional work to weave the two together.

System
Weaving
RV Weaving tool
Specification
tool instructions
SEPARATION OF CONCERNS:
ASPECT-ORIENTED PROGRAMMING

• AOP aims to increase modularity by allowing the

separation of cross-cutting concerns. It adds behaviour to
existing code without modifying the code itself, instead
separately specifying which code is modified via a
"pointcut" specification.
• This allows behaviors that are not central to the business
logic (such as logging) to be added to a program without
cluttering the code core to the functionality.
SEPARATION OF CONCERNS:
ASPECT-ORIENTED PROGRAMMING

• Consider adding a logging feature for all types of money transfers, and
which can be toggled on or off.
• Either we add a line to each transfer method:
void transfer() {
if (log.enabled) { log.write(“…”); }
...
}

• Or we can use an AOP tool and write:

@before call(*.transfer(..))
log.write(“…”);
 PROPERTY SPECIFICATION
LANGUAGES: AUTOMATA

• Finite state automata can be a good way of allowing the

specification of consequentiality behavior.

logout transfer

login
 PROPERTY SPECIFICATION
LANGUAGES: AUTOMATA

• Access to system data enhances expressiveness...

u.logout transfer(f) | f.isPrivate()

u.login | !u.isBlackListed()
 PROPERTY SPECIFICATION
LANGUAGES: AUTOMATA

• Access to system data enhances expressiveness and

interaction with the system…

u.logout transfer(f) | f.isPrivate() | u.logout()

u.login | !u.isBlackListed()
 PROPERTY SPECIFICATION
LANGUAGES: AUTOMATA

• Access to data enhances expressiveness and allows for more compact

descriptions:

badPin() | wpin=3
insertCard() | | wpin=0

withdrawCard()

goodPin()

badPin() | wpin < 3 | wpin++

 PROPERTY SPECIFICATION
LANGUAGES: AUTOMATA

• Parametrization allows us to have a separate monitor per user:

FOREACH
badPin() | wpin=3
USER
insertCard() | | wpin=0

withdrawCard()

goodPin()

badPin() | wpin < 3 | wpin++

A NOTE ON DETERMINISM

• Monitoring automata should be deterministic

• Transitions and automata are ordered
RV TOOL: LARVA

• A tool for RV in Java using Symbolic Timed Automata

COMPLETE PICTURE

Monitored

System

--------------
1 USER Specification
--------------
COMPLETE PICTURE

EVENTS VERIFYING
Monitored
AspectJ
SYSTEM
System
FEEDBACK

LARVA
2 Compiler

--------------
Specification
--------------
COMPLETE PICTURE

EVENTS VERIFYING
Monitored
AspectJ
SYSTEM
System
FEEDBACK

3 LOG

USER
PROPERTY SPECIFICATIONS

• DATEs (Dynamic Automata with Timers and Events):

• Automata-based
• Dynamic creation of new properties (automata)
• Replication of properties per object instance
• Triggering of transitions based on:
• Program method invocations
• Program exceptions
• Real-time events
• Channel communication between automata
PROPERTIES IN LARVA
login

logout

login

badpassword
badpassword
 PROPERTIES IN LARVA
login / / count=0

logout

badpassword / count > 2 /

badpassword / count < 3 / count++

 PROPERTIES IN LARVA
read / / reset timer
login / / count=0; reset timer

logout

timer@30 / /

badpassword / count > 2 /

badpassword / count < 3 / count++

 PROPERTIES IN LARVA
action(sessionID)? / /
login(sessionID) / /
reset timer
count=0;
reset timer;
open(sessionID)!

logout

timer@30 / /

badpassword / count > 2 /

badpassword / count < 3 / count++

 PROPERTIES IN LARVA
foreach
action(sessionID)? / /
user login(sessionID) / /
reset timer
count=0;
reset timer;
open(sessionID)!

logout

timer@30 / /

badpassword / count > 2 /

badpassword / count < 3 / count++

 EXAMPLE IN LARVA
FOREACH (User u) {
EVENTS {
... }
PROPERTY badlogins {
STATES {
BAD { bad }
NORMAL { loggedin }
STARTING { loggedout }
}
TRANSITIONS {
loggedout -> loggedin [ login(sessionID) // count=0; timer.reset(); open(sessionID)! ]
loggedin -> loggedin [ action(sessionID)? // timer.reset(); ]
loggedin -> loggedout [ logout ]
loggedout -> loggedout [ badpassword / count < 3 / count++; ]
loggedout -> bad [ badpassword / count > 2 / ]
loggedin -> loggedout [ timer@30 / / ]
}}}
GENERATING THE VERIFYING
SYSTEM
else if (_state_id_badlogins==1) {
if (1==0){}
else if ((_occurredEvent (_event, 2 /*actionevent*/ ))) {

_state_id_badlogins = 1;//moving to state loggedin

VERIFYING _goto_badlogins(_info);
}
SYSTEM
else if ((_occurredEvent (_event, 4 /*logout*/ ))){

_state_id_badlogins = 2;//moving to state loggedout

_goto_badlogins(_info);
}

else if ((_occurredEvent (_event, 8 /*cAT30*/ ))){

_state_id_badlogins = 2;//moving to state loggedout

_goto_badlogins(_info);
GENERATING ASPECTJ
before ( User u1) : (call(* User.logout(..)) &&
target(u1) && !cflow(adviceexecution())) {

User u;
u =u1 ;
_cls_badlogins31 _cls_inst =
_cls_badlogins31._get_cls_badlogins31_inst( u);
AspectJ
_cls_inst.u1 = u1;
_cls_inst._call(thisJoinPoint.getSignature().toString(), 4/*logout*/);
}

before ( int sessionID,User u1) : (call(* User.login(..)) && target(u1)

&& args(sessionID) && !cflow(adviceexecution())) {

User u;
u =u1 ;
_cls_badlogins31 _cls_inst =
_cls_badlogins31._get_cls_badlogins31_inst( u);
_cls_inst.sessionID = sessionID;
_cls_inst.u1 = u1;
LOG SAMPLE
AUTOMATON::> badlogins(user) STATE::>loggedout

MOVED ON METHODCALL: void main.Main.login() TO STATE::> loggedin

AUTOMATON::> badlogins(user) STATE::>loggedin

MOVED ON METHODCALL: void main.Main.logout() TO STATE::> loggedout

AUTOMATON::> badlogins(user) STATE::>loggedout

MOVED ON METHODCALL: void main.Main.badpassword() TO STATE::> loggedout

AUTOMATON::> badlogins(user) STATE::>loggedout

MOVED ON METHODCALL: void main.Main.badpassword() TO STATE::> loggedout

AUTOMATON::> badlogins(user) STATE::>loggedout

MOVED ON METHODCALL: void main.Main.badpassword() TO STATE::>

!!!SYSTEM REACHED BAD STATE!!! bad

aspects._asp_badlogin80.ajc$before$aspects__asp_badlogi(_asp_badlogin80.aj:30)
COMPLETE PICTURE

EVENTS VERIFYING
Monitored
AspectJ
SYSTEM
System
FEEDBACK

3 LOG

USER
PART IV

MODEL-BASED
TESTING
TESTING

Driver Oracle
Impleme
nt-ation
WHY MODEL-BASED TESTING?

• Much of testing is done manually

• Error prone
• Laborious
• Can we automate it?
TESTING

Driver Oracle
Impleme
nt-ation

Specification
TESTING

Driver Oracle
Impleme
nt-ation

manual manual

Specification
MODEL-BASED TESTING

Impleme
nt-ation
(Driver) (Oracle)

Model
WHERE ARE THE BUGS?

Impleme
Model nt-ation

Bugs are here

Specification
HOW CAN WE FIND THEM?

Impleme
Model nt-ation

Bugs are here

Specification
HOW CAN WE FIND THEM?

Check
correspondence Impleme
Model nt-ation

Specification
GENERIC MODEL-BASED
TESTING
TEST SUITE GENERATION

• Use the model to generate meaningful inputs/sequences

of inputs
• Decide strategy of traversing the model
PRIORITISING TEST CASES

• Random
• Greedy algorithm
• Lookahead
EXECUTING TESTS

• Usually this part is assumed to be automatic

• Might need some scaffolding/adapter
 TEST SUITE EXECUTION
• Offline – Generate once, execute many times
• Eg: regression testing

• Online – Generate on-the-fly

• Ideal for time-bounded testing
• Eg: test for a whole night
• Testing systems whose environment cannot be easily simulated
• Eg: Testing code connecting to 3rd party
THE ORACLE PROBLEM

• Models necessitate automation of oracle since model-

based testing is usually very fast and comprehensive (in
terms of system functionality)
• Eg: Assertions can be added to FSMs to check the system still
corresponds with the model
MODEL-BASED TESTING

system action System

System
after action

Model after
Model
model action action
 MODEL-BASED TESTING
Generating test
cases

system action System

System
after action

Model after
Model
model action action

Checking test
output
EXAMPLE
Dim

System system action System

(Normal) (Dim)

Model Model
(Normal) model action (Dim)

Correspond?
MEASURING TEST
EFFECTIVENESS

• Structural coverage criteria – aims to exercise code or

model concerning some coverage goal
 COVERAGE MEASURES
• States visited
• Transitions visited
• Transition pair
combinations
A TEST FAILURE

• Fix the model

• The model is not correct
• Fix the system
• The system is not correct
MBT TOOL: MODELJUNIT
Method call

Java system action Java

System System

Java Model Java Model

FSA Update variables FSA

Assertions
THE SYSTEM: VENDING
MACHINE

• Accepts money (€1 or €0.5 )

• Makes
• Tea [€1]
• Coffee [€1]
• Hot chocolate [€1.50]
• Has option to add milk, sugar or extra sugar
• Return Drink
Design a FSM for the Coffee Vending Machine
ABSTRACT MODEL: VENDING
MACHINE
MODEL JUNIT

• Define a Model
• Normally using an FSM with
• Variables
• Conditions
• Actions
• Build graph of possible transitions
• Generates abstract tests
• Add operation on SUT
• Update any tracked variables
• Add JUnit assertions on SUT after applying actions
MODEL IMPLEMENTS
FSMMODEL INTERFACE
DECLARE ACTIONS THAT CAN BE
CARRIED OUT WITH @ACTION
ANNOTATION
GUARDS CAN BE INTRODUCED

• You can add guard to indicate when it is possible to

choose a particular action
• Add a method that returns a boolean
• Should have the same name as the action method name and end
with the word Guard
GUARD EXAMPLE
GENERATING TEST CASES

• Random
• Greedy tester
• Decides at the current state which transitions are yet to be
visited

• Lookahead tester
• Looks ahead to find unvisited transitions
GENERATING TESTS IN
MODELJUNIT

Choose a test
Choose how many generation
tests you want algorithm
MEASURE COVERAGE Add coverage
metric

Print statistics
PART V

SYSTEM VERIFICATION
IN A NUTSHEEL
WHY TEST?

• Testing is a part of a wider effort:

Software Verification
• While testing is the most popular, there are
several other techniques
WHY VERIFY?

• The need for verification has long been

recognised:
• Safety-critical systems: Failures may lead to loss of life,
environmental damage, etc.
• Mission-critical systems: Failures may lead to failure in
achieving goal of the system.
• Business-critical systems: Failures may lead to high-
business or financial loss.
WHY VERIFY?

• Systems are growing:

• … in size
• … in complexity
• … in the ways they interact with real-life objects
• Leading to more opportunities for failure,
• and making the consequences of failure more serious
• Meaning we need to ensure our systems work correctly.
EXAMPLE: XOR

Function XOR (boolean x, boolean y)

{
boolean a = x AND y;
boolean b = x OR y;
return b AND NOT a;
}
 ~X, ~Y,
A, ~B

EXECUTION GRAPH
~X, ~Y,
~X, ~Y,
~A, B
A, B
X, Y
~A, B

X, Y ~X, Y X, ~Y ~X, ~Y,

X, Y
~A, ~B

X, Y ~X, Y X, ~Y ~X, ~Y,

A, B ~A, B ~A, B ~A, ~B
X, Y
A, ~B

RET ~RET
THE QUESTION

How do we check systems

work correctly?
LET’S START WITH TESTING

• Before deployment (and ideally throughout the

development phase)…
• try the program along different execution paths
to see whether the program works correctly.
TESTING

X, Y ~X, Y X, ~Y ~X, ~Y,

X, Y ~X, Y X, ~Y ~X, ~Y,

A, B ~A, B ~A, B ~A, ~B

RET ~RET
TESTING: THE CHALLENGES… (1)

• What constitutes a correct program?

• Functional correctness examples
• Blacklisted users may not start a transaction.
• Account balances may never go below zero.
• A user must have opened a transaction before discarding it.
• A single transaction may not involve more than $10,000 in fund
transfers.
• No transaction may last longer than 3 hours.
• Oracles used to check for correct functional behavior
TESTING: THE CHALLENGES… (1)

• What constitutes a correct program?

• Non-functional correctness
• The system load may not exceed 50% CPU time.
• No transaction may acquire more than 1Mb of memory at any point
in time.
• The system will not crash without dumping the state of the ongoing
transactions into the database.

• Use of profilers (in some cases in conjunction with

oracles).
TESTING: THE CHALLENGES… (2)

• What paths to test?

• Choices over inputs
• Non-determinism
TESTING: THE CHALLENGES… (2)

function transfer(act_source, act_dest, amount) {

balance = act_source.getBalance();
if (act_source.isWhiteListed() &&
balance >= amount) {
act_dest.deposit(amount);
act_source.withdraw(amount);
}
}
TESTING: THE CHALLENGES… (2)

function transfer(act_source, act_dest, amount) {

balance = act_source.getBalance();
if (act_source.isWhiteListed() &&
balance >= amount) {
act_dest.deposit(amount);
act_source.withdraw(amount);
}
}
• Question 1: What values and accounts should we test
the function on?
TESTING: THE CHALLENGES… (2)
thread1.execute(transfer(joe, peter, 1000));
thread2.execute(transfer(joe, peter, 5000));

• Question 2: What about non-determinism?

TESTING: THE CHALLENGES… (3)

• When do we stop testing?

• Data coverage: What percentage of possible data values have we
covered?
• Control path coverage: What percentage of possible control paths
(e.g. through the then or the else branch of a conditional) have we
tried?
• Property coverage: What percentage of property cases have we
covered e.g. “(if the user is whitelisted then the transaction should
pass) and (if the user is blacklisted then the transaction should be
frozen)” tested only for graylisted users or gray- and whitelisted users.
TESTING: THE CHALLENGES… (3)

• When do we stop testing?

• Data coverage: What percentage of possible data values have we
covered?
Intelligent
• Control coverage: test
What percentage casecontrol paths (e.g.
of possible

generation is challenging.
through the then or the else branch of a conditional) have we tried?
• Property coverage: What percentage of property cases have we
covered e.g. “(if the user is whitelisted then the transaction should
pass) and (if the user is blacklisted then the transaction should be
frozen)” tested only for graylisted users or gray- and whitelisted users.
SUMMARY: TESTING

• Pros:
• (Relatively) easy to set up
• Can be performed throughout the development phase
• (Usually) tests can be rerun across versions
• Cons:
• Difficult to generate paths intelligently
• Can only talk about what happens, not about what may happen
• We can (normally) never say “Correct”
WHAT ABOUT MODEL CHECKING?

• What is model checking?

• Build a model of the system
• Specify the properties you would like to hold
• Use an algorithm to confirm that the system
satisfies the properties
MODEL CHECKING

X, Y ~X, Y X, ~Y ~X, ~Y,

X, Y ~X, Y X, ~Y ~X, ~Y,

A, B ~A, B ~A, B ~A, ~B

RET ~RET
MODEL CHECKING: THE
CHALLENGES

• How do we specify the properties?

• Since we have the whole graph, we may talk about:
• Properties along infinite paths e.g. liveness properties
• Possibility e.g. “the system may choose to suspend a user at the end of a
transaction”
• Agglomerate path properties e.g. “The average number of interaction
points with the user during a single transaction is between 3 and 4.”

• None of which were possible to generally (and/or easily)

test for…
MODEL CHECKING: THE
CHALLENGES

• How do we check algorithmically?

• The challenge is that of keeping the description of the
graph compact, since it grows exponentially with the size
of the programs and/or variables
• Different algorithms for different types of logics, but for
useful logics, complexity of verification is at least NP
complete.
SUMMARY: MODEL CHECKING

• Pros:
• We can (possibly) say “Correct”
• We can talk about richer properties than in testing
• Cons:
• Does not scale up
• Despite the original intention, the tools are far from providing
push-button technology
• We require a formal semantics of the language
VERIFICATION SUMMARY

• In an ideal world, we would like to ensure that our

systems cannot fail…
• … but testing leaves out paths which may occur in
practice…
• … and model checking does not scale up.
SOME OBSERVATIONS…

• Observation 1:
a. Checking a whole system is usually computationally expensive…
b. But checking a single path is usually not (hence oracles in testing).
• Observation 2:
a. Generating a representative set of paths is tough…
b. because the system may go through some other path which we may
not have checked before… (hence complete coverage in model
checking).
A LOGICAL CONCLUSION…

• So why not:
• check properties only on certain execution paths
• but continue checking after deployment to ensure that any
execution paths followed by the live system do not violate the
property,
• and if they do fix the system or just stop it.
A LOGICAL CONCLUSION…
runtime monitoring
• So why not:
• check properties only on certain execution paths
• but continue checking after deployment to ensure that any
execution paths followed by the live system do not violate the
property,
• and if they do fix the system or just stop it.
A LOGICAL CONCLUSION…

• So why not:
• check properties only on certain execution paths
• but continue checking after deployment to ensure that any
execution paths followed by the live system do not violate the
property,
• and if they do fix the system or just stop it.

runtime verification
SO FINALLY, RUNTIME
VERIFICATION

• Monitor what the system is doing…

• verifying that it does not violate any property.
• If it does, stop the system or fix it.
RUNTIME VERIFICATION

X, Y ~X, Y X, ~Y ~X, ~Y,

X, Y ~X, Y X, ~Y ~X, ~Y,

A, B ~A, B ~A, B ~A, ~B

RET ~RET
RV: THE CHALLENGES… (1)

• How do we observe or monitor the system?

• How do we verify the system?
• Architecture-wise different options are possible:
• Include monitoring and verification code as part of the system.
• Add observable events in the code, keep verification separate.
• Use a tool to separate concerns (monitoring and verification).
RV: THE CHALLENGES… (2)

• How do we control interference with the system?

• Different types of interference:
• Monitoring/verification code can have bugs
• It may interfere with the functionality of the system (e.g. lock a
resource which the system wants to use)
• It may affect non-functional behaviour (e.g. memory usage,
temporal behaviour, information leaking)
SUMMARY: RUNTIME
VERIFICATION

• Pros:
• We can say “The system never continues after failure.”
• Scales up.
• Easy to adopt.
• Cons:
• We can never say “The system is correct.”
• Overheads (time and memory) may be prohibitive.
• Can only talk about finite traces.
• Can only be performed at runtime.
• Does not remove bugs, but stops their consequences.
SUMMARY

• Correctness in some systems is crucial, but ensuring it is not

trivial.
• Model checking is still a pipe-dream for most software systems.
• Runtime verification allows us to continue ‘testing’ after
deployment.
• Testing is essential and we can automate most of it via
modelling.