Designing Safety-Critical

Software
• Annotations (or program assertions)
• Safety invariants
• Connection between invariant and
annotations
• Use of annotations for fault-tolerance
• Examples
• Presentations
 Program State
• The collection of program variables values in
the given situation is called program state
(programtillstånd)
• Execution of a program fragment starts in the
initial state (starttillstånd) and finishes in the
final state (sluttillstånd)
• The set of all possible program states is
called program state space (tillståndrum)
 Example 1
Var x,y: num
B: Bool
Init x:=0 || y:= 0 || b:=T;
program state after initialization (x=0, y=0, b=T)
y:= y+3
program state after assignment (x=0, y=3, b=T)
if x ≤ y then b:= T else b:= F
program state after if (x=0, y=3, b=T)
do x ≤ 3
program state in the beginning of loop body (x=0, y=3, b=T)
x:= x+1 || y := y-1;
if x ≤ y then b:= T else b:= F
program state at the end of loop body 1st step (x=1, y=2, b=T)
od
Program state space Σ= (x=0, y=0, b=T), (x=0, y=3, b=T), (x=1, y=2,
b=T), (x=2, y=1, b=F), (x=3, y=0, b=F)
Program
Var x,y: num
B: Bool
Init x:=0 || y:= 0 || b:=T;
program state after initialization (x=0, y=0, b=T)

y:= y+3
program state after assignment (x=0, y=3, b=T)

if x ≤ y then b:= T else b:= F

program state after if (x=0, y=3, b=T)

do x ≤ 3
program state in the beginning of loop body (x=0, y=3, b=T)
x:= x+1 || y := y-1;
if x ≤ y then b:= T else b:= F
program state at the end of loop body 1step (x=1, y=2, b=T)
od
Program state space Σ= (x=0, y=0, b=T), (x=0, y=3, b=T), (x=1, y=2, b=T),
(x=2, y=1, b=F), (x=3, y=0, b=F)
 State Predicate
• A logical expression (condition) that says
something about a program state is called
state predicate (tillståndpredikat).
• For example 1
• (b ∧ (x>y)) is false in all states
• (b∨ (x≥ y)) is true in all states
• (x>y) is true in some states and false in the
others
 State Predicates
• (x>y) is true in
(x=2, y=1, b=F), (x=3, y=0, b=F)
• (x>y) is false in (x=0, y=0, b=T),
(x=0, y=3, b=T), (x=1, y=2, b=T)
• In other words a state predicate
describes the set of program states in
which the predicate is true
 Annotating program code
• An annotation (or assertion) is a logical
assertion (usually state predicate written in a
program text)
• Annotations are written within {...}
• Annotations show what facts or properties
should be true at that point of program
execution
• An annotation at the beginning of program
text describes precondition (förvillkor) for this
program (fragment)
• We can use x0 y0 values to denote unknown
initial values of variables x, y
 Example of program
annotation
var x,y, t: num
{x= x0 ∧y= y0}
t:= x;
{x= x0 ∧y= y0 ∧t= x0}
x:=y;
{x= y0 ∧y= y0 ∧t= x0}
y:=t;
{x= y0 ∧y= x0}
 Calculating New Annotations
• Annotations can be calculated automatically
from previous annotations
• {p} skip {p}
• {(x=n) ∧p} x:=e {p[x:=n] ∧x=e[x:=n]}
• {p} if b then {b ∧p} prog1
else {¬ b∧p} prog 2 fi
• if b then prog1 {p} else prog2 {q} fi {p∨q}
• p[x:=n] is p where all x’s are substituted with
n
 Example of calculating
annotations
• Example2annot.pdf
 Static and Dynamic Annotations
• Annotations can be used as logical comments
of program behaviors
• Calculating annotations allows us to check
that a program works exactly as it is supposed
to
• Annotations can be used for ”debugging”
too.We can implement annotations as macros
”assert”
assert (p, text) =
if p then skip
else print text; abort fi
 Invariants
• Cut sets of fault trees give us verbal
description of the situation which should be
avoided in the system
• We need to translate this description into the
condition (predicate) over the program
variables and check that it never occurs in
our system
• Or we can negate it and check that negation
of hazardous situation is always true in our
system
 Invariant
• A property which is always holds in the
system (program) is called invariant
• In safety-critical systems ”avoidance of
a hazard” should be an invariant
 Examples of formulating the invariant
• In the conveyor system (from the last lecture)
• Hazard: Object is dropped outside the
machines.
• Cut set: belt is running, object has arrived at
the end of belt and table is not ready to accept
the object
• Condition to be avoided: Belt is delivering
while the table is not ready
(Belt=Delivering ∧Table≠ReadyForLoading)
• Negation
¬(Belt=Delivering∧Table≠ReadyForLoading)
==
(Belt=Delivering ⇒ Table=ReadyForLoading)
 One more example
• Press and robot system (from your exercise)
• Hazard: Press jams robot arm1
• Cut set: Press is moving and robot arm 1 is
not retracted
• condition to be avoided:
(Press=Moving ∧ RobotArm1 ≠Retracted)
• Negation
Press=Moving ⇒RobotArm=Retracted
 Another example
• Safety boundaries: if physical parameter x
gets higher (or lower, or greater etc) than a
certain threshold then hazard occurs
• The invariant simply should be
• x≤ threshold
• Or keep certain parameter inside the
boundaries:
low_threshold ≤ x≤ high_threshold
 Verifying the invariant
• We need to check that the invariant is
true in certain places of the program
• Before executing a procedure (or
function, or program fragment)
• After executing a procedure (or function,
or program fragment)
 Life is beautiful again
• Because to verify an invariant we just
need to include the invariant in the
annotations
• before a procedure (i.e. make it a part of
precondition)
• and after a procedure (i.e. make it a part
of the postcondition)
• If it is that easy why does software still
have so many bugs???

• The devil is in the details

 Safety invariant and software
faults
• Safety invariant describes a property of a
correct software
• But what if software has bugs (overwrites
values of variables, runs in the infinite loops,
exceeds valid range of variables...)
• We might end up in the situation when
software still executes the proper algorithm
but with irrelevant data so we end up in a
hazardous state
 Software faults (bugs)
• Software specification faults
• coding faults
• logical errors within calculations
• stack overflows or underflows
• use of uninitialized variables
• infinite loops
 Annotating Programs to Cope with Bugs
• Software specification faults (partially, but
most often formal methods are needed)
• coding faults (good target. Include checking
for range validity, express which variables
should be affected and which unchanged
etc.)
• logical errors within calculations (sometimes,
but usually by diversity)
• stack overflows or underflows (good target)
• use of uninitialized variables (good target)
• infinite loops (good target)
 Annotating Loops: Loop Invariant
• The loop invariant is a special kind of annotations
containing information about loop behavior
• The invariant predicate should be true before the loop
and also after each loop execution step
• Example:
s:=0; i:=0;
{s=0 ∧ i=0}
do i<n →
{invariant i≤ n ∧ s=sum(a[0..i-1])}
s:= s+a[i]; i:= i+1
od
What is invariant: usually range and the progress
 Annotating Loops: Termination

• The loop variant (termination argument) is an

expression of natural number type which gets
smaller after each loop execution step
• It is needed to guarantee that loops terminates
s:=0; i:=0;
{s=0 ∧ i=0}
do i<n →
{invariant i≤ n ∧ s=sum(a[0..i-1])}
{variant n-i}
S:= s+a[i]; i:= i+1
od
 Annotation After Loop

• Annotation after loop can be constructed from the loop

invariant and the negated loop guard
s:=0; i:=0;
{s=0 ∧ i=0}
do i<n →
{invariant i≤ n ∧ s=sum(a[0..i-1])}
{variant n-i}
S:= s+a[i]; i:= i+1
Od
{i=n ∧ s=sum(a[0..n-1])}
 Annotation After Loop

• Annotation after loop is constructed by negation of the

loop guard and the loop invariant

• It is a good idea to check that this is what we want

• Probably even beforehand we just construct a this

conjunction and check it
 Full example

Var s, i: num; a: array[n] of num

{unchanged a}
s:=0; i:=0;
{s=0 ∧ i=0}
do i<n →
{invariant i≤ n ∧ s=sum(a[0..i-1])}
{variant n-i}
S:= s+a[i]; i:= i+1
od
{i=n ∧ s=sum(a[0..n-1])}
 Conclusions
• Annotations are useful things for coping
with “easy” bugs
• Cons: they complicate code
• Annotations is a way to show that the
safety invariant holds
• Of course a good discipline of
programming is still needed