SEN4013

Software Verification and

Validation

Lecture 2
A Framework for Test and Analysis
Learning Objectives
• Introduce dimensions and tradeoff between test
and analysis activities
• Distinguish validation from verification activities
• Understand limitations and possibilities of test and
analysis

2
Software Test
• Purpose of software test and analysis
• Assess software quality
• Improve software by finding defects
• Dependability of software
• Correctness
• Consistency of implementation with specification
• Reliability
• Likelihood of correct functioning
• Robustness
• Acceptable behavior in unusual circumstances
• Safety
• Absence of unacceptable behaviors
3
Software Test
• No perfect test or analysis techniques.
• Not a single “best” technique for all circumstances.
• Testing techniques
• exist in a complex space of trade-offs
• have complementary strengths and weaknesses.

4
First Computer Bug
• In 1947 Harvard University was operating a
room-sized computer called the Mark II.
• Mechanical relays
• Glowing vacuum tubes
• Technicians program the computer by reconfiguring it
• Technicians had to change the occasional vacuum
tube.

• A moth flew into the computer and was

zapped by the high voltage when it landed
on a relay.

• Hence, the first computer bug!

5
Bugs – Errors – Faults
• Pac-Man (1980)
• Has “Split Screen” at level 256
• Cause: Integer overflow
• 8 bits: maximum representable value

6
Bugs – Errors – Faults
• AT&T (1990)
• One switching system in
New York City experienced
an intermittent failure that
caused a major service
outage
• The first major network
problem in AT&T’s 114-year
history
• Cause: Wrong BREAK
statement in C Code
• Complete code coverage
could have revealed this bug
during testing
7
Bugs – Errors – Faults
• Ariane 5 flight 501 (1996)
• Destroyed 37 seconds after launch (cost: $370M)
• Cause: Arithmetic overflow
• Data conversion from a 64-bit floating point to 16-bit
signed integer value caused an exception
• The software from Ariane 4 was re-used for Ariane 5
without retesting

8
Bugs – Errors – Faults
• Mars Climate Orbiter (1998)
• Sent to Mars to relay signal from Mars Lander
• Smashed to the planet
• Cause: Failing to convert between different metric
standards
• Software that calculated the total impulse presented
results in pound-seconds
• The system using these results expected its inputs to be
in newton-seconds

9
Bugs – Errors – Faults
• THERAC-25 Radiation Therapy (1985)
• 3 cancer patients received fatal overdoses
• Cause:
• Miss-handling of race condition of the software in the
equipment

10
Software Failure, Fault &
Error
• Fault
• Incorrect portions of code.
• may involve missing code as well as incorrect code
• Necessary (not sufficient) condition for the occurrence of a failure.

• Failure
• Observable incorrect behavior of a program.

• Error
• Cause of a fault. something bad a programmer did (conceptual, typo,
etc.).

• Bug
• Informal term for fault/failure.
11
Approaches to Reduce
Faults
• Manual code review
• Manually review the code to detect faults
• Limitations:
• Hard to evaluate your progress
• Can miss many faults/bugs

12
Approaches to Reduce
Faults
• Manual code review

13
Approaches to Reduce
Faults

14
Approaches to Reduce
Faults

• Static analysis: Identify specific problems (e.g., memory

leak) in the software by scanning suspicious patterns
from the code
• Limitations: (1) Limited problem types, (2) False positives

15
Approaches to Reduce
Faults

• Testing: Feed input to software and run it to see

whether its behavior is as expected
• Limitations: (1) Impossible to cover all possible
execution, (2) Need test oracles
16
“Testing can show the presence of faults, but not their absence.”, Dijkstra
Approaches to Reduce
Faults

• Formal Verification: Consider all the possible program

executions, and formally prove that the program is
correct or not
• Limitations: (1) Difficult to have a formal specification,
(2) Most real-world programs are too expensive to
prove 17
Validation and Verification
• Validation:
Does the software system meet the user's real
needs?
are we building the right software?

• Verification:
Does the software system meet the requirements
specifications?
are we building the software right?

18
Validation
• Software meeting the user’s real needs???
• Fulfilling requirements is not the same as conforming to
a requirements specification.
• A specification is a statement about a particular
proposed solution to a problem, and that proposed
solution may or may not achieve its goals.
• Specifications are written by people, and therefore
contain mistakes.
• A system that meets its actual goals is useful, while a
system that is consistent with its specification is
dependable.
19
Verification
• Checking the consistency of an implementation
with a specification.
• Verification is a check of consistency between
implementation and specification, in contrast to
validation which compares a description (whether a
requirements specification, a design, or a running
system) against actual needs.

20
Validation and Verification
SW
Actual Specs
Requirements System

Validation Verification
Includes usability Includes testing,
testing, user feedback inspections, static
analysis

21
Validation and Verification
12345678
• Verification or validation depends on
the specification.
• Example: elevator response

Unverifiable (but validatable) spec: ... if a user presses a

request button at floor i, an available elevator must arrive
at floor i soon...

Verifiable spec: ... if a user presses a request button at floor i,

an available elevator must arrive at floor i within 30 seconds...
22
Validation and Verification
Activities

23
V Model
• Verification activities check consistency between
descriptions (design and specifications) at adjacent
levels of detail, and between these descriptions and
code.
• Validation activities attempt to gauge whether the
system actually satisfies its intended purpose.

24
Validation and Verification

25
Dependability
• A system that meets its actual goals is useful, while
a system that is consistent with its specification is
dependable.
• Dependability properties include correctness,
reliability, robustness, and safety.
• Correctness: Absolute consistency with a
specification, always and in all circumstances.

28
Dependability
• Reliability: Statistical approximation to correctness,
expressed as the likelihood of correct behavior in
expected use.
• Robustness: Weighs properties as more and less
critical. Distinguishes which properties should be
maintained even under exceptional circumstances
in which full functionality cannot be maintained.
• Safety: A kind of robustness in which the critical
property to be maintained is avoidance of
hazardous behaviors.

29
Degrees of Freedom
• Given a precise specification and a program, it seems that
one ought to be able to arrive at some logically sound
argument or proof that a program satisfies the specified
properties.
• After all, if a civil engineer can perform mathematical
calculations to show that a bridge will carry a specified
amount of traffic, shouldn’t we be able to similarly apply
mathematical logic to verification of programs?

30
Verification and
Undecidability
• Program testing is a verification technique and is as
vulnerable to undecidability as other techniques.
• Exhaustive testing, that is, executing and checking every
possible behavior of a program, would be a “proof by cases,”
which is a perfectly legitimate way to construct a logical
proof.
• BUT How long would this take?
• If we ignore implementation details such as the size of the
memory holding a program and its data, the answer is
“forever.”
• That is, for most programs, exhaustive testing cannot be
completed in any finite amount of time.
31
Exhaustive Testing
• Programs are executed on real machines with finite representations of
memory values.
• Consider the following trivial Java class:

class Trivial{
static int sum(int a, int b)
{ return a + b; }
}

• The Java language definition states that the representation of an int is 32

binary digits, and thus there are only 232 × 232 = 264  1021 different inputs on
which the method Trivial.sum() need be tested to obtain a proof of its
correctness.
• At one nanosecond (10−9 seconds) per test case, this will take approximately
1012 seconds, or about 30,000 years.
32
 Getting What You Need ...
Theorem proving:
Perfect verification of
Unbounded effort to
verify general
arbitrary properties by • optimistic inaccuracy: we may
logical proof or exhaustive
properties.
testing (Infinite effort) accept some programs that do not
Model checking:
possess the property (i.e., it may
Decidable but possibly
intractable checking of
not detect all violations).
simple temporal • testing
properties.
Data flow
analysis
• pessimistic inaccuracy: it is not
guaranteed to accept a program
Typical testing even if the program does possess
techniques
Precise analysis of
simple syntactic the property being analyzed
properties.
• automated program analysis
techniques
• simplified properties: reduce the
Simplified Optimistic
degree of freedom for simplifying
properties inaccuracy the property to check
Pessimistic
inaccuracy

Verification trade-off dimensions

33
Some Terminology
• Safe: A safe analysis has no optimistic inaccuracy,
i.e., it accepts only correct programs.
• Sound: An analysis of a program P with respect to a
formula F is sound if the analysis returns true only
when the program does satisfy the formula.
• Complete: An analysis of a program P with respect
to a formula F is complete if the analysis always
returns true when the program actually does satisfy
the formula.

34
Don’t know?
• Some analysis techniques may give a third possible
answer, “don’t know.”
• We can consider these techniques to be either
optimistic or pessimistic depending on how we
interpret the “don’t know” result.
• Perfection is unobtainable, but one can choose
techniques that err in only a particular direction.

35
Pessimistic
• A software verification technique that errs only in the pessimistic
direction is called a conservative analysis.
• It might seem that a conservative analysis would always be
preferable to one that could accept a faulty program.
• However, a conservative analysis will often produce a very large
number of spurious error reports, in addition to a few accurate
reports.
• A human may, with some effort, distinguish real faults from a few
spurious reports, but cannot cope effectively with a long list of
purported faults of which most are false alarms.
• Often only a careful choice of complementary optimistic and
pessimistic techniques can help in mutually reducing the different
problems of the techniques and produce acceptable results.
36
Third Dimension
• Substituting a property that is more easily checked or
constraining the class of programs that can be checked.
• Suppose we want to verify a property S, but we are not willing to
accept the optimistic inaccuracy of testing for S, and the only
messages that they are worthless.
• Suppose we know some property S’ that is a sufficient, but not
necessary, condition for S (i.e., the validity of S’ implies S, but not
the contrary).
• Maybe S’ is so much simpler than S that it can be analyzed with
little or no pessimistic inaccuracy.
• If we check S’ rather than S, then we may be able to provide
precise error messages that describe a real violation of S’ rather
than a potential violation of S.
37
Substitution Example
• Each variable should be initialized with a value
before its value is used in an expression.
• In the C language, a compiler cannot provide a
precise static check for this property, because of
the possibility of code like the following:

38
Substitution Example
(cont.)
• It is impossible in general to determine whether
each control flow path can be executed, and while a
human will quickly recognize that the variable sum
is initialized on the first iteration of the loop, a
compiler or other static analysis tool will typically
not be able to rule out an execution in which the
initialization is skipped on the first iteration.
• Java neatly solves this problem by making code like
this illegal; that is, the rule is that a variable must
be initialized on all program control paths, whether
those paths can ever be executed.
39
Summary
• Most interesting properties are undecidable, thus
in general we cannot count on tools that work
without human intervention
• Assessing program qualities comprises two
complementary sets of activities: validation (does
the software do what it is supposed to do?) and
verification (does the system behave as specified?)
• There is no single technique for all purposes: test
designers need to select a suitable combination of
techniques

40