15-May-20

Verification and Validation

Object-Oriented Programming

http://softeng.polito.it/courses/09CBI

Version 1.2.1 – June 2019

© Maurizio Morisio, Marco Torchiano, 2019

This work is licensed under the Creative Commons Attribution-

NonCommercial-NoDerivatives 4.0 International License.
To view a copy of this license, visit
http://creativecommons.org/licenses/by-nc-nd/4.0/.
You are free: to copy, distribute, display, and perform the work

Under the following conditions:

Licensing Note
 Attribution. You must attribute the work in the manner specified by
the author or licensor.

 Non-commercial. You may not use this work for commercial purposes.

 No Derivative Works. You may not alter, transform, or build upon this
work.

 For any reuse or distribution, you must make clear to others the
license terms of this work.
 Any of these conditions can be waived if you get permission from the
copyright holder.

Your fair use and other rights are in no way affected by the above.
2

1
 15-May-20

Development process
Requirements Requirement Requirement
engineering document VV requirements document

Design Design
document
VV design Design
document

Implement
unit VV unit
Unit Unit
Implement Unit Unit
unit VV unit

Integrate
units System VV system System

Project management
Configuration management
Quality management

V&V
 Validation
 is it the right software system?
 effectiveness
 external (vs. user)
 reliability
 Verification
 is the software system right?
 efficiency
 internal (correctness of transformations)
 correctness

2
 15-May-20

V&V Validation

What the customer

asked for What the customer
had in mind

How the analyst

saw it

How the system

was designed

Verification
As the programmer
wrote it

How it actually works

V&V
Requirement
document

Design
verification

document validation stakeholders

Unit
Unit

System

3
 15-May-20

TERMINOLOGY

Failure, fault, defect

 Error
 A mistake e.g. committed by a programmer
 Fault (Bug)
 The feature of software that causes a failure
 May be due to:
– An error in software
– Incomplete/incorrect requirements
 Failure
 An execution event where the software behaves
in an unexpected way
 Defect
 Typically a fault (sometimes a failure)

4
 15-May-20

Error-Fault-Failure

Error Causes

Fault
Causes

Failure

Insertion / removal
 Defect is characterized by
 Insertion activity (phase)
 Discovery
 Removal activity (phase)
insertion discovery

time
detection removal
delay

removal delay

10

5
 15-May-20

Cost of detection delay

11

11

Cost of detection delay

%$" €

&' ( ) *+' , ' - ./"

0+12*.' 1.) +' "
34- /.+) 154- "
$! "

#$"

!"
&' ( ) *+' , ' - ./" 0+12*.' 1.) +' " 34- /.+) 154- " 67/.' , "8' /." 94/.:&' ;' </' "

12

12

6
 15-May-20

Basic goals of VV
 Minimize number of defects inserted
 Cannot be zero due to inherent
complexity of software
 Maximize number of defects
discovered and removed
 Cannot prove 100% is achieved
 Minimize detection delay

13

V&V approaches
 Static
 inspections
 source code analysis
 Dynamic
 testing

14

7
 15-May-20

STATIC ANALYSIS

15

15

Static analysis techniques

 Compilation static analysis
 Control flow analysis
 Data flow analysis
 Symbolic execution
 Inspections

16

8
 15-May-20

Automatic code analysis

 It is performed
 without actually executing programs (at
compile time)
 On source code, or byte code
rules(customizable)

Source code, Automatic warnings

byte code code
analyzer

17

Code smells
 A code smell is a surface indication
that usually corresponds to a deeper
problem in the system
 Smells are certain structures in the
code that indicate violation of
fundamental design principles and
negatively impact design quality

Fowler et al., Refactoring, Improving quality of existing code. Addison-Wesley

18

18

9
 15-May-20

Technical Debt
 Technical debt reflects the extra
development work that arises when
code that is easy to implement in the
short run is used instead of applying
the best overall solution

19

19

Technical Debt
 “Shipping first time code is like going
into debt. A little debt speeds
development so long as it is paid back
promptly with a rewrite... The danger
occurs when the debt is not repaid.
Every minute spent on not-quite-right
code counts as interest on that debt.”
[W.Cunningham]

20

20

10
 15-May-20

Tools
 Static Analysis
 SonarQube
 Cast

21

21

TESTING

22

11
 15-May-20

Definition
The process of
operating a system or component
under specified conditions
observing and recording the results
to detect the differences between actual
and expected behavior (i.e. failures)

23

Purpose of test
 The purpose of testing process is to
find defects in software products
 A test process is successful if it is able to
detect failures

24

12
 15-May-20

Testing vs. debugging

 Defect testing and debugging are
different activities
 Often performed by different roles in
different times
 Testing tries to detect failures
 Debugging searches for the location of
the relative faults and removes them

25

Debugging

Test case
result Design
(failure) doc Test suite

Locate Design Repair

fault fault (code) Re-test
repair

26

13
 15-May-20

Test case
 A given stimulus applied to executable
(system or unit), consists in
 name
 input (or sequence of -)
 expected output
 With defined constraints/context
 E.g. version and type of OS, DBMS, GUI ..
 Test suite = set of related test cases

27

Good test case

 Reasonable chance of catching failure
 Does interesting things
 Doesn’t do unnecessary things
 Neither too simple nor too complex
 Non redundant w.r.t. other tests
 Makes failures obvious
 Mutually Exclusive
 Collectively Exhaustive

28

14
 15-May-20

Test case log

 Test case reference

+
 Time and date of application
 Actual output
 Result (pass / no pass)

29

Test cases examples

 Function add(int x, int y)
 Test case:
 T1: (1,1; 2)
 T2: (3,5; 8)
 Test suite
 TS1: {T1, T2}
 Test log
 T1, 16-3-2018 9:31, result 2, success
 T2, 16-3-2018 9:32, result 9, fail

30

15
 15-May-20

Test activities
Requirement doc, Code
design doc

Write Run Record

tests tests results

Test case, Test log

test suite

31

Oracle

Actual
Software output
under test

Test
Test Case Comparator result

Oracle
Expected
output

32

16
 15-May-20

Oracle
 The ideal condition would be to have
an automatic oracle and an automatic
comparator
 The former is very difficult to have
 The latter is available only in some cases
 A human oracle is subject to errors
 The oracle is based on the program
specifications (which can be wrong)

33

Oracle
 Necessary condition to perform testing:
 Know the expected behavior of a program
for a given test case (oracle)
 Human oracle
 Based on req. specification or judgment
 Automatic oracle
 Generated from (formal) req. specification
 Same software developed by other parties
 Previous version of the program (regression)

34

17
 15-May-20

Software peculiarities
 No ageing
 If function sum(2,3) works, it works
forever
– Supporting microprocessor will eventually fail
for age, not the software
 Not linear, not continuous
 If sum(2,3) works, may be sum(2,4) does
not

35

Exhaustive test
 function: Y = A + B
 A and B integers, 32 bit

 Total number of test cases:

232 * 232 = 264 ≈ 1020
 1 ns/test  ~ 3171 years

36

18
 15-May-20

Exhaustive test
 Exhaustive test is impossible
 Goal of test is finding defects, not
demonstrating that systems is defect
free
 Final objective of test (and VV in
general) is assuring a good enough
level of quality, confidence in sw

37

Dijkstra thesis
 Testing can only reveal the presence
of errors, never their absence

E. W. Dijkstra. Notes on Structured Programming.

In Structured Programming, O.-J. Dahl, E. W. Dijkstra, and C. A.
R. Hoare, Eds. Academic, New York, 1972, pp. 1–81.

38

19
 15-May-20

Test classification
 Per phase/granularity level
 Unit, integration, system
 Regression
 Per approach
 Black box (functional)
 White box (structural)
 Reliability assessment/prediction
 Risk based (safety security)

39

Test per granularity level/phase

 Unit tests
 Individual modules
 Integration tests
 Modules when working together
 System tests
 The system as a whole (usable system)
 Acceptance tests
 The system by customer

40

20
 15-May-20

Unit test
 Black box (functional)
 Random
 Equivalence classes partitioning
 Boundary conditions
 White Box (structural)
 Coverage of structural elements
– Statement
– Decision, condition (simple, multiple)
– Path
– Loop

41

Integration test
 Add one unit at a time, test the partial
aggregate
 Defects found, most likely, come by last
unit/interaction added
 Check dependencies on
– external services
– Third party components

42

21
 15-May-20

Stub, driver
 Driver
 Unit (function or class) developed to pilot
another unit
 Stub
 Unit developed to substitute another unit
(fake unit)

 Also called mockups

43

System test
 Is applied to the software system as a whole
 Aims at verifying the correspondence of the
system to the requirements
 Test of functional requirements
 Coverage of uses cases/scenarios as listed in
requirement document
 End-to-end testing: follow a real use case from
input to final result
 Test in conditions as far as possible close to
working conditions

44

22
 15-May-20

Regression testing
 Regression testing
 Tests previously defined are repeated
after a change
 To assure that the change has not
introduced defects
– Time0
– Element (unit, system ) in v0, test set t0 is defined
and applied, all tests pass
– Time1
– Element is changed to v1
– Test set t0 is re-applied, do all tests still pass?

45

Regression testing

Element v. x Element v. x+1

Run tests Run tests

Test suite

46

23
 15-May-20

References and Further Readings

 IEEE Std 829-2008: IEEE Standard for Software and
System Test Documentation
 ISTQB, Certified Tester Foundation Level Syllabus,
2001
 http://www.istqb.org/downloads/send/2-
foundation-level-documents/3-foundation-level-
syllabus-2011.html4
 Fowler et al., Refactoring, Improving quality of existing
code. Addison-Wesley
 E. W. Dijkstra. Notes on Structured Programming. In
Structured Programming, O.-J. Dahl, E. W. Dijkstra, and
C. A. R. Hoare, Eds. Academic, New York, 1972, pp. 1–
81.

47

47

24