Advanced Verification

Techniques for DO-254

Mike Bartley ([email protected])

Test and Verification Solutions

Delivering Tailored Solutions for
Hardware Verification and Software Testing

Agenda

Some background

Advanced Verification Techniques

Compliance
Hardware
(Software)

Combining Advanced Verification Techniques

with Compliance
Copyright TVS Limited | Private & Confidential | Page 2

A Quick Intro to your Speaker

PhD in Mathematical Logic

Worked in software testing and hardware verification for
over 25 years
IPL, Praxis, ST-Micro, Infineon, start-ups (Elixent and ClearSpeed)
TVS

Started TVS in 2008

Software testing and hardware verification products and services

Copyright TVS Limited | Private & Confidential | Page 3

TVS Leaders in Testing and Verification

Germany - 2011

UK - 2008
France - 2012
China
South Korea
India - 2011

Consistent
revenue
growth

2011-12
1.5M

2013-14
3.5M

2014-15
4M+

Continuous
geographical
Singapore - 2014 expansion

2012-13
2.5M

Deliver services closer to our customers

Services where costs & staff availability are important factors
Run projects on client sites or off-site
Help customers implement off-shore verification and testing
Copyright TVS Limited | Private & Confidential | Page 4

Customers and Customer Retention

Broadcom
2 years

Infineon
5 years

Intel
2 years

NVIDIA
4 years

NXP
2.5 years

ST
3 years

Copyright TVS Limited | Private & Confidential | Page 5

Some advanced verification techniques

Constrained Random
Verification Functional Coverage
it's all about
confidence
Code Coverage
Mike Bartley,
Formal Verification
SNUG 2001
Regression results metrics
Bug rate analysis
Analysis of open issues
Code review completion
Which ones to adopt?
Mutation analysis
Software running
Independent verification team
Are all requirements verified?
Copyright TVS Limited | Private & Confidential | Page 6

The Human Factor in Verification

Why do we need separate verification team?

Errors are introduced by (mis)interpretation.
Assumes you have a
specification!

Specification

RTL Coding

Interpretation

DANGER: When a designer verifies

her/his own design then she/he
is verifying her/his own
interpretation of the design

Interpretation

RTL

Verification

RTL Coding

Specification
Interpretation

RTL

Verification
Copyright TVS Limited | Private & Confidential | Page 7

Functional Verification Trends

Industry evolving its functional verification techniques
Wilson Research Group
and Mentor Graphics
2010 Functional
Verification Study,
Used with permission

37%

Assertions

69%

Constrained-Random
Simulation

41%
64%
2007
2010

48%

Code coverage

72%

Listen to the
2012 survey
Harry Foster at
DVClub April
8th

40%

Functional coverage

72%
0%

10%

20%

30%

40%

50%

60%

70%

80%

35%

29%

Median peak number of

verification engineers

30%

2007
2010

25%

19%

The adoption of
formal property
checking has
grown by 53%

20%
15%
10%
5%
0%
2007

2010
Copyright TVS Limited | Private & Confidential | Page 8

The mechanics of an advanced test bench

Functional
Coverage

Test

Coverage

Checker

Monitor

constraint
addr

Assertions

data

Stimulus
generator

Driver

Design
Under
Test
assert
Coverage

Active

Passive

Code Coverage

Copyright TVS Limited | Private & Confidential | Page 9

What is functional Coverage? Examples

The system may transfer packets of different
sizes
The test plan may require that transfer sizes with the
following size or range of sizes be observed:
1, 2, 3, 4 to 127, 128 to 252, 253, 254, or 255

Functional coverage also examines the

relationships between different objects
Cross coverage
An example of this would be examining whether an ALU has
done all of its supported operations with every different
input pair of registers
And if the ALU has written back to an input register
Copyright TVS Limited | Private & Confidential | Page 10

Adding value to your current test bench

Functional
Coverage

Checker

Coverage

Monitor

Assertions
Existing
Test
Bench

Design
Under
Test

Active

assert

Passive

Existing Test Bench

Existing
Test
Bench
Coverage

Code
Coverage

Copyright TVS Limited | Private & Confidential | Page 11

Add advanced techniques to your current test bench

Technique

Effort

Value

Code
Coverage

Low effort to start measuring

High effort to sign-off holes

Very useful when < 100%

When 100% - need other data

Functional
Coverage

High effort to define a full coverage

model
High effort to implement the
coverage model
High effort to sign-off holes

Check that major features are

fully verified

Assertions

Effort varies with number of

assertions

High value with well defined

assertions
High value for debug

Checker

Effort varies with sophistication of

the checker

High value can write tests

more quickly. Can consider
pseudo random

Constrained
random

High effort complex

Needs a checker and fnal coverage

Very high

Copyright TVS Limited | Private & Confidential | Page 12

The mechanics of finding a bug in simulation

Stimulate

Propagate

..01010101
01100101..

..01001101

11110101..
00010101..

..10011010
..01001101

Design Under Test

Mutation testing
adds value in terms
of test suite
qualification.

Actual
Results

Expected
Results

Observe

Compare

Copyright TVS Limited | Private & Confidential | Page 13

Add advanced techniques to your current test bench

Technique

Effort

Value

Code
Coverage

Low effort to start measuring

High effort to sign-off holes

Very useful when < 100%

When 100% - need other data

Functional
Coverage

High effort to define a full coverage

model
High effort to implement the
coverage model
High effort to sign-off holes

Check that major features are

fully verified

Assertions

Effort varies with number of

assertions

High value with well defined

assertions
High value for debug

Checker

Effort varies with sophistication of

the checker

High value - High value can

write tests more quickly. Can
consider pseudo random

Constrained
random

High effort complex

Needs a checker and fnal coverage

Very high

Mutation
Analysis

Low effort to adopt a tool

High effort to run and analyse output
Low effort for Do It Yourself

Very high if using tool

discover quality of you verif.
Copyright
DIY
useful
TVS will
Limited give
| Private &
Confidentialfeedback
| Page 14

The rise of design IP

External IP increase by 138% from 2007 to 2010
Wilson Research Group and Mentor Graphics
2010 Functional Verification Study

FPGA

FPGA

PCIe

PCIe
Into
the
lab

Simulation
VIP

FPGA
PCIe

PCIe
hard
ware

FPGA
PCIe

VIP

Copyright TVS Limited | Private & Confidential | Page 15

Functional Verification Approaches

Verification
Static
Reviews

Code
Analysis

Dynamic
Formal

Simulation Prototyping

Dynamic Formal
Linters
Equivalence
Checking

Silicon
FPGA

Model
Checking

Theorem
Proving

Emulation

Copyright TVS Limited | Private & Confidential | Page 16

The adoption of formal property

checking has grown by 53%

a_busy and b_busy are never both asserted on the

same cycle

if the input ready is asserted on any cycle, then the
output start must be asserted within 3 cycles

if an element with tag t and data value d enters the
block, then the next time that an element with tag t
leaves the block, its data value is the same as the
output of a reference piece of combinatorial logic
for which the input is d

stall cannot remain high indefinitely

Can be checked during

simulation (but not proved
by simulation)

Formal Verification: Some example properties

A liveness
property
Copyright TVS Limited | Private & Confidential | Page 17

Model Checking a brief introduction

Inputs to the tool

3 inputs to the tool

For example

A model of the design

Usually RTL

A property or set of
properties representing the
requirements

Items are transmitted to one of three

destinations within 2 cycles of being
accepted

(req_in && gnt_in) |-> ##[1;2]

(rec_a || rec_b || rec_c)
A set of assumptions,
expressed in the same
language as the properties
typically constraints on the
inputs to the design

The request signal is stable until it is

granted

(req_in && !gnt_out) |-> ##1

req_in
We would of course need a
complete set of constraints
Copyright TVS Limited | Private & Confidential | Page 18

Model Checking a brief introduction

Outputs from the tool

Proved
the property holds for all valid sequences of inputs

Failed(n)
there is at least one valid sequence of inputs of length n cycles, as
defined by the design clock, for which the property does not hold.
In this case, the tool gives a waveform demonstrating the failure.
Most algorithms ensure that n is as small as possible, but some more
advanced algorithms dont.

Explored(n)
there is no way to make the property fail with an input sequence of n
cycles or less
For large designs, the algorithm can be expensive in both time and
memory and may not terminate
Copyright TVS Limited | Private & Confidential | Page 19

The Strengths of Model Checking

Ease of set-up
No test bench required, add constraints as you go, VIP?

Flexibility of verification environment

Constraints can be easily added or removed

Full proof
Of the properties under the given constraints
(Can also prove completeness of the properties)

Intensive stressing of design

Explored(n) constitutes a large amount of exploration of the design
Judgement when the number of cycles explored in a run is sufficient
Significant bugs already found within a this number of cycles

Corner cases
Find any way in which a property can fail (under the constraints)

Copyright TVS Limited | Private & Confidential | Page 20

Potential issues with formal verification

False failures
Need constraints to avoid invalid behaviour of inputs

False proofs
Bugs may be missed in an over-constrained environment.

Limits on size of the model that can be analysed

Non-exhaustive checks: Explored(n)
Interpret the results
Can require significant knowledge and skill

Non-uniform run times

Often it cannot be predicted how long it will take for a check either to
terminate or to reach a useful stage

This can make formal unpredictable!

Copyright TVS Limited | Private & Confidential | Page 21

Safety-critical Systems

A safety critical system is a system where
human safety is dependent upon the correct
operation of the system

Elements of safety critical systems:
Computer hardware
Other electronic and electrical hardware
Mechanical hardware
Operators or users
Software

Traditionally associated with embedded

control systems
Copyright TVS Limited | Private & Confidential | Page 22

Safety Standards
 IEC61508: Functional Safety of
Electrical/Electronic/Programmable Electronic Safety-related
Systems
 IEC60880: Software aspects for computer-based systems
performing category A functions
 DO178: Software considerations in airborne systems and
equipment certification
 DO254: Design Assurance Guidelines for Airborne Electronic
Hardware
 EN50128: Software for railway control and protection systems
 IEC62304: Medical device software -- Software life cycle
processes
 ISO26262: Road vehicles Functional safety
Copyright TVS Limited | Private & Confidential | Page 23

Safety Standards

Process objectives and outputs

Integrity levels/classes
Picture from Kyle Beane http://www.noisefestival.com/node/14294
Copyright TVS Limited | Private & Confidential | Page 24

DO-254 identifies the following data items

Hardware Verification Plan

Validation and Verification Standards
Hardware Traceability Data
Hardware Review and Analysis Procedures
Hardware Review and Analysis Results
Hardware Test Procedures
Hardware Test Results
Hardware Acceptance Test Criteria
Problem Reports
Hardware Configuration Management Records
Hardware Process Assurance Records
Copyright TVS Limited | Private & Confidential | Page 25

DO-254 identifies the following data items

Hardware Verification Plan

Validation and Verification Standards
Hardware Traceability Data
Hardware Review and Analysis Procedures
Hardware Review and Analysis Results
Hardware Test Procedures
Hardware Test Results
Hardware Acceptance Test Criteria
Problem Reports
Hardware Configuration Management Records
Hardware Process Assurance Records
Copyright TVS Limited | Private & Confidential | Page 26

A closer look

Hardware Verification Plan
The hardware verification plan describes the procedures,
methods and standards to be applied and the processes and
activities to be conducted for the verification of the
hardware items.

Hardware Traceability Data

Hardware traceability establishes a correlation between the
requirements, detailed design, implementation and
verification data to support configuration control,
modification and verification of the hardware item

Copyright TVS Limited | Private & Confidential | Page 27

ISO 26262 Requirements Management

ISO 26262 Stipulates
The management of safety requirements includes managing requirements,
obtaining agreement on the requirements, obtaining commitments from those
implementing the requirements, and maintaining traceability.

Requirements

Stakeholder Requirements
(Customers and internal)

Upstream

Downstream

Product Requirements
Safety Requirements
Intent to
implement
System and Module Specs
Intent to
verify

Verification & Test Plans

Proof of
implementation

Verification & Test Results

Copyright TVS Limited | Private & Confidential | Page 28

REFINING THE REQUIREMENTS TO TEST DESCRIPTION LEVEL

Feature Level
Requirements
(Top-Level test Plan)

Refined
requirements
(sub-features)

Req1.1

Refined
requirements
(sub-features and
goals)

Req1.1.1

Measurable goals

Goal1.1.1.1

Goal1.1.1.2

Req1.1.2

Req1

Req1.2.1

Goal1.1.2.1

Req1.2
Goal1.2.2

Goal1.2.1.1

Copyright TVS Limited | Private & Confidential | Page 29

COMPLIANCE : HIERARCHICAL SET OF REQUIREMENTS

Copyright TVS Limited | Private & Confidential | Page 30

What are the implications for Requirements Signoff?

Just mapping a requirement to a directed test is NOT

sufficient

Requirements need to map to
Tests
Directed
Constrained random with a particular seed

Coverage
Code, functional and assertion

Checkers
Dynamic and Static

Proofs

Need to automate
Test pass and fail
Coverage collection and reporting
Checker pass and fail

All linked to configuration management data

Copyright TVS Limited | Private & Confidential | Page 31

Using Data From Advanced Verification

EDA

Require
ments

DB

UCIS API

Test
Plan

Copyright TVS Limited | Private & Confidential | Page 32

Using Data From Advanced Verification

EDA

Doors
Require
ments

DB

UCIS API

Doors
Test
Plan

This is
VERY hard
(DXL?)

Can be done.
But hard to update
with results
Copyright TVS Limited | Private & Confidential | Page 33

Using Data From Advanced Verification

EDA

Doors
Require
ments

UCIS API

DB

This is
Done

asureSIGN
Test
Plan

DB

Can be done
easily in
asureSIGN
Copyright TVS Limited | Private & Confidential | Page 34

MAPPING GOALS TO COVERAGE OR TESTS

Copyright TVS Limited | Private & Confidential | Page 35

PROJECT HISTORY GRAPH

Is a key indicator of the overall progress of the project.
Any dip or peaks indicate debugging and corrective actions may be required.

Summary

Increasing design complexity requires more

advanced verification techniques
DO254 will allow for such techniques BUT
We need to ensure we can still map
requirements to verification results
This is a lot more complex than for directed testing

Contact details

Mike Bartley
[email protected]
Mobile: +44 (0) 7796 307958
Fax: 0117 903 9001

Copyright TVS Limited | Private & Confidential | Page 38