Meet Certification Standards with Automated

Requirements Based Testing

Paul Urban
Verification and Validation Product Manager
Challenge to Deliver Complex Systems and Meet Standards

▪ Need to meet industry or customer’s standards

– DO-178C (Aero), ISO 26262 (Auto), IEC 62304 (Medical),
IEC 61508 (Industrial), MISRA, etc.

▪ Time and cost for safety critical projects estimated

20-30 times more costly*

▪ Finding defects late increases cost and time

*Source: Certification Requirements for Safety-Critical Software

2
ISO 26262-6:2018 notes Simulink and Stateflow as Suitable for
Software Architecture, Design and as basis for Code Generation

Table 2 Software Architecture Design Notations has similar suitability wording for use of Simulink and Stateflow

3
Qualify tools with IEC Certification Kit and DO Qualification Kit

▪ Qualify code generation and verification products

▪ Includes documentation, test cases and procedures

4
Qualify tools with IEC Certification Kit and DO Qualification Kit

▪ Qualify code generation and verification products

▪ Includes documentation, test cases and procedures

5
Qualify tools with IEC Certification Kit and DO Qualification Kit

▪ Qualify code generation and verification products

▪ Includes documentation, test cases and procedures

6
Conform to Certification Standards with Reference Workflow

Model Verification Code Verification

Discover design errors at design time Gain confidence in the generated code

Module and integration Back to Back

testing at the model level Testing
Prevention of
unintended functionality
Reviews and analysis
at model level

Model used for

Textual Executable Generated
production Object code
requirements specification code
code generation

Code Compilation
Modeling
generation and linking

7
Model Verification: Discover design errors at design time

Model Verification
• Manage requirements • Check standard compliance
• Systematically test • Detect design errors
• Measure model coverage • Prove model behavior compliance

Module and integration Back to Back

testing at the model level Testing
Prevention of
unintended functionality
Reviews and analysis
at model level

Model used for

Textual Executable Generated
production Object code
requirements specification code
code generation

Code Compilation
Modeling
generation and linking

8
Code Verification: Gain Confidence in the Generated Code

Code Verification
• Trace code to model and requirements • SIL/PIL equivalence testing
• Measure code coverage • Generate 100% coverage test vectors

Module and integration Back to Back

testing at the model level Testing
Prevention of
unintended functionality
Reviews and analysis
at model level

Model used for

Textual Executable Generated
production Object code
requirements specification code
code generation

Code Compilation
Modeling
generation and linking

9
Manage Requirements

Model Verification
• Manage requirements • Check standard compliance
• Systematically test • Detect design errors
• Measure model coverage • Prove model behavior compliance

Module and integration

testing at the model level

Model used for

Textual Executable Generated
production Object code
requirements specification code
code generation

Code Compilation
Modeling
generation and linking

10
Manage Requirements

External ▪ Ensure all requirements implemented

Requirements

▪ Verify the implementation is correct

▪ Respond quickly to requirement changes

Requirements
Managements
Tools

Model used for

Textual Executable Generated
production Object code
requirements specification code
code generation

Code Compilation
Modeling
generation and linking

11
Work with Requirements, Architecture and Design Together

External
Requirements Simulink Requirements

Implements
External Requirements
Architecture
/ Design

Authored Requirements
Verifies
Requirements
Tests
Managements
Tools

12
Demo: Requirements Perspective

13
Test and Requirements Traceability

Verification Status
Passed
Failed
Unexecuted
Missing

14
Review and Analyze Traceability with Traceability Matrix

Requirement is missing
link to Test Case

15
Review and Analyze Traceability with Traceability Matrix

• Review links between

different requirements,
model, test

• Filter view to manage large

sets of artifacts

• Highlight missing links

• Directly add links to

address gaps

16
Systematic Functional Testing of Model

Model Verification
• Manage requirements • Check standard compliance
• Systematically test • Detect design errors
• Measure model coverage • Prove model behavior compliance

Module and integration

testing at the model level

Model used for

Textual Executable Generated
production Object code
requirements specification code
code generation

Code Compilation
Modeling
generation and linking

17
Requirements Based Verification with Simulink Test
Implemented
FUNCTIONAL REQUIREMENTS By
The flight control system shall ... System Composer /
Simulink / Stateflow

Verified By

Test Case

Inputs Assessments
Test Harness

Signal Editor
MAT / Excel Test
MAT / Excel File (baseline) Assessments
file (input)

Test Sequence Simulink Test MATLAB Unit Test

19
Measure completeness of testing

Model Verification
• Manage requirements • Check standard compliance
• Systematically test • Detect design errors
• Measure model coverage • Prove model behavior compliance

Module and integration

testing at the model level

Model used for

Textual Executable Generated
production Object code
requirements specification code
code generation

Code Compilation
Modeling
generation and linking

20
Coverage Analysis to Measure Testing

Simulink • Identify testing gaps

Stateflow
• Missing requirements

• Unintended functionality

• Design errors
Code
Coverage Reports

21
Test and Requirements Traceability in Coverage Results

22
Scoping Model Coverage to Requirements-Based Tests

23
Scoping Model Coverage to Requirements-Based Tests

Missing
requirement
links identified

24
Test and Requirements Traceability in Coverage Results

25
Test and Requirements Traceability in Coverage Results

Decision covered but

not by linked test

26
Address missing Requirements Based Test Coverage

▪ Add missing implementation links to requirements

▪ Update test to increase target speed

27
100% Coverage but Testing Identified Error in Implementation

28
Additional Testing Identified Error in Implementation

29
Scoped Model Coverage to Requirements-Based Tests

Hit by linked RBT -- Satisfied

Hit, but not by linked RBT -- Unsatisfied

30
Check standard compliance

Model Verification
• Manage requirements • Check standard compliance
• Systematically test • Detect design errors
• Measure model coverage • Prove model behavior compliance

Reviews and analysis

at model level

Model used for

Textual Executable Generated
production Object code
requirements specification code
code generation

Code Compilation
Modeling
generation and linking

31
Verify Design to Guidelines and Standards

Check for:
• Readability and Semantics

• Performance and Efficiency

• Clones

• And more……
Model Advisor Analysis

Model used for

Textual Executable Generated
production Object code
requirements specification code
code generation

Code Compilation
Modeling
generation and linking

32
Built in checks for industry standards and guidelines

• DO-178/DO-331 • MISRA C:2012

• ISO 26262 • CERT C, CWE, ISO/IEC TS 17961

• IEC 61508 • MAB (MathWorks Advisory Board)

• IEC 62304 • JMAAB (Japan MATLAB Automotive Advisory Board)

• EN 50128

Model used for

Textual Executable Generated
production Object code
requirements specification code
code generation

Code Compilation
Modeling
generation and linking
Shift Verification Earlier With Edit-Time Checking

• Highlight violations as you edit

• Fix issues earlier

• Avoid rework
Edit-Time
Checking

Model used for

Textual Executable Generated
production Object code
requirements specification code
code generation

Code Compilation
Modeling
generation and linking

34
Detect Design Errors with Formal Methods

Model Verification
• Manage requirements • Check standard compliance
• Systematically test • Detect design errors
• Measure model coverage • Prove model behavior compliance

Reviews and analysis

at model level

Model used for

Textual Executable Generated
production Object code
requirements specification code
code generation

Code Compilation
Modeling
generation and linking

35
Detect Design Errors Using Formal Methods

▪ Find design errors

• Integer overflow
• Dead Logic
• Division by zero
• Array out-of-bounds
• Range violations

▪ Generate counter example to

reproduce error

36
Prove Model Behavior Compliance

Model Verification
• Manage requirements • Check standard compliance
• Systematically test • Detect design errors
• Measure model coverage • Prove model behavior compliance

Reviews and analysis

at model level

Model used for

Textual Executable Generated
production Object code
requirements specification code
code generation

Code Compilation
Modeling
generation and linking

37
Proving Model Meets Requirements

Safety Requirement: ▪ Need to ensure the design performs

correctly
When the brake is applied for three
consecutive steps, the throttle shall go to zero.

Model used for

Textual Executable Generated
production Object code
requirements specification code
code generation

Code Compilation
Modeling
generation and linking

38
Model functional and safety requirements

39
Link requirements to properties

40
Prove That Design Meets Requirements

41
Debugging Property Proving Violations

42
Resolve unexpected behavior in a model with Model Slicer
Isolate
Find the area of the model responsible
for unexpected behavior

Analyze dependencies
Understand data & control dependencies
in large or complex models

Inspect slice regions

Highlight model slices for time windows or Iterate
failure states & transitions for state flow.

Debug simulation behavior

Step through precompiled slices to
understand signal and port value propagation

Correct Model

43
Code Verification: Gain Confidence in the Generated Code

Code Verification
• Trace code to model and requirements • SIL/PIL equivalence testing
• Measure code coverage • Generate 100% coverage test vectors

Module and integration Back to Back

testing at the model level Testing
Prevention of
unintended functionality
Reviews and analysis
at model level

Model used for

Textual Executable Generated
production Object code
requirements specification code
code generation

Code Compilation
Modeling
generation and linking

44
Back-to-Back Testing

Tests

Embedded PC
Coder Compiler
Generated Object
Model
Code File
Desktop Simulation Object Code ▪ Automate SIL testing
(on PC) Execution (on PC) using Simulink Test

▪ Testing across releases

Compare

Results == ? Results

45
Automate Test Creation using Test Manager Wizard

▪ Guided steps to define component

to test, inputs, type of test and
format for output

▪ Wizard generates required test

harness

▪ Auto generate tests using Simulink

Design Verifier

46
Cross Release SIL/PIL Test Harness Generation

• Create a SIL/PIL test harness using code that was generated in a previous release

• Modify existing SIL/PIL test harnesses to store the build folder path information which
can be used for rebuild

47
Reference Workflow for Generated Code

Simulink Requirements*
IEC Cert Kit (for trace)
Simulink Test and Simulink Coverage (for MIL)*
Simulink Test and Simulink Coverage (for SIL)*

Simulink Check*
Simulink Design Verifier*
Simulink Test (for PIL)*

Model used for

Software Executable Generated
production Object code
requirements specification code
code generation

Code Compilation
Modeling
generation and linking
*Qualifiable

Simulink / Stateflow / AUTOSAR Blockset Embedded Coder*

48
Customer References and Applications

Airbus Helicopters Accelerates Development of DO-178B Certified Software

with Model-Based Design
Software testing time cut by two-thirds

LS Automotive Reduces Development Time for Automotive Component

Software with Model-Based Design
Specification errors detected early

Continental Develops Electronically Controlled Air Suspension for Heavy-

Duty Trucks
Verification time cut by up to 50 percent

More User Stories: www.mathworks.com/company/user_stories.html

49
Use reference workflow to conform to standards

▪ Shift verification earlier

▪ Automate manual verification tasks (coding, compiling, back-to-back)

▪ Measure completeness of Requirements Based Testing

Back to Back
Module and integration Testing
testing at the model level
Prevention of
unintended functionality
Reviews and analysis
at model level

Model used for

Textual Executable Generated
production Object code
requirements specification code
code generation

Code Compilation
Modeling
generation and linking

50
Learn More

▪ Verification, Validation, and Test Solution Page

▪ Requirements-Based Testing Workflow Example

▪ Verifying Models and Code for High-Integrity Systems

▪ Getting Started with Model Verification and Validation

51
Thank You!

52