Verification & Validation:

Automating Best Practices to

Improve Design Quality
Chuck Olosky

© 2015 The MathWorks, Inc.

1
 Growing Complexity of Embedded Systems

Battery Management
Stability Control Automatic Parking Infotainment
Emergency Braking Instrument Panel
Adaptive Cruise Control Smart Junction Box Airbag DC/DC Converter
Body Control Module Electric Power16 M
Steering Propulsion Motor Control
Voice Recognition Engine Management Navigation

Power Window Transmission Control

Vehicle-to- Forward Camera

Infrastructure
6M
Adaptive Front
2‐3M Lighting
Power Liftgate

Power Seat HVAC Control

Back-up Camera 2000 Lines of Code 2015 Vehicle-to-Vehicle

Smart Junction Box E-Call
Long-Range Radar Active Damping Keyless Entry
Stability Control
All-Wheel Drive 4-Wheel Steer Short-Range Radar
Siemens, “Ford Motor Company Case Study,” Siemens PLM Software, 2014
Tire Pressure Monitor Ultrasonic Sensor
McKendrick, J. “Cars become ‘datacenters on wheels’, carmakers become software companies,” ZDJNet, 2013
2
Growing Complexity
Challenges the Traditional Development Process

 Find requirements defects later in the process

 Find specification issues later in the process

 Find design issues later in the process

Requirements Specification C/C++

Hand code

Design & Code Verification

3
Using Simulink Models for Specification
 Find requirements defects
earlier in the process

 Find specification issues

earlier in the process

 Find design issues

later in the process

Executable
Requirements C/C++
Specification

Hand Code

Design & Code Verification

4
Complete Model Based Design
 Find requirements defects
earlier in the process

 Find specification issues

earlier in the process

 Find design issues

earlier in the process
Simulink Models
Model used for
Executable
Requirements production code C/C++
Specification
generation
Generated Code

Design Verification Code Verification

Code Generation
5
Model Based Design Verification Workflow

Equivalence
Component and Design Review and Testing
System Level Testing Static Analysis

Equivalence
Simulink Models Checking
Model used for
Executable
Requirements production code C/C++
Specification
generation
Generated Code

Design Verification Code Verification

6
Key Takeaways

 Author, manage requirements in Simulink

 Early verification to find defects sooner

 Automate static and dynamic verification

 Workflow that conforms to safety standards

“Reduce costs and project risk through early

verification, shorten time to market on a certified
system, and deliver high-quality production code that
was first-time right” Michael Schwarz, ITK Engineering

7
Why do 71% of Embedded Projects Fail?

Poor Requirements Management

Sources: Christopher Lindquist, Fixing the Requirements Mess, CIO Magazine, Nov 2005

8
Challenges with Requirements

Where are Are they How are

they consistent with they
implemented? the design? tested?

Simulink Models
Model used for
Executable
Requirements production code C/C++
Specification
generation
Generated Code

9
Gap Between Requirements and Design

Requirements

Simulink Models
Model used for
Executable
production code C/C++
Specification
generation
Generated Code

IBM Rational DOORS

10
Simulink Requirements

Link & Track

Author
Manage Updates

11
Requirements Editor

12
Requirements Editor

13
Import Requirements from External Sources
Import
Simulink Requirements Editor

IBM Rational
DOORS

14
Requirements Perspective View of Model

15
Requirements Perspective View of Model

16
Requirements Traceability
High Level Reqs

Derives
Low Level Req
REQ 3.1 ENABLING CRUISE CONTROL ENABLE SWITCH DETECTION
Cruise control is enabled If the Enable switch is
when ….. pressed ……

Implemented Verified
By By

Design Model
x
Generated Code
Test Case
17
Track Implementation and Verification Status

Implementation Status Verification Status

Implemented Passed
Failed
Justified
No Result
Missing
Missing

18
Respond to Requirements Change

Implements
Original Requirement
If the switch is pressed and the counter reaches 50
then it shall be recognized as a long press of the switch.

Updated Requirement

If the switch is pressed and the counter reaches 75

then it shall be recognized as a long press of the switch.

19
Verify Design to Guidelines and Standards

Is the design Too Ready for code

built right? complex? generation?

Review and
static analysis

Simulink Models
Model used for
Executable
Requirements production code C/C++
Specification
generation
Generated code

20
Automated Static Analysis of the Design

Check the model for

• Readability and Semantics

• Performance and Efficiency
• Clones
• And more……
Model Advisor Analysis

Simulink Models
Model used for
Executable
Requirements production code C/C++
Specification
generation
Generated code

21
Navigate to Problematic Blocks

Model Advisor Analysis

Simulink Models
Model used for
Executable
Requirements production code C/C++
Specification
generation
Generated code

22
Guidance Provided to Address Issues w/ Auto-Correct

Model Advisor Analysis

Simulink Models
Model used for
Executable
Requirements production code C/C++
Specification
generation
Generated code

23
Modeling Guidelines for High-Integrity Systems

 Leverage industry-best practices

and MathWorks tool expertise when
developing high-integrity systems

 Modeling Guidelines with

corresponding Model Advisor checks

 Mapped to the modeling standards

and guidelines objectives of industry
standards like ISO 26262 and
MISRA-C

http://www.mathworks.com/help/pdf_doc/simulink/hi_guidelines.pdf 24
Built in Checks for Industry Standards and Guidelines

• DO-178/DO-331 • MISRA C:2012

• ISO 26262 • CERT C, CWE, ISO/IEC TS 17961

• IEC 61508 • MAAB (MathWorks Automotive Advisory Board)

• IEC 62304 • JMAAB (Japan MATLAB Automotive Advisory Board)

• EN 50128

Simulink Models
Model used for
Executable
Requirements production code C/C++
Specification
generation
Generated code

25
Configure and Customize Static Analysis

Simulink Models
Model used for
Executable
Requirements production code C/C++
Specification
generation
Generated code

26
Generate Reports for Reviews and Documentation

Model Advisor Analysis Model Advisor Reports

Simulink Models
Model used for
Executable
Requirements production code C/C++
Specification
generation
Generated code

27
Checks for Standards and Guidelines are often
Performed Late

Rework
Static
Analysis

Simulink Models
Model used for
Executable
Requirements production code C/C++
Specification
generation
Generated code

28
Shift Verification Earlier with Edit-Time Checking

• Highlight violations as you edit

• Fix issues earlier

Static
Edit-Time
• Avoid rework
Checking
Analysis

Simulink Models
Model used for
Executable
Requirements production code C/C++
Specification
generation
Generated code

29
Find Compliance Issues while you Design

30
Modeling Standards Checking with Simulink Check

 Static analysis of models against

a set of checks

 Modeling Standards Checks

– MAAB Style Guidelines V3.0
– ISO 26262
– MISRA C:2012

 Additional Checks
– Model Metrics
– Tool Bug Reports (Cert Kit)
– Requirements Consistency

31
Assess Quality with Metrics Dashboard

• Consolidated view
of metrics
• Size
• Compliance
• Complexity

• Identify where
problem areas may
be

32
Grid View for Metrics Analysis

 Visualize Standards
Check Compliance
– Find Issues
– Identify patterns
– See hot spots

Legend:
Red: Fail
Orange: Warning
Green: Pass
Gray: Not run

33
Static Analysis for Detecting Design Errors
Integer Overflow Dead Logic

Static
Analysis

Simulink Models
Model used for
Executable
Requirements production code C/C++
Specification
generation
Generated code

 Find run-time design errors

 Generates a test case to reproduce the issue for debugging 34
Functional Testing

Does the Is it functioning

design meet correctly?
Is it completely
requirements?
tested?
Component and
System Level Testing

Simulink Models
Model used for
Executable
Requirements production code C/C++
Specification
generation
Generated code

35
Systematic Functional Testing
Test Case
Inputs Assessments

MAT file (input) MAT file (baseline)

Signal Builder MATLAB Unit Test

Test Harness

Test Sequence Test Assessment

and more! and more!

Excel file (input) Main Model

Excel file (baseline)

36
Test Execution and Results Analysis

37
Coverage Analysis to Measure Test Completeness

Simulink
• Identify testing gaps

• Missing requirements
Stateflow
• Unintended Functionality
Generated Code

• Design Errors

Coverage Reports

38
Test Case Generation for Functional Testing

Test Objective
 Specify functional test
objectives
– Define custom objectives that signals
must satisfy in test cases

Test Condition  Specify functional test

conditions
– Define constraints on signal values to
constrain test generator

Test Objective

39
Prove Design Meets Requirements

 Prove design properties using formal

requirement models

 Model functional and safety requirements

 Generates counter example for analysis and

debugging

Simulink Models
Model used for
Executable
Requirements production code C/C++
Specification
generation
Generated code

40
Static Code Analysis
Is integrated Is interface between
Is the code code free of generated and other
compliant run-time code fully tested?
to MISRA? errors?

Hand Code +
Simulink Models
Model used for C/C++
Executable
Requirements production code C/C++
Specification
generation
Generated Code

Generated Code is integrated

with Hand Code
41
Static Code Analysis with Polyspace

 Code metrics and standards

– Comment density, cyclomatic complexity,…
– MISRA and Cybersecurity standards
– Support for DO-178, ISO 26262, ….

 Bug finding and code proving

– Check data and control flow of software
– Detect bugs and security vulnerabilities
– Prove absence of runtime errors

Results from Polyspace Code Prover

42
Equivalence Testing

Is the code Is all the

functionally code tested?
equivalent to Equivalence
model? Testing

Equivalence
Checking
Simulink Models
Model used for
Executable
Requirements production code C/C++
Specification
generation
Generated code

43
Equivalence Testing

 Software in the Loop (SIL)  Re-use tests developed for model to test code
– Show functional equivalence, model to code
– Execute on desktop / laptop computer  Check for equivalent outputs model to code

 Processor in the Loop (PIL)  Collect code coverage, compare to model

– Numerical equivalence, model to target code coverage
– Execute on target board

Simulink Models SIL

Desktop
Model used for
Executable Computer
Requirements production code C/C++
Specification
generation
Generated code PIL

Target
Board 44
Qualify tools with IEC Certification Kit and
DO Qualification Kit
 Qualify code generation and verification tools

 Includes documentation, test cases and procedures

KOSTAL Asia R&D Center Receives ISO 26262 BAE Systems Delivers DO-178B Level A Flight
ASIL D Certification for Automotive Software Software on Schedule with Model-Based Design
Developed with Model-Based Design

45
Lear Delivers Quality Body Control Electronics
Faster Using Model-Based Design
Challenge
Design, verify, and implement high-quality automotive
body control electronics
Solution
Use Model-Based Design to enable early and
continuous verification via simulation, SIL, and HIL
testing
Lear automotive body electronic control
Results unit.
 Requirements validated early. Over 95% of “We adopted Model-Based Design not only to deliver better-
issues fixed before implementation, versus 30%
quality systems faster, but because we believe it is a smart
previously
choice. Recently we won a project that several of our
 Development time cut by 40%. 700,000 lines of competitors declined to bid on because of its tight time
code generated and test cases reused constraints. Using Model-Based Design, we met the original
throughout the development cycle delivery date with no problem."
 Zero warranty issues reported - Jason Bauman, Lear Corporation
Link to user story
46
Customer References and Applications

Airbus Helicopters Accelerates Development of DO-178B Certified Software

with Model-Based Design
Software testing time cut by two-thirds

LS Automotive Reduces Development Time for Automotive Component

Software with Model-Based Design
Specification errors detected early

Continental Develops Electronically Controlled Air Suspension for Heavy-

Duty Trucks
Verification time cut by up to 50 percent

More User Stories: www.mathworks.com/company/user_stories.html

47
Summary
1. Author and manage requirements within Simulink

2. Find defects earlier

3. Automate static and dynamic verification

4. Reference workflow that conforms to safety standards

Component Review and Equivalence

and system static analysis testing
testing
Equivalence
checking
Simulink Models
Model used for
Executable
Requirements production code C/C++
Specification
generation
Generated code
48
Learn More

Visit MathWorks Verification, Validation and Test Solution Page:

mathworks.com/solutions/verification-validation.html

49
Thank You!

50