CSC2125: Safety and

Certification of Autonomous
Vehicles
Lecture 3: Automotive Safety
Instructor: Prof. Marsha Chechik

University of Toronto, CSC2125, Lecture 3: Safety Analysis 1

Lecture Credits

• Research and talks by members of Safety Project (Toronto + McMaster)

• Mark Lawford
• Alan Wassyong
• Sahar Kokaly
• Alan Wassyong’s talk on hazards
• Toyota unintended acceleration analysis by Phillip Koopman (CMU)
• ISO 26262 process:
• Functional Safety Draft International Standard for Road Vehicles: Background,
Status, and Overview Barbara J. Czerny, Joseph D’Ambrosio, Rami Debouk (GM
R&D), Kelly Stashko (GM Powertrain)
• ISO 26262 introduction, Koen Leekens
• Functional Safety with Automated Functional Testing – QA Systems GmbH, 2017

University of Toronto, CSC2125, Lecture 3: Safety Analysis 2

Outline

• Why do safety analyses?

• Introduction to ISO 26262
• Spotlight: Hazard analysis
• Spotlight: Safety assurance cases
• Spotlight: Evolution of safety arguments
• Spotlight: Verification and validation
• Toyota unintended acceleration case study

University of Toronto, CSC2125, Lecture 3: Safety Analysis 3

Why Safety?

University of Toronto, CSC2125, Lecture 3: Safety Analysis 4

Modern Car

University of Toronto, CSC2125, Lecture 3: Safety Analysis 5

ISO 26262 – Fuctional Safety of
Road Vehicles

University of Toronto, CSC2125, Lecture 3: Safety Analysis 6

Goals of this part

• Exposure to the overall safety process

• Notion of hazard analysis
• Notion of risk assessment and ASIL determination
• Notion of safety case
• V&V methods

University of Toronto, CSC2125, Lecture 3: Safety Analysis 7

Functional Safety

• ISO 26262 (2011): Absence of unreasonable risk due to hazards caused

by malfunctioning behavior of E/E (electrical/electronic) system
• State of the art for automotive
• Developed with OEM (General Motors in particular)
• IEC 61508: Part of the overall safety related to the equipment under
control (EUC) that depends on the correct functioning of the safety-
related system

University of Toronto, CSC2125, Lecture 3: Safety Analysis 8

How do E/E systems fail?

University of Toronto, CSC2125, Lecture 3: Safety Analysis 9

ISO 26262 Principles

University of Toronto, CSC2125, Lecture 3: Safety Analysis 10

ISO26262 - Functional Safety of Road Vehicles
Standard has 10 parts
• Span across ~450 pages
• Require the production of ~120 work products
… that are the result of fulfilling a much larger number of requirements and recommendations

11
ISO 26262 follows a Safety Lifecycle

University of Toronto, CSC2125, Lecture 3: Safety Analysis 12

If you did ISO 26262 right

University of Toronto, CSC2125, Lecture 3: Safety Analysis 13

Terminology

University of Toronto, CSC2125, Lecture 3: Safety Analysis 14

University of Toronto, CSC2125, Lecture 3: Safety Analysis 15
University of Toronto, CSC2125, Lecture 3: Safety Analysis 16
University of Toronto, CSC2125, Lecture 3: Safety Analysis 17
University of Toronto, CSC2125, Lecture 3: Safety Analysis 18
Concept Phase

University of Toronto, CSC2125, Lecture 3: Safety Analysis 19

 Concept Phase
• OEM defines item – e.g., prevent use by unauthorized person by
mechanical lock
• Initiation of safety lifecycle
• Hazard analysis
• What can go wrong?
• Risk assessment
• How risky is that?
• Use ASILs to answer that

• Functional safety concept

University of Toronto, CSC2125, Lecture 3: Safety Analysis 20
Perform a Hazard Analysis. Determine ASIL

University of Toronto, CSC2125, Lecture 3: Safety Analysis 21

Consequence - Likelihood

University of Toronto, CSC2125, Lecture 3: Safety Analysis 22

Example ASILs

University of Toronto, CSC2125, Lecture 3: Safety Analysis 23

Example Outcome of Hazard Analysis

University of Toronto, CSC2125, Lecture 3: Safety Analysis 24

Identify Safety Goals

University of Toronto, CSC2125, Lecture 3: Safety Analysis 25

Identify Safety Goals - Combination

University of Toronto, CSC2125, Lecture 3: Safety Analysis 26

Identify Functional Safety Concept

University of Toronto, CSC2125, Lecture 3: Safety Analysis 27

System Level Development

University of Toronto, CSC2125, Lecture 3: Safety Analysis 28

Product Development at System Level

University of Toronto, CSC2125, Lecture 3: Safety Analysis 29

Product Development System Level

University of Toronto, CSC2125, Lecture 3: Safety Analysis 30

ISO 26262 Structure

University of Toronto, CSC2125, Lecture 3: Safety Analysis 31

Product Development Software Level

University of Toronto, CSC2125, Lecture 3: Safety Analysis 32

Safety Analyses

• Requirements decomposition w.r.t. ASIL tailoring

• Criteria for Coexistence of elements
• Dependent Failure Analysis
• Safety Analyses

University of Toronto, CSC2125, Lecture 3: Safety Analysis 33

Verification and Validation Order

University of Toronto, CSC2125, Lecture 3: Safety Analysis 34

Testing Methods by Safety Standard

University of Toronto, CSC2125, Lecture 3: Safety Analysis 35

ISO 26262

University of Toronto, CSC2125, Lecture 3: Safety Analysis 36

ISO 26262 Dynamic Testing Methods

University of Toronto, CSC2125, Lecture 3: Safety Analysis 37

Fault Injection - Simulation

University of Toronto, CSC2125, Lecture 3: Safety Analysis 38

Fault Injection - Interception

University of Toronto, CSC2125, Lecture 3: Safety Analysis 39

ISO 26262 – Dynamic Analysis – see Testing
Lecture

University of Toronto, CSC2125, Lecture 3: Safety Analysis 40

ISO 26262 – Static Analysis Verification
Methods

University of Toronto, CSC2125, Lecture 3: Safety Analysis 41

Adherence to MISRA Standards

University of Toronto, CSC2125, Lecture 3: Safety Analysis 42

ISO 26262 Recommendation - Summary

Identify obstacles to achieve

your goal

Goals

Requirements

43
HARA ISO 26262 Vehicle
Validation
Test

Safety
Concept

Verification
Test
Architecture

HW/SW Dev

Safety Case

University of Toronto, CSC2125, Lecture 3: Safety Analysis 44

Hazard Analyses
Credit: Alan Wassyong, McMaster University

University of Toronto, CSC2125, Lecture 3: Safety Analysis 45

 How do we construct correct, safe and
reliable software?
 Rigorous software engineering

Hazard analysis
is iterative over the
life of the project!

46
 How do we construct correct, safe and
reliable software?
 Rigorous software engineering

We have a defence-
in-depth approach to
the software development
process itself – driven by
identification of a single
point of failure (SPOF).

47
What is a ‘hazard’?

• It’s a property or condition in the system together with a condition in

the environment that has the potential to cause {harm or damage} =
loss [Nancy Leveson]

48
What is ‘hazard analysis’?

• Document hazards
• Document hazard controls – how to mitigate each hazard
• Hazard analysis introduces a different way of thinking about our systems
and processes
• There is lots of anecdotal evidence to suggest that we find and mitigate more
hazards when we follow some systematic process designed to perform the
hazard analysis
• Hazard analysis is mandatory in safety critical domains
• Nuclear, medical, chemical process industry, …
• And part of following standards in automotive domain

49
HA Flavours
• Lots to choose from
• Hazard & Operability Study (HAZOPS)
• Fault Tree Analysis (FTA)
• Failure Modes and Effects Analysis (FMEA)
• Failure Modes and Effects Criticality Analysis (FMECA)
• Functional Resonance Accident Model (FRAM)
• System-Theoretic Process Analysis (STPA)

50
Concept – ISO 14971

51
FTA

• Top down
• Process
• Define the TOP event to be analyzed
• Identify the lower level events which may lead to the TOP event and complete
the gates
• (optional) Find minimal cut sets (qualitative)
• (optional) Calculate the failure rate of TOP event (quantitative)

• Cut set = events that together cause the top event (sometimes
called a fault path)

• Good for identifying single points of failure

52
Example FTA: Insulin Pump

• Insulin Pump Extract – top level FTA

53
Example FTA: Insulin Pump

• Insulin Pump Extract – FTA - expand under-dosed

54
Example FTA: Insulin Pump

• Insulin Pump Extract – FTA - expand too low

55
FMEA

• Bottom up approach – need to know all details

• Was not designed to consider combination failure initiating events
• Performed on both processes and products
• Many people use RPN to prioritize – so mitigate only those hazards with
RPN > x
• RPN = Risk Priority Number
= Severity * Probability of Occurrence * Detection Rating

56
Example Process FMEA

57
Example FMEA: Insulin Pump
Functional decomposition of the insulin pump

58
 Example FMEA: Insulin Pump

59
 FMEA  Safety Requirements

60
 Motivation for STPA

• Many failures are traced back to interaction failures – components work

well, but put them together in a specific environment and we get
unanticipated failures

• STPA may help us find those hazardous interactions in the environment,

for example

• Question: Why do so many cars in Canada not have their rear lights on
at night?

61
STPA
From Nancy Leveson

• Four categories of control actions to consider

• A control action required for safety is not provided or is not followed
• An unsafe control action is provided that leads to a hazard
• A potentially safe control action is provided too early, too late, or out of
sequence
• A safe control action is stopped too soon (for a continuous or non-discrete
control action)
!! Some idea of “completeness” that helps us consider “all” possibilities

62
STPA

63
Example STPA: Insulin Pump
Viewed as a control system

64
Example STPA: Insulin Pump
Hazards included

65
Example STPA:
Insulin Pump

66
Safety Cases

University of Toronto, CSC2125, Lecture 3: Safety Analysis 67

Recall: ISO 26262 Recommendation

Identify obstacles to achieve

your goal

Goals

Requirements

68
Assurance Process

1. Completely and correctly identify goals (for safety / security / privacy)

2. Collect sufficient evidence that you have adequately dealt with each of
them

69
Assurance Case
• A.k.a. safety case, security case, privacy case, etc.
• An artifact that shows how each of the important
claims about the system can be argued for, ultimately
from evidence obtained about the system
• Evidence can come in many forms:
• test results
• analyses
• model checking results
• expert opinion
• etc.
• The argument is often informal
• “sufficient”
• “adequate”
…with some degree of confidence 70
Safety (Assurance) Case Types: Textual

71
Safety (Assurance) Case Types: Graphical

72
Assurance Case Modeling with GSN – Goal
Structuring Notation

73
Example: Power Sliding Door System

Safety goal SG1

Avoid activating the actuator when
vehicle speed > 15 kph

27
Power Sliding Door Safety Case

PSD

75
Incremental development: The answer to the
Complexity/Time Crunch?
• Idea: Reuse existing safety assurance arguments for minor
design changes (e.g. towing features)
• Sounds promising! What are the results?
Toyota Unintended Acceleration (UA)
• Brake Echo Check failsafe system for UA
only received “brake transitions”
• If your foot is already on the brake
…and then a UA event occurs,
you may have to completely take foot off
the brake & reapply to trigger failsafe
system!
2015 Ford Fusion vehicles equipped with a
mechanical key and dual screen cluster, 30
minutes after the ignition is turned off, the
Body Control Module (BCM) allows the key to
be removed when the transmission is not in
Park.
76
Part 573 Safety Recall Report 14V-736
Problem: Support for System and Safety
Evolution
• … correctly
• … quickly (via scalable automated tool support)
• … while facilitating product-level and product-line reuse

System Model System Model’

Change

R R’

Safety Case Safety Case’

?

77
System change: Removing Redundant
Switch in Power Sliding Door (PSD)
System S

Safety Goal SG1:

Avoid activating the actuator
when vehicle speed > 15 kph

Δ: removal of
redundant switch Safety goal remains
the same.
System S’
(evolved) How you achieve it &
why you believe it
changes.

78
 Naïve Evolution of Safety Case

PSD

Naïve evolution approach: Delete everything related to switch

79
Solution: Model Based Impact Assessment

System Megamodel
Human-in-the-loop refinement

Model-Based Impact
Assessment Algorithm Annotated Assurance Case
Traceability

Assurance Case

Delta (change) ✓ reuse recheck ! revise

Model Slicers
[MODELS16]: original approach
[SafeComp17]: improved approach, assurance case slicer, cost-savings analysis
80
Begin with original safety case

81
Based on traceability between system and
safety case…

82
Safety informed evolution of safety case
(after review and refinement by engineers)

By removing the redundant switch

we have change the ASIL of the AC
ECU to C

… which changes the acceptance

criteria for the evidence
84
 Assurance Case Templates

• Complete assurance case for a product line

• Complete assurance
case produced before
development starts
• Optional argument paths
• Evidence nodes specify
type of evidence
required and acceptance
criteria on that evidence
• Requires explicit
reasoning (not shown
here)
• Try to make it robust
with respect to change
• Assume it will be
developed by a
community in the same
way that standards are
• Could replace traditional
standards

85
ISO 26262 Assurance Case Template for ADAS

86
Even your keychain might be part of the
problem!
• A 1.6 mm difference in a $0.57 part!

87
Ignition Switch (keychain) Recall Revisited
• Explicit Safety Assurance
Case helps system
understanding
• Provides basis for
evaluating changes

88
Toyota Acceleration – from 2014
talk by Philip Koopman

University of Toronto, CSC2125, Lecture 3: Safety Analysis 89

Overview

University of Toronto, CSC2125, Lecture 3: Safety Analysis 90

Aug. 28, 2009, San Diego CA, USA

University of Toronto, CSC2125, Lecture 3: Safety Analysis 91

Recall and Investigation

University of Toronto, CSC2125, Lecture 3: Safety Analysis 92

May 25,
2010

University of Toronto, CSC2125, Lecture 3: Safety Analysis 93

NASA investigation

University of Toronto, CSC2125, Lecture 3: Safety Analysis 94

Toyota 2008 Electronic Traction Control System
(ETCS) – Two CPUs

University of Toronto, CSC2125, Lecture 3: Safety Analysis 95

ETCS is Safety-Critical

University of Toronto, CSC2125, Lecture 3: Safety Analysis 96

NASA Conclusions

University of Toronto, CSC2125, Lecture 3: Safety Analysis 97

$1.6B Economic Loss Class Action

University of Toronto, CSC2125, Lecture 3: Safety Analysis 98

University of Toronto, CSC2125, Lecture 3: Safety Analysis 99
The Bookout/Schwarz Results

University of Toronto, CSC2125, Lecture 3: Safety Analysis 100

US Criminal Investigation

University of Toronto, CSC2125, Lecture 3: Safety Analysis 101

The Technical Point of View

• NASA did not find a smoking gun but found plenty of questionable
practices, involving both hardware and software
• Jury found that ETCS defects caused a death
• We will look only at the software portion of the stack

University of Toronto, CSC2125, Lecture 3: Safety Analysis 102

Didn’t Vehicle Testing Make it Safe?

University of Toronto, CSC2125, Lecture 3: Safety Analysis 103

Testing Is Not Enough To Establish Safety

University of Toronto, CSC2125, Lecture 3: Safety Analysis 104

ETCS should probably be ASIL-C

University of Toronto, CSC2125, Lecture 3: Safety Analysis 105

What About Software Bugs

University of Toronto, CSC2125, Lecture 3: Safety Analysis 106

Toyota Source Code Quality

University of Toronto, CSC2125, Lecture 3: Safety Analysis 107

NASA ETCS Static Analysis Results

University of Toronto, CSC2125, Lecture 3: Safety Analysis 108

Code Complexity

University of Toronto, CSC2125, Lecture 3: Safety Analysis 109

Global Variables in Toyota

University of Toronto, CSC2125, Lecture 3: Safety Analysis 110

Concurrency Bugs / Race Conditions

University of Toronto, CSC2125, Lecture 3: Safety Analysis 111

ETCS Concurrency Bugs

University of Toronto, CSC2125, Lecture 3: Safety Analysis 112

Toyota ETCS and Recursion

University of Toronto, CSC2125, Lecture 3: Safety Analysis 113

Safety Culture

University of Toronto, CSC2125, Lecture 3: Safety Analysis 114

Other Issues

University of Toronto, CSC2125, Lecture 3: Safety Analysis 115

Some Legal Concepts

University of Toronto, CSC2125, Lecture 3: Safety Analysis 116