Coventry University

DOCTOR OF PHILOSOPHY

An accident data-driven scenario generation methodology for testing automated

driving systems in a mixed traffic environment

Perveen, Simon

Award date:
2023

Awarding institution:
Coventry University

Link to publication

General rights
Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners
and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.

• Users may download and print one copy of this thesis for personal non-commercial research or study
• This thesis cannot be reproduced or quoted extensively from without first obtaining permission from the copyright holder(s)
• You may not further distribute the material or use it for any profit-making activity or commercial gain
• You may freely distribute the URL identifying the publication in the public portal
Take down policy
If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately
and investigate your claim.

Download date: 22. May. 2024

 An accident data-driven scenario
generation methodology for testing
automated driving systems in a mixed
traffic environment

By
Ing. Simon Perveen

Doctor of Philosophy (PhD)

April 2023
 An accident data-driven scenario
generation methodology for testing
automated driving systems in a mixed
traffic environment

A thesis submitted in partial fulfilment of the

University’s requirements for the Degree of Doctor of
Philosophy

April 2023
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Abstract

To reduce road traffic accidents caused by human drivers, more and more vehicles
equipped with automated driving systems (ADSs) (SAE Level 3 ≥) are released. Since
ADSs replace human drivers, the expectation on their safety performance is very high. It
has been predicted that millions and billions of miles are required to be driven to estimate
the safety performance of ADSs. The industry aims to implement scenario-based test
strategies in virtual environments to reduce this effort. However, the parameter space for
scenario specification is multi-dimensional. Hence, a guided search is required to identify
solely the critical scenarios to avoid testing irrelevant scenario combinations.

This thesis proposes a non-simulation-based scenario generation method using the

STATS 19 accident data to identify critical scenarios that occurred under adverse
environmental conditions. The methodology is based on two studies and focuses on
mixed traffic scenarios in intersections. Adverse environmental conditions are known to
reduce the safety performance of ADSs. However, due to their low frequency, such
patterns are hidden in accident data. Hence, the first study uses the k-medoids clustering
algorithm to extract hidden patterns for scenario specification from the STATS 19 data.

Important information, such as vehicle dynamics, is missing in the STATS 19 data.

Vehicle dynamics, however, are essential to test the STATS 19 scenarios in a virtual
environment. Defining random vehicle dynamics would end in a large number of
irrelevant tests. Hence, the second study aims to optimise the search for relevant vehicle
dynamics parameters with high injury and collision risks. The second version of the Non-
dominated Sorting Genetic Algorithm (NSGA-II), a metaheuristics-based optimisation
algorithm, has been used to carry out this expensive search activity.

For scenario specification, the k-medoids algorithm successfully identified hidden

patterns in the STATS 19 data containing adverse environmental conditions (e.g., snow,
fog et al.). The NSGA-II algorithm has successfully identified injury critical vehicle
dynamics parameters with great coverage in the scenario design space. Moreover, by
automating the identification process of injury critical scenarios in a modular and time-
efficient way, this work has the potential to: a) accelerate verification and validation
activities and b) improve the safety of ADSs in mixed-vehicle fleet intersections.

i
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

This thesis is dedicated to my wonderful family.

ii
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Acknowledgements

I cannot begin to express my most profound appreciation to my supervisory team at

Coventry University. Especially my Director of Studies, Dr. Christophe Bastien, who
supported me from day one until the last day of completion of this thesis. A big thank you
goes to Dr. Jesper Christensen, who has been part of my supervisory team; he contributed
exceptionally with his valuable advice to succeed with my research.

A big thank you goes to Dr. Karthikeyan Ekambaram who has been my Director of
Studies prior leaving Coventry University, he has put a lot of time into my academic
development, a great thank you for your valuable time. I also wish to thank Prof. Dr.
Stratis Kanarachos and Prof. Dr. Dobrilla Petrovic who have been part of my supervisory
team and supported me during my PhD studies. I also want to thank the Doctoral College
for supporting me during my studies.

I also want to thank the examiners team Dr. Alexandro Badea Romero and Dr. Olivier
Haas for giving me valuable suggestions for improving the proposed work in this thesis.

A big thank you goes to Dr. James Lenard who has helped me technically during my data
collection.

A huge thank you to Alexander Diederich, who gave me valuable advice during my PhD
studies.

I also want to thank Prof. Dr. Jörg Hoffmann and Prof. Dr. Groh for their support.

A huge thank you goes to my line manager Dr. Philipp Schnetter at CARIAD who
supported me during my studies.

Thank you to the Dooley family for their valuable support during my studies.

Finally, a great thank you goes to my family for unconditionally supporting me during
these challenging times. A special thank you goes to my mother.

iii
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Table of content

Chapter 1. Introduction .............................................................................................. 21

1.1 Background information .................................................................................. 21

1.2 Challenges related to scenario-based testing ................................................... 25

1.3 Dissertation outline .......................................................................................... 29

Chapter 2. Literature Review ..................................................................................... 31

2.1 Introduction ...................................................................................................... 31

2.2 Sensors and their impact on ADSs safety ........................................................ 32

2.3 Scenario generation data and methods for Level 0 to Level 5 systems ........... 38

2.3.1 Data sources for critical scenario generation ............................................ 38

2.3.2 Algorithms for scenario space construction using accident data .............. 45

2.3.3 Algorithms for scenario space exploration ............................................... 51

2.4 Summary of literature review .......................................................................... 62

Chapter 3. Methodology............................................................................................. 65

3.1 Introduction ...................................................................................................... 65

3.2 Study 1: Scenario space construction using accident data ............................... 67

3.2.1 STATS 19 data ......................................................................................... 68

3.2.2 Layer 1: High frequency functional scenarios .......................................... 78

3.2.3 Scenario Ranking ...................................................................................... 84

3.2.4 Layer 2: rare event functional scenarios ................................................... 86

3.3 Study 2: Identification of injury critical concrete scenarios ............................ 88

3.3.1 Introduction to the intersection optimisation problem ............................. 89

3.3.2 Step 1: Mathematical intersection model ................................................. 90

iv
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

3.3.3 Step 2: Identification of potential injury critical concrete scenarios using

NSGA-II ............................................................................................................... 104

3.3.4 Step 3: Identification of potential injury critical concrete scenarios using

LHS 108

3.3.5 Step 4: Validation of injury critical concrete scenarios .......................... 111

3.4 Summary of the methodology chapter ........................................................... 114

Chapter 4. Results and Discussion ........................................................................... 116

4.1 Introduction .................................................................................................... 116

4.2 Study 1: Specification of rare event accident scenarios using STATS 19 data
117

4.2.1 Algorithm settings layer 1 ...................................................................... 117

4.2.2 High frequency functional scenarios ...................................................... 120

4.2.3 Low frequency functional scenarios ....................................................... 127

4.2.4 Limitations of study 1 ............................................................................. 131

4.3 Identification of injury critical concrete scenarios ........................................ 132

4.3.1 Validation of injury criticality of concrete scenarios ............................. 132

4.3.2 Analysing scenario space coverage ........................................................ 138

4.3.3 Limitations and assumptions of study 2 ................................................. 144

4.4 Presentation of scenario library ..................................................................... 148

4.5 Summary of results and discussion ................................................................ 150

Chapter 5. Conclusions and future work .................................................................. 152

5.1 Introduction .................................................................................................... 152

5.2 Conclusions .................................................................................................... 152

5.3 Future work .................................................................................................... 155

5.4 Summary of conclusion and future work ....................................................... 156

v
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

List of figures

Figure 1-1: Six levels of driving automation introduced by SAE (Copyright © 2021 SAE
International) (SAE International 2021) ......................................................................... 22

Figure 1-2: Number of miles required to showcase certain percentage of improvement of

ADSs over human drivers (Kalra and Paddock 2016) ................................................... 24

Figure 1-3: Overall SOTIF Goal minimizing area 3 and maximizing area 1 scenarios
adapted from ISO 21448. ............................................................................................... 26

Figure 2-1: Placement of different sensors for environment perception (Wendt and Cook
2018). .............................................................................................................................. 32

Figure 2-2: Link between TCs, FISs and hazardous behaviour at vehicle level
(International Standardisation Organisation 2022)......................................................... 33

Figure 2-3: Systematic 6-layer model for scenario parametrisation accordingly to the
PEGASUS method (Eberle, Wachenfeld, and Schittenhelm 2019) ............................... 36

Figure 2-4: Workflow from functional to logical and logical to concrete scenario (Menzel,
Bagschik, and Maurer 2018) .......................................................................................... 37

Figure 2-5: Knowledge and data-driven scenario generation workflow adopted from
Riedmaier et al. (2020) ................................................................................................... 39

Figure 2-6: Summary of different existing accident data sources in literature applied for
scenario generation. ........................................................................................................ 40

Figure 2-7: Workflow of Genetic Algorithms and their genetic operators adopted from
(Bi and Hu 2021, Liu et al. 2019) ................................................................................... 56

Figure 3-1: Overall proposed scenario generation method. ........................................... 65

Figure 3-2: Rare event functional scenario generation workflow based on STATS 19 data.
........................................................................................................................................ 67

Figure 3-3: STATS 19 data structure. ............................................................................ 68

Figure 3-4: Structure of supporting document for looking up codes of labels. .............. 69

vi
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Figure 3-5: Data shape of merged STATS 19 data on vehicle level. ............................. 71

Figure 3-6: STATS 19 data sample size after conducting data preparation steps reported
in this thesis. ................................................................................................................... 78

Figure 3-7: Parameter interactions required to cover 100% of software failure for different
engineering sectors (Kuhn, Wallace, and Gallo 2004). .................................................. 84

Figure 3-8: Diagram showing the breakdown of a high frequency scenario into multiple
low frequency scenarios by applying clustering iteratively. .......................................... 87

Figure 3-9: Scenario generation workflow for the identification of injury critical concrete
scenarios. ........................................................................................................................ 88

Figure 3-10: Generic pictogram of T-intersection scenario and the additional parameters
required to run it in a simulation environment. .............................................................. 90

Figure 3-11: Impact speed in function of Injury (MAIS3+) probability for different
scenarios (Jurewicz et al. (2016)). .................................................................................. 95

Figure 3-12: Number of conflict points for different road geometries and manoeuvres
(European Commission 2015). ....................................................................................... 98

Figure 3-13: Relationship between TA, conflicting speed, and accident injury (Hyden
1987 cited in Nitsche (2018)). ........................................................................................ 99

Figure 3-14: NSGA-II workflow based on the principle of non-dominated sorting and
crowding distance (Jiang et al. 2021). .......................................................................... 105

Figure 3-15: Serious injury model expressed in function of Delta V and serious injury
probability (Gabauer and Gabler (2006)). .................................................................... 113

Figure 4-1: Mean Silhouette value plot for the estimation of optimal k. ..................... 117

Figure 4-2: Mean Davies Bouldin Value plot for the estimation of optimal k. ............ 118

Figure 4-3: Silhouette plot for square Euclidean distance method. .............................. 119

Figure 4-4: Criticality ranking of the 24 high frequency functional scenarios. ........... 124

Figure 4-5: Distribution of weather conditions in layer 2. ........................................... 127

Figure 4-6: Probability for serious injury outcome of the generated concrete scenarios by
the NSGA-II and the LHS. ........................................................................................... 133

vii
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Figure 4-7: Delta V values for generated concrete scenarios by the NSGA-II and LHS
algorithm. The red line represents the performance boundary of 10% to classify the
scenarios as relevant or not relevant for serious injury outcome in T-intersections. ... 136

Figure 4-8: Parameter distribution in design space of Ego velocity (vehicle A) by LHS
(left) and NSGA-II (right) algorithm. ........................................................................... 138

Figure 4-9: Parameter distribution in design space of Target vehicle (vehicle B) velocity
by LHS (left) and NSGA-II (right) algorithm. ............................................................. 139

Figure 4-10: Parameter distribution in design space of impact angle by LHS (left) and
NSGA-II (right) algorithm. .......................................................................................... 141

Figure 4-11: Parameter distribution in design space of Ego vehicle (vehicle A) road
friction by LHS (left) and NSGA-II (right) algorithm. ................................................ 142

Figure 4-12: Parameter distribution in design space of Target vehicle (vehicle B) road
friction by LHS (left) and NSGA-II (right) algorithm. ................................................ 143

Figure 4-13: Transition from functional to logical and from logical to concrete scenarios
adopted from (Menzel, Bagschik, and Maurer 2018). .................................................. 148

Figure_APDX A: Schematic PEGASUS workflow for the assessment of SAE Level 3 or

greater systems. ............................................................................................................ 174

viii
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

List of tables

Table 2-1: Summary of past studies using optimisation for test scenario generation. The
table shows the aim, optimisation objectives, number of objectives and optimisation
algorithm applied to solve the optimisation problem. .................................................... 58

Table 3-1: Selected parameters using the PEGASUS 6-Layer model and rationale for
parameter selection ......................................................................................................... 72

Table 3-2: Summary of the selected parameters from STATS 19 and its observations after
the data preparation. ....................................................................................................... 76

Table 3-3: Summary of potential triggering conditions and their observations used to
parametrize the rare event scenarios. .............................................................................. 85

Table 3-4: Svensson's (1998) risk model for critical accident parametrization. ............ 91

Table 4-1: Functional scenarios identified in the first layer with the highest frequency in
the STATS 19 data sample. The highlighted observations show how cluster 1 and cluster
4 differ. ......................................................................................................................... 121

Table 4-2: Parametrisation of the two highest ranked high frequency functional scenarios
3 and 9. ......................................................................................................................... 125

Table 4-3: Five highest ranked critical high frequency functional scenarios among the 24
clusters. ......................................................................................................................... 126

Table 4-4: Low frequency and rare event functional scenarios.................................... 128

Table 4-5: Set of high injury scenarios identified by the NSGA-II but not by the LHS
algorithm....................................................................................................................... 134

Table 4-6: High injury critical concrete scenario identified by LHS but not by NSGA-II.
...................................................................................................................................... 135

Table_APDX 1: Clustering results for layer 1 for cluster 1 to cluster 7 ...................... 176

Table_APDX 2: Clustering results for layer 1 for cluster 8 to cluster 14 .................... 180

ix
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Table_APDX 3: Clustering results for layer 1 for cluster 15 to cluster 21 .................. 184

Table_APDX 4: Clustering results for layer 1 for cluster 22 to cluster 24 .................. 188

Table_APDX 5: Re-clustering results of cluster 3 showing clusters 3.1 to 3.6. .......... 190

Table_APDX 6: Re-clustering results of cluster 3 showing clusters 3.7 and 3.8......... 194

x
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

List of code listings

Listing 3-1: Command for selection of serious injury accidents only. (Note: ‘d’ is the
STATS 19 data sample and 16 the column where the injury information is stored.) .... 73

Listing 3-2: Command for selecting only accidents containing strictly two vehicles. ... 74

Listing 3-3: Command to realise step (2) deleting unwanted information and only keeping
T-intersections. ............................................................................................................... 74

Listing 3-4: Step (f): Commands for merging data, recoding the observations, and deleting
the unwanted data. .......................................................................................................... 75

Listing 3-5: Command for omitting accidents with missing observations. .................... 77

Listing 3-6: Command to investigate optimal k in a range fro [1:50] for STATS 19 data.
........................................................................................................................................ 83

Listing 3-7: Command to implement k-medoids algorithm in MATLAB. Note that X, in

this case, is the data to be clustered = prepared STATS 19 data. ................................... 83

Listing 3-8: Implementation of mathematical intersection model in ‘pymoo’ coded as a

MOO problem. ............................................................................................................. 106

Listing 3-9: Implementation of NSGA-II algorithm in ‘pymoo’ to solve the MOO problem
defined in the class ‘MyProblem(Problem)’. ............................................................... 107

Listing 3-10: LHS implementation in MATLAB for the generation of 10.000 random
concrete scenarios. ........................................................................................................ 109

Listing 3-11: Computation of constraints equations from the mathematical model. ... 110

Listing 3-12: Application of bounds of constraints as a filter to omit the non-valid

scenarios in context of mathematical intersection model. Only 500 scenarios are kept as
shown in last command. ............................................................................................... 110

Listing 3-13: MATLAB implementation to compute serious injury probability using

equation 3-24 ................................................................................................................ 112

xi
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

List of appendices

APPENDIX A: PEGASUS framework ........................................................................ 173

APPENDIX B: STATS 19 data preparation code ........................................................ 175

APPENDIX C: Layer 1 high frequency scenarios ....................................................... 176

APPENDIX D: Layer 2 low frequency scenarios and rare event scenarios ................. 190

xii
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

List of abbreviations and acronyms

ADS Automated driving system

AEB Automated emergency braking

AIS Abbreviated Injury Scale

AV Automated vehicle

BASt Bundesanstalt für Straßenwesen

CA Clustering algorithm

CS Conflicting speed

DB Davies Bouldin method

DDT Dynamic driving task

DMV Department of Motor Vehicles

DOE Design of experiments

DoT Department of Transports

DSS Driver Support System

EDR Event-data-recorder

EDV Extended-Delta-V

FHA Federal Highway Administration

FIS Functional insufficiency

GA Genetic Algorithm

GIDAS German In-Depth Accident Study

GSO Global Simplex Optimisation

HC Hierarchical clustering

IoT Internet of things

IS Importance sampling

xiii
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

ISO International Standardisation Organisation

ITF International Transport Forum

ITRAD International Road Traffic and Accident Database

LCC Latent class clustering

LHS Latin Hyper Cube Sampling

MAIS3 + Maximum Abbreviated Injury Scale 3 + (serious injury ≥)

MC Monte Carlo

MCMC Markov Chain Monte Carlo

MHH Medizinische Hochschule Hannover

MOEA Multi-Objective Evolutionary Algorithm

MOO Multi-objective-optimisation

NASS National Automotive Sampling System

NDD Naturalistic driving data

NDS Naturalistic driving studies

NFOTs Naturalistic field operational tests

NHTSA National Highway Traffic Safety Administration

NLP Natural Language Processing

NN Neuronal networks

NSGA Non-dominating Sorting Genetic Algorithm

NSGA-II Non-dominated Sorting Genetic Algorithm - version 2

ODD Operational design domain

OEDR Object and event detection and response

OI Output insufficiencies

OTS On-the-spot

PAM Partitioning around medoids

xiv
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

RAIDS Road Accident In-Depth Studies

PEGASUS Project for the Establishment of Generally Accepted

quality criteria, tools and methods, as well as Scenarios and
Situations

PMAIS3 + Probability of Maximum Abbreviated Injury Scale 3 +

(serious injury ≥)

PSO Particle Swarm Optimisation

PYMOO Multi-Objective-Optimisation in Python

RQ (1) Research question (1)

RQ (2) Research question (2)

RQ (3) Research question (3)

SA Simulated Annealing

SAE Society of automotive engineers

SBST Search-based software testing

SOTIF Safety of the intended functionality

SPMD Safety Pilot Model Deployment Program

TA Time-to-Accident

TC Triggering condition

TCT Swedish Traffic Conflict Technique

TTC Time-to-Collision

V&V Verification and Validation

VSN Variable Neighbourhood Search

WHO World Health Organisation

xv
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

List of symbols

Symbol Description Page

k Numbers of clusters 79

𝑑𝐸 Squared Euclidean distance 80

⃗⃗⃗
𝑃 Population vector of a cluster 80

⃗⃗⃗
𝑄 Population vector of a cluster 80

𝑖 Index 80

𝑛 ⃗⃗⃗ and 𝑄
Size of cluster 𝑃 ⃗⃗⃗ 80

𝑝(𝑖) Element in cluster vector ⃗⃗⃗

𝑃 80

𝑞(𝑖) Element in cluster vector ⃗⃗⃗

𝑄 80

𝑠(𝑗) Silhouette value 81

j Object in cluster A or B 81

𝑏(𝑗) average dissimilarity of 𝑗 compared to all other 81

objects within cluster 𝐵

𝑎(j) average dissimilarity of 𝑗 compared to all other objects 81

within cluster 𝐴

𝑑𝑏(𝑟) Davies Bouldin value 82

𝐶(𝑟) A random cluster 82

𝑇(𝑟) Size of cluster 𝐶(𝑟) 82

𝑙 Index element for 𝑋(𝑙) 82

xvi
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

𝑋(𝑙) 𝑋(𝑙) is an n-dimensional vector assigned to 𝐶(𝑟) 82

𝐴(𝑟) 𝐴(𝑟) is the centroid of the cluster 𝐶(𝑟) 82

𝑝 Power of 𝑑𝑏(𝑟) equation 82

𝑛𝑟𝑒𝑝𝑙. Number of cluster replicates 82

∞ Infinite 82

𝑣𝐴 (𝑋) Speed of vehicle A 92

𝑣𝐵 (𝑋) Speed of vehicle B 92

𝜇𝐴 (𝑋) Road tyre friction coefficient for vehicle A 92

𝜇𝐵 (𝑋) Road tyre friction coefficient for vehicle A 92

𝛼𝑖𝑚𝑝𝑎𝑐𝑡 (𝑋) Impact angle between vehicle A and B 92

𝑇𝐴𝐴 Time-to-accident vehicle A 92

𝑇𝐴𝐵 Time-to-accident vehicle B 93

∆𝑡𝑐𝑜𝑛𝑓𝑙𝑖𝑐𝑡 Crash condition to ensure closeness in time for 93

vehicle A and vehicle B

𝑠𝐴 Braking distance to collision point vehicle A 93

𝑠𝐵 Braking distance to collision point vehicle B 93

𝐸_𝛥𝑣𝐴 Extended Delta V for vehicle A 93

𝑔 Gravity 99

𝑚𝐴 Mass of vehicle A 103

𝑚𝐵 Mass of vehicle B 103

xvii
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

𝑃𝑡 Initial population of NSGA-II solutions 104

𝑌 Size of 𝑃𝑡 104

𝑄𝑡 Offspring population of 𝑃𝑡 104

Rt Combined solution pool consisting of 𝑃𝑡 and 104

𝑄𝑡

𝑃𝑀𝐴𝐼𝑆3+ Probability of Maximum Abbreviated Injury Scale 112

3 plus (serious injury or greater)

𝑑𝑣 Speed change during collision 112

xviii
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Chapter 1. Introduction
1.1 Background information

Despite many improvements in today’s vehicle safety, traffic accidents are still common.
A statistic published by the World Health Organisation (WHO) reports that
approximately 1.3 million people die yearly from road traffic accidents (World Health
Organization 2022). Furthermore, injuries from traffic accidents are the leading cause of
death for children and young adults aged 5 to 29 years. According to National Highway
Traffic Safety Administration’s (NHTSA) statistics, 94% of accidents in the United
States of America (USA) occur due to poor human driver performance (NHTSA n.d.).

The NHTSA is dedicated to improving road safety by implementing new vehicle

technologies, which are taking more and more responsibility from the human driver. The
automotive and research industry believes that introducing vehicle driving automation
systems, which perform all or part of the dynamic driving task (DDT), will decrease road
traffic accidents (Mercedes-Benz Group 2019, Petrovic, Mijailović, and Pešić 2020).

According to the definition of the society of automotive engineers (SAE), the DDT
comprises of the lateral and longitudinal control of the vehicle, the object and event
detection and response (OEDR), the manoeuvre planning and the enhancement of the
conspicuity (via lighting, sounding the horn, signalling, gesturing et al.) (SAE
International 2021).

The SAE J3016 has defined six levels of driving automation from Level 0 (no driving
automation), Level 1 (driver assistance), Level 2 (partial driving automation), Level 3
(conditional driving automation), Level 4 (high driving automation) to Level 5 (full
driving automation) (SAE International 2021). Figure 1-1 gives detailed information
regarding the relationship between these six levels and the role of the human driver.

21
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

This item has been removed due to 3rd Party Copyright. The unabridged version of the thesis can be found in
the Lanchester Library, Coventry University.

Figure 1-1: Six levels of driving automation introduced by SAE (Copyright © 2021 SAE
International) (SAE International 2021).

Level 0 systems provide momentary assistance to the driver. They only intervene briefly
during the DDT when necessary (e.g. in a safety critical situation). They do not
significantly impact the driver’s role in performing part or all of the DDT and have no
role in automating the DDT. Hence, Level 0 systems are classified as no driving
automation. Note that systems which are active during vehicle operation but only provide
warnings to the driver are classified as Level 0 (no driving automation), as they do not
automate the DDT or replace the driver’s role in executing the DDT.

Level 1 to Level 2 systems can be classified as Driver Support Systems (DSSs). DSSs
only perform part of the DDT supporting the driver. DSSs require the human driver to
steadily supervise the DDT and provide support for lateral or/and longitudinal vehicle
control.

Automated Driving Systems (ADSs) represent Level 3, Level 4 and Level 5 systems,
highlighted in green in Figure 1-1. ADSs are capable of performing the entire DDT.

22
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

However, for Level 3, when requested, the human driver must take over the DDT in given
situations. For Level 4 and Level 5 ADSs, there is no takeover request for the human
driver. Levels 4 and 5 differ because Level 5 has no operational design domain (ODD)
limits, whereas Level 4 is restricted to a specific ODD. The ODD defines the boundaries
for which an ADS has been designed and its operational environment (SAE Industry
Technologies Consortia 2020). Since, for SAE Level 3 and greater systems, the human
driver is replaced by technology, safety requirements for these systems are very strict.

Evidence exists that Level 0 to Level 2 systems have previously contributed positively to
road accident statistics by lowering traffic accidents (Spicer et al. 2018, Tan et al. 2020,
Bill 2020).

The first vehicles with Level 3 and 4 ADSs were introduced in 2022 (Mercedes-Benz
Group 2022, Bosch 2022). Hence, there is insufficient evidence to confidently assess
whether ADSs have reduced the number of road accidents. However, early predictions
have been made that ADSs will positively impact road accident statistics (Federal
Ministry of Transport and Digital Infrastructure 2015).

A more recent statistic published by Östling, Jeppsson, and Lubbe (2019) predicted that
although ADSs are likely to reduce road traffic accidents in the future, ADSs will cause
an increase in mixed-vehicle fleet accidents at intersections. Mixed-vehicle fleet
accidents are collisions between vehicles with different levels of driving automation,
ranging from Level 0 to Level 5.

Moreover, recent analyses of autonomous vehicle collision reports, ranging from Level 3
to Level 5, gathered by the California Department of Motor Vehicles (DMV) have clearly
shown that most accidents in mixed-vehicle fleet situations occurred at intersections
where the human driver has been in the majority of the cases deemed to be at fault (Favaro
et al. 2017, Pokorny and Høye 2022).

For ADSs to succeed in the future, one necessary activity is to address their safety
performance in mixed-vehicle fleet intersection scenarios. Consequently, more evidence
is required to showcase the safety benefits of ADSs. Providing evidence for such safety
argumentation is one of the most significant challenges the automotive industry is facing
(Cieslik et al. 2022).

23
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Evidence, in general, is generated by iteratively conducting different types of tests during

the verification and validation (V&V) phase of ADSs. Since ADSs take over DDT from
human drivers, the criteria for these tests are highly stringent. The tests must provide
evidence that ADSs decrease the number of road accident injuries and fatalities compared
to human drivers (Di Fabio 2017, Cerny et al. 2020). However, according to statistics
based on American accident data, the fatality rate of human drivers is 1.34 fatalities per
100 million miles driven, which is exceptionally low. To evidence that, for ADSs to
compete against this low fatality rate, millions and billions of miles must be driven
(Wachenfeld and Winner 2015, Kalra and Paddock 2016). Kalra and Paddock (2016)
estimated that 5 billion miles must be driven to show a 20% improvement in automated
vehicles (AVs) compared to human drivers in reducing road fatalities (see Figure 1-2).
This assumption was based on 1.34 fatalities per 100 million miles and statistical analysis.
This item has been removed due to 3rd Party Copyright. The unabridged version of the thesis can be found in the
Lanchester Library, Coventry University.

Figure 1-2: Number of miles required to showcase certain percentage of improvement of

ADSs over human drivers (Kalra and Paddock 2016).

Distance-based testing is time-consuming and expensive and can also put other road
traffic participants at risk. To complete a 5 billion miles test at a speed of 25 miles per
hour, 1000 AVs must function and collect evidential data constantly for 23 years. This
calculation shows how challenging such a distance-based test strategy can be.

24
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

To reduce the number of testing miles, the industry is moving towards scenario-based test
strategies in virtual environments (Eberle, Wachenfeld, and Schittenhelm 2019,
Levermore and Peters 2020). Note that scenario-based testing is not a replacement for
distance-based testing but can reduce the effort required for distance-based testing.

Eberle, Wachenfeld, and Schittenhelm (2019) introduced a widely accepted definition for
the term scenario: “A scenario describes the temporal development between several
scenes in a sequence of scenes. Every scenario starts with an initial scene. Actions &
events, as well as goals & values, may be specified to characterize this temporal
development in a scenario. Other than a scene, a scenario spans a certain amount of
time.”

A scene has been defined as: “A scene describes a snapshot of the environment including
the scenery and dynamic elements, as well as all actors’ and observers’ self-
representations, and the relationships among those entities. Only a scene representation
in a simulated world can be all-encompassing (objective scene, ground truth). In the real
world, it is incomplete, incorrect, uncertain, and from one or several observers’ points of
view (subjective scene).”

1.2 Challenges related to scenario-based testing

From the definitions provided by Eberle, Wachenfeld, and Schittenhelm (2019), it can be
witnessed that scenarios have a multidimensional parameter space. Testing ADSs against
every parameter combination of scenarios can lead to millions and billions of tests, which
is not cost-efficient and computationally complex (Amersbach and Winner 2019).
Moreover, not each scenario in the full factorial design space is relevant for the safety
assessment of ADSs because it is not sure whether they have a potential for human harm.
Hence, to reduce the vast scenario space, a strategy is necessary to classify scenarios as
relevant or irrelevant for safety assessment of ADSs.

To tackle the challenge of relevant scenarios for safety assessment, the ISO 26262
(International Standardisation Organisation 2012) and ISO 21448 (International
Standardisation Organisation 2022), two safety standards frequently applied in the
automotive industry, propose to assess the safety of electrical and

electronic (E/E) systems based on hazards. However, both standards have a different
focus. The malfunctioning of E/E systems defines the root cause for the hazards in ISO

25
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

26262, thus addressing functional safety. ISO 21448 covers the interaction of the E/E
systems with the environment (e.g. weather and lighting conditions but also human
machine interface), which can cause a hazardous behaviour of ADSs leading to hazards
and human harm. The hazardous behaviour of the E/E system in ISO 21448 does not lead
directly to a malfunction but can reduce the performance of an ADS to such an extent that
it causes a safety problem. As a result, ISO 21448 is concerned with the safety of the
intended functionality (SOTIF) of Level 0 to Level 5 systems. This thesis will focus on
hazards resulting from ISO 21448 rather than from ISO 26262.

For scenario classification, to reduce the scenario parameter space to a relevant set of
scenarios, ISO 21448 suggests classifying the scenario space into four different scenarios
areas:

• Known, not hazardous scenarios (SOTIF area 1)

• Known, hazardous scenarios (SOTIF area 2)
• Unknown, hazardous scenarios (SOTIF area 3)
• Unknown, not hazardous scenarios (SOTIF area 4)

Figure 1-3 shows how these four scenario categories can contribute to easing the safety
assessment of ADSs by following activities suggested in ISO 21448.

Figure 1-3: Overall SOTIF Goal minimizing area 3 and maximizing area 1 scenarios adapted from
ISO 21448.

26
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Figure 1-3 gives an overview of the assumed initial scenario distribution and the desired
scenario distribution after following activities proposed by ISO 21448.

ISO 21448 recommends minimising the hazardous scenario space (areas 2 and 3) and
maximising the known not hazardous scenario space (area 1). The desired scenario
distribution can be achieved through continuous product development improvement. To
do so, unknown and known hazardous scenarios must be identified and documented.
Moreover, safety measures must be specified to optimise the ADS under development to
transfer the hazardous scenarios to the not hazardous scenario space.

The parametrisation of area 2 and 3 scenarios is one of the automotive industry's most
significant challenges (Beringhoff et al. 2022). The identification of unknown and known
hazardous scenarios is not straightforward. Moreover, ISO 21448 suggests linking
scenarios with triggering conditions (TCs) which can initiate a hazardous behaviour of
the system. Environmental conditions, such as adverse weather (rain, wind, storms, snow,
etc.), lighting conditions (darkness, shades etc.), or other traffic participants, are known
potential TCs (Zhang et al. 2023). Identifying and documenting TCs can be very
challenging due to the large parameter space ADSs are exposed to (Qidong et al. 2022).

TCs are defined in the ISO 21448 standard as a specific condition of a scenario, which
can initiate a system reaction of Level 0 to Level 5 systems contributing to:

a. a hazardous behaviour
b. or an inability to prevent or detect and mitigate a reasonably foreseeable indirect
misuse.

An example of indirect misuse could be a driver that must monitor the DDT, but is
sleeping, reading the newspaper or watching a movie on a tablet, and is therefore not
using a Level 2 system as intended. This thesis does not cover misuse within the proposed
scenario generation method.

ISO 21448 suggests different activities for identifying potential TCs for scenario
parametrisation. Amongst these activities, it is recommended to analyse accident data to
identify potential TCs. One of the databases suggested in ISO 21448 is the STATS 19
data, which records accident circumstances recorded by the British police leading to
personal road traffic injuries in Great Britain since 1979 (Department for Transport 2021).

27
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

ISO 21448, however, does not provide any method to extract such mixed-vehicle fleet
scenarios containing TCs from accident data. It only suggests the identification of TCs
from accident data as one possible activity for safe ADSs development. Jinkang et al.
(2022) observed that there is a potential for patterns to remain undiscovered due to
performance limitations of the application of CAs on accident data. Hence, the qualitative
framework of ISO 21448 needs to be converted into quantitative steps to identify hidden
patterns representing potential TCs in accident data. To realise these quantitative steps
using the STATS 19 data, the author of this thesis built a hypothesis based on two claims
introduced next.

First, the author claims that accident patterns that occurred under adverse environmental
conditions, which can be used for ADSs testing in mixed-vehicle fleets, are hidden in
accident data. Across different engineering fields, clustering algorithms (CAs) have been
used frequently to identify patterns in large datasets (Todorovic and Simic 2019). CAs
are unsupervised machine learning algorithms that identify patterns in datasets and group
similar data points into clusters (Kaufman and Rousseeuw 1990, Al-Jabery et al. 2020).In
the opinion of this thesis's author, hidden patterns can be extracted by applying CAs on
accident data in combination with a systematic data preparation technique. It is claimed
that these hidden patterns can be used to specify hazardous scenarios as required in ISO
21448 for testing ADSs.

The challenge of using police recorded accident data, such as STATS 19, is that this
information is collected on the macro level, i.e., describing environmental conditions and
static information of the accident circumstances. Micro level information, which is
important to test the scenarios extracted from accident data in a virtual environment, such
as the vehicle kinematics, is missing. Extending STATS 19 scenarios with random vehicle
kinematics, or evaluating the full factorial parameter space of possible vehicle kinematics,
would result in millions and billions of tests (Amersbach and Winner 2019). Such a large
number of tests will also cover unrealistic vehicle dynamics combinations, making the
search for realistic scenarios very expensive.

The second claim made by the author of this thesis is that optimisation algorithms based
on metaheuristics can be used to explore the scenario design space systematically to
generate probabilistic, but theoretical vehicle dynamics parameters for a more efficient

28
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

extension of the STATS 19 scenarios. Metaheuristic methods solve different optimisation

problems and provide a set of solutions near the global optimum (Chopard and Tomassini
2018).

This thesis proposes a scenario generation methodology based on the STATS 19 data,
which has been divided into two studies to create a scenario library consisting of scenarios
for mixed-vehicle fleet testing.

The first study introduces a 2-layer application of the k-medoids clustering algorithm
aiming to identify hidden patterns that occurred under adverse environmental conditions
(snow, fog, et al.) in the STATS 19 data. The k-medoids is an unsupervised machine
learning algorithm based on the partitioning around medoids (PAM) algorithm, which
can partition the STATS 19 data into k clusters, respectively patterns (Jin and Han 2017).

The second study reports extending the STATS 19 data by exploring the scenario design
space. It introduces the application of a metaheuristics based genetic algorithm, namely
the Non-dominated Sorting Genetic Algorithm – II (NSGA-II) (Deb et al. 2002). GAs are
optimisation algorithms which solve single-objective and multi-objective problems,
inspired by Darwin’s theory: the survival of the fittest (Goldberg 1989).

1.3 Dissertation outline

The dissertation is organised as follows:

Chapter 1 – Introduction: The introduction chapter introduces the problem of identifying

hidden patterns in accident data or rare event scenario parametrisation to test ADSs.
Moreover, it reports a hypothesis, defined by the author of this thesis, to solve the
challenge of identifying hidden patterns in accident data using clustering and optimisation
algorithms.

Chapter 2 – Literature review: Chapter 2 of this thesis introduces the importance of

considering adverse environmental conditions for scenario parametrisation and the
impact of adverse environmental conditions on the safety of ADSs. Moreover, in chapter
2, methods for scenario generation are introduced, and their limitations are highlighted.
From these limitations, three research questions are derived.

29
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Chapter 3 – Methodology: Chapter 3 proposes a scenario generation methodology based

on the k-medoids and NSGA-II algorithms using the STATS 19 accident data. The
methodology is divided into two studies to answer research questions 1, 2, and 3.

Chapter 4 – Results and discussions: In chapter 4, the results for the application of the k-
medoids and NSGA-II algorithms on the STATS 19 are presented and discussed.
Moreover, a statement regarding the research questions defined in chapter 2 is provided.

Chapter 5 – Conclusion and future work: Chapter 5 concludes this thesis's outcome and
highlights the proposed work's main contributions. Moreover, it suggests future work to
enhance this thesis's proposed scenario generation methodology.

30
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Chapter 2. Literature Review

2.1 Introduction

This chapter aims to review best practices for scenario generation in the field of ADSs
testing. Moreover, it aims to identify gaps in existing methods for identifying TCs from
accident data. These gaps will contribute to the design of this thesis's proposed scenario
generation method. This chapter is organised as follows:

Section 2.2 raises the impact of adverse environmental conditions on sensors realising
ADS technology. Furthermore, section 2.2 aims to show the relation between TCs and
how these can initiate human harm. Showing this relation is essential to understand the
impact of TCs on ADS safety. Section 2.2 also discusses adverse environmental
conditions representing potential TCs impacting ADSs safety and negatively impacting
human driver performance. Knowing parameters that negatively impact ADSs and human
drivers is essential to parametrise critical mixed-vehicle fleet scenarios. Finally, section
2.3 presents a method for documenting scenarios and potential TCs.

Section 2.3 introduces scenario generation methods for testing Level 0 to Level 5 systems.
To give a more precise overview of existing literature, section 2.3 has been divided into
two sub-sections. Sub-section 2.3.1 discusses different data sources available from which
scenarios have been extracted in the past. Since this thesis focuses on using accident data,
sub-section 2.3.1 has covered different accident data types and their advantages and
disadvantages for scenario generation in more detail.

Sub-section 2.3.2 covers existing algorithms that can be potentially used for scenario
generation. More precisely, the first algorithms discussed serve to construct a scenario
search space utilising accident data. This scenario search space can also be interpreted as
functional and logical scenarios. Second, algorithms able to explore scenario search
spaces for identifying critical concrete scenarios are reviewed. The review in sub-section
2.3.2 also highlights the current limitations of these algorithms.

Finally, section 2.4 summarises the literature review chapter to draw an overall
conclusion about the existing methods and data for scenario generation.

31
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

2.2 Sensors and their impact on ADSs safety

Sensors are an important part of ADS technology and provide essential information to
conduct the DDT. Figure 2-1, retrieved from (Wendt and Cook 2018), illustrates an
example of available sensors, their positioning, coverage and functions they realise.

The functionality of the sensors, depicted in Figure 2-1, is discussed in more detail by
Yeong et al. (2021) and Ignatious and Khan (2022).

This item has been removed due to 3rd Party Copyright. The unabridged version of the thesis can be found in the
Lanchester Library, Coventry University.

Figure 2-1: Placement of different sensors for environment perception (Wendt and Cook 2018).

Sensor data deliver essential information for the perception algorithms. The perception
activity can be divided into two categories: 1) semantics and 2) localisation (Li, Moreau,
and Ibanez-Guzman 2022, Petrovai, Oniga, and Varga 2022). Semantics refers to
perceiving the vehicle environment and clustering objects by their semantic meaning
(vehicles, buses, road signs, Pedestrians, et al.). Localisation refers to the vehicle's ability
to compute its position within the environment accurately.

Sensor data used for the perception can be very sensitive to different TCs and are, hence,
the most crucial part of ADSs technology (Sharath and Mehran 2021, Zhang et al. 2023).
Before delving into the actual characteristics indicating potential TCs impacting ADSs
sensors, it is critical to understand the safety impact of TCs on ADSs and the reaction

32
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

chain they can trigger in a vehicle equipped with Level 0 to Level 5 driving automation
systems.

Figure 2-2 shows the concept of TCs presented by ISO 21448 standard. Triggering
conditions can originate from the ODD and represent one specific or multiple condition(s)
of a scenario (e.g. weather or lighting conditions). The safety performance of the sense,
plan and act chain of an ADS can be negatively impacted by TCs. More precisely, as seen
in Figure 2-2, TCs can initiate functional insufficiencies (FISs) of a component of the
sense plan act chain and lead to output insufficiencies (OIs) on the system level
(functional view). Such an OI is interpreted in this thesis for simplicity as a FIS on the
system level with the potential to initiate a hazardous behaviour of the vehicle. If this
hazardous behaviour has a potential for human harm, it needs to be addressed during the
development. This shows the importance of identifying TCs to enhance the safety
performance of ADSs. Note that the TCs can include misuse, which is not covered in this
thesis.

According to ISO 21448, FISs can be classified into two categories (International
Standardisation Organisation 2022):

• Performance insufficiency: Technical performance limitations of hardware and

software leading to hazardous behaviour when triggered by one or more TCs.
• Insufficiency of specification: This can be an incomplete specification leading to
hazardous behaviour at the vehicle level when triggered by one or more TCs.
This item has been removed due to 3rd Party Copyright. The unabridged version of the thesis can be found in the
Lanchester Library, Coventry University.

Figure 2-2: Link between TCs, FISs and hazardous behaviour at vehicle level (International
Standardisation Organisation 2022).

33
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Accident data is mentioned in ISO 21448 for the identification of potential TCs.
Identifying TCs from accident data influencing both the human driver and ADSs is of
significant importance for parametrising mixed-vehicle fleet scenarios. Accident data
allows forecasting of unexpected situations in which ADSs can potentially be exposed in
the future in mixed-vehicle fleet scenarios (Szénási 2021).

Environmental conditions impacting sensor safety are discussed next to understand which
parameters within the environment can be classified as potential TCs for ADSs. Note that
this thesis will not provide an exhaustive review of existing TCs but will instead guide
the reader in understanding the overall impact of TCs and why their identification is
necessary. Adverse weather conditions have been identified in the past as potential TCs
leading to sensor performance limitations:

Snowflakes have been identified to be challenging for camera sensors, as they can
negatively impact the intensity of a recorded image. Snowflakes raise the intensity of an
image to the point where different patterns can no longer be distinguished. As a result,
the accuracy of the image recognition algorithm decreases drastically, which could result
in the hazardous behaviour of an ADS on a higher level. (Kurihata et al. 2005, Zang et al.
2019). The same effect has been observed for raindrops. Single raindrops on a camera
lens reduce image quality significantly (Yoneda et al. 2019).

Moreover, snow on the roads can potentially be projected onto the camera lenses and
prevent accurate detection of objects or cause physical damage to the lens. Rain also has
an impact on the radar sensors' performance. More precisely, the back-scatter effect,
which is caused by raindrops, leads to noise and thus has a negative impact on object
detection (Vargas et al. 2021).

In combination with low temperatures, snow and rain can lead to unfavourable road
surface conditions, which can negatively impact the performance of ADSs actuators, such
as the braking system. Due to the increased braking distance required under unfavourable
road surface conditions, the requirements for actuators are challenging to maintain safety
(Abdi et al. 2018). As a result, snow and rain can negatively affect ADSs and, thus,
represent potential TCs that can initiate FISs leading to OIs.

Foggy conditions have been classified as problematic for LiDAR and camera sensors, as
they reduce detection range, limiting the performance of sensors (Zhan and Northrop

34
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

2021, Ronen, Agassi, and Yaron 2021). However, according to Yoneda et al. (2019),
ADSs rarely encounter foggy scenarios.

Different lighting conditions significantly impact cameras and LiDAR sensors (Zhang et
al. 2021). For instance, a strong light source from any artificial source, or the sunlight,
can lead to camera blindness and cause so called ‘black spots’ in point clouds generated
by LiDAR sensors. Another issue related to lighting conditions is the sunlight reflection
on objects or wet road surfaces. In such cases, the camera performance is limited (Rosique
et al. 2019, Martí et al. 2019).

Snow, fog, rain and different lighting conditions not only reduce the performance of
ADSs but also negatively impact the performance of human drivers (Druta et al. 2020).
Hence, it is essential to model adverse environmental conditions into scenarios for mixed-
fleet testing of ADSs.

Creating such models can be challenging, as the parameter space is multi-dimensional for
scenario description. Hence, the PEGASUS project (Project for the Establishment of
Generally Accepted quality criteria, tools and methods, as well as Scenarios and
Situations framework) has provided a method for the systematic parameterisation of
scenarios, which can also be applied to document scenarios containing TCs. PEGASUS
addresses the dilemma of shifting from distance-based testing to scenario-based testing.
The entire PEGASUS method for assessing driving automation systems can be found in
APPENDIX A.

Eberle, Wachenfeld, and Schittenhelm (2019) introduced a systematic six-layer model to

ease scenarios' parametrisation for Level 0 to Level 5 systems testing (see Figure 2-3).

35
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

This item has been removed due to 3rd Party Copyright. The unabridged version of the thesis can be found in the
Lanchester Library, Coventry University.

Figure 2-3: Systematic 6-layer model for scenario parametrisation accordingly to the PEGASUS method
(Eberle, Wachenfeld, and Schittenhelm 2019).

Moreover, PEGASUS not only provided the six-layer method but also introduced a
method to represent scenario parameters in three different levels of abstractions:
functional, logical, and concrete scenarios (Menzel, Bagschik, and Maurer 2018) (see
Figure 2-4).

Functional scenarios are expressed in human-readable form (mostly requirements

expressed in sentences) to delimit the parameter space in an initial state roughly. Logical
scenarios represent the state space values of these functional scenarios. For instance, a
functional scenario, as depicted in Figure 2-4, can include the information of a scenario
taking place on a bending road. The logical scenario, on the other hand, gives a parameter
space for the radius of this bend. The functional and logical scenarios can be used to
construct a scenario search space systematically.

A concrete scenario represents a single combination of continuous data points within this
multidimensional scenario search space. Note that simulation tools require concrete
scenarios as input. When moving from functional to concrete scenarios, the level of
abstraction decreases, but the number of scenarios increases. Figure 2-4 gives an

36
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

overview with an example of a traffic jam scenario depicting the three levels of
abstractions and how they interact.
This item has been removed due to 3rd Party Copyright. The unabridged version of the thesis can be found in the
Lanchester Library, Coventry University.

Figure 2-4: Workflow from functional to logical and logical to concrete scenario (Menzel,
Bagschik, and Maurer 2018).

37
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

2.3 Scenario generation data and methods for Level 0 to Level 5 systems

To meet ISO 21448, the hazardous scenarios must have a potential for human harm to
serve as valid test scenarios for the safety argumentation of ADSs. The term ‘critical
scenarios’ is assumed, in this thesis, to be widely used in the industrial and research
sectors and to refer to hazardous scenarios for the safety assessment of ADSs. An in-
depth systematic review has concluded that critical scenario generation can be divided
into two activities: (i) critical scenario space construction and (ii) scenario space
exploration for the identification of critical scenarios (Zhang et al. 2021).

In the PEGASUS scenario generation framework context, the critical scenario space
construction can be referred to as the development of functional and logical scenarios.
Scenario space exploration can be referred to as identifying concrete scenarios in the
logical scenario space.

Furthermore, Zhang et al. (2021) observed that activities (i) and (ii) are, in most cases,
accompanied by a data source. As a result, the following section will first cover data
sources for critical scenario development, then methodologies for scenario space
construction, and finally, scenario space exploration.

2.3.1 Data sources for critical scenario generation

Traditionally, two different data strategies are used as a foundation for generating
functional, logical, and concrete scenarios (Riedmaier et al. 2020). Figure 2-5 shows that
the first strategy is based on a knowledge-based approach, whereas the second is based
on a data-driven approach.

38
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Figure 2-5: Knowledge and data-driven scenario generation workflow adopted from
Riedmaier et al. (2020).

The knowledge-based approach mainly considers data generated by expert knowledge.

However, knowledge of regulations, standards and development guidelines can also be
considered. In ADSs development, expert knowledge has been stored frequently in
ontologies (Bagschik, Menzel, and Maurer 2018, Gelder et al. 2021). Ontologies define
concepts to share and store information based on specific domain know-how
(Chandrasekaran and Josephson 1999).

In a knowledge-based scenario generation approach, field experts specify tests based on

experiences made in the past, in projects with similar scope and/or on their expert
knowledge. However, each expert has a unique background and understanding of
criticality, which is a limitation of the knowledge-based approach (Khastgir et al. 2018).
More precisely, the information which stems from expert knowledge is biased. Hence,
there is a probability that some areas of the critical scenario space are unintentionally not
considered due to a lack of knowledge. Riedmaier et al. (2020) claim that data-driven
scenario generation strategies achieve a higher coverage when constructing a critical
scenario space. However, the author of this thesis suggests using both methods to increase
scenario coverage. This thesis is solely concerned with an accident data-driven strategy.
As a result, various data sources are examined next to determine which are most suited to
represent mixed-vehicle fleet scenario development for ADSs testing.

39
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Figure 2-6 lists the most common data sources for data-driven critical scenario generation
used in the past: historical accident data, naturalistic driving data (NDD), and data
collected from internet of things (Abdulhafedh 2017, Chand, Jayesh, and Bhasi 2021).

Figure 2-6: Summary of different existing accident data sources in literature applied for scenario
generation.

Historical accident data stores road accidents that occurred in the past. It was introduced
to understand factors negatively impacting road safety. Understanding these factors
allows for proposing counter measures to enhance future road accident statistics (Ifthikar
and Hettiarachchi 2018, Szénási 2021). Moreover, in different safety standards and
official safety reports, historical accident data has been suggested for the specification of
critical scenarios for ADSs testing (Mercedes-Benz Group 2019, International
Organization for Standardization 2020, Naujoks 2021, International Standardisation
Organisation 2022).

Historical accident data can be collected on two different levels: macro level and micro
level. Macroscopic accident data serves to give a broad picture of accident circumstances
in a large geographical region (Junietz 2019). It seeks to collect as many accidents as
possible to create statistics on fatality rates in relation to other environmental
circumstances, such as weather, road type, lighting conditions, et al.

On the other hand, data entries in microscopic accident data describe a specific accident
scene (Yannis et al. 2020). Although microscopic data contains information, such as the

40
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

exact vehicle dynamics or meteorological conditions at the moment of the accident, it is

challenging to use for summarising accident patterns for a whole country or a wide
geographic area.

The collection of microscopic data is conducted by accident investigation teams visiting

the accident spot. This makes the collection of microscopic data expensive in resources
and time. Hence, the microscopic data can only be collected for smaller geographical
areas of a country. Macroscopic data, on the other hand, can be collected by different
police departments or other official bodies by filling out reports and storing them in a
collaborative database (Abdulhafedh 2017). Consequently, macroscopic data can be
collected for larger geographical areas but with fewer details regarding the specific
accident scenes.

Macroscopic data, for the majority, describes accident records on a categorical level,
whereas microscopic accident data provides concrete values in the form of continuous
data for the said collected accidents. For instance, in macroscopic data, the road friction
can be coded as wet or dry. In contrast, the exact road friction value of the accident spot
is provided in microscopic data.

Macroscopic data has the advantage that it can be used to create an initial scenario design
space using categorical data. This categorical data can then be translated into functional
scenarios. However, one must remember that assumptions based on other statistics must
be made to build logical and concrete scenarios, bringing uncertainty into the scenarios.

Microscopic data is more helpful for analysing what happened during an accident to the
different actors and objects. For instance, it can be seen which body segment of the driver
has been injured or which part of the vehicle's body has been damaged. To grasp such
linkages, microscopic data can be highly valuable in designing safer cars. However,
microscopic data is smaller than macroscopic data; deriving a broader picture of accident
patterns is more complicated. Both macroscopic and microscopic accident data have
accuracy limitations to a certain extent because they are collected and reported by
humans, which can lead to unintended reporting mistakes.

41
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Examples of macroscopic accident data are:

STATS 19: The STATS 19 accident database provides accident circumstances that led to
personal road traffic injuries in Great Britain since 1979. Accidents are recorded by the
police using STATS 19 forms, which are then stored in a national database and made
available online for public use (Department for Transport 2021).

FARS: The Fatality Analysis Reporting System records vehicle-to-vehicle and vehicle-
to-non-vehicle accidents in which a fatality occurred within 30 days of the collision. It
has recorded data since 1975 for the 50 states of the United States of America and is
available online for public use, managed by the NHTSA (NHTSA-FARS 2023).

IRTAD road safety database: The International Road Traffic and Accident Database
(ITRAD) of the International Transport Forum (ITF/OECD) was created in 1988. It stores
validated accident data for 32 countries and aims to serve as an international comparison
of road safety and the specification of more effective road safety policies. It is accessible
online, and most accident circumstances can be looked up in their annual reports (OECD
n.d.).

Examples of Microscopic accident data:

RAIDS: The Road Accident In-Depth Studies (RAIDS) data has been collected by the
Department for Transport of the United Kingdom since 2012. Unlike the STATS 19
database, which determines collision responsibility, the RAIDS data serves the in-depth
investigation of injury causation. RAIDS contains data from two different investigations:
the crash scene investigation and the backwards-looking investigation. The crash scene
investigation includes the collection of non-injury and injury accidents onsite. On the
other hand, the backwards-looking strategy collects data from vehicles with serious
damage recovered from a crash scene and the occupants brought to the hospital due to
their injuries (Department for Transport 2013).

GIDAS: The German In-Depth Accident Study (GIDAS) database represents accidents
collected by an accident investigation team on the spot-on German roads. It has recorded
accidents since 1999 and serves accident research scientists to identify road traffic issues.
It has been initiated by the Bundesanstalt für Straßenwesen (BASt) and is collected by
the accident investigation team of the Technical University of Dresden and the

42
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Medizinische Hochschule Hannover (MHH) (GIDAS 2023). The data source aims to
enhance regulations and vehicle systems so that road traffic accidents can be statistically
reduced.

Autonomous vehicles collision reports: The state of California Department of Motor

Vehicles (DMV) has collected, since 2014, accident reports from manufacturers testing
their AVs in public. They need to report crashes with property damage and slight, serious
and fatal crashes. Even though this data is solely collected in the state of California, this
information can still be categorised as macroscopic due to the vast extent of the state's
surface area. The DMV issued accident report sheets to AV testing companies for them
to fill in the paperwork within ten days of an incident occurring. These forms are openly
available on the DMV website (DMV 2023). Although it is macroscopic data, only a few
accidents are stored in the database (May 12th, 2022, = 463 accidents).

Naturalistic driving data can be divided into two classes: naturalistic driving studies
(NDS) and naturalistic field operational tests (NFOTs) (Bärgman 2015). Van Schagen
and Sagberg (2011) defined NDS as: “a study undertaken to provide insight into driver
behaviour during everyday trips by recording details of the driver, the vehicle and the
surroundings through unobtrusive data gathering equipment and without experimental
control”. Field operational tests have been defined as: “a study undertaken to evaluate a
function, or functions, under normal operating conditions in environments typically
encountered by the host vehicle(s) using quasi-experimental methods” (Barnard and
Carsten n.d.). These definitions clearly show the difference between NDS and NFOTs.
NDS focus on recording the behaviour of all the road traffic participants, including
environmental conditions. In contrast, NFOTs focus on recording ADSs' behaviour in
specific driving situations and the respective environmental conditions. Data gathered
from both methods are overly complex. NDD is more suitable for extracting normal
driving statics rather than the critical behaviour of other road users with low exposure in
NDD. It has been reported that inaccurate data processing methods to detect critical
scenarios from NDD lead to false positive classifications (Hjälmdahl 2009, Cabrall et al.
2018).

Furthermore, it is unclear how many critical scenarios are missed due to inaccurate
classification models (Hjälmdahl 2009). Another challenge when using NFOTs for

43
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

automated vehicles is to assess what ADS of the multiple ADSs fitted into the vehicle
lead to a critical scenario (Bärgman 2015). It has also been reported that identifying
critical scenarios in NDS is complicated, as critical scenarios are rare (Leitner 2019).
Moreover, identifying scenario parameters covering the ODD sufficiently is challenging
(Roesener et al. 2017). NDS have been conducted to overcome such challenges using
simulation environments with human drivers in the loop. Although this technique allows
to scale up the number of vehicle fleets and drivers by decreasing costs, it has been
observed that drivers act differently compared to the real-world environment (Ekanayake
et al. 2013). This leads to inaccurate data collection and hence, a negative impact on the
quality of NDS collected in simulation environments with a human driver in the loop. It
can be concluded that it is challenging to extract critical scenarios from NDD. However,
NDD could be used to understand traffic patterns and normal driving behaviours for the
area in which it has been collected.

Examples of naturalistic driving data are:

SHRP2 NDS: The SHRP2 NDS database has been introduced to increase highway safety
and reduce traffic congestion. More than 3,400 drivers have recorded more than
5,400,000 trips in the US (Virginia Tech Transportation Institute 2020).

UDRIVE: UDRIVE is the first large-scale NDS conducted in Europe. It aims to record
naturalistic driving behaviour in cars, trucks and motorbikes. The evaluation of the
UDRIVE data served to compare everyday driving behaviour across different European
regions. Moreover, it aimed to detect and analyse near collisions and collisions. Finally,
the UDRIVE data also intends to analyse the influence of driving behaviour on emissions
and fuel consumption (European Commission 2017).

Internet of things (IoT) allows communication between data sources and computer
devices. In case of accidents, information regarding the circumstances can be shared on
digital maps, social media, or other sources on the internet. For instance, Google maps or
the social media platform Twitter are common media where accident events are posted.
Accident data collected from IoT have been reported to be unreliable in extracting
accident scenarios. More precisely, these are also unreliable, biased, and often difficult to
interpret (Gutierrez-Osorio 2019). However, accident data collected from IoT has been

44
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

used successfully for traffic flow optimisation in smart cities. Consequently, IoT is not a
reliable source to systematically generate mixed-vehicle fleet scenarios for ADSs testing.

It can be concluded that NDD and accident data collected from IoT are unreliable sources
for critical scenario generation – especially for mixed-vehicle fleet applications. More
precisely, NDD do not populate a high frequency of critical scenarios, but rather normal
driving scenarios. Historical accident data is more reliable for critical scenario generation.
It keeps track of numerous accidents that resulted in human injuries.

2.3.2 Algorithms for scenario space construction using accident data

This section investigates the different methods applied to accident data to construct a
scenario space.

Different algorithms have been applied, in the past, to different available accident data
sources (Gutierrez-Osorio 2019, Riedmaier et al. 2020). Natural Language Processing
(NLP), classification models and clustering algorithms are among these algorithms.

NLP algorithms are used to extract systematically target information from large natural
language databases. NLP can be divided into speech recognition, natural-language
generation and natural-language understanding (Collobert et al. 2011). The latter, natural-
language understanding, is the method which can be applied to natural-language accident
data to extract target accident information computationally.

Classification algorithms are supervised machine learning algorithms which use pre-
categorised training datasets to label input data. More precisely, classification algorithms
identify the likelihood that a particular observation corresponds to a specific class
(Aggarwal and Zhai 2012).

It can be observed that algorithms applied in the past for data-driven scenario generation
stem from unsupervised and supervised machine learning. Consequently, this section has
been divided into two sub-sections supervised machine learning for scenario space
construction and unsupervised machine learning for scenario space construction.

2.3.2.1 Supervised machine learning for scenario space construction

Applying NLP algorithms to construct scenario spaces on accident data is very
challenging (Lu et al. 2018, Dabiri and Heaslip 2019). Past studies have used NLP
algorithms on accident data derived from IoT, such as Twitter data. Social media data is

45
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

believed to be unreliable and a biased source for extracting injury relevant accident
scenarios (Gutierrez-Osorio 2019). Compared to historical accident data collected
systematically by the police or other institutions, Twitter data is random and inconsistent.
Therefore, it is challenging to label Twitter posts as accident relevant posts. In Gu, Qian,
and Chen (2016), it has been stated that only 5% of the used data has been reliable in
extracting critical scenarios. This underlines that deriving large pools of scenarios from
social media data using NLP algorithms to define a scenario space is unsuitable.

Classification algorithms have been used to classify data points in accident datasets as
critical or non-critical scenarios (Dozza and González 2013, Bakhit, Guo, and Ishak 2018,
Paredes et al. 2022). To be more precise, classification algorithms have been applied to
NDD to identify collision scenarios. The challenge for this activity is that first, a large
recording time of NDD is required to encounter critical scenarios, and second, only a
small number of concrete scenarios can be identified. Moreover, classification algorithms
require apriori knowledge of the problem, which makes it challenging to identify
previously unknown patterns in NDD or other accident data sources. The identified
scenarios will strongly depend on the classification models' accuracy (Hjälmdahl 2009),
making identifying SOTIF area 3 scenarios challenging as their patterns are unknown.

Based on these findings, it is possible to conclude that using supervised machine learning
to identify hidden patterns in accident data is difficult. However, supervised machine
learning algorithms can be applied when reliable data is available to design classification
models to identify known scenarios.

2.3.2.2 Unsupervised machine learning for scenario space construction

In contrast to supervised machine learning methods, CAs, which are part of the
unsupervised machine learning family, need no apriori information to generate scenarios
from accident data. Clustering algorithms have been used frequently to identify patterns
across different engineering fields (Saini, Kaur, and Lamba 2021). Also for scenario
generation for Level 0 to Level 5 systems testing they have been considered frequently
(Riedmaier et al. 2020, Nalic and Mihalj 2020, Zhang et al. 2021, Jinkang et al. 2022).
This section gives an overview of the different CAs used in the past on accident data for
scenario generation.

46
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

The following clustering methods amongst CAs have been used on accident data, and
their applications will be reviewed next:

• Density-based methods (Ester 2018): Density-based methods first plot the data

points on a map, then organise data points close to each other into clusters.

• Grid-based methods (Akodjènou-Jeannin, Salamatian, and Gallinari 2007):

Grid-based methods plot the data points on a map and then lay a grid over the

map. Data points lying within the same grid of the map are then assigned to the

same cluster.

• Hierarchical clustering (HC) method (Nielsen 2016): This technique generates a

dendrogram of a dataset that can be interpreted as a tree representing hierarchies.

Data points assigned to the same branch of this tree are assigned to the same

cluster.

• Partitioning around methods (PAM) (Xu and Tian 2015): Partitioning around

methods randomly partition the dataset into initial clusters by using similarity

metrics to iteratively assign similar data points into their corresponding clusters.

Density-based and grid-based methods have been applied to identify city accident
hotspots (Ganjali Khosrowshahi et al. 2021, Almjewail et al. 2018). Both methods
identify so-called hotspots for accidents. These are geographic areas where accidents are
highly likely to happen. According to Agrawal et al. (2018), providing this information
can reduce crash rates. However, the application of density-based and grid-based methods
is insufficient to identify scenarios containing adverse environmental conditions. It is
unknown whether the hot spot scenarios cover all accidents that occurred under adverse
environmental conditions.

A more promising strategy observed in the past is using HC, PAM and LCC algorithms
on historical accident data to extract high frequency accidents. The little available
research on applying these algorithms for scenario synthesis on historical accidents is
thoroughly evaluated next.

47
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Lenard, Badea-Romero, and Danton (2014) have applied HC on STATS 19 and on-the-
spot (OTS) microscopic databases, which is now part of the RAIDS database (see sub-
section 2.3.1). They aimed to derive pedestrian scenarios for automated emergency
braking (AEB) testing. In total, 22 clusters from the macroscopic accident data and 17
from the microscopic have been identified. The authors found that applying HC on both
datasets, focusing on pedestrian accidents, derived remarkably similar scenarios. Their
results indicate that the application of HC has shown some robustness for their datasets.
However, the general accident clusters identified by Lenard, Badea-Romero, and Danton
(2014) did not contain some of the TCs discussed in section 2.2 of this chapter. Moreover,
it has been observed that the HC algorithm has not identified high injury scenarios but
identified mainly slight injury scenarios. This might be due to the higher exposure of
slight injuries compared to serious injuries and fatalities in the database.

Nitsche et al. (2017) proposed an advanced hybrid method for clustering accident data
using a PAM algorithm (k-medoids) and association mining rules. The method proposed
in Nitsche et al. (2017) was also a 2 layer application of two different algorithms. The
RAIDS database has been used to identify functional scenarios representing intersection
accidents for ADSs testing. In total, six 4-legged intersection clusters (population size:
368) and thirteen 3-legged intersections (T-intersection) clusters (population size: 930)
were derived. Association mining rules were applied to the clusters to identify additional
parameters leading to intersection accidents. Although diverse clusters were identified in
their study, which are useful for testing ADSs under general accident circumstances, no
cluster containing adverse environmental conditions was identified linked to high
injuries. In total, none of the derived scenarios represented high injury scenarios. Nitsche
et al. (2017) assumed that the low diversity in the scenarios was due to the simultaneous
analysis of several use cases in one cluster analysis. For instance, the dataset comprised
vehicle-to-vehicle intersection use cases and vehicle-to-cycle/pedestrian. The accident
pattern for each use case differs. Moreover, each pattern has different parameters
correlating to an accident's injury outcome. Accident patterns for each use case change,
especially the circumstances leading to injuries.

Sander and Lubbe (2018) benchmarked different clustering methods to assess their
performance for intersection scenario generation for AEB testing. To assess the

48
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

performance, they investigated whether it was possible to reduce the diversity of

intersection accidents into a set of test scenarios using the GIDAS database. The
benchmarked algorithms were PAM, HC, and latent class clustering (LCC). It was
concluded that applying the benchmarked algorithms was unsuitable for narrowing down
the scenario space into a set of representative clusters. They found that the results for each
benchmark algorithm varied strongly. For instance, each algorithm generated clusters
expressing a different number of clusters and different accident patterns. According to
Sander and Lubbe (2018), applying CAs for scenario generation was challenging, and
obtaining robust results was difficult; however, not impossible.

Esenturk et al. (2022) claimed that the entropy-based COOLCAT clustering algorithm
was more suitable for scenario generation when using the STATS 19 data than PAM and
HC algorithms. They clustered the STATS 19 accident database from 2016 – 2018 using
the COOLCAT algorithm to identify functional scenarios for testing ADSs. A total of six
clusters were derived. One cluster containing potential triggering conditions, such as fog,
was identified within these clusters. Another cluster represented an accident pattern
describing fatal accident circumstances. These were not discovered in past studies.
However, six clusters seem low for representing accident patterns for various use cases
nationwide. No initial filter was applied to the accident data to separate, for instance,
intersection accidents from highway accidents. The author of this thesis is convinced that,
for analysing accident data using clustering algorithms, the data should be prepared first
only to populate accidents of a specific use case rather than mixing all accidents in one
analysis.

2.3.2.3 Critical evaluation of supervised and unsupervised machine learning for

accident data processing
Supervised machine learning is problem-dependent, making identifying previously
unknown and hidden patterns from accident data challenging. Supervised machine
learning is more beneficial for the identification of known accident patterns.
Unsupervised machine learning is more applicable for identifying unknown accident
patterns because they do not need any apriori information. However, some limitations for
identifying unknown patterns using unsupervised machine learning were revealed in this
research.

49
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Among the unsupervised learning methods, density-based and grid-based methods were
identified as helpful in identifying patterns in accident hotspots. However, these methods
do not identify low frequency scenarios from accident data that occurred under adverse
environmental conditions. Thus, these methods do not allow a high coverage of the
scenario design space.

Past studies have successfully applied PAM and HC algorithms to accident data for
scenario generation (see sub-section 2.3.2.2). However, the literature review of this thesis
has observed that the application of CAs on accident data focussed on deriving general
accident patterns rather than on accident patterns representing adverse environmental
conditions. The general accident patterns derived in previous studies were more likely to
represent accident patterns that occurred under favourable environmental conditions.
Moreover, it has been observed that the derived accident patterns were more likely to
represent slight injury accidents rather than serious injury accidents or even fatalities.
Although general accident patterns are essential for test scenario development, they
cannot ensure optimal coverage of the scenario design space. Ideally, an accident data-
driven scenario generation method for ADSs testing will provide general and rare event
accident patterns that occurred under adverse environmental conditions to achieve better
coverage in the scenario design space.

Applying the COOLCAT algorithm has shown the best results in identifying potential
TCs from the STATS 19 data. Moreover, the COOLCAT algorithm also did find
scenarios with diverse injury outcomes. However, insufficient scientific evidence exists
to state which CAs (COOLCAT, PAM or HC) reviewed in this thesis perform best for
clustering accident data. Jinkang et al. (2022) confirmed that it is likely that accident data
clusters derived from the application of CAs hide some essential patterns, which need
further analysis to be revealed.

The author of this thesis believes that the core challenge in identifying hidden patterns
containing adverse environmental conditions is not in choosing the appropriate clustering
algorithm but in preparing the accident data. Past studies did not elaborate on the data
preparation steps conducted on accident data. However, the author of this thesis strongly
believes that data preparation can significantly enhance identifying hidden patterns in
accident data. For instance, by applying the COOLCAT algorithm on STATS 19,

50
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Esenturk et al. (2022) derived six patterns to describe accidents occurring within different
junction types (e.g., intersections and roundabouts). Accident patterns for roundabout
accidents and intersection accidents, however, are different. Focussing only on one
junction type could reduce the data complexity and potentially reveal more hidden
patterns. Nitsche et al. (2017) confirmed this observation. They assumed that, due to
mixing attributes, such as accident type, junction type and injury outcome, they had not
revealed higher injury patterns and patterns that occurred under adverse environmental
conditions. Focussing only on one group or type of accident is suggested so that the
complexity of the dataset, regardless of its size, can be managed by CAs. In industry,
when test scenarios are generated for ADSs safety assessment, they argue a specific
function for a specific ODD. For example, if an ADS is solely meant to be used on
highways, only highway testing scenarios will be accepted as evidence for safety
arguments. This shows the importance of splitting the dataset during the data preparation
so as not to obtain scenarios occurring on other road types or junctions that do not include
the ODD.

Moreover, it has been observed that past studies have analysed datasets with mixed injury
outcomes. Accident patterns for different injury outcomes can be vastly different. Hence,
splitting the datasets into different injury classes to reduce the complexity of accident data
and obtain satisfying clustering results is strongly recommended in this thesis.

In summary for this section, it has been observed that CAs have more focussed on general
accident patterns. Hence, this thesis investigates whether CAs also reveal hidden accident
patterns from accident data. The following research question has been derived:

RQ (1): Can clustering algorithms identify rare event accidents in accident data for
the specification of mixed-vehicle fleet scenarios containing potential TCs?

2.3.3 Algorithms for scenario space exploration

This section covers search-based methods for the exploration of critical scenario design
spaces. Batsch et al. (2020) classified search-based methods for scenario generation for
Level 0 to Level 5 systems testing into three categories: (i) random search, (ii) guided

51
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

random search and (iii) optimisation algorithms. The same observation has been made by
Nalic and Mihalj (2020), Riedmaier et al. (2020) and Zhang et al. (2021). Hence, this
section focuses on reviewing and discussing these three categories.

2.3.3.1 Random search and guided random search algorithms for scenario generation

The Monte Carlo (MC) simulation method is a well-known technique in random search.
It generates random data points in predefined multidimensional design spaces (Dupree
and Fraley 2002). In the context of Level 0 to Level 5 systems, testing the MC simulation
application allows for generating random parameters. These can be used to parametrise
concrete scenarios and conduct simulations. A more efficient random search algorithm
compared to MC is the Latin Hyper Cube Sampling (LHS) algorithm. LHS was
introduced by McKay, Beckman, and Conover (1979). Rather than randomly sampling
points in the design space, it divides the design space into intervals and then generates
random data points within these intervals. Splitting the design space into intervals allows
the LHS to cover the entire design space more efficiently than the MC method.

Nitsche (2018) used the LHS method to generate critical concrete scenarios for
intersection testing in mixed-vehicle fleet environments. From Nitsche (2018), it can be
observed that the application of MC and LHS can quickly generate many concrete
scenarios. However, it was also noticed that concrete scenarios generated by LHS are
naïve and need to be validated using time-consuming simulations. Scenarios generated
by random search methods, such as LHS and MC, need these expensive simulation
validations because they are not guided by apriori information (Batsch et al. 2020). More
precisely, LHS and MC simulation assumed that the distribution of the parameters and
the covariance of the parameters are independent. However, in reality, the parameters
describing a critical scenario space are dependent and represent specific correlations and
patterns (Raffaelli et al. 2016). The MC method is mainly used as a benchmark to
highlight the efficiency of other algorithms due to its pure randomness.

It can be observed that the random search needs some guidance in the design space to
overcome its limitations due to pure randomness. A priori information regarding the
problem is required to guide the random search.

52
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Importance sampling (IS) is a guided random search technique that has been applied
frequently to generate concrete scenarios (Gietelink, De Schutter, and Verhaegen 2006,
De Gelder and Paardekooper 2017, Huang, Lam, and Zhao 2018, Zhang et al. 2018).
Compared to MC and LHS algorithm, it requires apriori information regarding the
parameter distribution (Chopin and Papaspiliopoulos 2020). Hence, it does not generate
uniformly distributed parameters like MC and LHS but random parameters representing
the distribution of the concrete scenario generation problem. Models describing the
distribution of parameters representing critical scenarios can be very complex and cannot
be derived accurately (Riedmaier et al. 2021). Therefore, when applying IS, the scenarios
need to be validated in a simulation environment to assess whether they are critical or not.

Furthermore, when IS is used, and the output equation is unknown, real-world data must
be acquired to create an output distribution which correlates to the real world (Beck and
Zuev 2016). This makes applying IS highly challenging for such an extensive search task.
Nonetheless, evidence has shown that IS requires fewer iterations than MC simulation to
uncover the same number of crucial scenarios (Zhang et al. 2018, Xia et al. 2021). Hence,
IS can potentially reduce V&V efforts compared to random search.

Markov Chain Monte Carlo (MCMC) simulations are more advanced compared to IS
when it comes to guiding random searches. MCMC simulations do not only output the
required distribution of the problem to solve but also imitate the covariance of the
parameters (van Ravenzwaaij, Cassey, and Brown 2018).

Zhang et al. (2018) compared MCMC simulations and IS for concrete scenario
generation. The stochastic input model for the MCMC and IS came from the Safety Pilot
Model Deployment Program (SPMD) dataset. Only cut-in cases were considered to build
the stochastical model. Their study demonstrated no significant performance deviation
when comparing MCMC to IS for scenario generation using the SPMD dataset but
covered the same critical scenario space when compared to each other.

Briefly, MC simulation and the LHS methods required enormous validation efforts using
simulations. Since the scenario design space for a given use case can be huge, these
random search methods are inefficient. Guided search approaches, such as IS and MCMC
simulations, were employed to reduce validation efforts for ADSs compared to random
search. However, the challenge was that MCMC and IS methods require a priori

53
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

information to replicate the parameters' distribution and intra-relationship during the

guided search. This fact limited these methods only to already known scenarios. The
identification of previously unknown patterns is more challenging using guided random
search.

2.3.3.2 Optimisation algorithms for scenario generation

Optimisation algorithms are considered for safety critical systems when the search space
becomes too large, and many solutions need to be verified to identify global optimum or
near global optimum solutions (Corso et al. 2021). An optimisation problem can be
formulated as a single-objective or multi-objective-optimisation (MOO) problem (Bhatti
2000). In single-objective optimisation, the search task is to identify solutions for one
search criterion. The aim of a MOO search problem is to identify solutions meeting
multiple criteria at the same time.

Optimisation problems can be classified into three classes: (1): enumerative search, (2)
deterministic optimisation and (3) stochastics (Juan et al. 2015, Paredes et al. 2022).

Enumerative search analyses each solution in the full factorial space to identify the global
optimum. This time-consuming activity can be compared to the random and supported
random searches presented in sub-section 2.3.3.1. Moreover, in a random search, the
probability of missing good solutions during the search is high for large search spaces.

Deterministic optimisation algorithms aim to find solutions for a problem representing

the global optimum, guaranteeing that the solutions are the global best in reality
(Cavazzuti 2013). The reason deterministic algorithms can find the global best solution
to a problem is that the algorithms already know what the true solution is, owing to a
priori known input information. Hence, deterministic optimisation problems are problem-
dependent, and not each algorithm can be applied to solve each deterministic problem
(Peres and Castelli 2021). As a result, deterministic algorithms have problems solving
black-box problems. Note that a deterministic algorithm returns the same global best
solutions when the input to the algorithm remains the same, which is an essential
advantage for specific applications (Lin, Tsai, and Yu 2012, Framinan, Perez-Gonzalez,
and Fernandez-Viagas 2019).

Stochastic optimisation does not rely on a priori input data and is based on randomness.
As a result, the solutions given by stochastic optimisation techniques will differ for each

54
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

optimisation run. Moreover, due to their randomness, stochastic optimisation provides

solutions near the global optimum. Therefore, one of the drawbacks of using stochastic
optimisation is that a certain level of uncertainty and solutions also need to be validated
using real-world data of the search problem (Kvasov and Mukhametzhanov 2018).
Nevertheless, stochastic optimisation is a good option for problems where random search
becomes too time exhaustive and deterministic optimisation cannot be used due to the
nature of the optimisation problem (Peres and Castelli 2021).

It can be concluded that, compared to the deterministic approach, the stochastic approach
provides solutions that are only close to the exact global optimum. Nevertheless, for some
problems, approximated solutions are still valuable if they lead in real-world by trial and
error to exact solutions. It should be noted that some problems might take years to solve;
hence, trading time for approximate solutions is a better option in such circumstances.

Amersbach and Winner (2019) discussed that testing the full factorial design space would
result in millions and billions of test scenarios – especially in cases where the number of
parameters can become very exhaustive. Moreover, in many cases, the search problem
for identifying scenarios is unknown. In such cases, deterministic optimisation is
ineffective because no apriori information regarding the search problem is available.
Hence, the scenario generation problem can be classified as a stochastic problem (Juhnke,
Tichy, and Houdek 2021).

Stochastic algorithms can be heuristic-based and metaheuristic-based. Heuristics can only

be used for one specific problem, not a series of problems (Gomez 2019).

Metaheuristics, conversely, can solve many problems and are problem independent. They
can be used efficiently without significant changes to the algorithms to solve large search
problems by providing estimations of the global optimum (Chopard and Tomassini 2018).

Abdel-Basset, Abdel-Fatah, and Sangaiah (2018) classified metaheuristics into two

categories: (i) metaphor-based metaheuristics and non-metaphor-based metaheuristics.
Metaphor-based metaheuristics are algorithms that simulate nature-inspired biological
processes. Metaphor-based algorithms are Genetic Algorithms (GA), Particle Swarm
Optimisation (PSO) (Yang 2021), Simulated Annealing (SA) (Guilmeau, Chouzenoux,
and Elvira 2021), et al.

55
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Non-metaphor-based metaheuristics do not use any simulation to define their search

strategy for near-global optimum solutions. Popular non-metaphor-based algorithms are
Scatter Search (Martí, Corberán, and Peiró 2016), Global Simplex Optimisation (GSO)
(Karimi and Siarry 2012), and Variable Neighbourhood Search (VSN) (Hansen and
Mladenović 2018) et al.

The GA has been utilised the most common across various engineering domains for the
selection of test cases to perform search-based software testing (SBST) (Afzal, Torkar,
and Feldt 2009, Ali et al. 2010, Silva, Senger de Souza, and Lopes de Souza 2017,
Coello Coello et al. 2020, Sarhani, Voß, and Jovanovic 2022). There are many different
GA reported in the past. Some are useful for solving single-objective optimisation
problems, whereas others are for MOO problems (Whitley and Sutton 2012, Katoch,
Chauhan, and Kumar 2021). However, there is insufficient evidence that GAs works best
and are suitable for ADSs test scenario generation. As a result, this thesis will investigate
whether GAs can be used to identify injury critical scenarios for ADSs testing.

For a successful design of any GA based on Darwin’s theory (the survival of the fittest),
the underlying basic steps in Figure 2-7 must be implemented (Bi and Hu 2021, Liu et al.
2019):

Figure 2-7: Workflow of Genetic Algorithms and their genetic operators adopted
from (Bi and Hu 2021, Liu et al. 2019).

56
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Figure 2-7 shows that the first step requires computing an initial set of solutions
represented by individuals, also known as a population. Traditionally, this initial
population is generated randomly using an MC computation in the parameter design
space. In the second step, the fitness of each individual, representing the initial population,
is evaluated. The fitness of an individual expresses the satisfaction of a solution in
percentage. The higher the fitness level, the better the solution satisfies the search
conditions.

In the third step, the genetic operators of the GA process the population:

• Selection: The best individuals are picked for the crossover to build a new and
more robust population set.
• Crossover: A new individual is generated while parents (solutions generated in
the last iteration) pass on their best characteristics to their child to achieve higher
fitness.
• Mutation: The mutation is used as a natural process to vary the inheritance
information randomly to generate fitter individuals representing a new set of
stronger populations. Since the mutation brings randomness in the generation
process, it potentially can also generate solutions in the population which are
significantly weaker. As a result, these solutions need to be omitted in the next
iteration during the selection activity.

Note that different implementations are available to compute the different genetic
operators (Yang 2014, Katoch, Chauhan, and Kumar 2021). After computing the genetic
operators, a new population is generated, which can be assessed by a fitness function(s).
Steps 2 and 3 are repeated iteratively until a termination criterion for the algorithm is
achieved. Different termination criteria are discussed in Ravber et al. (2022).

In past studies, when optimisation algorithms have been used for test scenario generation
in the context of ADSs testing, GAs have been the first choice. However, very few studies
have been conducted on the suitability of metaheuristics for test scenario development for
Level 0 to Level 5 system testing. Table 2-1 organises studies conducted in the past where
GAs have been used to generate critical test scenarios for assessing Level 0 to Level 5
systems. The table illustrates the purpose of each study, the optimisation objective(s), and
which GA was employed to address the problem. Each of the studies are discussed as
well in more detail.

57
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Table 2-1: Summary of past studies using optimisation for test scenario generation. The table shows
the aim, optimisation objectives, number of objectives and optimisation algorithm applied to solve
the optimisation problem.

Author(s) Aim Optimisation No. of Algorithm

objectives objectives

Abdessalem et al. Safety assessment Time-to-Collision, 3 NSGA-II

(2016) of pedestrian distance between and NN
detection system. vehicle and pedestrian,
distance of pedestrian
to critical area on the
road.

Klück et al. AEB test scenario Time-to-Collision 1 GA

(2019) generation

Felbinger et al. AEB test scenario Time-to-collision 1 GA

(2019) generation

Kaufmann et al. Scenario Distance from ego 1 GA

(2021) generation for vehicle to other actors
ADSs testing in
the following use
cases: (1) Lane
change/low-speed
merge, (2) vehicle
following, (3)
move out of travel
lane/park.

Abdessalem et al. (2016) generated concrete scenarios using the NSGA-II in combination
with a surrogate model based on neuronal networks (NN), focussing on critical

58
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

pedestrian-to-vehicle conflict scenarios. The surrogate model has been modelled using
output simulation data of critical pedestrian-to-vehicle conflicts to train a neuronal
network for population classification. In the context of their study, the neuronal network
aims to support the NSGA-II algorithm in the identification of critical scenarios. The
search objectives for the optimisation have been: (1) the time-to-collision (TTC), (2) the
distances between the pedestrian and the vehicle and (3) the distances between the
pedestrian and a defined collision risk area. These three functions were used to define a
critical scenario. The performance of the surrogate NSGA-II was compared to a simple
NSGA-II and a random search algorithm (MC). Their analysis showed that the NSGA-II,
in combination with the surrogate model, performs better than the NSGA-II and the
random search variants. The neuronal network and NSGA-II based approach generated
the same number of critical scenarios faster than the other methods. As a result, it can be
concluded that the NSGA-II performs better when supported through other models to
guide the search.

Klück et al. (2019) compared the performance of genetic algorithms to simulated

annealing and random search for critical scenario generation for AEB testing. In their
study, crash or near-crash scenarios have been defined to be ’Critical’. The generated
scenarios represented vehicle-to-vehicle rear-end scenarios for AEB testing, where
pedestrians were located on the roadside and could potentially cross the road. The Time-
to-Collision to the lead vehicle was the objective function that was minimised during the
optimisation activity. According to their findings, the GA was more efficient than the
benchmark algorithms, such as SA and random search. The GA generated the lowest TTC
scenarios with the lowest number of executions. Therefore, the GA allowed a faster
verification of the system properties for AEB testing in this case.

Felbinger et al. (2019) discussed the application of genetic algorithms and combinatorial
testing for AEB test case generation. Combinatorial testing is a technique that narrows
down the full factorial test space by identifying T-pairs of combinations that lead to
system malfunctions (Wu et al. 2020). The GA and the combinatorial testing method were
applied to a vehicle-to-pedestrian scenario to stress an AEB system in a co-simulation
environment. The GA was used to identify concrete scenarios, and the combinatorial
testing was used to identify critical parameter interactions. Moreover, in this study, the

59
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

TTC was minimised as a single objective to identify critical scenarios. Their results
showed that both techniques (combinatorial testing and GA) were very promising for the
identification of concrete scenarios. They claimed that the controllability of the GA
allowed to generate, when initialised appropriately, at each iteration, a potential crash
scenario, which is not possible with random search techniques. Furthermore, the study
highlighted that combinatorial testing was more suitable for the identification of
parameter interactions leading to critical situations, whereby GA was more suitable for
searching concrete values to parametrise critical scenarios.

Kaufmann et al. (2021) proposed a simulation-based method for identifying critical

scenarios for different use cases. They analysed the following use cases: (1) lane
change/low-speed merge, (2) vehicle following, (3) move out of travel lane/park. Within
these use cases, the EGO vehicle equipped with ADSs was required to detect and respond
to (a) school buses, (b) encroaching oncoming vehicles and (c) to pedestrians. A single-
objective GA was utilised to obtain critical concrete scenarios in these employed use cases
to assess whether the EGO vehicle meets requirements (a), (b), and (c). The optimisation
objective to search for critical scenarios was to minimise the distance between the EGO
vehicle and the different actors required to be detected. The scenarios were generated in
the first step, and in the second step, they were executed in a virtual environment. Lastly,
the scenarios were evaluated by considering the critical ones with small distances. It was
claimed that they discovered circumstances in which the EGO vehicle did not react as
expected, as well as scenarios in which the other participants in the scenarios exhibited
unexpected behaviour. However, the authors claimed a lack of reproducibility of the
results when simulating the same scenarios repeatedly. This indicated that simulation
environments dealing with determinism issues are not the best platforms for validating
metaheuristics-generated scenarios.

2.3.3.3 A critical evaluation of the application of genetic algorithms for scenario

generation

Although there have been limited investigations into the use of GAs for test scenario
generation for Level 0 to Level 5 system testing, some limitations have been observed
that must be addressed.

60
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

First, it was discovered that there had been a lack of validation of the generated critical
scenarios by the GA to classify them as injury critical. Each study reviewed in Table 2-1
applied simulation environments to classify the generated scenarios by the GA as critical.
However, no mention was made of how well the simulation environments correlate to
reality in terms of sensor and vehicle models. It is known that simulation environments
today are not accurate enough to replicate real-world scenarios for ADSs testing
(Figueiredo et al. 2009, Beringhoff et al. 2022). Validation of the generated scenarios
using metaheuristics like GAs is essential, as solutions for an optimisation problem
generated by GAs only provide solutions close to the global optimum. The solutions
require validation using empirical data to classify the generated solution population into
relevant and non-relevant solutions. From the few studies, it can be concluded that no
empirical evidence proves that GAs can generate critical scenarios for safety assessment
of Level 0 to Level 5 systems which are close to reality. Moreover, there is no evidence
that the generated scenarios are probable to result in a human-related injury, which is
essential for safety assessment. Hence, the following research question has been derived
for this thesis:

RQ (2): Can Genetic Algorithms be used for injury critical scenario generation for
ADSs testing or not?

The second limitation has been discovered in the formulation of the optimisation
problems. From Table 2-1, it can be seen that the majority of the studies have minimised
the TTC to search for critical scenarios. However, using only time- or distance-based
indicators to guide the search to identify injury-relevant scenarios is insufficient. More
information must be modelled into the optimisation problems to guide the search to
identify injury critical scenarios. Laureshyn, Svensson, and Hydén (2010) suggested that
the severity of a traffic event depends on two different risk states, a) collision risk and b)
injury risk. Optimising towards minimum TTC is sufficient for achieving high collision
risk; however, a low TTC does not automatically results in a high probability of injury
risk. This highlights the lack of accurate problem formulation for critical scenario
generation for ADS testing. It can be concluded that existing models must be extended
with more information to achieve better results, which will be done in this thesis.

61
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Moreover, it can be said that the search for injury critical scenarios is a MOO problem
rather than a single-objective optimisation problem.

The third limitation concerns performance comparisons between GAs and other
algorithms for scenario generation for Level 0 to Level 5 systems testing. Performance
comparisons, in the past, focussed only on how fast GAs can generate a certain number
of scenarios. No questions were raised in the studies regarding the scenario coverage in
the design space achieved by the benchmarked algorithms, which is essential to enhance
the safety of ADSs. ISO 21448 requires exploring the scenario space adequately to
achieve a high coverage before the release of ADSs. However, there is no mention in ISO
21448 of how to measure the scenario coverage in the design space. Some studies suggest
comparing optimisation algorithms to random search for a performance benchmark.
Hence, in this thesis, the NSGA-II is compared to the random search algorithm LHS due
to its ability to spread its generated data points well in the design space. This comparison
aims to state whether GAs are better than random simulation algorithms for scenario
space coverage. The following research question has been derived:

RQ (3): Do GAs have better coverage in the scenario design space compared to
random simulation algorithms?

2.4 Summary of literature review

The literature review has raised the importance of understanding how TCs impact the
safety of ADSs. It has been found that fog, snow, and rain have a major impact on ADSs
sensor safety, leading to FISs on both component and system levels. Moreover, it has
been raised that these adverse environmental conditions also impact human driver
performance. Hence, for the parametrisation of mixed-vehicle fleet scenarios, it is
essential to incorporate such adverse environmental conditions into the scenarios for
ADSs testing.

Sub-section 2.3.1 discussed different accident databases available for scenario generation.
The use of NDD and accident data originating from IoT have been classified as ineffective

62
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

for extracting critical scenarios. It has been found that macroscopic accident data, such as
the STATS 19, is a data source which can be used to construct a scenario space initially.

Unsupervised and supervised machine learning algorithms have been used frequently for
scenario space construction using accident data, which comprises the identification of
functional and logical scenarios. However, it has been identified in sub-section 2.3.2 that
using supervised machine learning requires a priori information which is not the case for
supervised machine learning. As a result, supervised machine learning algorithms are
inefficient for identifying unknown hidden patterns in accident data. For identifying
unknown hidden patterns in accident data, the use of clustering algorithms has been
widespread. However, no clustering study concentrating solely on creating scenarios from
accident data comprising TCs, such as poor environmental conditions (snow, fog, et al.),
has been conducted yet. Moreover, existing CAs studies derived general accident patterns
under favourable environmental conditions rather than rare event accident patterns.
Hence, in this thesis, RQ (1) investigates whether CAs can be used to identify rare event
accidents from accident data for the parametrisation of TCs leading to FISs.

Search-based algorithms were often deployed for scenario space exploration, known as
concrete scenario generation in the context of the PEGASUS project. Random simulation
was classified as too exhaustive to solve the scenario generation problem. Moreover,
guided random simulations, such as MCMC and IS, were considered inefficient as they
require a priori information to generate scenarios that are challenging to provide for
unknown scenarios. Hence, to solve the scenario generation problem, which is
computationally expensive, optimisation algorithms based on metaheuristics were used
frequently, especially for Level 0 to Level 5 test scenario generation, when GAs were
used as the most frequent algorithm. However, it has been highlighted that GAs, based
on metaheuristics, only provide solutions near the global optimum. Hence, the generated
scenarios need to be validated against real-world data to provide evidence for their
effectiveness for critical scenario generation for ADS testing. This validation activity was
not performed in existing studies using information from real-world data.

Moreover, the optimisation problem for critical scenario generation was formulated in
past studies as a single-objective optimisation problem, whereas it has been mentioned in
past literature that for the injury classification of accidents (also scenarios), multiple

63
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

indicators are necessary. As a result, RQ (2) was established, which examines whether
GAs can be utilised to identify important injury critical scenarios. Finally, it was
determined that using GAs for scenario generation for ADSs testing does not achieve high
coverage of the scenario design space. Hence, RQ (3) has been defined, analysing the
scenario space coverage in the scenario design space compared to random search.

64
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Chapter 3. Methodology
3.1 Introduction

From the previous chapter, challenges for the identification of scenarios from accident
data have been discussed. Moreover, shortcomings in identifying critical scenarios using
optimisation have been raised in the previous chapter. This chapter provides detailed
information regarding the tools and data used in this thesis to tackle the previously
identified challenges.

Figure 3-1 gives an overview of how this chapter has been organised. The chapter is
divided into two studies. Section 3.2 introduces the first study: ‘Scenario space
construction using accident data’, aiming to answer RQ (1). Section 3.3 introduces the
second study: ‘Scenario space exploration using optimisation’, aiming to answer RQ (2)
and RQ (3). Note that the proposed workflow has been based on the PEGASUS
framework (see section 2.2), which anticipates deriving functional, logical and concrete
scenarios.

Figure 3-1: Overall proposed scenario generation method.

65
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Within study 1, in sub-section 3.2.1, a detailed data preparation and selection activity is
first introduced. The previous chapter highlighted in sub-section 2.3.1 that macroscopic
data, such as the STATS 19 data, is a valuable source to draw an overall picture of
accident circumstances for larger regions. Hence, STATS 19 data from 2010 to 2018 was
considered to demonstrate the proposed scenario generation method in this thesis. Using
clustering methods, a 2-layer k-medoids application on the STATS 19 data is proposed
to overcome the challenge of extracting hidden patterns from accident data.

The first layer, representing step 1.1 from Figure 3-1, is introduced in sub-section 3.2.2.
The first layer aims to derive high frequency clusters using the 𝒌-medoids algorithm from
the STATS 19 data. High frequency functional scenarios can be specified based on the
high frequency clusters. Sub-section 3.2.3 proposes a criticality metric to rank the high
frequency scenarios according to their relevance for testing ADSs under adverse
conditions. In sub-section 3.2.4, step 1.2 is introduced. Step 1.2 clusters a highly ranked
high frequency scenario to obtain rare event functional scenarios. This breakdown of the
high frequency clusters answers RQ (1).

Study 2 fills the gap for missing injury critical vehicle dynamics parameters in the STATS
19 dataset, which is required as input for different virtual test suites. A rare event
functional scenario identified from study 1, introduced in sub-section 4.2.3 in chapter 4,
serves as a first input for study 2. In sub-section 3.3.2, step 2 of the scenario generation
method is introduced. Step 2 comprises designing a mathematical intersection model
formulated as a MOO problem correlating to the rare event functional scenario introduced
in sub-section 4.2.3 in chapter 4. Different accident indicators and relevant critical
parameter boundaries have been used to formulate the MOO problem, which is
simultaneously the logical scenario.

In step 3 (sub-section 3.3.3 and sub-section 3.3.4), the NSGA-II algorithm and the LHS
algorithm, which use the mathematical intersection model as an input, are used to identify
potential injury critical concrete scenarios.

Sub-section 2.3.3.2 of the literature review chapter has raised the lack of validation of
generated scenarios by metaheuristics, which is addressed in step 4 by sub-section 3.3.5.
To validate the concrete scenarios to classify them as injury-relevant and non-injury-
relevant, real-world accident models based on event-data-recorder (EDR) accident data

66
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

are used. Moreover, the design space coverage of the NSGA-II is compared to random
sampling using the standard deviation. The results of step 4 answer RQ (2) and RQ (3).

Finally, section 3.4 summarises the methods, tools and data used to develop this chapter.

3.2 Study 1: Scenario space construction using accident data

Figure 3-2 illustrates the precise workflow of the 2-Layer clustering method to extract the
rare accident scenarios from the STATS 19 data. The method to construct an initial
scenario space is divided into four steps.

Figure 3-2: Rare event functional scenario generation workflow based on STATS 19 data.

The data preparation is introduced in step 1. The data preparation plays an essential role
in reducing the complexity of the multidimensional space of the STATS 19 data (see sub-
section 2.3.2.3 in chapter 2). Step 2, in the first layer, partitions the prepared STATS 19
data into 𝒌 initial high frequency clusters using the k-medoids algorithm. Step 3
introduces a method to rank the high frequency clusters accordingly to their criticality for
ADSs testing under adverse conditions. The higher the occurrence of potential TCs in a
cluster is, the higher its relevance and, thus, its criticality rank. After ranking the clusters,
the highest ranked cluster can be selected for further break down into smaller sub-clusters.

67
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

The higher-ranked clusters are assumed to hide sub-pattern containing relevant TCs not
identified in the first layer. The further breakdown of the high frequency clusters is
accomplished in step 4 to identify rare event scenarios.

3.2.1 STATS 19 data

For this study, it has been decided to use the STATS 19 database. Not only has its use
been suggested in ISO 21448, but it is also a macroscopic database, which provides a
broader picture of the existing accident data patterns (see sub-section 2.3.1 in chapter 2).
In order to use a representative (large) pool of data, it has been decided to consider STATS
19 data from 2010 to 2018. The data is freely accessible and can be downloaded in CSV
format from the UK government's website (comma-separated values) files.

Figure 3-3 shows the data structure of the STATS 19 data, consisting of three files – the
accident, vehicle and casualties’ files. The accident files store general information about
the accident, such as the accident’s environmental conditions, time, location, et al.

Figure 3-3: STATS 19 data structure.

The vehicle file stores information regarding the vehicles involved in the accidents, such
as vehicle type, vehicle make, manoeuvre, et al. The casualties file stores information

68
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

regarding the accident victims and the type of vehicle involved (bus, car, pedestrian, et
al.).

The accidents in the STATS 19 data are tagged with accident IDs. Figure 3-3 shows that
the ID occurs only once in the accident file but can occur many times in the vehicle and
casualties files. For instance, if more than one vehicle is involved in one accident, the
same accident ID is assigned multiple times to these involved vehicles. By doing so,
traceability can be ensured, which is essential for merging all the information of one
accident.

The STATS 19 data is a categorical dataset coded using numerical categories (e.g., slight
injury = ‘1’, serious injury = ‘2’ et al.). Hence, to support the interpretation of the STATS
19 database, alongside the files illustrated in Figure 3-3, the UK Department for Transport
provides a dataset guide in the form of a lookup table. Figure 3-4 shows a snapshot of this
lookup table.

Figure 3-4: Structure of supporting document for looking up codes of labels.

In this study, the ‘field name’ is referred to as the parameter, while the ‘label’ is the
observation the parameter can take.

3.2.1.1 Data preparation

The data preparation in this study has been divided into two activities: data selection and
data cleaning. The data selection was conducted in SPSS, whereas the data cleaning was
conducted in MATLAB. The proposed data preparation procedure consists of six steps,
(a) and (b) representing the data selection and (c) to (g) representing the data cleaning.
The steps are summarised below:

69
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

(a) Merging of accident and vehicle files.

(b) Select parameters according to the 6-Layers model of the PEGASUS
framework.
(c) Select only serious injury accident entries.
(d) Select only vehicle-to-vehicle accidents.
(e) Select only T-intersection accidents.
(f) Merge similar observations to reduce complexity.
(g) Omit accidents with missing data entries.

The underlying sub-sections provide detailed information regarding each step and the
code listings for replication of this study.

3.2.1.2 Data selection

Step (a): For data processing, it is necessary to merge the STATS 19 files from 2010 -
2018 into one single file (see Figure 3-3). This study requires information from the
accident and the vehicle file. The casualties file can be ignored because this thesis only
focuses on vehicle-to-vehicle (mixed-vehicle fleets) accidents. Hence, no information, for
instance, regarding injured bus passengers or pedestrians, is required.

For this study, the STATS 19 data have been merged in SPSS. Two different types of
merge commands are required to combine the STATS 19 data. First, the accident files
ranging from 2010 – 2018 have been concatenated vertically. The same has been done
for the vehicle files. As a result, all accident files are merged into a single large file, as
are the vehicle files. Second, the concatenated accident file has been merged horizontally
with the concatenated vehicle file. To realize this horizontal merge, the one-to-many
merge command in SPSS was used due to the data structure presented in Figure 3-3. The
one-to-many merge has been used because the accident ID is represented in the accident
file once per data entry and in the vehicle file multiple times per involved vehicle.

Following these two merging activities (vertical and horizontal), each vehicle in the
vehicle file has been assigned its accident circumstances from the accident file. As a
result, the data is now at the vehicle level. Figure 3-5 shows the data shape at the vehicle
level for better understanding.

70
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Figure 3-5: Data shape of merged STATS 19 data on vehicle level.

Step (b): In step (b), relevant parameters from the STATS 19 have been selected to
parametrise intersection scenarios. The selection of parameters has been conducted in
SPSS. The 6-Layer model introduced in section 2.2 (chapter 2) has been used to guide
the PEGASUS selection process. For example, knowing which police department
reported the accident is not required to parametrise mixed-vehicle fleet intersection
scenarios for this thesis. As a result, this unnecessary information has been omitted.
Parameters have been chosen to make the scenarios' reconstruction in a virtual
environment as accurate as possible. In total, 13 parameters have been selected,
summarized in Table 3-1 alongside a rationale.

71
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Table 3-1: Selected parameters using the PEGASUS 6-Layer model and rationale for parameter
selection.

PEGASUS Layer Selected parameter Parameter rationale

1st Road Class These parameters are
2nd Road class required to understand the
Layer 1: Road-Level Road Type road geometry of the
Speed limit scenario. They can be used
Road surface condition to model the road
Area infrastructure in a
simulation environment.
Layer 2: Traffic Junction control Allows to understand in
infrastructure case a violation of the
junction control is
observed.
Layer 3: Temporary Information is not
manipulation of Layer 1 - available in STATS 19.
and Layer 2
Layer 4: Objects 1st point of impact (on These parameters allow
vehicle) parametrising the involved
Vehicle manoeuvre actors of the scenario.
Vehicle type
Injury outcome (of vehicle
occupants)

Layer 5: Environment Weather condition These environmental

Lighting condition conditions can potentially
trigger a deviation from
the intended functionality
of ADS, as well as have a
negative impact on human
driver performance.

72
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Layer 6: Digital - Information is not

information available in STATS 19.

3.2.1.3 Data cleaning

Steps (c) to (g) to clean the STATS 19 data have been conducted in MATLAB. The
STATS 19 data sample is imported in CSV format and converted into an array. The code
listings and the exact steps for the data cleaning are reported within the underlying
paragraphs.

Step (c): In step (c), only one specific injury class was considered. Injury classes are coded
as ‘1’, ‘2’ or ‘3’ for slight, serious, or fatal injury. Since injury patterns for fatal accidents
are likely to differ from slight injury accidents, it is proposed to consider only the data
from one injury class to obtain clean patterns from the clustering. Step (c) has the potential
to increase the homogeneity of the dataset and thus will result in more efficient
understanding of the correlation between high injury accidents in intersections and
environmental conditions. Code Listing 3-1 shows the line of code required to realize step
(c).

Listing 3-1: Command for selection of serious injury accidents only. (Note: ‘d’
is the STATS 19 data sample and 16 the column where the injury information
is stored.)

Step (d): In step (d), the STATS 19 data is reduced to only cover the use case vehicle-to-
vehicle intersection accident scenarios. Injury outcomes, for instance, for vehicle-to-
pedestrian accidents, differ from vehicle-to-vehicle accidents. Hence, it is suggested to
analyse them separately. Therefore, buses, vans, motorcycles, pedestrians, and other road
users have been omitted. To be more specific, only vehicle-to-vehicle accidents involving
two vehicles were considered. Listing 3-2 shows the implementation on MATLAB of
step (d).

73
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Listing 3-2: Command for selecting only accidents containing strictly two vehicles.

Step (e): Generally, intersection accidents can be divided into 3-way (T-intersection) and
4-way intersection conflicts. Nitsche et al. (2018) concluded that accident patterns for T-
intersections differ from 4-way intersection accidents. Hence, to further reduce the
complexity of the dataset and increase the representativeness of the resulting clusters, T-
intersection accidents are suggested to be analysed separately from 4-way intersection
accidents. Nitsche et al. (2018) found that in the UK, T-intersection accidents have a
higher number of patterns (13) compared to 4-way intersection accidents (6). To
demonstrate the proposed 2-Layer clustering method, it has been decided to choose T-
intersection scenarios only. Listing 3-3 shows the commands in MATLAB to realise step
(e).

Listing 3-3: Command to realise step (2) deleting unwanted information and only
keeping T-intersections.

Step (f): Previously, in step (b), the parameters to parametrise the functional scenarios in
this thesis were selected. In step (f), the observations of these parameters are processed.
Step (f) consists of merging similar observations and omitting unwanted information. The
parameter ‘weather condition’, for instance, can be coded as ‘heavy rain’ or ‘light rain’,

74
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

both of these codes are merged into one, namely ‘rain’. The merging steps have been
implemented further to reduce the complexity of the STATS 19 data. The implementation
in MATLAB of the merging step can be seen in Listing 3-4. This reduces the complexity
of the multidimensional space once more, resulting in better clustering results. Note that
later when the logical scenarios are built, it is still possible to choose different logical
ranges for heavy or light rain if needed.

The second step in step (f) was to delete unwanted observations irrelevant to the scenario
parametrisation. The resulting list of parameters and their observations are presented in
Table 3-2. Listing 3-4 shows how the steps of step (f) can be realised in MATLAB code.
The entire code for the data preparation to look up how each observation has been
processed in step (f) is available in APPENDIX B.

Listing 3-4: Step (f): Commands for merging data, recoding the observations, and deleting the
unwanted data.

75
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Table 3-2: Summary of the selected parameters from STATS 19 and its observations after the data
preparation.

Parameters Observations
1st
Road class Motorway
A(M)
A
B
C
Unclassified
2nd Road class Motorway
A(M)
A
B
C
Unclassified
Road type Roundabout
One-way street
Dual carriageway
Single carriageway
Slip road
Speed limit 20-30
[mph] 40-50
60-70
Junction control Auto traffic signal
Stop sign
Give way or uncontrolled
Light condition Daylight
Darkness
Weather Fog or mist
condition Fine
Raining
Snow

76
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Road surface Dry

condition Wet or damp
Frost, ice, or snow
Oil or diesel
Mud
Flood
Vehicle Turning left
Maneuver Reversing
U-turn
Turning right
Going ahead
1st point of Front
impact Back
Offside
Nearside
Area Urban
Rural

Step (g): Finally, accident records in the STATS 19 dataset with missing data entries have
been deleted. As a result, only accidents with complete information have been considered.
Due to the large data size ranging from 2010 - 2018, this data-omitting strategy is
acceptable. Variables with missing observations are coded with ‘-1’ in the STATS 19
data. The command loop in Listing 3-5 has been used to delete data entries with missing
observations.

Listing 3-5: Command for omitting accidents with missing observations.

77
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

3.2.1.4 STATS 19 data sample

This section gives key information regarding the data size obtained after the STATS 19
data preparation. Figure 3-6 shows that after conducting the data preparation activities
from step (a) to (f), 29579 vehicle entries represent serious injury Vehicle-to-Vehicle T-
intersection accidents.

Figure 3-6: STATS 19 data sample size after conducting data preparation steps reported in this
thesis.

After applying step (g), the missing data strategy, 28555 vehicle entries were left. As a
result, 5% of the data has been omitted due to missing values, which is acceptable for the
large data size available from the STATS 19 data. This STATS 19 data sample has been
used as input for the clustering activities in study 1. Note again that these are vehicle
observations within the STATS 19 data, not the number of accident records. Since only
accidents with two vehicles have been considered, approximately the 28555 vehicle
28555
entries represent 14277 ( = 14277) accident records. Some of the vehicle entries
2

have been omitted in step (g). Hence the number of accidents can be slightly above or
under 14277. Analysing the dataset on the vehicle level has the advantage that the
accident circumstances for each vehicle can be extracted in combination with the
vehicle’s manoeuvre. As a result, ADSs can be exposed to these circumstances derived
from the STATS 19 they could face in the future.

3.2.2 Layer 1: High frequency functional scenarios

This section describes step 2 of Figure 3-2, which relates to mining high frequency
clusters from the STATS 19 data using the k-medoids algorithm. In sub-section 2.3.2.3
(chapter 2), it has been highlighted that no particular CA is known to be the best for

78
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

accident data partitioning. This thesis uses the k-medoids algorithm because it is known
for its good performance on heterogenous data (Harikumar 2015, Dixon et al. 2019).
Since the STATS 19 data sample consists of categorical and discrete data, the k-medoids
is considered an acceptable choice for study 1.

The k-medoids algorithm consists of two steps (build and swap step) and has the
following inputs and outputs (Helm 2021):

Input:

• Dataset
• Distance metric
• Number of clusters k

Build step:

• Initialization of initial k medoids, which serve as a cluster centre.

• Computation of (dis)similarity matrix for the input dataset using the distance metric.
• Partitioning: Observations within the dataset are assigned to the closest medoid.

Swap step:

The main goal of the algorithm is to minimise its cost function to find the best medoids.
The cost for the k-medoids algorithm is the sum of overall distances to the non-medoids
within a cluster. As a result, the swap step iteratively checks whether the distance metric
changes (reduces) when swapped against the initial medoid for all non-medoids in one
cluster. The medoid is swapped if the sum of the distances to other observations becomes
less than one. If medoids are changed, the partitioning from the Build and Swap steps is
repeated until no changes are observed.

Output:

• k clusters of the input dataset

The algorithm structure shows that, for the initialisation of the STATS 19 data sample, it
is necessary to define both an appropriate distance metric and select the number for k for
the dataset. Hence, the distance metric selection and number for k are discussed below.

79
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

3.2.2.1 Selection of distance metric

The distance metric in machine learning applications expresses the (dis)similarity of data
points in a dataset. In the case of clustering algorithms, the dis(similarity) is used to assign
similar data points into the same clusters to separate different data points from each other.
Many algorithms can compute a data set's (dis)similarity matrix (Arora, Khatter, and
Tushir 2019). Not every distance algorithm works well for categorical data. However, for
categorical data where patterns are unknown, the Euclidean distance is a well-established
option (Sammon 1969, Lourenço, Lobo, and Bação 2004). The squared Euclidean method
computes the dissimilarity faster than the simple Euclidean method, as it does not
consider the root. The squared Euclidean distance can be expressed as (Hossain and
Abufardeh 2019):

2
𝑛
⃗⃗⃗ , 𝑄
𝑑𝐸 (𝑃 ⃗ ) = √∑(𝑝(𝑖) − 𝑞(𝑖))2 3-1
𝑖=1

Where,

• 𝑃 and 𝑄 are population vectors of clusters,

• 𝑝(𝑖) and 𝑞(𝑖) the 𝑖 𝑡ℎ element of 𝑃 and 𝑄 and
• 𝑛 the size of observations in 𝑃 and 𝑄.

3.2.2.2 Selection of number for k

The selection of k, when the number of patterns in the dataset is unknown, is an incredibly
challenging task (Ikotun et al. 2023). Suppose no suitable value of k can be assigned from
prior knowledge of the dataset, which is the case for the STATS 19 data sample. In that
case, the literature suggests the application of different metrics to estimate optimal k (Piao
Tan and A. Floudas 2009, Patil and Baidari 2019). To estimate k, for the prepared STATS
19 data sample in this study, it has been decided to use two different metrics, allowing to
estimate k with higher confidence.

80
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

The first indicator applied to estimate the optimal k is the Silhouette method introduced
by (Rousseeuw 1987). Silhouette values indicate how well a data point has been assigned
into its cluster compared to its neighbouring cluster. The Silhouette values range from -1
to 1. The closer Silhouette values are to 1, the better the data points fit into their assigned
cluster. Negative Silhouette values indicate that the data points are not assigned to the
correct cluster. In such cases, a reconsideration for selecting k and the distance metric is
required. The Silhouette values can be computed using the following equation
(Rousseeuw 1987):

𝑏(𝑗) − 𝑎(j)
𝑠(𝑗) = 3-2
max{𝑎(𝑗), 𝑏(j)}

Where,

• 𝑠(𝑗) = silhouette value between -1 and 1;

• 𝑗 = object within cluster 𝐴 or 𝐵;
• 𝑎(𝑗) = average dissimilarity of 𝑗 compared to all other objects within cluster 𝐴
and
• 𝑏(𝑗) = average dissimilarity of 𝑗 compared to all other objects within cluster 𝐵.

The second metric chosen in this study to estimate k was the Davies Bouldin (DB) method
(Davies and Bouldin 1979). The DB values, like the Silhouette values, indicate how well
a data point has been assigned into its cluster compared to neighbouring clusters. It ranges
between 0 and 1. In contrast to the Silhouette method, the closer the DB values are to 0,
the better the data points suit their clusters. The Davies-Bouldin value is expressed
through the following equation (Davies and Bouldin 1979):

𝑇(𝑟) 3-3
1 1
𝑑𝑏(𝑟) = ( ∑(|𝑋(𝑙) − 𝐴(𝑟)|)𝑝
𝑇(𝑟)
𝑙=1

Where,

• 𝐴(𝑟) is the centroid of the cluster 𝐶(𝑟);

81
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

• 𝑋(𝑙) is an n-dimensional vector assigned to 𝐶(𝑟);

• 𝑇(𝑟) is the size of 𝐶(𝑟) and
• 𝑝 is set to 2 to build the Euclidean distance between the centroid and the
member of vector 𝑋(𝑙).

Since the number of patterns for the STATS 19 data sample is unknown, an investigation
range for optimal k needs to be performed. In this thesis, an investigation range between
1 and 50 is chosen. It is important to note that if no satisfying results are found within an
investigation range, increasing the investigation range iteratively is recommended until
the Silhouette and DB values are both satisfied. Because this step is computationally
costly, it was decided, in this study, first to investigate the range 1 to 50 for optimal k. To
estimate k, the STATS 19 data sample was clustered using the k-medoids algorithm
iteratively in the defined investigation range. The results for each kth iteration have been
assessed using the Silhouette values and DB values to identify the kth iteration giving the
best results. The best kth iteration was then chosen as optimal k in this study for
partitioning the STATS 19 data sample.

Furthermore, there is the possibility that during the clustering process, the k-medoids
algorithm converges to a local minimum. A converging to a local minimum can occur if
the total sum of distances increases instead of decreasing during the swap step. To avoid
this phenomenon, it is suggested to replicate the build and swap step for each kth iteration
𝑛𝑟𝑒𝑝𝑙. times whereby, 𝑛𝑟𝑒𝑝𝑙. can range from [1 ∞].

The replicate solution showing the smallest sum of distances between cluster observations
is then selected to represent the final solution. Note that the smaller the distances between
the data points in a cluster, the higher their similarity is. The replicate built-in function of
the MATLAB library pool has been used in this study to avoid a local (non-global)

82
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

minimum. For this study 𝑛𝑟𝑒𝑝𝑙. has been set to 5. Following MATLAB, commands have
been applied to identify optimal k for the STATS 19 data sample:

Listing 3-6: Command to investigate optimal k in a range for [1:50] for STATS 19 data.

3.2.2.3 Application of k-medoids on the first layer

The STATS 19 data sample was clustered in the first layer using the underlying
MATLAB commands in Listing 3-7.

Listing 3-7: Command to implement k-medoids algorithm in MATLAB. Note that X, in this case, is
the data to be clustered = prepared STATS 19 data.

83
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

3.2.3 Scenario Ranking

Theoretically, the clusters populating the highest number of accidents could be classified
as the most representative clusters. However, it cannot be said whether the highest
populated clusters also have the highest population of hidden rare event accidents that
occurred under adverse environmental conditions. Hence, to rank the scenarios in this
thesis, a criticality ranking method based on potential TCs having an impact on ADSs
safety is proposed in step 3. By ranking the high frequency clusters from layer 1, one can
strategically select the critical high frequency clusters for further analysis in layer 2.

To rank the clusters derived in layer 1, the frequency distribution for potential TCs within
each cluster has been computed. The clusters having the highest occurrence of TCs have
been assigned the highest criticality ranks amongst the total number of clusters. However,
before ranking, it must be determined which parameters and observations in STATS 19
are relevant potential TCs affecting ADSs and human driver performance negatively. The
TCs identified from the literature review in section 2.2 have been chosen. Note that these
are TCs only referring to sensor performance and actuation performance. They do not
cover misuse cases.

The number of combinations of TCs required to trigger FISs in ADS is not well known.
Kuhn, Wallace, and Gallo (2004) reported that 100% of safety-critical software
application failures could be discovered by testing three to four parameter interactions
(blue and green curves in Figure 3-7.
This item has been removed due to 3rd Party Copyright. The unabridged version of the
thesis can be found in the Lanchester Library, Coventry University.

Figure 3-7: Parameter interactions required to cover 100% of software failure for
different engineering sectors (Kuhn, Wallace, and Gallo 2004).

84
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Moreover, a study conducted by Klück et al. (2021) revealed that a combination of four
parameters is sufficient to impact the safety of Level 0 to Level 2 systems (valid for
DSSs). Based on this evidence, this study assumes for ADSs that a minimum combination
of four potential TCs should be designed into a scenario to obtain critical functional
scenarios. These four potential TCs have been the vehicle manoeuvre (representing the
behaviour of other road users, which can also be a potential TC according to ISO 21448),
the weather, lighting and road surface conditions. Table 3-3 summarizes the observations
used from the STATS 19 data to define the potential TCs and gives a short rationale.

Table 3-3: Summary of potential triggering conditions and their observations used to parametrize
the rare event scenarios.

Potential Critical Rationale

triggering observation in
conditions STATS 19
Manoeuvre Turning left The vehicle manoeuvre allows modelling the
Reversing behaviour of other road users to which ADS
U-turn can be exposed. It correlates the accident
Turning right scene and the human driver's behaviour.
Going ahead
Weather Fog/mist These triggering conditions statistically have
conditions Raining an impact on the sensor's field of view, as
Snow well as on the human driver’s eyesight.
Hence, modelling these parameters into the
SOTIF relevant scenarios allows testing the
intended functionality of ADSs in mixed-
vehicle fleets.
Lighting Darkness For this framework, it has been decided to
conditions choose darkness as a triggering condition
since specific sensors and the human driver
have a lower performance in the context of
recognition for such a lighting condition.

85
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Road surface Wet/damp Unfavourable road surface conditions lead to

conditions Frost/ice greater braking distances and can hence,
Snow represent TCs leading to FISs of actuators.
Flood
Oil/Diesel
Mud

For the ranking, a criticality metric was defined. It first counts the sum of total critical
observations listed in Table 3-3 for each high frequency cluster. Then it divides this tally
by the total number of cluster populations to obtain the percentage of critical observations
within this population. The higher this criticality percentage score is, the higher the cluster
ranking. It is then assumed that the clusters having the highest rank hide the rarest event
accidents compared to the lower ranked clusters. In layer 2 hence, the highest ranked
clusters can be used to further explore for rare event accidents.

3.2.4 Layer 2: rare event functional scenarios

Step 4 aims to break down the high frequency clusters into low frequency clusters to
identify rare event scenarios containing potential TCs. For the clustering, the k-medoids
algorithm was used again. For the initialisation of the k-medoids in the second layer,
equally as for the first layer, the distance metric and a number for k must be defined. A
high frequency scenario of choice was selected as an input dataset for the second layer.
Since the clusters in layer 1 were ranked according to their criticality for TCs, the highest
ranked high frequency scenario was picked in this study to demonstrate the breakdown
to answer RQ (1). Figure 3-8 illustrates the concept of the breakdown. Amongst the low
frequency scenarios, it is expected to discover rare event accidents to parametrise TCs.

86
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Figure 3-8: Diagram showing the breakdown of a high frequency scenario into multiple low frequency
scenarios by applying clustering iteratively.

Since the input data in layer 2 is also categorical, the Squared Euclidean metric was
applied again to compute the dis(similarity) matrix (see sub-section 3.2.2.1).

However, for the estimation of k, a different strategy compared to layer 1 was used. More
precisely, optimal k, to satisfy the needs of this study, was identified as follows: the input
high frequency cluster from layer 1 was clustered for k iteratively until each weather
condition was assigned into its own cluster. Since it is known that adverse weather
conditions are potential TCs to ADSs, such a breakdown of these TCs into their own
cluster can be forced to identify hidden sub-patterns.

Applying this optimisation step for the identification ok optimal k from the beginning in
layer 1 would bias the number for k, resulting in the loss of essential clusters due to failure
in identification. After finding a suitable number for k, which splits the weather conditions
into their clusters, the clustering in layer 2 is stopped. The same coding was used for layer
1 to cluster the dataset (see Listing 3-7). However, the input has been a high frequency
scenario rather than the entire prepared STATS 19 data.

87
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

3.3 Study 2: Identification of injury critical concrete scenarios

Study 2 reports using the NSGA-II algorithm to identify injury critical vehicle dynamics
parameters to extend the STATS 19 functional scenarios.

Figure 3-9: Scenario generation workflow for the identification of injury critical concrete
scenarios.

The workflow in Figure 3-9 reports the specific steps to conduct the parameter search.
Step 1 of the workflow has been divided into two sub-steps, step 1.1 and step 1.2. Both
steps represent the formulation of the multi-objective optimisation problem.

Step 1.1 uses information from a high ranked rare event functional scenario and accident
indicators from accident literature to build a mathematical model to support the search
for potential injury critical concrete scenarios.

Step 1.2 derives a logical scenario from the rare event functional scenario using relevant
parameter distributions correlating to serious injury accidents. The activity in step 1.2
contributes to the definition of the search space for the model built in step 1.1 and
represents, at the same time, the logical scenario.

Step 2 reports the generation of potential injury critical concrete scenarios using the
NSGA-II algorithm. In this study, a python library of the NSGA-II called ‘pymoo’ was
applied. The python library was developed by Blank and Deb (2020) and represents the

88
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

NSGA-II algorithm suggested by Deb et al. (2002). The entire code for identifying injury-
critical concrete scenarios is reported in section 3.3.3. This study will elaborate on the
code step-by-step.

In step 3, the LHS algorithm is applied to generate potential injury critical concrete
scenarios. The LHS scenarios serve to benchmark the NSGA-II algorithm and answer RQ
(2) and RQ (3).

Step 4 reports the validation strategy of the generated concrete scenarios. The validation
filters out only the injury relevant concrete scenarios using injury probability models.
Moreover, the standard deviations in step 4, for the generated parameters by the NSGA-
II and the LHS, are computed to verify which of both has better coverage in the scenario
design space.

3.3.1 Introduction to the intersection optimisation problem

A rare event functional scenario mined from the STATS 19 data is first required to
formulate the optimisation problem. The rare event functional scenario serves to
formulate the boundaries and decide which variables must be included in the optimisation
problem. The rare event functional scenario 3.4 obtained from the second layer of
clustering (see sub-section 4.2.3 in chapter 4 results and discussion for more information)
has been used to demonstrate the proposed methodology:

Car A is travelling on an A class road with a speed limit of 20 - 30 mph and is exposed
to a frontal impact with car B while turning right, the merging road is an unclassified
road. The driver/passenger of car A suffers from a serious injury. The accident
happens in the dark and snowy weather conditions. The road type is a single carriage
way where the road surface condition is wet. The scenario tends to occur in an urban
area within an uncontrolled junction.

Figure 3-10 visualizes a generic pictogram of the rare event functional scenario 3.4
derived from the STATS 19 data. Note that the information gathered from STATS 19
only addresses vehicle A because STATS 19 has been analysed at the vehicle level.
Vehicle B has been added to the scene to replicate a crash. Vehicle B is also meant to be

89
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

the vehicle equipped with ADSs. Doing so, the reaction of ADSs can be assessed when
exposed to vehicle A in a mixed environment.

Figure 3-10: Generic pictogram of T-intersection scenario and the additional

parameters required to run it in a simulation environment.

The missing vehicle dynamics, which must be generated in this study, such as the vehicle
speeds and distances to the potential conflict points (the crash bomb in Figure 3-10), are
highlighted in red in Figure 3-10. The optimisation task in this study is to search for these
missing parameters. Moreover, the missing vehicle dynamics must be traceable to the rare
event functional scenario 3.4. The traceability is ensured by generating the vehicle
dynamics parameters, matching the environmental conditions and the injury outcome of
rare event functional scenario 3.4.

3.3.2 Step 1: Mathematical intersection model

Step 1 reports with step 1.1 the development of the mathematical intersection model and
its search space in step 1.2. The model is based on the principles of Svensson's (1998)
risk model, which suggests parametrising critical accidents using multiple accident
indicators. His model defines two essential pillars for critical accident parametrisation:

90
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

collision risk and injury risk. These pillars indicate that multiple optimisation objectives
are required to formulate the MOO problem. Table 3-4 lists examples of accident
indicators accordingly to Svensson (1998) to parametrise collision and injury risk.

Table 3-4: Svensson's (1998) risk model for critical accident parametrisation.

Collision risk Injury risk

Closeness in space Speed differences
Closeness in time Mass differences
Speeds of the involved road users Relative angle
- The fragility of the involved road users

It can be witnessed from Table 3-4 that vehicle dynamics, such as vehicle speed, can also
be used to estimate injury risk. Moreover, Table 3-4 highlights that it must be ensured
that the parameter describing an injury critical scenario must correlate to injury risk, as
well as to collision risk. Note that a collision can also happen without any injury outcome.
Therefore, it is essential to have both aspects in a scenario: injury risk and collision risk.
The mathematical model has been designed respecting the aspects of Table 3-4.

Since the mathematical model is an input for the NSGA-II algorithm, it must be
formulated as a multi-objective optimisation problem, which requires defining objectives,
variables, constraints, and upper and lower boundaries.

Step 1.2 defines the upper and lower boundaries for the constraints and variables. The
boundaries defined in step 1.2 represent the search space, which at the same time is
referred to as the logical scenario.

First, a summary of the mathematical model ensuring injury and collision risk is given,
followed by detailed explanations:

Objectives and variables:

Maximise:

91
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

𝑣𝐴 (𝑋)
3-4

𝑣𝐵 (𝑋)
3-5

Minimise:

𝜇𝐴 (𝑋)
3-6

𝜇𝐵 (𝑋)
3-7

𝛼𝑖𝑚𝑝𝑎𝑐𝑡 (𝑋)
3-8

In subject to:

Upper boundaries as well as lower boundaries for variables:

𝑚 𝑚
8 ≥ 𝑣𝐴 (𝑋) ≥ 20 3-9
𝑠 𝑠

𝑚 𝑚
8 ≥ 𝑣𝐵 (𝑋) ≥ 20 3-10
𝑠 𝑠

0.15 ≥ 𝜇𝐴 (𝑋) ≥ 0.4

3-11

0.15 ≥ 𝜇𝐵 (𝑋) ≥ 0.4

3-12

60 ° ≥ 𝛼𝑖𝑚𝑝𝑎𝑐𝑡 (𝑋) ≥ 90 °
3-13

Constraints

1.3 𝑠 ≥ 𝑇𝐴𝐴 ≥ 2.75 𝑠

3-14

92
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

1.3 𝑠 ≥ 𝑇𝐴𝐵 ≥ 2.75 𝑠

3-15

0 𝑠 ≥ ∆𝑡𝑐𝑜𝑛𝑓𝑙𝑖𝑐𝑡 ≥ 0.3 𝑠
3-16

1 𝑚 ≥ 𝑠𝐴 ≥ 50 𝑚
3-17

1𝑚 ≥ 𝑠𝐵 ≥ 50 𝑚
3-18

𝑚
𝐸_𝛥𝑣𝐴 ≥ 0 3-19
𝑠

Where,

• 𝑣𝐴 (𝑋) and 𝑣𝐵 (𝑋) describe the vehicle speeds of vehicle A and vehicle B,
• 𝜇𝐴 (𝑋) and 𝜇𝐵 (𝑋) describe the road friction coefficient for vehicles A and B and
• 𝛼𝑖𝑚𝑝𝑎𝑐𝑡 (𝑋) describes the impact angle.

Moreover,

• 𝑇𝐴𝐴 and 𝑇𝐴𝐵 represent the Time-to-Accident of each vehicle to a conflict point
(see sub-section 3.3.2.2 for more information),
• ∆𝑡𝑐𝑜𝑛𝑓𝑙𝑖𝑐𝑡 represents the crash condition so that both vehicles meet at the same
time at the same point (see sub-section 3.3.2.2 for more information),
• 𝑠𝐴 and 𝑠𝐵 represent the braking distances to the conflict point (see sub-section
3.3.2.2 for more information) and
• 𝐸_𝛥𝑣𝐴 represents Extended_Delta_V for vehicle A (see sub-section 3.3.2.2 for
more information).

3.3.2.1 Justification for objectives, variables and bounds

The proposed mathematical model consists of five objectives, six constraints and five
variables. The aim was to create a search model to identify critical parameters for mixed-
vehicle fleet intersection testing. The risk model in Table 3-4 has been followed to do so.

The risk model in Table 3-4 suggests that vehicle speeds are essential to describe high
collision and high injury risk scenarios. Furthermore, Figure 3-10 shows the necessity to

93
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

generate the vehicle speeds to simulate rare event functional scenario 3.4 in a simulation
environment. Hence, equations 3-2 and 3-4 were added to the model as optimisation
objectives. More precisely, 𝑣𝐴 (𝑋) represents the vehicle speed profile represented by the
STATS 19 data, whereas 𝑣𝐵 (𝑋) represents the speed profile of the ADS equipped vehicle.
Past studies have shown that high vehicle speeds correlate with high injury outcomes in
vehicle-to-vehicle accidents (Doecke et al. 2020, 2021). Hence, to obtain high-speed
concrete scenarios, 𝑣𝐴 (𝑋) and 𝑣𝐵 (𝑋) were maximised.

The objectives, which simultaneously represent the variables in this study, require a
design space. This design space is part of the logical scenario represented by the lower
and upper bounds. Equations 3-9 and 3-10 of the model describe upper and lower bounds
for the speed profiles. The requirement on the upper and lower bound for 𝑣𝐴 (𝑋) and
𝑣𝐵 (𝑋) is to cover speed profiles causing serious injury outcomes. To choose the bounds
for 𝑣𝐴 (𝑋) and 𝑣𝐵 (𝑋) , a model proposed by Jurewicz et al. (2016) depicted in Figure 3-11
was considered. It expresses the correlation between the impact speed and the probability
of a MAIS3+ injury outcome for the bullet vehicle. The bullet vehicle is the vehicle
causing the accident, which in this study is vehicle A. Jurewicz et al. (2016) applied the
NASS/CDS database (see sub-section 2.3.1 discussing accident data in chapter 2) to
express the correlation between impact speeds and serious injuries. Note that the
Abbreviated Injury Scale (AIS) is widely used in the medical field to classify the severity
of an injury. The AIS severity score ranges from 1 to 6, where 1 indicates a minor injury
and 6 indicates maximal unsurvivable injury (Wong 2011). A severity with a score of 3
or higher, namely a MAIS3+ injury, can be classified as clinically serious injured (Nunn
et al. 2018). Since the STATS 19 data pool used in this thesis only covers serious injury
accidents, it has been decided to use models correlating to a MAIS3+ injury outcome.

94
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

This item has been removed due to 3rd Party Copyright. The unabridged version of the thesis can be
found in the Lanchester Library, Coventry University.

Figure 3-11: Impact speed in function of Injury (MAIS3+) probability for

different scenarios (Jurewicz et al. (2016)).

The graph in Figure 3-11 represents five different models for different collision scenarios.
Since Sub-Scenario 3.4 represents a frontal collision experienced by vehicle A, the ‘Head
on (frontal)’ model was chosen to define the search space for the speed profiles of both
vehicles.

The red-boxed area in Figure 3-11 was chosen as the design area for 𝑣𝐴 (𝑋) and 𝑣𝐵 (𝑋).
ISO 26262 suggests in the appendix of part 3 (concept phase) that scenarios which have
a potential for human harm shall represent a minimum probability of 10% for a specific
injury class (no injury, slight injury, serious injury or fatality) (ISO 2011). Consequently,
the lower bound for the impact speed was chosen from a 10% probability for a MAIS3+
injury outcome in this thesis. A probability of 100% for a MAIS3+ was chosen for the
upper boundary. As a result, the design space for the impact speed was 28 km/h to 72
km/h or 8 m/s to 20 m/s, respectively. This design area is represented in the mathematical
model by equations 3-9 and 3-10.

The third and fourth objectives 𝜇𝐴 (𝑋) and 𝜇𝐵 (𝑋) representing equations 3-6 and 3-7 of
the model, relate to the road/tire friction coefficient. The road/tire friction coefficient can

95
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

act as a potential triggering condition reducing the performance of actuators such as

braking and steering components. Furthermore, human drivers fail to adopt an adequate
speed for snowy conditions, impacting vehicle dynamics and the stopping distance (Abdi
et al. 2018). Hence, adding 𝜇𝐴 (𝑋) and 𝜇𝐵 (𝑋) to the search model is essential to compute
accurate results for the vehicle dynamics parameters. Furthermore, the consideration of
the road/tyre friction builds traceability to the rare event functional scenario 3 by
matching the road surface condition.

The rare event functional scenario 3.4 represents a pattern for T-intersection accidents
under snowy and wet road surface conditions. However, for the design space of 𝜇𝐴 (𝑋)
and 𝜇𝐵 (𝑋) , it was decided to use the upper and lower boundaries for snowy road
surfaces. By using a range for 𝜇𝐴 (𝑋) and 𝜇𝐵 (𝑋) for snowy conditions, first, the scenario's
complexity increases, and second, the range for 𝜇𝐴 (𝑋) and 𝜇𝐵 (𝑋) matches with the
weather conditions of the rare event functional scenario 3.4. Since lower friction values
increase the required stopping distance to avoid a collision, they are more critical. As such
𝜇𝐴 (𝑋) and 𝜇𝐵 (𝑋) was minimised to capture low friction scenarios. An in-depth study
analysing road friction values under snowy conditions conducted by Müller (2017)
suggests a range of 0.15 and 0.4 for snowy roads. This range was considered for equation
3-11 and equation 3-12.

Finally, the impact angle 𝛼𝑖𝑚𝑝𝑎𝑐𝑡 (𝑋) was chosen as an optimisation objective in this
study. Equation 3-8 represents 𝛼𝑖𝑚𝑝𝑎𝑐𝑡 (𝑋) and is referred to the angle built by the
colliding speed vectors. To compute these speed vectors with high accuracy, physics-
based simulations for vehicle dynamics require information regarding the 𝛼𝑖𝑚𝑝𝑎𝑐𝑡 (𝑋).
This will enhance the accuracy of the generation of the vehicle dynamics parameters to
parametrise the injury critical concrete scenarios.

Table 3-4 suggests that, according to Svensson (1998), the relative angle between
colliding vehicles is required to calculate the injury risk of an accident. Since speed
directly correlates to the injury severity outcome of an accident, considering 𝛼𝑖𝑚𝑝𝑎𝑐𝑡 (𝑋)
in the mathematical model will lead to more accurate speed profiles and, thus, a more
accurate injury classification of the concrete scenarios in step 4 of study 2.

96
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

For this study, the 𝛼𝑖𝑚𝑝𝑎𝑐𝑡 (𝑋) is assumed to be equal to the road skew angle α in Figure
3-10, both vehicles approach the conflict point with the same angle, which equals the
road skew angle. The skew angle α, when small, can have a negative impact on the field
of view of sensors and a negative impact on the human eye’s field of view (Distefano and
Leonardi 2018). Hence, it has been decided to minimise 𝛼𝑖𝑚𝑝𝑎𝑐𝑡 (𝑋). The smaller, the
𝛼𝑖𝑚𝑝𝑎𝑐𝑡 (𝑋) values, the more challenging it is for the ADS to detect vehicle B and take
early action to reduce speed for collision avoidance. For the upper and lower bounds of
𝛼𝑖𝑚𝑝𝑎𝑐𝑡 (𝑋) represented by equation 3-13, values suggested from the Federal Highway
Administration (FHA) to design safe skewed road intersections were chosen. This choice
ensures that the road intersection geometries are realistic and represent the real-world
geometry of road networks. The FHA suggests that intersection angles should range
between 60° to 90° to reduce collision risks in intersections (Federal Highway
Administration 2004). These value ranges have been adopted in this work to describe the
upper and lower boundary for 𝛼𝑖𝑚𝑝𝑎𝑐𝑡 (𝑋) in equation 3-13.

3.3.2.2 Justification for constraints

Most of the constraints used to design the mathematical model serve to ensure that the
scenarios identified in step 2 have a high risk of collision. Without a collision, there is no
high probability of injury. Hence, these constraints have high importance.

The scenarios developed in this study are vulnerable to T-intersection accidents; the
scenario actors' time proximity is a common conflict point in the intersection. The
pictogram in Figure 3-10 illustrates conflict points, represented by a ‘bomb’, that can be
placed anywhere on the road legs. Figure 3-12 depicts potential conflict points in T-
intersections, which can be used as a reference when setting up a simulation environment.

97
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment
This item has been removed due to 3rd Party Copyright. The unabridged version of the thesis can be found in
the Lanchester Library, Coventry University.

Figure 3-12: Number of conflict points for different road geometries and manoeuvres
(European Commission 2015).

Sub-section 2.3.3.2 in chapter 2 discussed that, in the past, time-based accident indicators,
such as the 𝑇𝑇𝐶 , have been applied frequently to claim the generation of critical
scenarios. However, the 𝑇𝑇𝐶 should only be applied when the vehicle speeds are constant
(Hydén 1987, Minderhoud and Bovy 2001). This study's speed profiles are not constant
for the mixed-vehicle fleet scenarios. It is assumed that both vehicles detect each other,
eventually, to perform an evasive manoeuvre for crash avoidance. When an evasive
manoeuvre occurs to avoid a collision, the 𝑇𝑇𝐶 equation becomes inaccurate. The time
remaining for a collision to happen after an evasive manoeuvre, such as braking or
steering, is defined by the Time-to-Accident (𝑇𝐴) (Hydén 1987). The 𝑇𝐴 is the
appropriate time-based method to compute critical time gaps between the vehicles and
the conflict point in this study. To consider 𝑇𝐴, it is necessary to model the deceleration
and acceleration behavior into the time equation to represent the evasive maneuver. Since,
the variables to compute the 𝑇𝐴 are already available in the model, such as the vehicle
speeds 𝑣𝐴 (𝑋) and 𝑣𝐵 (𝑋) and road surface conditions 𝜇𝐴 (𝑋) and 𝜇𝐵 (𝑋), it is ensured that
the covariance between these variables and the 𝑇𝐴 will correlate to critical T-intersection
accidents. The 𝑇𝐴 , for both vehicles, was expressed by following equation:

𝑣 (𝑋) 3-20
𝑇𝐴 =
2 ∗ 𝜇 (𝑋) ∗ 𝑔

98
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Where,

• 𝑣 (𝑋) is the vehicle speed at the timestamp of braking,

• 𝜇 (𝑋) the road/tire friction coefficient and
• 𝑔 the braking power applied to the vehicle.

Note that 𝑔 has been chosen in this study as 9.81 m/s2 in order to generate scenarios where
both vehicles, when applying full brakes, would still be in a critical 𝑇𝐴 range. In equation
3-20, 𝑔 is multiplied with the corresponding road/tire coefficient 𝜇 (𝑋), consequently, not
the full braking power of one 𝑔 can be transferred to the road surface, which makes the
scenarios more realistic.

Hydén (1987) described the relationship between 𝑇𝐴 and serious, as well as non-serious
injury conflicts. This relationship is depicted in Figure 3-13 and has served to define the
upper and lower constraint for 𝑇𝐴. The x-axis represents the vehicle conflict speed (𝐶𝑆)
in km/h whereas, the y-axis represents 𝑇𝐴 values.
This item has been removed due to 3rd Party Copyright. The unabridged version of the thesis can be
found in the Lanchester Library, Coventry University.

Figure 3-13: Relationship between TA, conflicting speed, and

accident injury (Hyden 1987 cited in Nitsche (2018)).

99
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

The red line partitions Hydén's (1987) graph into serious and non-serious conflicts. The
functional scenarios in this thesis represent serious T-intersection accidents. Hence the
threshold values for the constraints in equations 3-14 and 3-15 were chosen accordingly.

The range for variables 𝑣𝐴 (𝑋) and 𝑣𝐵 (𝑋) (28km/h – 72km/h) is framed by the blue dashed
lines in Figure 3-11. As a result, one can conclude that the respective thresholds for the
possible 𝑇𝐴 values, in time unit for both vehicles to the conflict point, should vary
between 1.3 seconds to 2.75 seconds for serious injury accidents. Note that 𝐶𝑆 is the
vehicle speed at the time an evasive manoeuvre is taken. The upper and lower bounds of
𝑣𝐴 (𝑋) and 𝑣𝐵 (𝑋) are the only possible values for 𝐶𝑆 for the computed concrete scenarios.

After ensuring that the vehicles have a serious injury potential TA value to the conflict
point, it is necessary to ensure that they meet simultaneously at the collision point for a
collision to occur. As a result, to increase the probability that the vehicle A and B crash,
the following crash condition was added to the mathematical model:

∆𝑡𝑐𝑜𝑛𝑓𝑙𝑖𝑐𝑡 = |𝑇𝐴𝐴 − 𝑇𝐴𝐵 | = 0

3-21

∆𝑡𝑐𝑜𝑛𝑓𝑙𝑖𝑐𝑡 is computed using the 𝑇𝐴 equations, respecting the upper and lower boundaries
of 𝑇𝐴, which enhances the search for collisions resulting in serious injuries. This crash
condition ∆𝑡𝑐𝑜𝑛𝑓𝑙𝑖𝑐𝑡 ensures that vehicle A and B meet at the same time and place because
their time difference to the conflict point is 0. Guiding the search towards this condition
aims to ensure that the generated variables will represent colliding vectors in a simulation
environment. However, a limitation of ∆𝑡𝑐𝑜𝑛𝑓𝑙𝑖𝑐𝑡 is that only vectors are generated
assuming the vehicle as single mass points. A crash, however, can be experienced with
the entire car body and not only with one point on the car body. Hence, upper and lower
constraints for ∆𝑡𝑐𝑜𝑛𝑓𝑙𝑖𝑐𝑡 were added as can be witnessed by equation 3-16. These
constraints aim to increase the impact coverage on the vehicle, thus increasing the
probability of identifying diverse serious injury accidents.

The thresholds for the constraints in equation 3-16 were derived using the following logic:

100
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Figure 3-10 shows that vehicle B offers vehicle A a collision length of 4.5 m (average
vehicle length has been assumed). According to this study's analysed rare event functional
scenario 3.4, vehicle A suffers from a frontal serious injury collision. Supposing, for the
most critical case, that vehicle A is exactly in front of the conflict point, for a maximum
speed of 20m/s (upper velocity bound for 𝑣 (𝑋)), vehicle B would only have 0.225s to
exit the conflict point without causing a crash. The impact locations on vehicle B would
theoretically no longer be limited to a single mass point but to the entire vehicle length.
The 0.225s were rounded up to 0.3s for the constraints. As a result, for the upper and
lower constraints of ∆𝑡𝑐𝑜𝑛𝑓𝑙𝑖𝑐𝑡 , a value range between 0 and 0.3s was fixed.

Equations 3-17 and 3-18 represent the last two constraints aimed at satisfying the
requirement for the closeness in time of the vehicles for the generated scenarios.
Closeness in time is intended to be achieved by generating braking distances in
combinations with speeds such that the vehicles meet at the same time at the same point.
Hence, modelling the braking distance into the mathematical model has been considered.
Only when having the right relationship between distance and speed to a conflict point
can it be ensured that the crash is likely to happen. To compute the braking distance, the
following equation was used:

𝑣(𝑋)2
𝑠(𝑋) = 3-22
2 × 𝑔 × 𝜇 (𝑋)

Where,

• 𝑠(𝑋) is the braking distance to the conflict point,

• 𝑣 (𝑋) is the vehicle speed at the timestamp of braking,
• 𝜇 (𝑋) the road/tire friction coefficient and
• 𝑔 the braking power applied to the vehicle.

It was necessary to add lower and upper bounds for the braking distance, which the
optimization algorithm must avoid violating. All the accidents recorded within the
STATS 19 database occurred within 20m of the T-junctions. However, for snowy road
surface conditions, a braking distance of 20m for a maximum possible speed of 20m/s
seemed exceptionally low. It is assumed that the probability of missing out on essential

101
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

scenarios would be high when constraining the search to a maximum braking distance of
20m. Hence, for this study, it was decided to use a widely accepted generic equation to
compute the maximal braking distance required to stop the vehicle before reaching the
potential conflict point. The equation can be expressed as vehicle speed in km/h divided
by ten to the power of 2 (Rechtien 2020). Usually, for critical cases, the resulting braking
distance from the first equation is divided by 2 for ideal road conditions. However, this is
not the case for snowy road surface conditions. The generic equation in this study results
in a maximum braking distance of 49m for 20 m/s, rounded up in this study to 50m. As a
result, for the lower and upper boundary of the constraints in equations 3-17 and 3-18, it
was decided to apply a range for the braking distance between 1m and 50m. By knowing
this information, one can also design the minimum length of the road legs using
simulation.

The last accident indicator incorporated, as equation 3-19 into the mathematical model,
to guide the search towards injury critical vehicle dynamics parameters was the Extended-
Delta-V (EDV). Delta-V is referred to as the velocity change of two objects after a
collision. Car occupants partially experience the velocity change in the context of a
vehicle-to-vehicle crash. Furthermore, the vehicle's crash structure partly absorbs the
velocity change, known as the kinetic energy. Hence, Delta-V can be directly correlated
to the injury outcome of an accident (Johnsson, Laureshyn, and Ceunynck 2018).
Laureshyn et al. (2017) introduced an accurate method to calculate Delta-V. For the first
time, a Delta_V equation, as a function of the impact angle, was introduced to estimate
the injury risk of an accident. This equation was reported as the Extended Delta-V (EDV).
The impact angle allows more accurate computation of the speed change experienced
during an accident.

From STATS 19, only the injury information of vehicle A is known (serious injury).
Hence, it is only intended to assess how the EDV changes when vehicle A experiences a
crash with vehicle B, which is equipped with ADS. As a result, for this specific
assessment, only the EDV of vehicle A must be modelled into the mathematical model.

EDV can be computed using the parameters suggested in Svenson’s (1999) model for
accident risk modelling (Laureshyn et al. 2017):

102
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

𝑚𝐵 3-23
𝐸_𝛥𝑣𝐴 = ∗ √𝑣𝐴2 + 𝑣𝐵2 − 2 ∗ 𝑣𝐴 ∗ 𝑣𝐵 ∗ cos 𝛼𝑖𝑚𝑝𝑎𝑐𝑡
𝑚𝐴 + 𝑚𝐴

Where,

• 𝑚𝐴 and 𝑚𝐵 are the masses of the vehicles

• 𝑣𝐴 and 𝑣𝐵 their impact speeds
• and 𝛼𝑖𝑚𝑝𝑎𝑐𝑡 represents the impact angle between both speed vectors.

The vehicle masses are essential for estimating injury risk and should be considered when
generating injury critical concrete scenarios. Constraint 3-19 of the mathematical
intersection model shows that EDV scenarios greater or equal to 0 shall be identified
during the optimisation task.

Since little research exists providing EDV values correlating to serious injury T-
intersection accidents, it has been decided not to set constraints for the upper and lower
boundaries for 𝐸_𝛥𝑣𝐴 to avoid biased results. There is a probability of missing out on
serious injury scenarios when choosing inaccurate constraints for 𝐸_𝛥𝑣𝐴 . However, the
validation part of this study conducted in step 4 will assess the injury criticality of the
generated concrete scenarios. Step 4 will allow the filtering of non-serious injury concrete
scenarios. Moreover, the use of 𝐸_𝛥𝑣𝐴 into the mathematical model ensures that there is
a covariance between the generated variables, the injury probability, and the vehicle
masses.

103
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

3.3.3 Step 2: Identification of potential injury critical concrete scenarios using NSGA-
II

The NSGA-II uses the mathematical model defined in sub-section 3.3.2 as an input.

The NSGA-II used in this thesis is an improved version of the non-dominating sorted
genetic algorithm (NSGA) and was designed to apply multi-objective search problems,
whereby the objectives can be conflicting. Non-dominating solutions express a set of
solutions where no solution is better, and no other is worst. More precisely, they represent
a compromise without degrading any search objectives, also known as pareto-optimal
solutions (Deb et al. 2002).

The NSGA-II is the first Multi-Objective Evolutionary Algorithm (MOEA), which deals
with a) the computational complexity, b) the challenge of the non-elitism approach and
c) the need for specifying a sharing parameter. However, caution must be taken with
solutions provided by MOEA, as they only represent approximations. Hence, in step 4, it
has been proposed to validate the solutions using real-world data.

The NSGA-II proposed by Deb et al. (2002) is based on the basic steps of a simple GA
(see sub-section 2.3.3.2):

• Input: Objective functions, design variables, population size 𝑛𝑝𝑜𝑝 ,

inequality/equality constraints and lower and upper boundaries of variables and
constraints.
• Output: Pareto-optimal solutions for design variables.

NSGA-II main algorithm:

The main loop is depicted in Figure 3-14, which has been adopted from Jiang et al. (2021).
The NSGA-II algorithm generates first a random initial population 𝑃𝑡 of size 𝑌 and
directly creates an offspring population 𝑄𝑡 by performing a selection, a recombination,
and a mutation (same as in a simple GA) on the individuals of 𝑃𝑡 . 𝑃𝑡 and 𝑄𝑡 are then
combined to 𝑅𝑡 as shown in Figure 3-14. Using the combined population 𝑅𝑡 for further
steps ensures the elitism of the NSGA-II algorithm. More precisely, strong individuals
from the parent population can be inherited by the next iteration. After the initialization,
the individuals in 𝑅𝑡 are sorted accordingly to their non-domination.

104
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

The best individuals from the first front are chosen accordingly to their non-domination
to create 𝑃𝑡+1. However, if the individuals of the first front build a smaller population size
compared to 𝑌, then individuals from the subsequent fronts are added to 𝑃𝑡+1until the size
of 𝑌 is achieved. Individuals of the remaining fronts are rejected. Finally, a new offspring
population 𝑄𝑡+1 can be generated by performing selection, crossover, and mutation on the
individuals of 𝑃𝑡+1 . The crowding distance is used for the selection, which is computed
using the Manhattan distance (Szabo 2015). This ensures that the computational
complexity can be reduced and that an efficient spread within the solutions is maintained.
This item has been removed due to 3rd Party Copyright. The unabridged version of the thesis can be found in the
Lanchester Library, Coventry University.

Figure 3-14: NSGA-II workflow based on the principle of non-dominated sorting

and crowding distance (Jiang et al. 2021).

Computing the distance between the solutions, the NSGA-II ensures that similar
population individuals are rejected to achieve diversification within the proposed
solutions. This loop is carried out until a stopping criterion is met.

The ‘pymoo’ code, to implement the optimisation problem from sub-section 3.3.2, and
the NSGA-II has been organised as follows: First, the problem has been coded as seen in
Listing 3-8 and second, the NSGA-II solver has been chosen as can be seen in Listing
3-9.

105
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Listing 3-8: Implementation of mathematical intersection model in ‘pymoo’ coded as a MOO

problem.

106
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Listing 3-9: Implementation of NSGA-II algorithm in ‘pymoo’ to solve the MOO problem defined
in the class ‘MyProblem(Problem)’.

Note that for the code presented in Listing 3-8, the vehicle masses to compute the mass
relation for 𝐸_𝛥𝑣𝐴 have been assumed to be 1700kg for vehicle A and 1200kg for vehicle
B. These numbers for the masses have been chosen to represent an urban intersection
accident scenario between two small cars. These numbers, however, can be replaced by
other masses if desired.

In Listing 3-9, it has been decided to use the default settings provided by the ‘pymoo’
library for the genetic operators. For the population size, the number 500 has been chosen
arbitrarily. As a result, 500 concrete scenarios can be generated. The population size can
be chosen according to the required number of concrete scenarios.

107
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

3.3.4 Step 3: Identification of potential injury critical concrete scenarios using LHS

In this study, the LHS algorithm has been used as a benchmark to compare the
performance of the NSGA-II to a random simulation algorithm. Moreover, the benchmark
allows answering RQ (2) and RQ (3).

The following parameters have been generated using the LHS algorithm:

• Vehicle velocities (𝑣𝐴 (𝑋) and 𝑣𝐵 (𝑋))

• Impact angle (𝛼𝑖𝑚𝑝𝑎𝑐𝑡 (𝑋))
• Road/Tire friction coefficients (𝜇𝐴 (𝑋) and 𝜇𝐵 (𝑋))

The LHS algorithm generated the potential injury critical concrete scenarios using the
mathematical model developed in steps 1.1 and 1.2. The models served as a filter. Using
the models is essential to compare the NSGA-II results to the LHS under the same
conditions. The MATLAB implementation presented by (Budiman 2023) has been used
as a basis and has been extended to carry out the following loop:

First, 10.000 random concrete scenarios have been generated using the LHS algorithm,
as seen in Listing 3-10.

Second, for each of these random 10.000 scenarios, the constraints of the mathematical
intersection model have been computed – see Listing 3-11.

Third, in Listing 3-12, the constraints bounds have been used as a filter to omit the random
concrete scenarios, which do not meet the scope of the mathematical intersection model.

Finally, in Listing 3-12, only 500 potential injury critical concrete scenarios have been
selected for comparison with the 500 NSGA-II scenarios.

108
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Listing 3-10: LHS implementation in MATLAB for the generation of 10.000 random concrete
scenarios.

109
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Listing 3-11: Computation of constraints equations from the mathematical model.

Listing 3-12: Application of bounds of constraints as a filter to omit the non-valid scenarios in
context of mathematical intersection model. Only 500 scenarios are kept as shown in last command.

110
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

3.3.5 Step 4: Validation of injury critical concrete scenarios

In step 4, the validation of the generated potential injury critical concrete scenarios is
reported. The first activity, in step 4, represents the classification of the concrete scenarios
generated by the NSGA-II algorithm and the LHS algorithm into serious or non-serious
injury accidents. The second activity computes the standard deviations of the generated
parameters by both algorithms to assess their spread in the scenario design space. Note
that, in sub-section 2.3.3.2 of chapter 2, the need for the validation of generated concrete
scenarios.

For the validation of the potential injury critical concrete scenarios, two different injury
probability models were applied in step 4. The models express the correlation between
the injury outcome of an accident and the Delta-V. Note that the mathematical model in
equation 3-23 computes the 𝐸_𝛥𝑣𝐴 . These 𝐸_𝛥𝑣𝐴 values can be used as an input for the
validation models used in this thesis for the injury classification of the generated concrete
scenarios. Both models were developed using real world accident data, which allows an
injury classification close to real-world behaviour.

Both models used in this thesis to estimate the injury probability of generated concrete
scenarios were also successfully used by Nitsche et al. 2018 to predict the injury
probability of randomly generated intersection accidents.

The first serious injury classification model used in this thesis was reported by the US
department of transports (DoT) (US DoT 2001). The model predicts the probability of a
MAIS3+ injury outcome (serious injury) for a given Delta-V value. It was developed
using the American NASS (National Automotive Sampling System) database. The NASS
database records Delta-V alongside its correlating injury outcomes collected during real
world accidents from EDR. From this information, different injury probability models
were developed. Since these models are based on real world data, they allow us to
empirically estimate the injury outcome of the generated concrete scenarios in this study.
Note that the injury prediction models provided in US DoT (2001) are only valid for
accidents where an evasive manoeuvre, such as braking, has taken place. This is
consistent with the concrete scenarios developed in this study, in which the vehicles are
expected to brake if they detect each other. The following equation represents the
probability model for serious injury outcome (US DoT 2001):

111
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

𝑒 0.1292∗𝑑𝑣−5.5337
𝑃𝑀𝐴𝐼𝑆3+ = 100 ∗ 3-24
1 + 𝑒 0.1292∗𝑑𝑣−5.5337

Where, 𝑃𝑀𝐴𝐼𝑆3+ is the probability for serious injury outcome, and 𝑑𝑣 represents the speed
change experienced by vehicle A during a collision, which is in this study referred to as
the 𝐸_𝛥𝑣𝐴 . The implementation of equation 3-24 in MATLAB is shown in

Listing 3-13.

Listing 3-13: MATLAB implementation to compute serious injury probability using equation 3-24

The second model stems from Gabauer and Gabler (2006), who developed an injury
regression model as a function of Delta-V, indicating the probability of a serious injury
(MAIS 3+) outcome for frontal collisions. To develop their model, only data for frontal
collisions from the NHTSA EDR database were used to design the injury model. These
collisions align with the frontal collision experienced by the target vehicle (Figure 3-10)
in rare event functional scenario 3.4.

Figure 3-15 represents in blue the injury regression model proposed by Gabauer and
Gabler (2006), expressing the correlation between the probability of a serious injury
outcome and Delta-V for frontal collisions.

112
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

This item has been removed due to 3rd Party Copyright. The unabridged version of the thesis can
be found in the Lanchester Library, Coventry University.

Figure 3-15: Serious injury model expressed in function of Delta V and serious
injury probability (Gabauer and Gabler (2006)).

Figure 3-15 shows that a longitudinal DV of 10 m/s is sufficient to ensure that a frontal
vehicle-to-vehicle crash has a 10 % probability of resulting in a serious injury outcome.
This has been used as a threshold in this study to determine whether or not a specific
scenario has the potential to cause serious injury. More precisely, the concrete scenarios
with a 𝐸_𝛥𝑣𝐴 value under 10 m/s are classified as non-relevant. Note again that the 10%
threshold was chosen from ISO 26262 suggestion on scenario classification with a
probability for human harm.

113
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

3.4 Summary of the methodology chapter

This chapter has proposed a data-driven scenario generation methodology to create a

scenario library for mixed-vehicle fleet testing of ADSs in T-intersections.

In study 1, a detailed data processing strategy for the STATS 19 data ranging from 2010-
2018 was proposed to reduce the complexity of the dataset. The data processing strategy
comprises the data preparation and the data selection based on the six levels of
PEGASUS. The data strategy has been realised in MATLAB and partially in SPSS.

Moreover, in study 1, the challenges for applying the k-medoids were introduced to
perform the 2-layer clustering. Since the number of patterns in the STATS 19 data is
unknown, identifying optimal k has been raised as a significant challenge. To
approximate optimal k in this thesis, it was proposed to use the Silhouette and the Davies
Bouldin method in the first layer. After identifying optimal k, the STATS 19 data is
partitioned into k clusters using the k-medoids algorithm and the Squared Euclidean
distance metric. The resulting high frequency clusters from the clustering in layer 1 will
be ranked using the criticality metric based on the proposed TCs concept. The higher the
occurrence of potential TCs within a cluster, the higher it is rank and relevance for ADS
testing. Subsequently, one high frequency scenario of choice will be re-clustered in study
2 using the k-medoids algorithm and the Squared Euclidean distance metric. In the
second layer, however, optimal k will be chosen by clustering the high frequency cluster
so often until each weather condition has been assigned into its own cluster. The resulting
low-frequency scenarios serve to answer RQ (1). The entire 2-layer application of the k-
medoids was realised in MATLAB.

The second study reports how to create a mathematical intersection model for rare event
functional scenario 3.4, which can be used as an input to the NSGA-II algorithm to
identify injury critical vehicle dynamics parameters. The mathematical intersection model
consists of five optimisation objectives, five variables and six constraints. The model aims
to identify concrete scenarios in the scenario design space representing high collision and
injury risks. For instance, the vehicle speeds were maximised during the optimisation task
to achieve high injury risk. Moreover, to ensure high collision risk, the crash constraint
∆𝑡𝑐𝑜𝑛𝑓𝑙𝑖𝑐𝑡 was introduced, which aimed to ensure that the vehicle dynamics generated by
the NSGA-II result in collisions. The crash constraint ∆𝑡𝑐𝑜𝑛𝑓𝑙𝑖𝑐𝑡 also uses the 𝑇𝐴 instead

114
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

of the 𝑇𝑇𝐶, which gives more accurate values for the purpose of this thesis. The
mathematical intersection model was also applied as a filter on the LHS random search
algorithm for benchmark purposes.

For the validation of the generated concrete scenarios by the NSGA-II and the LHS
algorithm, injury classification models based on real-world data were suggested. The
classification models give the correlation between Delta-V and the serious injury
probability of the potential collision. The classification of the generated potential injury
critical scenarios serves to answer RQ (2). Finally, the standard deviation for the
generated vehicle dynamics parameter is suggested to be computed to analyse whether
the NSGA-II algorithm has a better spread in the scenario design space compared to the
LHS algorithm or not. The latter activity allows answering RQ (3).

For the NSGA-II, a python implementation named ‘pymoo’ was used, and a MATLAB
implementation was considered for the LHS algorithm. The injury classification models
were incorporated in MATLAB, as well as the computation of the standard deviations.

115
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Chapter 4. Results and Discussion

4.1 Introduction

This chapter aims to introduce and discuss the results generated in study 1 and study 2.
More precisely, the results answer RQ (1) to tell whether the 2-layer application of the k-
medoids algorithm can identify rare event functional scenarios containing adverse
weather conditions. Moreover, the results serve to answer RQ (2) and RQ (3), analysing:
a) the capability of GAs to identify injury critical scenarios in a large search space and b)
how well GAs can cover the scenario design space compared to random search. This
chapter also discusses the overall impact of the used methods and tools for designing
scenario generation methodologies.

In section 4.2, the results for study 1 are introduced. Section 4.2 shows the specific results
for the settings of the k-medoids algorithm in sub-section 4.2.1. Moreover, in sub-section
4.2.2, the most relevant patterns identified in layer 1 are introduced and discussed. In sub-
section 4.2.3, the results for layer 2 are presented, showing how rare event accidents have
been mined in this thesis from the STATS 19 data. These results provide evidence to
answer RQ (1). Finally, in sub-section 4.2.3, the limitations of the 2-layer k-medoids
application are introduced to the reader and discussed.

Section 4.3 shows and discusses the results obtained in study 2. In sub-section 4.3.1, the
validation results of the potential injury critical concrete scenarios generated by the
NSGA-II and the LHS algorithm are discussed. Sub-section 4.3.1 allows answering RQ
(2). In sub-section 4.3.2, the scenario space coverage of the NSGA-II generated
parameters compared to the LHS algorithm is discussed to answer RQ (3).

Finally, in section 4.4, a snapshot of how the scenario library consisting of functional,
logical, and concrete scenarios can be represented is illustrated. Moreover, the scenario
library snapshot gives an overall view of the scenario generation methodology and
discusses the limitation discovered in this thesis for using the PEGASUS scenario
workflow for safety argumentation.

116
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

4.2 Study 1: Specification of rare event accident scenarios using STATS 19 data
4.2.1 Algorithm settings layer 1

The Silhouette and DB methods have been applied to identify optimal k for the STATS
19 data sample.

Figure 4-1 shows the results for the Silhouette method. The mean silhouette has been
computed for each iteration within the investigation range of [1 50].

Figure 4-1: Mean Silhouette value plot for the estimation of optimal k.

The x-axis represents the number of clusters k investigated, and the y-axis the mean
Silhouette values. At first glance, it can be observed that the Silhouette values for each
investigated k are greater than 0.6. According to the Silhouette theory, 0.6 indicates a
satisfying result for the STATS 19 data sample created.

Moreover, the silhouette value for k = 4 equals 1, indicating that k = 4 is amongst the best
candidates for selecting k. Furthermore, the graph also highlights that the silhouette
values between [16 - 26] are very close to 1, with the mean silhouette value for k = 24
equal to 1. MATLAB’s algorithm automatically suggests k = 24 as optimal k (see the red
dot in Figure 4-1) because it is the highest number for k having a mean Silhouette value
equal to 1. If more than one option has a k with the same Silhouette value, then the highest
number for k within the investigated range is suggested – hence, 24.

117
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Figure 4-2 shows the results for the DB values, which has been the second method to
estimate optimal k for the STATS 19 data sample. The x-axis represents the investigated
range for k, and the y-axis the mean DB values.

Figure 4-2: Mean Davies Bouldin Value plot for the estimation of optimal k.

It can be observed in Figure 4-2 that the Davies Bouldin values strongly indicate that k =
24 is the only solution. In comparison, in Figure 4-1, the Silhouette values suggested two
options, k = 4 and k = 24. However, the results for k = 4 in Figure 4-2 also give satisfying
DB values close to 0.

Considering both methods, choosing k = 24 is the better option for this thesis's STATS
19 data sample. Twenty-four clusters give more options compared to 4 for identifying
rare event functional scenarios. Moreover, more clusters will express different accident
patterns for k = 24 than for k = 4. As explained, the number of clusters k must satisfy the
user's needs. Hence, k = 4 is not a realistic option, but k = 24 is.

As a result, k = 24 has been chosen for the first layer of clustering in study 1. However,
this does not indicate that there are no further hidden patterns within these 24 clusters.
Note that the assumption has been that patterns will be hidden within the high frequency
clusters as data outliers that still need to be extracted in layer 2.

118
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

In this study, the square Euclidean distance has been chosen to compute the (dis)similarity
matrix based on recommendations from past studies. A silhouette plot has been used to
assess whether the square Euclidean distance performed well for the STATS 19 data
sample. The Silhouette plot, which should not be confused with the mean Silhouette
scatter plot in Figure 4-1, assesses the quality of the clustering results visually. More
precisely, the graphical representation of the Silhouette values highlights how well data
points have been assigned to their clusters. For instance, the wider the shape of a cluster
with a Silhouette value close to 1 within the graphical representation, the better the data
points assignment. Figure 4-3 depicts the computed Silhouette graphical plot using for k
= 24 and the k-medoids algorithm. The x-axis represents the Silhouette values, whereas
the y-axis the cluster number ranging from 1 to 24. Note again that Figure 4-3 only serves
as a first visual indicator to assess whether the distance metric chosen is acceptable for
the STATS 19 data sample before deriving functional scenarios.

Figure 4-3: Silhouette plot for square Euclidean distance method.

Each bar in Figure 4-3 represents one cluster. Hence, there are 24 bars illustrated in Figure
4-3. Each cluster has a Silhouette mean of 1 which indicates that, theoretically, each
object has been assigned to its right cluster. The visual assessment of bars number 16 –
19 and 22 – 24 in Figure 4-3 shows that they have a wide cluster shape (bar), hence the
higher the number of data populated within the cluster. It can be said that at least 7 highly

119
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

populated clusters have been identified for the T-intersection STATS 19 data. This
indicates that the square Euclidean distance performed well in building clusters.
However, to assess whether the square Euclidean distance is adequate for this thesis, it
must be evaluated whether interesting patterns have been identified from layer 1 and layer
2 for testing ADSs.

4.2.2 High frequency functional scenarios

The first layer of cluster analysis resulted in 24 high frequency functional scenarios
representing T-intersection accident patterns. The 24 clusters have been organised into
tables, which have been attached in APPENDIX C due to their large size. The most
valuable results amongst the 24 clusters for the parametrisation of scenarios containing
TCs are discussed in this section.

Firstly, two highly populated clusters have been identified amongst these 24 clusters in
the first layer. More precisely, cluster 1 and cluster 4 represent 41 % of the prepared
STATS 19 data sample vehicle records. Table 4-1 gives an overview of the
parametrisation of both clusters and the occurrence of the observations within these
clusters.

120
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Table 4-1: Functional scenarios identified in the first layer with the highest frequency in the STATS
19 data sample. The highlighted observations show how cluster 1 and cluster 4 differ.

Parameter Observations Occurrence Observations Occurrence

(cluster 1) cluster 1 [%] (cluster 4) cluster 1 [%]

1st road class unclassified 46 % A 42%

2nd road class unclassified 86 % unclassified 87%

Road type single 92 % single 95%

carriageway carriageway

Speed limit 20 – 30 mph 100 % 20 – 30 mph 100%

Area Urban 83% Urban 87%

Junction Uncontrolled 90 % Uncontrolled 93%

control junction junction

Light Daylight 100 % Daylight 100%

condition

Weather Fine 90 % Fine 92%

condition

Road surface Dry 80 % Dry 83%

condition

Vehicle Going ahead 100 % Turning right 100%

manoeuvre

1st point of Front 69% Front 45%

impact

The parametrisation in Table 4-1 shows that clusters 1 and 4 are very similar. Cluster 1
has a population size of 5200 (18% of total pop. size), whereas cluster 4 has 6656 (23%
of total pop. size) vehicle observations of the STATS 19 data samples. Both clusters
represent accidents that occurred under ideal weather and lighting conditions, which
generally have a lower probability of impacting ADS's safety than adverse weather
conditions. However, the importance of cluster analysis for parametrising functional

121
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

scenarios can be observed by the highlighted (in grey) observations in Table 4-1. In
cluster 1, the road class was ‘unclassified’, and the vehicle manoeuvre was ‘going ahead’.

In contrast, for cluster 4, the road class was an ‘A class road’, and the vehicle was ‘turning
right’. Discovering such co-variances between the parameters in the multidimensional
accident data space is essential for specifying unknown and unsafe scenarios. In this
particular case of clusters 1 and 4, test setups in virtual environments can be varied to
recreate the accident circumstances that occurred in reality when testing ADSs.

Cluster 3 (size = 2817) and cluster 6 (size = 137) both represent functional scenarios
occurring under ‘darkness’, ‘fine’ weather conditions and ‘dry’ road surface conditions.
Moreover, cluster 3 represents a ‘right turn’ scenario resulting in a ‘frontal’ crash. Cluster
6 represents, on the other hand, a ‘U-turn’ and results in an ‘offside’ crash. Again, it can
be observed that the cluster analysis gives essential insights to model functional scenarios
for ADSs testing. Both clusters, 3 and 6, could potentially be used for ADSs testing since
they represent ‘dark’ lighting conditions, which can negatively impact both ADSs and
the human driver performance (see sub-section 2.2 of chapter 2 for more information).
Cluster 9 (size = 2153) describes the same pattern as cluster 3, but with the vehicle ‘going
ahead’.

Interesting patterns have been found in clusters 14 (size = 48), 16 (size = 629) and 19
(size = 5). These patterns represent T-intersection accidents occurring under ‘darkness’
and ‘fine’ weather conditions but in ‘wet/damp’ road surface circumstances. The
correlation between not raining (‘fine’ weather) and ‘wet/damp’ road surface conditions
was representative in the parametrisation of T-intersection accidents. Such correlations
between the parameters within the STATS 19 data can be used to specify scenarios
containing potential TCs. Wet roads generally lower human driver capabilities due to
miscalculating required braking distances for crash avoidance (see sub-section 2.2 of
chapter 2).

Furthermore, wet roads can lead to sunlight reflection, hence reducing the field of view
of ADSs and human drivers. For instance, such a sunlight reflection can act as a potential
TC initiating a FIS preventing object detection, which impacts road safety. Clusters 14,
16 and 19 show ideal candidates already in the first layer of clustering for the
parametrisation of mixed-vehicle fleet scenarios, where both the human driver and ADSs

122
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

can have performance issues leading to human harm. Moreover, such scenarios
parametrised with adverse road surface conditions can be used to test camera-based road
surface estimation algorithms, which are an essential component of ADSs
(Roychowdhury et al. 2018). Cluster 14 represents a ‘left turn’, cluster 16 a ‘going ahead’
scenario and cluster 19 a ‘U-turn’ scenario. Hence, each cluster describes a new
correlation between the discovered TCs and vehicle manoeuvres, which allows for
specifying various tests.

In general, no high diversification in the parametrisation for the road infrastructure has
been discovered among the 24 clusters. More precisely, all the clusters for the 1st road
class represent ‘A class roads’ roads except clusters 1 and 11, which represent
‘unclassified roads’. It was, however, observed that for the 2nd road class, the entire 24
clusters represent ‘unclassified roads’. Moreover, amongst the 24 clusters, only one
cluster (cluster 23) represents the road type ‘dual carriage way’, whereas the remaining
clusters represent ‘single carriage way’ accident scenarios. Furthermore, the 24 clusters
represent only ‘give way/uncontrolled’ junction control accidents. No clusters for ‘stop
sign’ or auto ‘traffic signal’ junctions have been identified. Hence, from these results, it
can be stated that serious injury T-intersection accidents are likely to happen under the
above-discussed road infrastructure circumstances.

Moreover, amongst the 24 clusters, it can be observed that no adverse weather condition
scenarios have been identified, such as snowy or foggy scenarios. This finding highlights
that a further breakdown of the 24 clusters is required to reveal rare event scenarios that
occurred under adverse weather conditions.

The criticality metric has been applied to identify which of the high frequency functional
scenarios could include hidden patterns describing adverse environmental conditions.
Figure 4-4 shows the ranking results of the 24 clusters. The x-axis represents the number
of high frequency scenarios, whereas the y-axis the occurrence of TCs within a cluster as
a percentage.

123
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Figure 4-4: Criticality ranking of the 24 high frequency functional scenarios.

At first glance, it can be observed that cluster 3 (size = 2817) and cluster 9 (size = 2153)
represent scenarios having the highest frequency of TCs defined for this study. Table 4-2
shows the main differences between cluster 3 and cluster 9.

124
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Table 4-2: Parametrisation of the two highest ranked high frequency functional scenarios 3 and 9.

Parameter Observations Observations

(cluster 3) (cluster 9)

1st road class A A

2nd road class unclassified unclassified

Road type single single

carriageway carriageway

Speed limit 20 – 30 mph 20 – 30 mph

Area Urban Urban

Junction Uncontrolled Uncontrolled

control junction junction

Light condition Darkness Darkness

Weather Fine Fine

condition

Road surface Dry Dry

condition

Vehicle Turning right Going ahead

manoeuvre

1st point of Front Front

impact

Comparing clusters 3 and 9 more closely, it can be noticed that both have the same
parameter observations; however, their manoeuvre is different (see highlighted in grey in
Table 4-2). The vehicle in cluster 3 undertakes a right turn manoeuvre while the vehicle
in cluster 9 continues going ahead within the intersection. From this observation, it can
be stated that T-intersection accidents that occurred under adverse environmental
conditions are more likely to happen for right turn accidents rather than going ahead
accident scenarios. Note that this is only true for the STATS 19 data sample used in this
thesis. Although for clusters 3 and 9, the highest frequent observation of the weather is
‘fine weather’, it can be observed using the criticality metric that both clusters hide hidden

125
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

patterns for adverse weather conditions. Table 4-3 summarises the top 5 ranked high
frequency clusters.

Table 4-3: Five highest ranked critical high frequency functional scenarios among the 24 clusters.

Ranking Cluster with High Occurrence of relevant TCs

frequency scenario [%]

1 3 53.34

2 9 40.83

3 4 20.71

4 1 17.84

5 16 13.17

Since cluster 3 is the highest ranked cluster, it has been used for further analysis to
demonstrate the 2-layer application. However, it is suggested when replicating this work
to investigate all clusters from the first layer t obtain optimal results.

The process will now look in the following sub-section in detail at cluster 3 and re-cluster
it to extract hidden patterns for rare event accident scenario specification.

126
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

4.2.3 Low frequency functional scenarios

The results of the re-clustering of cluster 3 for identifying rare event functional scenarios
are reported and discussed in this section. Section 3.2.4 of the methodology chapter
highlighted that the k-medoids, in the second layer, partitions the high frequency
scenarios iteratively until each weather condition has been assigned to its own cluster.
Note that this condition has been chosen due to the relevant impact of weather conditions
on ADSs sensors (see section 2.2 in chapter 2). This condition was achieved for k = 8, as
witnessed by the bar graph in Figure 4-5. Figure 4-5 shows how each observation for the
weather condition has been assigned into its own cluster for k = 8.

Figure 4-5: Distribution of weather conditions in layer 2.

Figure 4-5 shows how remarkably low the observations for ‘snow’ and ‘fog/mist’
accidents are within Cluster 3. This low frequency in the database might be why such
accident circumstances that occurred under adverse weather conditions are challenging
to reveal in the first instance within layer 1 using clustering methods. However, this

127
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

limitation of the small population size of some of the derived sub-clusters from cluster 3
is discussed in this sub-section's last paragraph.

Since k = 8 has been used, 8 low frequency functional scenarios have been derived from
cluster 3. The low frequency functional scenarios are summarized in Table 4-4. Moreover,
the numerical clustering results of layer 2 have been organized into tables and can be seen
in APPENDIX D of this thesis.

Table 4-4: Low frequency and rare event functional scenarios.

Number Rare event functional scenario

3.1 Car A is travelling on an A-class road with a speed limit of 20 - 30 mph and
is exposed to an offside impact with car B while turning right, the merging
road is an unclassified road. The driver/passenger of Car A suffers from a
serious injury. The accident happens during darkness and fine weather
conditions. The road type is a single A-class where the road surface
condition is dry. The scenario tends to occur in an urban area within an
uncontrolled junction.

3.2 Car A is travelling on an A-class road with a speed limit of 20 - 30 mph and
is exposed to a frontal impact with car B while turning right, the merging
road is an unclassified road. The driver/passenger of Car A suffers from a
serious injury. The accident happens during darkness and rainy weather
conditions. The road type is a single A-class where the road surface
condition is wet. The scenario tends to occur in an urban area within an
uncontrolled junction.

3.3 Car A is travelling on an unclassified road with a speed limit of 20 - 30

mph and is exposed to an offside impact with car B while turning right, the
merging road is an unclassified road. The driver/passenger of Car A suffers
from a serious injury. The accident happens during darkness and fine
weather conditions. The road type is a single A-class where the road surface
condition is dry. The scenario tends to occur in an urban area within an
uncontrolled junction.

3.4 Car A is travelling on an A-class road with a speed limit of 20 - 30 mph and
is exposed to a frontal impact with car B while turning right, the merging
road is an unclassified road. The driver/passenger of Car A suffers from a
serious injury. The accident happens during darkness and snowy weather
conditions. The road type is a single A-class where the road surface

128
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

condition is wet. The scenario tends to occur in an urban area within an

uncontrolled junction.

3.5 Car A is travelling on an A-class road with a speed limit of 20 - 30 mph and
is exposed to a frontal impact with car B while turning right, the merging
road is an unclassified road. The driver/passenger of Car A suffers from a
serious injury. The accident happens during darkness and fine weather
conditions. The road type is a single A-class where the road surface
condition is dry. The scenario tends to occur in an urban area within an
uncontrolled junction.

3.6 Car A is travelling on an A-class road with a speed limit of 20 - 30 mph and
is exposed to a frontal impact with car B while turning right, the merging
road is an unclassified road. The driver/passenger of Car A suffers from a
serious injury. The accident happens during darkness and fine weather
conditions. The road type is a dual A-class where the road surface condition
is dry. The scenario tends to occur in an urban area within an uncontrolled
junction.

3.7 Car A is travelling on an A-class road with a speed limit of 20 - 30 mph and
is exposed to a frontal impact with car B while turning right, the merging
road is an unclassified road. The driver/passenger of Car A suffers from a
serious injury. The accident happens during darkness and fog. The road
type is a single A-class where the road surface condition is wet. The
scenario tends to occur in an urban area within an uncontrolled junction.

3.8 Car A is travelling on an unclassified road with a speed limit of 20 - 30 mph

and is exposed to a frontal impact with car B while turning right, the
merging road is an unclassified road. The driver/passenger of Car A suffers
from a serious injury. The accident happens during darkness and fine
weather conditions. The road type is a single A-class where the road surface
condition is dry. The scenario tends to occur in an urban area within an
uncontrolled junction.

129
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

The breakdown of Cluster 3, which resulted in 8 low frequency scenarios, revealed three
rare event accident scenarios, namely clusters 3.2, 3.4, and 3.7, that occurred under
adverse weather conditions. Note that the first layer analysis has not discovered such
parameter combinations containing adverse weather conditions. This finding justifies the
author's assumption in this thesis that potentially rare event accident scenarios are hidden
in high frequency clusters and need to be revealed in further processing. The parameter
combinations describing these rare event functional scenarios can be used to define test
scenarios containing potential TCs to stress ADSs. Moreover, these scenarios can also
negatively impact human driver performance, making them valuable for mixed-vehicle
fleet testing.

To be precise, rare event functional scenario 3.2 (size = 535) occurs under rainy
conditions. A snowy scenario has been discovered in rare event functional scenario 3.4
(size = 13) and a foggy scenario in 3.7 (size = 18). Scenarios 3.4 and 3.7 have a very low
frequency in the STATS 19 data pool. They represent 31 scenarios from the total 24
clusters population (size = 28555). This finding again shows why CAs have difficulty
extracting such rare event scenarios from large accident databases, such as the STATS 19
data.

There is the potential to classify the rare event functional scenarios as non-relevant due
to their low population in the database. The reason for the small population may be that
the accidents occurred under circumstances (darkness and snow/fog/rain) where traffic
participation is lower than usual (Roh 2020). As a result, accidents that occur under
adverse environmental conditions are not frequent in accident data and can be considered
rare.

Despite the low population size, in this thesis, it is strongly believed that the findings of
such low-populated clusters containing potential TCs are very valuable in specifying
scenarios which have been unknown previously. The identified rare event accident
scenarios can be interpreted as forecast scenarios to which ADSs will be exposed in the
future. ADSs are expected to react safely in such scenarios. Hence, the findings can be
classified as essential for testing ADSs, and RQ (1) can be answered with a clear yes. It
is possible to use CAs for rare event accident identification in accident data containing
potential TCs. However, the author of this thesis believes that these satisfying results for

130
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

study 1 have not only been obtained due to the two-layer clustering but also due to the
analytical steps to prepare the dataset. It is assumed that an essential factor for the
satisfying results has been the split of the dataset into serious injury class and analysing
only one use case (T-intersection accidents) at the time.

Accidents that occurred during fine weather conditions could have been deleted during
the data preparation steps only to identify adverse weather condition accident patterns in
the STATS 19 data. However, this thesis aimed to describe the co-variance within the
multidimensional parameter space of the STATS 19 data. Doing so, for instance, high
frequency clusters 14, 16 and 19, which happened during fine weather but included poor
road surface conditions, would not be discovered. These combinations would then stay
undiscovered, leading to a decrease in the scenario space coverage.

4.2.4 Limitations of study 1

This sub-section introduces the limitations that need to be considered when analysing the
results of study 1.

One of the main limitations of the two-layer clustering approach is the criticality measure
proposed to rank the high frequency scenarios. To compute and define the criticality
measure, the author of this thesis has chosen potential generic TCs identified in section
2.2 of chapter 2, impacting the safety of ADSs. However, for industrial applications,
choosing the potential TCs by analysing the system architecture of a specific ADS under
development is strongly suggested. For instance, if it is known that raindrops are potential
TCs for radar sensors, but sun rays are not, then only the parameter raindrop should be
considered as a potential TC for the criticality ranking of the high frequency scenarios. In
this thesis, potential TCs for different sensors known in general have been included in the
analysis, which is acceptable to demonstrate the proposed methodology in this thesis.

A limitation of using the STATS 19 dataset is that it is unknown whether any Level 0 to
Level 5 systems have been active during the T-intersection accidents. In this study, the
assumption has been that the accident patterns derived from the STATS 19 data are
representative for vehicles with no driving automation. Hence, when modelling the
scenarios, one must remember that it is not known whether the accident in the real world

131
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

has been triggered by the hazardous behaviour of a Level 0 to Level 5 system due to any
environmental condition or even due to human driver failure.

Another limitation is that it has been assumed that the environmental conditions recorded
in the accident data could be the root cause causing the T-intersection accidents. However,
it might be possible that the real root cause has not been covered by the STATS 19 data
and thus remains unknown. For instance, if a brake malfunction or a steering mechanism
malfunction caused the accidents, this information would remain unknown in STATS 19.
The results obtained from this study describe the environmental conditions of the
accidents and behaviour of the road users (such as manoeuvres) but cannot reveal the
actual root cause. However, when launching the scenarios in simulation, it might be that
parameters describing the accident patterns revealed from STATS 19 can be identified as
TCs impacting the performance of ADSs negatively.

Another limitation of the functional scenario derivation in layers 1 and 2 has been the
author's bias during the parameter selection. The author of this thesis has chosen the
parameters from the STATS 19 data, which, to the best of his knowledge, are required to
build a simulation setup in a virtual environment and can be potential TCs. However, it
might be that some parameters being the root cause of an accident have been omitted due
to a lack of the author’s experience and expertise. For instance, it might be that the driver's
age might correlate to the injury outcome of the scenarios. However, this parameter was
not added to the parameter pool for the parametrisation of the mixed-vehicle fleet
scenarios in this study.

Finally, one also must keep in mind that the recorded accidents in STATS 19 might
contain data collection errors. Studies have shown that there is a chance that the police
officers recording the accident circumstances might wrongly capture some parameters
(Ahmed, Sadullah, and Yahya 2019). Furthermore, accident witnesses may
unintentionally make wrong statements, leading to inaccurate data collection.

4.3 Identification of injury critical concrete scenarios

4.3.1 Validation of injury criticality of concrete scenarios

Figure 4-6 illustrates the MAIS3+ injury probabilities obtained by equation 3-24 reported
in sub-section 3.3.5 to validate the concrete scenarios.

132
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

The y-axis represents the frequency of the concrete scenarios, whereby the x-axis shows
the probability value for a MAIS3+ (serious) injury in [%].

Figure 4-6: Probability for serious injury outcome of the generated concrete scenarios by the
NSGA-II and the LHS.

Results generated by the NSGA-II are illustrated in blue; the orange highlighted bars
represent the LHS results. The red line shows the 10% threshold, suggested by ISO 26262,
to classify scenarios as injury relevant or not. This threshold (represented by the red line)
divides the graph in Figure 4-6 into two performance areas. The right side represents the
concrete scenarios relevant to serious injuries, while the left represents the non-relevant
area to serious injuries. The results for the injury probability in Figure 4-6 show that the
NSGA-II dominates the serious injury relevant area compared to the LHS algorithm. The
NSGA-II algorithm generated 33% of its 500 concrete scenarios in the serious injury
relevant area, whereby the LHS algorithm generated only 19% of its 500 concrete
scenarios within the serious injury relevant area.

133
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Figure 4-6 illustrates that, according to the validation model, the NSGA-II has also
identified high serious injury probability scenarios, which have not been discovered by
the LHS method. This can be observed by the bins representing the 30 to 34 % range for
MAIS3+ injury probability. Table 4-5 shows the high injury critical concrete scenarios in
more detail, identified by the NSGA-II but not by the LHS.

Table 4-5: Set of high injury scenarios identified by the NSGA-II but not by the LHS algorithm.

PMAIS3+ 𝑣𝐴 (𝑋) 𝑣𝐵 (𝑋) [𝑚⁄𝑠] 𝜇𝐴 (𝑋) 𝜇𝐵 (𝑋) 𝛼𝑖𝑚𝑝𝑎𝑐𝑡 (𝑋) [°]

[%] [𝑚⁄𝑠] [/] [/]

33.87 19.05 20.00 0.35 0.4 87.77

31.42 19.27 18.98 0.36 0.37 87.49

30.07 17.81 19.91 0.33 0.39 87.42

From Table 4-5, it can be observed that the high injury scenarios, having a high
probability for serious injury outcome, cover the upper design space of 𝑣𝐴 (𝑋) and 𝑣𝐵 (𝑋).
Hence, from this observation, it can be stated that the NSGA-II additionally identified
high speed and high injury scenarios, which have not been identified by the LHS
algorithm. Although the speeds have been in the critical bound, the road friction values
have been in the less critical bound of the design space. However, identifying such
scenarios must be taken with caution. Having such high speeds and low friction values
can be very rare in reality and is rather unlikely to happen. Nevertheless, the rare event
functional scenario matching to the concrete scenarios in Table 4-5 has a potential impact
on the field of view of ADSs sensors and human drivers due to snowy and dark
environmental conditions. Hence, in such a scenario where both vehicles do not detect
each other due to the adverse environmental condition while having a high speed, such
high speed crashes are possible and worth testing.

Also, the LHS covered one range in the scenario design space, which the NSGA-II has
not covered. Table 4-6 summarises the parameter combinations of this one specific injury
critical concrete scenario.

134
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Table 4-6: High injury critical concrete scenario identified by LHS but not by NSGA-II.

PMAIS3+ 𝑣𝐴 (𝑋) 𝑣𝐵 (𝑋) [𝑚⁄𝑠] 𝜇𝐴 (𝑋) 𝜇𝐵 (𝑋) 𝛼𝑖𝑚𝑝𝑎𝑐𝑡 (𝑋) [°]

[%] [𝑚⁄𝑠] [/] [/]

28.12 15.23 15.27 0.28 0.32 88.56

Table 4-6 shows that the discovered injury critical concrete scenario also represents a
scenario of higher velocities. However, comparing the NSGA-II scenarios from Table
4-5, the LHS scenario has a lower velocity but also has a lower friction value.

Table 4-5 and Table 4-6 show that the NSGA-II has slightly better covered the higher
injury design space. However, according to the results shown in Figure 4-6, from
validation model 1, the NSGA-II identifies a higher number of injury concrete scenarios
than the LHS.

To validate the injury criticality using validation model 2 of the generated concrete
scenarios, it was required to compute 𝐸𝛥𝑣𝐴 using equation 3-23. Using the 𝐸𝛥𝑣𝐴 , it can be
verified whether the concrete scenarios are over or under the critical threshold of 10 m/s
for 𝐸𝛥𝑣𝐴 , which has been defined to be critical. Figure 3-15, in section 3.3.5, showed that
a collision speed of 10m/s was sufficient to classify frontal vehicle-to-vehicle accidents
into serious injury scenarios accordingly to the 10% suggestions made by ISO 26262. The
results of the computation of 𝐸𝛥𝑣𝐴 are presented in the Figure 4-7 on the x-axis. The blue
bins represent the results for the NSGA-II algorithm, while the orange ones highlight the
results for the LHS. The y-axis represents the frequency of the Extended-Delta-V values
assigned to the concrete scenarios. The bar graph has been divided again into two areas
(by the red line) representing the non-serious (left) and serious (right) injury areas.

135
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Figure 4-7: Delta V values for generated concrete scenarios by the NSGA-II and LHS
algorithm. The red line represents the performance boundary of 10% to classify the
scenarios as relevant or not relevant for serious injury outcome in T-intersections.

It can be observed from Figure 4-7 that the NSGA-II algorithm has populated its concrete
scenarios into the serious injury relevant area. The LHS algorithm, however, dominates
the left side, which is less relevant to serious injuries, accordingly to validation model 2.
More precisely, 63% of the total NSGA-II concrete scenarios have been generated on the
right side, while 56% of the LHS scenarios have been generated on the left side.

More precisely, Figure 4-7 shows that the LHS algorithm generates most of its scenarios
for a bin range within 9m/s and 10m/s for Extended-Delta-V, which is irrelevant to serious
injuries. Conversely, the NSGA-II generated the most scenarios within the bin range
representing 10m/s and 11m/s for Extended-Delta-V values. Hence, the LHS method
dominates the lower Extended-Delta-V range, while the NSGA-II dominates the range
with a probability for higher Extended-Delta-V values. As a result, the LHS tends to
identify most of its scenarios in the less injury critical design space compared to the
NSGA-II, which performs efficiently in the more injury critical design space. These

136
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

observations align with the results obtained by validation model 1, which showed that the
NSGA-II tends to identify a higher number of injury critical scenarios than the LHS.

Furthermore, the bar graph in Figure 4-7 highlights that the NSGA-II has generated its
concrete scenarios in a range EDV from 5m/s – 17m/s, whereas the LHS covers a scenario
space for 5m/s 16 m/s for EDV. This again confirms the findings made by the results in
the first model that the NSGA-II covers the high injury areas more efficiently compared
to the LHS algorithm.

From both models, it can be observed that the NSGA-II, first, identifies a higher number
of injury critical concrete scenarios in the scenario design space and, second, identifies
concrete scenarios that the LHS algorithm has not identified. However, the empirical
validation using these models based on real-world accident data show that not all the 500
generated concrete scenarios are relevant to serious injury accidents. According to model
1, only 33% of the concrete scenarios suggest serious injury relevant, whereas model 2
classifies 63% as serious injury relevant. When using MOO algorithms, these findings
show that it is essential to validate the solutions against real-world information for the
injury classification of concrete scenarios.

One reason the NSGA-II outperforms the LHS in injury-critical scenario generation is its
ability to optimise its search activity. For instance, maximising 𝑣𝐴 and 𝑣𝐵 allows the
NSGA-II to cover the more critical areas more efficiently compared to the LHS, which
randomly samples the values for 𝑣𝐴 and 𝑣𝐵 . Coding the LHS algorithm, such that it
maximises or minimises objectives, is technically impossible. Hence, the NSGA-II
algorithm is a better choice for identifying injury concrete scenarios because it can
optimise its search within the scenario design space towards the more critical areas.

Note that the functional scenarios derived from the STATS 19 database correlate to real-
world circumstances. However, to extend the functional scenarios, the vehicle dynamics
generated by the NSGA-II are only theoretical variations, which theoretically correlate to
serious injury outcomes for a collision. Moreover, the covariance of the generated
parameters by the NSGA-II algorithm does not represent realistic circumstances in each
of the generated concrete scenarios. Some of the parameter combinations might represent
serious injury scenarios which are very rare in reality. For instance, Table 4-5 shows the
identification of high injury scenarios that occurred in the higher bounds of the scenario

137
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

velocity design space in combination with low friction values. Such combinations can
instead be classified as rare and thus ‘theoretical edge case scenarios’. However, they still
have importance for testing to assess system risk within the boundaries of the scenario
design space. Only by testing an ADS against the theoretically generated concrete
scenarios in a virtual environment it can be identified which theoretical parameter
variations impact the safety of ADSs and which do not. Note, that the simulation of the
scenarios was not in the scope of this thesis.

Research question 2 can be answered with a yes. However, it has been discovered that it
is essential to apply validation models to classify the NSGA-II population into relevant
and non-relevant scenarios.

4.3.2 Analysing scenario space coverage

4.3.2.1 Speeds of the vehicles

This section compares the generated velocities for both vehicles in study 1 using the
NSGA-II and the LHS algorithm.

The left bar graph in Figure 4-8 shows the velocities generated for the ego vehicle
(𝑣𝐴 (𝑋)) by using the LHS method. The right bar graph in Figure 4-8 depicts the results
generated by the NSGA-II algorithm. The y-axis represents, on each bar graph, the
frequency of the parameters 𝑣𝐴 (𝑋) , while the x-axis, as mentioned previously, represents
the velocity in m/s generated for the ego vehicle by both benchmarked algorithms.

Figure 4-8: Parameter distribution in design space of Ego velocity (vehicle A) by LHS (left) and
NSGA-II (right) algorithm.

138
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

By visually comparing the right graph (NSGA-II) to the left (LHS), one can note that the
velocities generated from the NSGA-II are uniformly distributed, which is not the case
for the LHS generated velocities. The LHS method, in comparison to the NSGA-II
method, is more likely to distribute a higher percentage of its population 𝑣𝐴 (𝑋) in the
range of 8– m/s 16.3m/s (see the red encircled area in Figure 4-8). Hence, the LHS covers
the high-speed scenario range from 16.3m/s to 20 m/s less efficiently than the NSGA-II.

When analysing the upper boundary of the design space (bin 17.6m/s to 20m/s), the
NSGA-II algorithm has generated 106 high-velocity scenarios. Whereby the LHS only
identified 28 in the identical range. Accordingly, to the generated data points, the NSGA-
II populates almost 4 times more data points in the upper boundary of the design space of
𝑣𝐴 (𝑋) compared to the LHS method. Note that this not only indicates a better coverage
of the scenario space but also indicates that the NSGA-II identifies more scenarios which
could be injury critical due to the higher speeds compared to the LHS.

For both benchmarked algorithms, the standard deviation for 𝑣𝐴 (𝑋) has been computed.
The NSGA-II population has a standard deviation of 3.6341, while the LHS population
shows a standard deviation of 2.9106. According to the computed standard deviations,
the NSGA-II achieves a better spread for 𝑣𝐴 (𝑋) compared to the LHS algorithm.

Figure 4-9 depicts the results generated for 𝑣𝐵 (𝑋) using the LHS and NSGA-II methods.
Again, the y-axis represents the frequencies of the generated velocities; the x-axis
represents the velocity design space in m/s.

Figure 4-9: Parameter distribution in design space of Target vehicle (vehicle B) velocity by
LHS (left) and NSGA-II (right) algorithm.

139
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Similar observations for 𝑣𝐵 (𝑋) can be made as for 𝑣𝐴 (𝑋). First, the LHS algorithm
concentrated its search again in the lower bounds of the design space of 𝑣𝐵 (𝑋) (see the
red encircled area in Figure 4-9) and neglected the higher bounds. Moreover, it can be
seen from Figure 4-9 that the NSGA-II again filled the design space uniformly, which has
not been the case for the LHS.

The NSGA-II has generated in the higher range (last two bins of each graph in Figure
4-9) 86 scenarios, while the LHS has generated 33 scenarios. This again highlights that
the NSGA-II covers areas which the LHS has not covered within the design space for
𝑣𝐵 (𝑋). Hence these results suggest that the NSGA-II has a better spread in the higher
design space of 𝑣𝐵 (𝑋) compared to the LHS.

The NSGA-II population achieved a standard deviation of 3.4917, and the LHS
population a standard deviation of 2.9779. Hence, it can be stated, again, that the NSGA-
II achieves a better spread in the design space of 𝑣𝐵 (𝑋) compared to LHS.

It was identified that the NSGA-II algorithm is more likely to identify injury critical
scenarios compared to the LHS algorithm. This can be confirmed by the findings made
during benchmark goal 2. The NSGA-II has a better spread in the design space for vehicle
speeds and identifies more high-speed scenarios. Note that high-speed scenarios
correlated directly to higher injury outcomes. The reason that the NSGA-II covers the
higher areas better is that it can concentrate its search in the higher bounds of the velocity
design space due to the maximization of 𝑣𝐴 (𝑋) and 𝑣𝐵 (𝑋). The LHS, however, is not
guided through the search space, as it randomly samples its data points in the design space
of 𝑣𝐴 (𝑋) and 𝑣𝐵 (𝑋).

140
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

4.3.2.2 Impact angle

This section discusses the generated impact angles for the concrete scenarios. Figure 4-10
illustrates the generated impact angles using the LHS and the NSGA-II method. The
design space for the impact angle is presented on the x-axis, whereby the y-axis represents
the frequency distribution.

Figure 4-10: Parameter distribution in design space of impact angle by LHS (left) and NSGA-II
(right) algorithm.

Both algorithms show uniformly distributed data points by visually assessing the bar
graphs in Figure 4-10. A standard deviation of 8.7549 for the LHS and 9.0248 for the
NSGA-II were computed. Since the standard deviation for the NSGA-II is greater
compared to the one of the LHS method, it can be stated that the NSGA-II method spreads
its impact angle data more efficiently in the design space.

Moreover, the bar graphs show that the LHS generated 51 low impact angle scenarios in
the range of [60 63] °, while the NSGA-II sampled 63 in the same range. Low impact
angles are essential for the scenarios generated in this study to challenge the field of view
of the human driver and ADSs sensors. The NSGA-II explored lower boundaries more
efficiently, which are assumed to be more critical in this study.

It must be said that the NSGA-II has not significantly outperformed the LHS in the impact
angle design space; however, NSGA-II has a better spread in the design space.

141
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

4.3.2.3 Road surface values

This section reports the generated road friction values for both vehicles. Figure 4-11
illustrates the generated road friction values for the Ego vehicle (vehicle A). The y-axis
represents the sample frequency of the road friction values; the x-axis represents the road
friction design space.

Figure 4-11: Parameter distribution in design space of Ego vehicle (vehicle A)

road friction by LHS (left) and NSGA-II (right) algorithm.

It can be observed that the NSGA-II has a more uniform spread in the design space,
whereby the LHS method concentrated its sampling activity in the less critical area
(higher friction values) of the design space. Moreover, it can be observed that the LHS
has not covered the entire design space compared to the NSGA-II. To be precise, the
lower design space for the road friction has not been filled with data points. The LHS's
minimum friction coefficient is 0.1592, whereas the NSGA-II has covered the whole
design space from [0.15 0.4]. In the lower search boundary of [0.15 0.2], the NSGA-II
identified 108 scenarios, while the LHS method only identified 28 scenarios. This again
demonstrates that the NSGA-II has a better spread in the design space and identifies, in
theory, more critical scenarios than the LHS. The standard deviation for the NSGA-II ego
friction value population is 0.0701, whilst the LHS population represents a standard
deviation of 0.0597. This indicates that the NSGA-II is more efficient than the LHS in
the context of design space coverage for the ego vehicle friction values.

142
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Figure 4-12 illustrates the generated friction values for the road segment on which the
target vehicle (vehicle B) travels. The y-axis represents the sample frequency of the road
friction coefficient, whereas the x-axis represents the road friction coefficient for the
Target vehicle.

Figure 4-12: Parameter distribution in design space of Target vehicle (vehicle B) road
friction by LHS (left) and NSGA-II (right) algorithm.

By assessing the graph in Figure 4-12 visually, it can be witnessed that the NSGA-II
spreads its data points uniformly in the design space, whereby the LHS method tends to
generate its data points in the less critical upper design space. This observation is similar
for 𝜇𝐴 (𝑋). However, the LHS algorithm has covered the entire design space for 𝜇𝐵 (𝑋) ,
which has not been the case for 𝜇𝐴 (𝑋). The NSGA-II generated in the lower design space
in the range of [0.15 0.2], 91 scenarios. In comparison, the LHS only covered the same
lower design space with 38 scenarios.

The standard deviation for the NSGA-II population is 0.07; the standard deviation for the
LHS population is 0.0608. It can be concluded that the NSGA-II achieves a better spread
in the design space as compared to the LHS method. For 𝜇𝐵 (𝑋) it can be stated that the
NSGA-II covers the entire design space more efficiently than the LHS method. The LHS
does not achieve an appropriate coverage in the lower road friction space.

143
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

4.3.3 Limitations and assumptions of study 2

4.3.3.1 Limitations of validation models

The validation of the potential injury critical concrete scenarios in this study includes
some limitations, which need to be considered when interpreting the results reported in
study 2.

The first model used to validate the injury probability of the concrete scenarios,
introduced by the US Department of Transport (DoT), only considered accidents from
1995 to 1999. As a result, only old vehicles have been considered to derive the prediction
model represented by equation 3-24 in sub-section 3.3.5. However, vehicle structures
today (2023) have more optimized crashworthiness performance, leading to a reduced
injury risk compared to older vehicles. Consequently, when considering the injury
probabilities derived from validation model 1, it must be considered that higher Delta-V
values would be required nowadays for accidents to result in serious injuries compared
to 1995 to 1999. Moreover, the models are limited to intersection accidents, other use
cases, and collision types (near-side, frontal, rear collision, et al.).

The injury prediction model Gabauer and Gabler (2006) provided is based on frontal
collisions with airbag deployment. Note that the impact location of the low frequency
functional scenario 3.4, which has been used for further investigation, also represents
frontal impact accidents. However, the frontal collisions considered by Gabauer and
Gabler (2006) are not only limited to T-intersections but can comprise different use cases.
Hence, it must be considered that the injury models represent not only intersection
accidents. Another essential aspect is that the data comprised unbelted and belted
occupants. Unbelted accidents, in theory, need a lower Delta-V value compared to belted
accidents to result in serious injury. The frequency distribution of the vehicle make year
is unknown in the data sample used by Gabauer and Gabler (2006). However, there are
potentially the same limitations as for the first model regarding the correlation between
newer and older vehicle structures.

Another limitation, which sets the boundary of interpretation for the derived injury
criticality concrete scenarios, is the computation of 𝐸_𝛥𝑣𝐴 . More precisely, in equation
3-23, which has been used for the computation of 𝐸_𝛥𝑣𝐴 , the vehicle speeds at the impact
are required. However, the impact speeds for the scenarios can only be witnessed when

144
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

executing the scenarios in simulation environments. The proposed method in this study
only provides critical initial speeds, which potentially can also be the impact speeds in
case the vehicles do not brake at all due to no detection in simulation. Hence, in this study,
for the computation of 𝐸_𝛥𝑣𝐴 , the initial vehicle speeds have been used instead of the
impact speeds. Using the initial speeds allows assigning the maximum potential injury
outcome to the scenarios because the impact speed is likely inferior or equal to the impact
speed.

Despite the limitations discussed above, the validation models and the computation of the
Delta-V values are appropriate to classify the accidents into not serious or serious injury
accidents.

4.3.3.2 Limitation of mathematical intersection model

The mathematical intersection model reported in sub-section 3.3.2 also has some
limitations, which need to be considered when interpreting the results presented in study
2.

The mathematical model used to solve the constrained MOO problem defined in this
thesis for injury critical concrete scenario generation also consists partly of the constraints
defined in section 3.3.2. Past research has found that applying Genetic Algorithms to
solve constrained MOO problems was very challenging (Andersson 2000). More
precisely, one of the biggest challenges using all types of Genetic Algorithms, which can
handle constraints, is the implementation and parametrisation of the search problem. The
NSGA-II requires in-depth mathematical expertise to formulate the problem
appropriately. Choosing accurate bounds for the constraints of the mathematical
intersection model was challenging in this thesis. The first limitation in this context,
which needs to be considered when interpreting the results of the injury critical concrete
scenario, are the upper and lower bounds of 𝑣𝐴 (𝑋) and 𝑣𝐵 (𝑋). To define the search space
for 𝑣𝐴 (𝑋) and 𝑣𝐵 (𝑋) , the ’Head-on (frontal)’ collision model, to predict the MAIS3+
(serious) injury outcome, has been chosen from Jurewicz et al. (2016) (see Figure 3-11 in
sub-section 3.3.2.1). However, when considering Figure 3-10, representing the generic
pictogram of the T-intersection scenarios treated in this study, it can be observed that
vehicle B has a high potential for frontal collision, while vehicle A would more likely
experience a side impact. Since the rare event functional scenario 3.4 represents a frontal

145
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

collision accident for vehicle B, the ’Head-on’ collision model was the best choice
provided by Jurewicz et al. (2016). Concisely, the bounds for vehicle A (not for vehicle
B) could be different for the mathematical model in reality compared to the bounds in
equation 3-9 and equation 3-10.

Another minor limitation related to the mathematical model is that the impact angles have
been set equal to the road skew angle of the T-intersection. In reality, both parameters
may differ from each other. Since there is less research on identifying critical impact
angles available in intersections, this assumption has been adopted.

Finally, a potential source for limitations can also be the bounds chosen for 𝑇𝐴𝐴 and 𝑇𝐴𝐵
derived from Hydén (1987). The 𝑇𝐴 is used to compute critical time gaps for ∆𝑡𝑐𝑜𝑛𝑓𝑙𝑖𝑐𝑡
in equation 3-21. The model to select 𝑇𝐴 values stems from 1987. However, vehicle
systems, responsible for steering and braking have higher capabilities in terms of reaction
and processing time compared to the vehicle systems in 1987. As a result, there is the
potential that the time remaining for a collision to happen after an evasive manoeuvre
could nowadays be shorter. Nevertheless, the Hydén (1987) model is still applied today
to calculate the Swedish Traffic Conflict Technique (TCT) (Laureshyn et al. 2017) and
has been found appropriate for the application in this study. Note that the Swedish TCT
is a technique based on the 𝑇𝐴 and the conflicting speed(𝐶𝑆) serving the classification of
accidents into serious or not serious collisions.

The limitations mentioned above are contextualised for applying the NSGA-II algorithm
and the formulation of the MOO problem to identify injury critical scenarios. The
limitations and assumptions made in this study are acceptable and have led accordingly
to the author of this thesis to acceptable results for identifying injury critical concrete
scenarios. However, if the MOO problem cannot be formulated, objective functions need
to be computed physically in a simulation environment, requiring very high
computational efforts (Coello Coello et al. 2020, Andersson 2000). In such cases, the
Latin Hypercube sampling method to perform random simulations in the search space
might be more promising concerning time and costs than the NSGA-II.

146
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

4.3.3.3 NSGA-II algorithm settings

The settings of the NSGA-II algorithm are subject to limitations too. However, these
limitations have no significant impact on the results but, if addressed, could improve the
performance of the NSGA-II algorithm. To be precise, the default settings of the NSGA-
II python implementation for the genetic operators have been used. Blank and Deb (2020)
investigated the optimal settings for the used python implementation in this thesis by
benchmarking different optimisation problems. However, the settings should be
investigated for each problem individually by performing design of experiments (DOE)
and evaluating each DOE iteration with performance indicators developed for this
purpose (Ji and Dai 2020). The advantage of using the LHS algorithm is that choosing
the optimal settings is not required, which is time consuming.

147
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

4.4 Presentation of scenario library

Figure 4-13 shows a snapshot of the generated scenario library in this thesis. It can be
witnessed that the snapshot shows the transition from a rare event functional scenario
which stems from the STATS 19 data to a logical scenario and a concrete scenario.

Figure 4-13: Transition from functional to logical and from logical to concrete scenarios adopted
from (Menzel, Bagschik, and Maurer 2018).

As discussed in Menzel, Bagschik, and Maurer (2018), under the PEGASUS umbrella,
the transition between the different scenario categories (functional, logical, and concrete),
the level of abstraction decreases, and the number of scenarios increases. In this thesis,
from one rare event functional scenario (sub-cluster 3.4), 500 potential injury critical
concrete scenarios have been generated. While evaluating the results generated in this
thesis, it has been observed that with increasing information added to the scenarios, the
number of scenarios increases, but also, the inaccuracy of the scenarios increases.
Initially, the STATS 19 dataset allows to specify functional scenarios that are close to
reality; however, due to assumptions and limitations added step by step to the scenarios

148
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

(logical and concrete), the accuracy of the scenarios has the potential to decrease. This
essential aspect discussed in this thesis needs to be considered when using the PEGASUS
model and the STATS 19 data. Knowing this limitation is necessary because it will impact
the accuracy of the risk assessment of ADSs during the safety argumentation process.

It is also crucial to understand that, in this research, the proposed scenario generation
method has been used: a) to mine meaningful scenarios from the STATS 19 data for
mixed-vehicle fleet testing and b) to extend the information gained from the STATS 19
data. Hence, not all the parameters from the functional scenario have been converted into
a logical scenario representation. Instead, the research has been focussed on creating a
scenario design space (logical scenario) for adding vehicle dynamics parameters
correlating to the functional scenario. The key parameter to ensure the correlation
between functional, logical and concrete scenarios has been the potential injury output of
the collision scenario.

As a result, to complete the scenario library, the remaining parameters from the functional
scenario, which have not been converted into a logical and concrete scenario, must be
considered. Therefore, analysing the intended ODD, for which the ADS under test is
developed and sample random data points, is suggested to extend the potential injury
critical concrete scenarios generated in this thesis.

149
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

4.5 Summary of results and discussion

The results for study 1 revealed that the Silhouette and Davies Bouldin methods suggested
using k = 24 to cluster the STATS 19 data in the first layer. Cluster 3 was identified by
the criticality metric as the most critical cluster and was hence re-clustered in the second
layer. The clustering in the second layer resulted in eight low frequency scenarios, from
which three were identified as rare event functional scenarios. These three rare event
functional scenarios represent T-intersection accidents that occurred under snow, fog,
rain, and dark lighting conditions. Furthermore, the first layer has revealed several
interesting patterns that can be used to specify TCs. For instance, accident patterns
occurring in good weather but with wet surface conditions. This finding validates the
assumption made in this thesis that accident patterns for adverse environmental
conditions are hidden in accident data. The 2-layer application of the k-medoids allows
answering RQ (1) with a ’yes’. However, limitations were raised, which must be
considered when interpreting the results. The most critical limitation is linked to the
criticality metric, where general potential TCs from the literature were chosen to compute
the criticality metric instead of defining TCs for an existing system architecture of an
ADS.

In study 2, a total of 500 potential injury concrete scenarios were generated by the NSGA-
II algorithm and the LHS algorithm. Both injury validation models based on real-world
data used in this thesis confirmed the importance of validating data generated by
metaheuristics. Figure 4-6 from Validation Model 1 shows that just 33% of the NSGA-II
generated concrete scenarios can be classed as serious injury significant, but only 19% of
the LHS created concrete scenarios can. Figure 4-7, from Validation Model 2, shows that
63% and 56% of the NSGA-II and LHS scenarios were significant to serious harm,
respectively. These findings show how important it is to validate the generated concrete
scenarios. As a result, the NSGA-II may be utilised to identify injury important concrete
scenarios and performs more effectively than the LHS according to the validation model
employed in this thesis. However, a validation must be conducted to omit the non-relevant
scenarios. In short, RQ (2) can be answered with a ‘yes’.

The computation of the standard deviations for the generated parameters has clearly
shown that the NSGA-II showed a better spread compared to the LHS in the scenario

150
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

design space. Moreover, it was witnessed from Table 4-5 that the NSGA-II covered the
critical design space in areas where the LHS did not perform its search activity. As a
result, RQ (3) can be answered with a ‘yes’ also. Nevertheless, it is important to
understand the limitations linked with the results (see sub-section 4.3.3). For instance, the
accuracy of the classification models is not up to date and needs reconsideration. A major
limitation is also linked to the design of the mathematical intersection model due to
missing available models.

For the overall scenario generation method, it was observed that extracting functional
scenarios from the STATS 19 data and designing a logical scenario to identify concrete
scenarios works well to create a mixed-vehicle fleet library. The workflow from the
PEGASUS framework has contributed to the design of the proposed method in this thesis.
However, the inaccuracy of the scenarios grows when shifting from functional to concrete
scenarios due to various assumptions made during the creation of the scenario generation
method (see

Figure 4-13). This needs to be considered when using the generated scenarios from this
thesis for the safety assessment of ADSs.

151
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Chapter 5. Conclusions and future work

5.1 Introduction

This chapter aims to conclude the proposed scenario generation method for testing ADSs
in mixed-vehicle fleet intersections. The previous chapter discussed the results obtained
relative to the specification of rare event accidents and the generation of injury critical
vehicle dynamics parameters, using the k-medoids and the NSGA-II algorithm to answer
RQ (1), RQ (2), and RQ (3). Based on these findings in chapter 4, this chapter brings all
the results together and provides an impact assessment of this thesis on the automotive
industry. Additionally, this chapter aims to give valuable suggestions for future work on
improving scenario-based testing using accident data.

This chapter is structured as follows: Section 5.2 provides a conclusion for this thesis's
proposed scenario generation method, while section 5.2 introduces the essential
contributions of this thesis to the automotive industry for scenario-based testing. In
section 5.3, based on the limitations of this thesis, suggestions for future work are made.
Finally, section 5.4 summarises this chapter to close the thesis.

5.2 Conclusions

The need to identify mixed-vehicle fleet scenarios from accident data that occurred under
adverse environmental conditions has been highlighted in this thesis. Moreover, it has
been raised that identifying such rare event accidents can be challenging because they are
likely to be hidden in accident data. Furthermore, accident data needs to be extended with
more information (vehicle kinematics) to be reused for testing ADSs in a virtual world.

In this thesis, it has been claimed that hidden patterns for rare event accidents could be
identified from the STATS 19 accident database using a systematic approach based on
the k-medoids clustering algorithm. To extend the STATS 19 scenarios with injury
critical vehicle dynamics parameters, it has been claimed that the metaheuristics-based
NSGA-II algorithm can be successful. Hence, a scenario generation methodology based
on these both algorithms has been proposed for mixed-vehicle fleet testing of ADSs
intersections.

152
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

For the application of the k-medoids and the NSGA-II algorithm for the scenario
generation problem, essential research questions (RQ (1), RQ (2) and RQ (3)) have been
raised in chapter 2.

The research questions have been answered based on the evidence generated in chapter
4, i.e., by applying the proposed scenario generation methodology to the STATS 19 data.

It has been discovered that the k-medoids can extract hidden rare event accident patterns
containing potential TCs when combined with a systematic data preparation strategy. For
the NSGA-II it has been discovered that it can be used to identify injury critical scenarios;
however, results generated by the NSGA-II need to be validated. Finally, it has been
observed that the NSGA-II has a better ability to cover the scenario design space
compared to the LHS algorithm. As a result, the proposed scenario generation method
based on the combination of both algorithms, the k-medoids and the NSGA-II, actively
contributes to addressing current challenges the automotive industry faces, which are
introduced next.

The parameter space of scenarios can be endless, resulting in millions, if not billions, of
test combinations. Hence, engineers face the problem of deciding which scenarios to test
and where to obtain these critical scenarios to test ADSs. This is where the proposed
methodology based on accident data can contribute to testing mixed-vehicle fleet
functions at T- intersections. More precisely, the 2-layer application can help scan the
STATS 19 accident database systematically for rare event accidents and highlight the
most relevant test scenarios for testing ADSs. This thesis's essential contribution and
innovation is the 2-layer approach, which brings automation into the scenario
specification process using accident data. Thanks to the automated mining of scenarios,
important rare event scenarios can be precisely selected from the large STATS 19 data,
which human engineers could not anticipate.

Moreover, in past studies, it has been observed that scenarios mined from accident data
had a higher potential for slight injuries than for serious injuries or even fatalities. The
proposed method in this thesis has the benefit of generating specific scenarios for a target
injury outcome; consequently, a higher coverage for high injury risk scenarios can be
achieved. This allows the automotive industry to focus more efficiently in the early stage
of ADSs development and reduce the injury risk in mixed-vehicle fleet intersections.

153
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Furthermore, the early identification of high injury scenarios allows engineers to test them
in a virtual environment before conducting physical tests on public roads. As a result, the
probability of identifying high-risk scenarios during public road testing has the potential
to be reduced, resulting in a safer test regime.

The STATS 19 scenarios containing the potential TCs can serve the automotive industry
to predict which situations ADSs may be exposed to and specify safety measures to
handle these forecasted situations safely. The identification of the previously unknown
patterns from STATS 19 data conducted in this thesis may also be used for the
specification of the unknown hazardous scenarios (SOTIF area 3 scenarios). This thesis's
proposed methods are not limited to ISO 21448 but can also be applied to fulfil other
standards needing a scenario-based testing approach for ADSs.

This thesis faced a core challenge: bringing the raw information from accident data into
virtual environments due to missing information, such as vehicle dynamics. Generating
such information randomly can end in scenarios with no collision and no injury risk. This
challenge is addressed by the optimisation problem formulated in this thesis to search for
injury critical intersection scenarios. Solving the optimisation problem using the NSGA-
II algorithm allowed extending the STATS 19 data with theoretical data for vehicle
dynamics parameters representing serious injury collisions at intersections. Past studies
have used similar approaches to generate additional vehicle dynamics data. However, in
this work, additional constraints have been added to the optimisation problem to
efficiently target the search for injury-critical intersection scenarios. Generating such
missing information using a guided search technique allows quickly taking macroscopic
accident data such as the STATS 19 into simulation environments.

It is important to note that the scenarios in this thesis have been generated without using
a simulation environment. Doing so, the scenarios are not biased by the capabilities of the
simulator and the ADS models. This has the advantage of generating scenarios which can
be used as a library to test different ADSs and not limited to only one ADS.

The listed contributions made within this thesis show that the proposed methodology can
accelerate V&V activities for ADSs development. More precisely, the method allows for
the automation and acceleration of some scenario generation steps needed for the test
specification of ADSs. An acceleration of such important V&V activities also allows for

154
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

estimating the risks related to ADSs in a quicker fashion, which has the potential to reduce
virtual testing efforts. A reduced testing effort for ADSs can result in quicker deployment
of ADSs for road testing and save valuable resources and costs.

It has therefore been established that the proposed modular data-driven scenario
generation method can positively contribute to addressing existing challenges faced by
the automotive industry. Since the methodology is modular and fits into the PEGASUS
scenario generation workflow, the automotive industry can adopt the proposed method.

5.3 Future work

The modularity of the proposed method also offers a great opportunity to give suggestions
for future work where required:

This thesis does not claim that the k-medoids algorithm is the best option to cluster the
STATS 19 data. Although the k-medoids algorithm satisfied the needs of this thesis, it is
suggested to benchmark other clustering algorithms to conclude which algorithms are the
most efficient for identifying hidden patterns in the STATS 19 data. However, it is
recommended to follow the suggestions made in this thesis: splitting the accident dataset
into different injury classes to reduce the multidimensional complexity of the data.
Moreover, this thesis only focused on T-intersection accidents; consequently, in the
future, it is suggested to focus on other use cases, such as roundabout or highway
scenarios.

Another aspect, which could be addressed in the future, is implementing the proposed
scenario method for the development phase of one specific ADS and investigating
whether any adjustments in the framework are necessary.

For the optimisation activity, as future work, it is suggested to update the injury
classification models to validate the concrete scenarios by the NSGA-II algorithm. The
classification models utilized in this thesis for the validation phase are 15 to 20 years old.
Vehicle structures were different back then, impacting the accidents' injury outcomes.
Structures have been optimised today; hence future studies are advised to construct injury
classification models based on the latest vehicle architectures.

Moreover, it is suggested to focus further on designing a more accurate representation of

the logical scenario. The data used to specify the design space for the vehicle speeds

155
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

𝑣𝐴 (𝑋) and 𝑣𝐵 (𝑋) and the impact angle 𝛼𝑖𝑚𝑝𝑎𝑐𝑡 (𝑋) could be improved, as they have been
selected from existing studies. It is suggested, as future work, to investigate the design
space more accurately to match real-world circumstances.

This thesis does not claim that the NSGA-II is the best option to solve the proposed MOO
problem. However, it is suggested to benchmark other optimisation algorithms based on
metaheuristics to investigate the most efficient method to solve the MOO defined in this
thesis. Also, in the context of the NSGA-II algorithm, it is suggested to research different
algorithm settings on the genetic operators to verify whether these affect the identification
of critical injury scenarios. This investigation was not undertaken, as it was not within the
scope of this thesis.

Finally, it is recommended to incorporate the proposed scenario generation method into

a simulation environment, for instance, to benchmark different sensor settings in
intersections.

5.4 Summary of conclusion and future work

The need for efficient methods to identify rare event accident scenarios has been raised
in this thesis. This thesis has proposed a new scenario generation method based on the k-
medoids and the NSGA-II algorithms to fulfil this need.

It can be concluded from the evidence generated that the proposed work contributes to
understanding the application of the k-medoids and the NSGA-II algorithm for scenario
development using accident data. A data-driven scenario generation methodology has
been proposed, which aligns with current industry standards, such as ISO 21448.
Moreover, it can be concluded that the proposed method not only contributes to
accelerating V&V activities but also provides a mean of improving the safety of ADSs at
intersections. Automating the proposed scenario generation methodology contributes to
easing the scenario specification process for engineers. Engineers will be able to select
scenarios resulting from the 2-layer clustering and quickly assess ADSs’ performance.

For future work, it is strongly advised to benchmark different clustering and optimisation
algorithms to identify more efficient scenario generation methods. Since the proposed
work in this thesis is modular, such a benchmark can be conducted without impacting the
entire proposed methodology. Moreover, using more accurate data to design the MOO

156
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

problem and developing injury validation models based on recent data is suggested.
Finally, it is suggested to use the proposed scenario generation methodology in a
simulation environment for future work.

157
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Bibliography
Abdel-Basset, M., Abdel-Fatah, L., and Sangaiah, A.K. (2018) ‘Metaheuristic
Algorithms: A Comprehensive Review’. in Computational Intelligence for
Multimedia Big Data on the Cloud with Engineering Applications.
Abdessalem, R. Ben, Nejati, S., Briand, L.C., and Stifter, T. (2016) ‘Testing Advanced
Driver Assistance Systems Using Multi-Objective Search and Neural Networks’.
ASE 2016 - Proceedings of the 31st IEEE/ACM International Conference on
Automated Software Engineering 63–74
Abdi, A., Rahmani, O., Saman, A., and Nasiri, A. (2018) Effect of Adverse Weather
Conditions on Vehicle Braking Distance of Highways. 4 (1)
Abdulhafedh, A. (2017) ‘Road Traffic Crash Data: An Overview on Sources, Problems,
and Collection Methods’. Journal of Transportation Technologies 07 (02), 206–219
Afzal, W., Torkar, R., and Feldt, R. (2009) ‘A Systematic Review of Search-Based
Testing for Non-Functional System Properties’. Information and Software
Technology [online] 51 (6), 957–976. available from
<http://dx.doi.org/10.1016/j.infsof.2008.12.005>
Aggarwal, C.C. and Zhai, C.X. (2012) ‘A Survey of Text Classification Algorithms’. in
Mining Text Data. vol. 9781461432234
Agrawal, K., Ruth, V.M., Nandini, Y., and Sravani, K. (2018) ‘Analysis of Road Accident
Locations Using DBSCAN Algorithm’. IJRST 4 (8)
Ahmed, A., Sadullah, A.F.M., and Yahya, A.S. (2019) ‘Errors in Accident Data, Its
Types, Causes and Methods of Rectification-Analysis of the Literature’. in Accident
Analysis and Prevention. vol. 130. 3–21
Akodjènou-Jeannin, M.-I., Salamatian, K., and Gallinari, P. (2007) ‘Flexible Grid-Based
Clustering’. in Kok, J.N., Koronacki, J., de Mantaras, R., Matwin, S., Mladenič, D.,
and Skowron, A. (eds.) Knowledge Discovery in Databases: PKDD 2007. held 2007
at Berlin, Heidelberg. Springer Berlin Heidelberg, 350–357
Al-jabery, K.K., Obafemi-Ajayi, T., Olbricht, G.R., and Wunsch II, D.C. (2020) ‘3 -
Clustering Algorithms’. in Computational Learning Approaches to Data Analytics
in Biomedical Applications [online] ed. by Al-jabery, K.K., Obafemi-Ajayi, T.,
Olbricht, G.R., and Wunsch II, D.C. Academic Press, 29–100. available from
<https://www.sciencedirect.com/science/article/pii/B9780128144824000036>
Ali, S., Briand, L.C., Hemmati, H., and Panesar-Walawege, R.K. (2010) ‘A Systematic
Review of the Application and Empirical Investigation of Search-Based Test Case
Generation’. IEEE Transactions on Software Engineering 36 (6), 742–762
Almjewail, Alaa, Almjewail, Aljoharah, Alsenaydi, S., ALSudairy, H., and Al-Turaiki, I.
(2018) ‘Analysis of Traffic Accident in Riyadh Using Clustering Algorithms’. in
Advances in Intelligent Systems and Computing. held 2018

158
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Amersbach, C. and Winner, H. (2019) ‘Defining Required and Feasible Test Coverage
for Scenario-Based Validation of Highly Automated Vehicles’. 2019 IEEE
Intelligent Transportation Systems Conference, ITSC 2019 425–430
Andersson, J. (2000) ‘A Survey of Multiobjective Optimization in Engineering Design’.
Optimization (January 2000), 34
Arora, J., Khatter, K., and Tushir, M. (2019) ‘Fuzzy C-Means Clustering Strategies: A
Review of Distance Measures’. in Hoda, M.N., Chauhan, N., Quadri, S.M.K., and
Srivastava, P.R. (eds.) Software Engineering. held 2019 at Singapore. Springer
Singapore, 153–162
Bagschik, G., Menzel, T., and Maurer, M. (2018) ‘Ontology Based Scene Creation for
the Development of Automated Vehicles’. IEEE Intelligent Vehicles Symposium,
Proceedings 2018-June, 1813–1820
Bakhit, P.R., Guo, B.B., and Ishak, S. (2018) ‘Crash and Near-Crash Risk Assessment of
Distracted Driving and Engagement in Secondary Tasks: A Naturalistic Driving
Study’. Transportation Research Record 2672 (38)
Bärgman, J. (2015) On the Analysis of Naturalistic Driving Data. (January 2015)
Barnard, Y. and Carsten, O. (n.d.) Field Operational Tests: Challenges and Methods.
323–332
Batsch, F., Kanarachos, S., Cheah, M., Ponticelli, R., and Blundell, M. (2020) ‘A
Taxonomy of Validation Strategies to Ensure the Safe Operation of Highly
Automated Vehicles’. in Journal of Intelligent Transportation Systems: Technology,
Planning, and Operations.
Beck, J.L. and Zuev, K.M. (2016) ‘Rare-Event Simulation’. in Handbook of Uncertainty
Quantification [online] ed. by Ghanem, R., Higdon, D., and Owhadi, H. Cham:
Springer International Publishing, 1–26. available from
<https://doi.org/10.1007/978-3-319-11259-6_24-1>
Beringhoff, F., Greenyer, J., Roesener, C., and Tichy, M. (2022) Thirty-One Challenges
in Testing Automated Vehicles : Interviews with Experts from Industry and
Research.
Bhatti, M.A. (2000) ‘Optimization Problem Formulation’. in Practical Optimization
Methods.
Bi, L. and Hu, G. (2021) ‘A Genetic Algorithm-Assisted Deep Learning Approach for
Crop Yield Prediction’. Soft Computing 25 (16)
Bill, V. (2020) SAE 2020 Government/Industry Meeting: Level-2 Driver Assistance
Systems May Be Working [online] available from
<https://www.sae.org/news/2020/01/nhtsa-and-iihs-sae-level-2-findings> [17
February 2023]
Blank, J. and Deb, K. (2020) ‘Pymoo: Multi-Objective Optimization in Python’. IEEE
Access 8
Bosch (2022) Weltpremiere: Fahrerloses Parksystem von Bosch Und Mercedes-Benz

159
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Erhält Genehmigung Für Serieneinsatz - Bosch Media Service [online] available

from <https://www.bosch-presse.de/pressportal/de/de/weltpremiere-fahrerloses-
parksystem-von-bosch-und-mercedes-benz-erhaelt-genehmigung-fuer-
serieneinsatz-248960.html> [21 February 2023]
BS ISO 26262 (2012) BSI Standards Publication Road Vehicles — Functional Safety
Part 10 : Guideline on ISO 26262.
BS ISO 26262 (2011) BSI Standards Publication Road Vehicles — Functional Safety
Part 3 : Concept Phase.
Budiman, M. (2023) Latin Hypercube Sampling - File Exchange - MATLAB Central
[online] available from
<https://www.mathworks.com/matlabcentral/fileexchange/4352-latin-hypercube-
sampling> [20 February 2023]
Cabrall, C.D.D., Lu, Z., Kyriakidis, M., Manca, L., Dijksterhuis, C., Happee, R., and de
Winter, J. (2018) ‘Validity and Reliability of Naturalistic Driving Scene
Categorization Judgments from Crowdsourcing.’ Accident; Analysis and
Prevention 114, 25–33
Cavazzuti, M. (2013) ‘Deterministic Optimization’. in Optimization Methods: From
Theory to Design Scientific and Technological Aspects in Mechanics [online] Berlin,
Heidelberg: Springer Berlin Heidelberg, 77–102. available from
<https://doi.org/10.1007/978-3-642-31187-1_4>
Cerny, D., Danhaer, J., Devillier, N., and Johansson, V. (2020) Ethics of Connected and
Automated Vehicles : Recommendations on Road Safety , Privacy , Fairness ,
Explainability and Responsibility Independent Expert Report.
Chand, A., Jayesh, S., and Bhasi, A.B. (2021) ‘Materials Today : Proceedings Road
Traffic Accidents : An Overview of Data Sources , Analysis Techniques and
Contributing Factors’. Materials Today: Proceedings [online] 47, 5135–5141.
available from <https://doi.org/10.1016/j.matpr.2021.05.415>
Chandrasekaran, B. and Josephson, J.R. (1999) What Are Ontologies , and Why Do We
Need Them ? (May 2016)
Chopard, B. and Tomassini, M. (2018) ‘An Introduction to Metaheuristics for
Optimization’. in Natural Computing Series.
Chopin, N. and Papaspiliopoulos, O. (2020) ‘Importance Sampling’. in An Introduction
to Sequential Monte Carlo [online] Cham: Springer International Publishing, 81–
103. available from <https://doi.org/10.1007/978-3-030-47845-2_8>
Cieslik, I., Expósito Jiménez, V.J., Martin, H., Scharke, H., and Schneider, H. (2022)
‘State of the Art Study of the Safety Argumentation Frameworks for Automated
Driving System’. in Trapp, M., Schoitsch, E., Guiochet, J., and Bitsch, F. (eds.)
Computer Safety, Reliability, and Security. SAFECOMP 2022 Workshops. held 2022
at Cham. Springer International Publishing, 178–191
Coello Coello, C.A., González Brambila, S., Figueroa Gamboa, J., Castillo Tapia, M.G.,
and Hernández Gómez, R. (2020) ‘Evolutionary Multiobjective Optimization: Open

160
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Research Areas and Some Challenges Lying Ahead’. Complex & Intelligent Systems
[online] 6 (2), 221–236. available from <https://doi.org/10.1007/s40747-019-0113-
4>
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., and Kuksa, P. (2011)
‘Natural Language Processing (Almost) from Scratch’. Journal of Machine
Learning Research 12
Corso, A., Moss, R.J., Koren, M., Lee, R., and Kochenderfer, M.J. (2021) ‘A Survey of
Algorithms for Black-Box Safety Validation of Cyber-Physical Systems’. Journal
of Artificial Intelligence Research 72
Dabiri, S. and Heaslip, K. (2019) ‘Developing a Twitter-Based Traffic Event Detection
Model Using Deep Learning Architectures’. Expert Systems with Applications 118
Davies, D.L. and Bouldin, D.W. (1979) ‘A Cluster Separation Measure’. IEEE
Transactions on Pattern Analysis and Machine Intelligence PAMI-1 (2)
Deb, K., Pratap, A., Agarwal, S., and Meyarivan, T. (2002) ‘A Fast and Elitist
Multiobjective Genetic Algorithm: NSGA-II’. IEEE Transactions on Evolutionary
Computation 6 (2)
Department for Transport (2021) Road Safety Data [online] available from
<https://data.gov.uk/dataset/cb7ae6f0-4be6-4935-9277-47e5ce24a11f/road-safety-
data> [20 November 2021]
Department for Transport (2013) Road Accident In-Depth Studies (RAIDS) - GOV.UK
[online] available from <https://www.gov.uk/government/publications/road-
accident-investigation-road-accident-in-depth-studies/road-accident-in-depth-
studies-raids> [2 February 2023]
Distefano, N. and Leonardi, S. (2018) ‘A List of Accident Scenarios for Three Legs
Skewed Intersections’. IATSS Research [online] 42 (3), 97–104. available from
<http://dx.doi.org/10.1016/j.iatssr.2017.07.003>
Dixon, M.-G., Genov, S., Hnatyshin, V., and Thayasivam, U. (2019) ‘Accuracy of
Clustering Prediction of PAM and K-Modes Algorithms’. in Arai, K., Kapoor, S.,
and Bhatia, R. (eds.) Advances in Information and Communication Networks. held
2019 at Cham. Springer International Publishing, 330–345
DMV (2023) Autonomous Vehicle Collision Reports - California DMV [online] available
from <https://www.dmv.ca.gov/portal/vehicle-industry-services/autonomous-
vehicles/autonomous-vehicle-collision-reports/> [2 February 2023]
Doecke, S.D., Baldock, M.R.J., Kloeden, C.N., and Dutschke, J.K. (2020) ‘Impact Speed
and the Risk of Serious Injury in Vehicle Crashes’. Accident Analysis and
Prevention 144
Doecke, S.D., Dutschke, J.K., Baldock, M.R.J., and Kloeden, C.N. (2021) ‘Travel Speed
and the Risk of Serious Injury in Vehicle Crashes’. Accident Analysis and
Prevention 161
Dozza, M. and González, N.P. (2013) ‘Recognising Safety Critical Events: Can
Automatic Video Processing Improve Naturalistic Data Analyses?’ Accident

161
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Analysis and Prevention 60

Druta, C., Kassing, A., Gibbons, R., and Andrew, V.A. (2020) ‘Assessing Driver
Behavior Using Shrp2 Adverse Weather Data’. Journal of Safety Research [online]
73, 283–295. available from <https://doi.org/10.1016/j.jsr.2020.03.013>
Dupree, S.A. and Fraley, S.K. (2002) ‘Monte Carlo Sampling Techniques’. in A Monte
Carlo Primer: A Practical Approach to Radiation Transport [online] Boston, MA:
Springer US, 21–56. available from <https://doi.org/10.1007/978-1-4419-8491-
3_2>
Eberle, U., Wachenfeld, W., and Schittenhelm, H. (2019) Pegasus Method [online]
available from <https://www.pegasusprojekt.de/en/pegasus-
method%0Ahttps://www.pegasusprojekt.de/en/>
Ekanayake, H.B., Backlund, P., Ziemke, T., Ramberg, R., Hewagamage, K.P., and
Lebram, M. (2013) ‘Comparing Expert Driving Behavior in Real World and
Simulator Contexts’. International Journal of Computer Games Technology [online]
2013, 891431. available from <https://doi.org/10.1155/2013/891431>
Esenturk, E., Wallace, A.G., Khastgir, S., and Jennings, P. (2022) ‘Identification of
Traffic Accident Patterns via Cluster Analysis and Test Scenario Development for
Autonomous Vehicles’. IEEE Access 10, 6660–6675
Ester, M. (2018) ‘Density-Based Clustering’. in Encyclopedia of Database Systems
[online] ed. by Liu, L. and Özsu, M.T. New York, NY: Springer New York, 1053–
1058. available from <https://doi.org/10.1007/978-1-4614-8265-9_605>
European Commission (2017) Final Report Summary - UDRIVE (EUropean Naturalistic
Driving and Riding for Infrastructure & Vehicle Safety and Environment) | FP7 |
CORDIS | European Commission [online] available from
<https://cordis.europa.eu/project/id/314050/reporting> [20 February 2023]
Di Fabio, U. (2017) Ethics Commission Automated and Connected Driving.
Favaro, F.M., Nader, N., Eurich, S.O., Tripp, M., and Varadaraju, N. (2017) Examining
Accident Reports Involving Autonomous Vehicles in California. 1–20
Federal Highway Administration (2004) Chapter 3 - Signalized Intersections:
Informational Guide [online] available from
<https://www.fhwa.dot.gov/publications/research/safety/04091/03.cfm> [2 October
2021]
Federal Ministry of Transport and Digital Infrastructure (2015) Stratgey for Automated
and Connected Driving [online] available from
<https://bmdv.bund.de/SharedDocs/EN/publications/strategy-for-automated-and-
connected-driving.pdf?__blob=publicationFile> [21 February 2023]
Felbinger, H., Kluck, F., Li, Y., Nica, M., Tao, J., Wotawa, F., and Zimmermann, M.
(2019) ‘Comparing Two Systematic Approaches for Testing Automated Driving
Functions’. in 2019 8th IEEE International Conference on Connected Vehicles and
Expo, ICCVE 2019 - Proceedings. held 2019
Figueiredo, M.C., Rossetti, R.J.F., Braga, R.A.M., and Reis, L.P. (2009) ‘An Approach

162
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

to Simulate Autonomous Vehicles in Urban Traffic Scenarios’. in IEEE Conference

on Intelligent Transportation Systems, Proceedings, ITSC. held 2009
Framinan, J.M., Perez-Gonzalez, P., and Fernandez-Viagas, V. (2019) ‘Deterministic
Assembly Scheduling Problems: A Review and Classification of Concurrent-Type
Scheduling Models and Solution Procedures’. European Journal of Operational
Research [online] 273 (2), 401–417. available from
<https://www.sciencedirect.com/science/article/pii/S0377221718303369>
Gabauer, D.J. and Gabler, H.C. (2006) ‘Comparison of Delta-V and Occupant Impact
Velocity Crash Severity Metrics Using Event Data Recorders’. in Annual
Proceedings - Association for the Advancement of Automotive Medicine. held 2006
Ganjali Khosrowshahi, A., Aghayan, I., Kunt, M.M., and Choupani, A.-A. (2021)
‘Detecting Crash Hotspots Using Grid and Density-Based Spatial Clustering’.
Proceedings of the Institution of Civil Engineers - Transport [online] 1–13. available
from <https://www.icevirtuallibrary.com/doi/10.1680/jtran.20.00028>
Gelder, E. De, Paardekooper, J., Saberi, A.K., Elrofai, H., and Camp, O.O. Den (2021)
Towards an Ontology for Scenario Definition for the Assessment of Automated
Vehicles : An Object-Oriented Framework. 1–16
De Gelder, E. and Paardekooper, J.P. (2017) ‘Assessment of Automated Driving Systems
Using Real-Life Scenarios’. IEEE Intelligent Vehicles Symposium, Proceedings (Iv),
589–594
Gietelink, O., De Schutter, B., and Verhaegen, M. (2006) ‘Adaptive Importance Sampling
for Probabilistic Validation of Advanced Driver Assistance Systems’. in
Proceedings of the American Control Conference. held 2006
Goldberg, D.E. (1989) ‘Genetic Algorithms in Search, Optimization, and Machine
Learning, Addison-Wesley, Reading, MA, 1989’. NN Schraudolph and J.. 3
Gomez, J. (2019) ‘Stochastic Global Optimization Algorithms: A Systematic Formal
Approach’. Information Sciences [online] 472, 53–76. available from
<https://www.sciencedirect.com/science/article/pii/S0020025517305248>
Gu, Y., Qian, Z., and Chen, F. (2016) ‘From Twitter to Detector: Real-Time Traffic
Incident Detection Using Social Media Data’. Transportation Research Part C:
Emerging Technologies 67
Guilmeau, T., Chouzenoux, E., and Elvira, V. (2021) ‘Simulated Annealing: A Review
and a New Scheme’. in IEEE Workshop on Statistical Signal Processing
Proceedings. held 2021
Gutierrez-osorio, C. (2019) ScienceDirect Modern Data Sources and Techniques for
Analysis and Forecast of Road Accidents : A Review. 7 (14076)
Hansen, P. and Mladenović, N. (2018) ‘Variable Neighborhood Search’. in Handbook of
Heuristics. vol. 1–2
Harikumar, S. (2015) ‘K-Medoid Clustering for Heterogeneous DataSets’. Procedia
Computer Science [online] 70, 226–237. available from
<https://www.sciencedirect.com/science/article/pii/S187705091503241X>

163
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Helm, M. (2021) A Deep Dive into Partitioning around Medoids [online] available from
<https://towardsdatascience.com/a-deep-dive-into-partitioning-around-medoids-
a77d9b888881>
Hjälmdahl, M. (2009) ‘Incidents in SeMiFOT’. in Co-Operation on Theories and
Concepts in Traffic Safety (ICTCT). held 2009
Hossain, M.K. and Abufardeh, S. (2019) ‘A New Method of Calculating Squared
Euclidean Distance (SED) Using PTreE Technology and Its Performance Analysis’.
Proceedings of 34th International Conference on Computers and Their
Applications, CATA 2019 58 (1), 45–54
Huang, Z., Lam, H., and Zhao, D. (2018) ‘An Accelerated Testing Approach for
Automated Vehicles with Background Traffic Described by Joint Distributions’.
IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC 2018-
March, 933–938
Hydén, C. (1987) ‘The Development of a Method for Traffic Safety Evaluation: The
Swedish Traffic Conflicts Technique’. LTH Lund
Ifthikar, A. and Hettiarachchi, S. (2018) ‘Analysis of Historical Accident Data to
Determine Accident Prone Locations and Cause of Accidents’. in Proceedings -
International Conference on Intelligent Systems, Modelling and Simulation, ISMS.
held 2018
Ignatious, H.A. and Khan, M. (2022) ‘ScienceDirect ScienceDirect An Overview of
Sensors Belgium Vehicles An Overview of Sensors in Autonomous Vehicles’.
Procedia Computer Science [online] 198 (2021), 736–741. available from
<https://doi.org/10.1016/j.procs.2021.12.315>
Ikotun, A.M., Ezugwu, A.E., Abualigah, L., Abuhaija, B., and Heming, J. (2023) ‘K-
Means Clustering Algorithms: A Comprehensive Review, Variants Analysis, and
Advances in the Era of Big Data’. Information Sciences [online] 622, 178–210.
available from
<https://www.sciencedirect.com/science/article/pii/S0020025522014633>
International Organization for Standardization (2020) ISO - ISO/TR 4804:2020 - Road
Vehicles — Safety and Cybersecurity for Automated Driving Systems — Design,
Verification and Validation [online] available from
<https://www.iso.org/standard/80363.html> [19 November 2021]
International Standardisation Organisation (2022) ISO 21448:2022 - Road Vehicles —
Safety of the Intended Functionality [online] available from
<https://www.iso.org/standard/77490.html> [1 February 2023]
Ji, H. and Dai, C. (2020) ‘A Simplified Hypervolume-Based Evolutionary Algorithm for
Many-Objective Optimization’. Complexity [online] 2020, 8353154. available from
<https://doi.org/10.1155/2020/8353154>
Jiang, R., Ci, S., Liu, D., Cheng, X., and Pan, Z. (2021) ‘A Hybrid Multi-Objective
Optimization Method Based on NSGA-II Algorithm and Entropy Weighted TOPSIS
for Lightweight Design of Dump Truck Carriage’. Machines [online] 9 (8). available
from <https://www.mdpi.com/2075-1702/9/8/156>

164
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Jin, X. and Han, J. (2017) ‘K-Medoids Clustering’. in Encyclopedia of Machine Learning

and Data Mining.
Jinkang, C., Weiwen, D., Hoaran, G., Ying, W., Jiankun, L., and Juan, D. (2022) ‘A
Survey on Data-Driven Scenario Generation for Automated Vehicle Testing’. MDPI
Johnsson, C., Laureshyn, A., and Ceunynck, T. De (2018) ‘In Search of Surrogate Safety
Indicators for Vulnerable Road Users : A Review of In Search of Surrogate Safety
Indicators for Vulnerable Road Users : A Review of Surrogate Safety Indicators’.
Transport Reviews [online] 0 (0), 1–21. available from
<https://doi.org/10.1080/01441647.2018.1442888>
Juan, A.A., Faulin, J., Grasman, S.E., Rabe, M., and Figueira, G. (2015) ‘A Review of
Simheuristics: Extending Metaheuristics to Deal with Stochastic Combinatorial
Optimization Problems’. Operations Research Perspectives [online] 2, 62–72.
available from
<https://www.sciencedirect.com/science/article/pii/S221471601500007X>
Juhnke, K., Tichy, M., and Houdek, F. (2021) ‘Challenges Concerning Test Case
Specifications in Automotive Software Testing: Assessment of Frequency and
Criticality’. Software Quality Journal [online] 29 (1), 39–100. available from
<https://doi.org/10.1007/s11219-020-09523-0>
Junietz, P.M. (2019) ‘Microscopic and Macroscopic Risk Metrics for the Safety
Validation of Automated Driving. Ph.D. Thesis’. TU Darmstadt, Darmstadt
Jurewicz, C., Commission, T.A., Sobhani, A., Woolley, J., and Dutschke, J.K. (2016)
Exploration of Vehicle Impact Speed – Injury Severity Relationships for Exploration
of Vehicle Impact Speed – Injury Severity Relationships for Application in Safer
Road Design. (April)
Kalra, N. and Paddock, S.M. (2016) Driving to Safety: How Many Miles of Driving Would
It Take to Demonstrate Autonomous Vehicle Reliability?
Karimi, A. and Siarry, P. (2012) ‘Global Simplex Optimization—A Simple and Efficient
Metaheuristic for Continuous Optimization’. Engineering Applications of Artificial
Intelligence [online] 25 (1), 48–55. available from
<https://www.sciencedirect.com/science/article/pii/S095219761100159X>
Katoch, S., Chauhan, S.S., and Kumar, V. (2021) ‘A Review on Genetic Algorithm: Past,
Present, and Future’. Multimedia Tools and Applications 80 (5)
Kaufman, L. and Rousseeuw, P.J. (1990) ‘Finding Groups in Data: An Introduction to
Cluster Analysis (Wiley Series in Probability and Statistics)’. in Eepe.Ethz.Ch. vol.
66
Kaufmann, D., Klampfl, L., Kl, F., and Zimmermann, M. (2021) Critical and
Challenging Scenario Generation Based on Automatic Action Behavior Sequence
Optimization 2 . Method for Automatic Test Scenario Gen- Eration Based on Action
Behavior Sequence Optimization. 118–127
Khastgir, S., Birrell, S., Dhadyalla, G., and Jennings, P. (2018) ‘The Science of Testing:
An Automotive Perspective’. SAE Technical Papers 2018-April, 10–12

165
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Klück, F., Wotawa, F., Neubauer, G., Tao, J., and Nica, M. (2021) ‘Analysing
Experimental Results Obtained When Applying Search-Based Testing to Verify
Automated Driving Functions’. in 2021 8th International Conference on
Dependable Systems and Their Applications (DSA). held 2021. 213–219
Klück, F., Zimmermann, M., Wotawa, F., and Nica, M. (2019) ‘Performance Comparison
of Two Search-Based Testing Strategies for ADAS System Validation’. in Lecture
Notes in Computer Science (Including Subseries Lecture Notes in Artificial
Intelligence and Lecture Notes in Bioinformatics). held 2019
Kuhn, D.R., Wallace, D.R., and Gallo, A.M. (2004) ‘Software Fault Interactions and
Implications for Software Testing’. IEEE Transactions on Software Engineering 30
(6), 418–421
Kurihata, H., Takahashi, T., Ide, I., Mekada, Y., Murase, H., Tamatsu, Y., and Miyahara,
T. (2005) ‘Rainy Weather Recognition from In-Vehicle Camera Images for Driver
Assistance’. in IEEE Intelligent Vehicles Symposium, Proceedings. held 2005
Kvasov, D.E. and Mukhametzhanov, M.S. (2018) Metaheuristic vs . Deterministic Global
Optimization Algorithms : The Univariate Case. 318, 245–259
Laureshyn, A., Goede, M. de, Saunier, N., and Fyhri, A. (2017) ‘Cross-Comparison of
Three Surrogate Safety Methods to Diagnose Cyclist Safety Problems at
Intersections in Norway’. Accident Analysis and Prevention 105
Laureshyn, A., Svensson, Å., and Hydén, C. (2010) ‘Evaluation of Traffic Safety, Based
on Micro-Level Behavioural Data: Theoretical Framework and First
Implementation’. Accident Analysis and Prevention [online] 42 (6), 1637–1646.
available from <http://dx.doi.org/10.1016/j.aap.2010.03.021>
Leitner, A. (2019) Testing and Validation of Highly Automated Systems [online] available
from
<https://www.tugraz.at/fileadmin/user_upload/Institute/IHF/Projekte/ENABLE-
S3_SummaryofResults_May2019.pdf> [14 June 2022]
Lenard, J., Badea-Romero, A., and Danton, R. (2014) ‘Typical Pedestrian Accident
Scenarios for the Development of Autonomous Emergency Braking Test Protocols’.
Accident Analysis and Prevention 73 (June 2016), 73–80
Levermore, T. and Peters, A. (2020) ‘Test Framework and Key Challenges for Virtual
Verification of Automated Vehicles : The VeriCAV Project’. 39th International
Conference on Computer Safety, Reliability and Security (SAFECOMP) [online]
available from <https://hal.archives-ouvertes.fr/hal-02931723>
Li, Y., Moreau, J., and Ibanez-guzman, J. (2022) Unconventional Visual Sensors for
Autonomous Vehicles. 1–20
Lin, M.-H., Tsai, J.-F., and Yu, C.-S. (2012) ‘A Review of Deterministic Optimization
Methods in Engineering and Management’. Mathematical Problems in Engineering
[online] 2012, 756023. available from <https://doi.org/10.1155/2012/756023>
Liu, Y., Xiao, G., Wang, M., and Li, T. (2019) ‘A Method for Flight Test Subject
Allocation on Multiple Test Aircrafts Based on Improved Genetic Algorithm’. in

166
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Aerospace Systems. vol. 2 (2)

Lourenço, F., Lobo, V., and Bação, F. (2004) ‘Binary-Based Similarity Measures for
Categorical Data and Their Application in Self-Organizing Maps’. Measurement
(Kohonen 2001), 1–18
Lu, H., Zhu, Y., Shi, K., Lv, Y., Shi, P., and Niu, Z. (2018) ‘Using Adverse Weather Data
in Social Media to Assist with City-Level Traffic Situation Awareness and Alerting’.
Applied Sciences (Switzerland) 8 (7)
Martí, E., De Miguel, M.Á., García, F., and Pérez, J. (2019) ‘A Review of Sensor
Technologies for Perception in Automated Driving’. IEEE Intelligent
Transportation Systems Magazine 11 (4), 94–108
Martí, R., Corberán, Á., and Peiró, J. (2016) ‘Scatter Search’. in Handbook of Heuristics
[online] ed. by Martí, R., Panos, P., and Resende, M.G.C. Cham: Springer
International Publishing, 1–24. available from <https://doi.org/10.1007/978-3-319-
07153-4_20-1>
McKay, M.D., Beckman, R.J., and Conover, W.J. (1979) ‘Comparison of Three Methods
for Selecting Values of Input Variables in the Analysis of Output from a Computer
Code’. Technometrics [online] 21 (2), 239–245. available from
<https://doi.org/10.1080/00401706.1979.10489755>
Menzel, T., Bagschik, G., and Maurer, A.M. (2018) ‘Scenarios for Development, Test
and Validation of Automated Vehicles’. IEEE Intelligent Vehicles Symposium,
Proceedings 2018-June, 1821–1827
Mercedes-Benz Group (2022) Mercedes-Benz Ist Vorreiter Bei Automatisierten Fahr-
Und Sicherheitstechnologien | Mercedes-Benz Group > Innovation >
Produktinnovation > Autonomes Fahren [online] available from
<https://group.mercedes-benz.com/innovation/case/autonomous/drive-pilot.html>
[21 February 2023]
Mercedes-Benz Group (2019) “Safety First for Automated Driving” (SaFAD) |
Mercedes-Benz Group > Innovation > Product Innovation > Autonomous Driving
[online] available from <https://group.mercedes-
benz.com/innovation/case/autonomous/safety-first-for-automated-driving-2.html>
[17 February 2023]
Minderhoud, M.M. and Bovy, P.H.L. (2001) ‘Extended Time-to-Collision Measures for
Road Traffic Safety Assessment’. Accident Analysis and Prevention 33 (1), 89–97
Müller, G. (2017) How Important Are Water, Snow and Ice Layers for the Friction
Measurement? [online] available from <https://www.lufft.com/blog/en/how-
important-are-water-snow-and-ice-layers-for-the-friction/> [2 October 2021]
Nalic, D. and Mihalj, T. (2020) Scenario Based Testing of Automated Driving Systems:
A Scenario Based Testing of Automated Driving Systems: A Literature Survey.
(October)
Naujoks, F. (2021) Deliverable D2 . 3 / Code of Practice for the Development of
Automated Driving Functions. (July)

167
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

NHTSA-FARS (2023) Fatality Analysis Reporting System (FARS) | NHTSA [online]

available from <https://www.nhtsa.gov/research-data/fatality-analysis-reporting-
system-fars> [2 February 2023]
NHTSA (n.d.) Automated Vehicle Safety | NHTSA [online] available from
<https://www.nhtsa.gov/technology-innovation/automated-vehicles-safety> [17
February 2023]
Nielsen, F. (2016) ‘Hierarchical Clustering BT - Introduction to HPC with MPI for Data
Science’. in Springer.
Nitsche, P. (2018) Safety-Critical Scenarios and Virtual Testing Procedures for
Automated Cars at Road Intersections.
Nitsche, P., Thomas, P., Stuetz, R., and Welsh, R. (2017) ‘Pre-Crash Scenarios at Road
Junctions: A Clustering Method for Car Crash Data’. Accident Analysis and
Prevention 107, 137–151
Nunn, J., Barnes, J., Morris, A., Petherick, E., Mackenzie, R., and Staton, M. (2018)
‘Identifying MAIS 3+ Injury Severity Collisions in UK Police Collision Records’.
Traffic Injury Prevention [online] 19 (sup2), S142–S144. available from
<https://doi.org/10.1080/15389588.2018.1532205>
OECD (n.d.) International Traffic Safety Data and Analysis Group (IRTAD) | ITF
[online] available from <https://www.itf-oecd.org/IRTAD> [2 February 2023]
Östling, M., Jeppsson, H., and Lubbe, N. (2019) ‘Predicting Crash Configurations in
Passenger Car to Passenger Car Crashes to Guide the Development of Future
Passenger Car Safety’. in Conference Proceedings International Research Council
on the Biomechanics of Injury, IRCOBI. held 2019
Paredes, J.J., Yepes, S.F., Salazar-Cabrera, R., Pachón de la Cruz, Á., and Madrid Molina,
J.M. (2022) ‘Intelligent Collision Risk Detection in Medium-Sized Cities of
Developing Countries, Using Naturalistic Driving: A Review’. Journal of Traffic
and Transportation Engineering (English Edition) 9
Patil, C. and Baidari, I. (2019) ‘Estimating the Optimal Number of Clusters k in a Dataset
Using Data Depth’. Data Science and Engineering [online] 4 (2), 132–140. available
from <https://doi.org/10.1007/s41019-019-0091-y>
Peres, F. and Castelli, M. (2021) Applied Sciences Combinatorial Optimization Problems
and Metaheuristics :
Petrovai, A., Oniga, F., and Varga, R. (2022) Enhanced Perception for Autonomous
Driving Using Semantic and Geometric Data Fusion.
Petrovic, D., Mijailović, R., and Pešić, D. (2020) ‘Traffic Accidents with Autonomous
Vehicles: Type of Collisions, Manoeuvres and Errors of Conventional Vehicles’
Drivers’. in Transportation Research Procedia. held 2020
Piao Tan, M. and A. Floudas, C. (2009) ‘Determining the Optimal Number of
ClustersDetermining the Optimal Number of Clusters’. in Encyclopedia of
Optimization [online] ed. by Floudas, C.A. and Pardalos, P.M. Boston, MA:
Springer US, 687–694. available from <https://doi.org/10.1007/978-0-387-74759-

168
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

0_123>
Pokorny, P. and Høye, A. (2022) ‘Descriptive Analysis of Reports on Autonomous
Vehicle Collisions in California: January 2021–June 2022’. Traffic Safety Research
2 (January 2021), 000011
Qidong, Z., Tong, Z., Yunshuang, Z., Chao, C., Qingyu, Z., Shuai, Z., and Zhibin, D.
(2022) ‘The Research on the Identification of ACC SOTIF Triggering Conditions
Based on Scenario Analysis’. in 2022 IEEE International Conference on Real-Time
Computing and Robotics (RCAR). held 2022. 263–266
Raffaelli, L., Vallée, F., Fayolle, G., Souza, P. De, Pfeiffer, M., Géronimi, S., Pétrot, F.,
Ahiad, S., Raffaelli, L., Vallée, F., Fayolle, G., Souza, P. De, and Rouah, X. (2016)
Facing ADAS Validation Complexity with Usage Oriented Testing To Cite This
Version : HAL Id : Hal-01277494.
Ravber, M., Liu, S.-H., Mernik, M., and Črepinšek, M. (2022) ‘Maximum Number of
Generations as a Stopping Criterion Considered Harmful’. Applied Soft Computing
[online] 128, 109478. available from
<https://www.sciencedirect.com/science/article/pii/S1568494622005804>
van Ravenzwaaij, D., Cassey, P., and Brown, S.D. (2018) ‘A Simple Introduction to
Markov Chain Monte–Carlo Sampling’. Psychonomic Bulletin & Review [online] 25
(1), 143–154. available from <https://doi.org/10.3758/s13423-016-1015-8>
Rechtien, T. (2020) Calculating Stopping Distance: Braking Is Not a Matter of Luck -
Mobilityblog [online] available from <https://mobilityblog.tuv.com/en/calculating-
stopping-distance-braking-is-not-a-matter-of-luck/> [20 November 2021]
Riedmaier, S., Ponn, T., Member, S., Ludwig, D., Schick, B., and Diermeyer, F. (2020)
Survey on Scenario-Based Safety Assessment of Automated Vehicles. 8
Riedmaier, S., Schneider, J., Danquah, B., Schick, B., and Diermeyer, F. (2021)
‘Simulation Modelling Practice and Theory Non-Deterministic Model Validation
Methodology for Simulation-Based Safety Assessment of Automated Vehicles ✩’.
Simulation Modelling Practice and Theory [online] 109 (January), 102274. available
from <https://doi.org/10.1016/j.simpat.2021.102274>
Roesener, C., Sauerbier, J., Zlocki, A., Fahrenkrog, F., Wang, L., Várhelyi, A., Breunig,
S., Mejuto, P., -Galician, C.T.A.G., Tango, F., and Lanati, J. (2017) ‘A
Comprehensive Evaluation Approach for Highly Automated Driving’. 25th
Enhanced Safety of Vehicles 2017 [online] 1–13. available from <https://www-
esv.nhtsa.dot.gov/Proceedings/25/25ESV-000259.pdf>
Roh, H.-J. (2020) ‘Assessing the Effect of Snowfall and Cold Temperature on a
Commuter Highway Traffic Volume Using Several Layers of Statistical Methods’.
Transportation Engineering [online] 2, 100022. available from
<https://www.sciencedirect.com/science/article/pii/S2666691X20300233>
Ronen, A., Agassi, E., and Yaron, O. (2021) ‘Sensing with Polarized Lidar in Degraded
Visibility Conditions Due to Fog and Low Clouds’. Sensors 21 (7)
Rosique, F., Navarro, P.J., Fernández, C., and Padilla, A. (2019) ‘A Systematic Review

169
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

of Perception System and Simulators for Autonomous Vehicles Research’. Sensors

(Switzerland) 19 (3)
Rousseeuw, P.J. (1987) ‘Silhouettes: A Graphical Aid to the Interpretation and Validation
of Cluster Analysis’. Journal of Computational and Applied Mathematics 20 (C),
53–65
Roychowdhury, S., Zhao, M., Wallin, A., Ohlsson, N., and Jonasson, M. (2018) ‘Machine
Learning Models for Road Surface and Friction Estimation Using Front-Camera
Images’. in 2018 International Joint Conference on Neural Networks (IJCNN). held
2018. 1–8
SAE Industry Technologies Consortia (2020) ‘AVSC Best Practice for Describing an
Operational Design Domain: Conceptual Framework and Lexicon’.
Avsc00002202004
SAE International (2021) ‘SAE J3016 Levels of Driving Automation’. Society of
Automotive Engineers
Saini, P., Kaur, J., and Lamba, S. (2021) ‘A Review on Pattern Recognition Using
Machine Learning’. in Manik, G., Kalia, S., Sahoo, S.K., Sharma, T.K., and Verma,
O.P. (eds.) Advances in Mechanical Engineering. held 2021 at Singapore. Springer
Singapore, 619–627
Sammon, J.W. (1969) ‘A Nonlinear Mapping for Data Structure Analysis’. IEEE
Transactions on Computers C–18 (5)
Sander, U. and Lubbe, N. (2018) ‘The Potential of Clustering Methods to Define
Intersection Test Scenarios: Assessing Real-Life Performance of AEB’. Accident
Analysis and Prevention [online] 113 (July 2017), 1–11. available from
<https://doi.org/10.1016/j.aap.2018.01.010>
Sarhani, M., Voß, S., and Jovanovic, R. (2022) ‘Initialization of Metaheuristics:
Comprehensive Review, Critical Analysis, and Research Directions’. International
Transactions in Operational Research 0, 1–37
Van Schagen, I. and Sagberg, F. (2011) Towards a Large Scale European Naturalistic
Driving Study : Final Report of PROLOGUE.
Sharath, M.N. and Mehran, B. (2021) A Literature Review of Performance Metrics of
Automated Driving Systems for On-Road Vehicles. 2 (November), 1–19
Silva, R.A., Senger de Souza, S. do R., and Lopes de Souza, P.S. (2017) ‘A Systematic
Review on Search Based Mutation Testing’. Information and Software Technology
81
Spicer, R., Vahabaghaie, A., Bahouth, G., Drees, L., Martinez von Bülow, R., and Baur,
P. (2018) ‘Field Effectiveness Evaluation of Advanced Driver Assistance Systems’.
Traffic Injury Prevention 19 (sup2)
Svensson, Å. (1998) A Method for Analysing the Traffic Process in a Saftey Perspective.
University of Lund
Szabo, F.E. (2015) ‘The Linear Algebra Survival Guide’. in The Linear Algebra Survival

170
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Guide [online] ed. by Szabo, F.E. Boston: Academic Press, 219–233. available from
<https://www.sciencedirect.com/science/article/pii/B9780124095205500205>
Szénási, S. (2021) ‘Analysis of Historical Road Accident Data Supporting Autonomous
Vehicle Control Strategies’. PeerJ Computer Science 7, 24–25
Tan, H., Zhao, F., Hao, H., Liu, Z., Amer, A.A., and Babiker, H. (2020) ‘Automatic
Emergency Braking (AEB) System Impact on Fatality and Injury Reduction in
China’. International Journal of Environmental Research and Public Health 17 (3)
Todorovic, M. and Simic, M. (2019) ‘Clustering and Pattern Recognition in
Bioengineering and Autonomous Systems’. Procedia Computer Science [online]
159, 2364–2373. available from
<https://www.sciencedirect.com/science/article/pii/S1877050919316151>
US DoT (2001) ‘Tire Pressure Monitoring System Fmvss No. 138 Preliminary Economic
Assessment’. Traffic Safety (138)
Vargas, J., Alsweiss, S., Toker, O., Razdan, R., and Santos, J. (2021) An Overview of
Autonomous Vehicles Sensors and Their. 1–22
Virginia Tech Transportation Institute (2020) SHRP2 Naturalistic Driving Study [online]
available from <https://insight.shrp2nds.us/> [20 February 2023]
Wachenfeld, W. and Winner, H. (2015) ‘Autonomes Fahren’. Autonomes Fahren (April
2016)
Wendt, Z. and Cook, J.S. (2018) Saved by the Sensor: Vehicle Awareness in the Self-
Driving Age | Machine Design [online] available from
<https://www.machinedesign.com/mechanical-motion-
systems/article/21836344/saved-by-the-sensor-vehicle-awareness-in-the-
selfdriving-age> [31 March 2023]
Whitley, D. and Sutton, A.M. (2012) ‘Genetic Algorithms --- A Survey of Models and
Methods’. in Handbook of Natural Computing [online] ed. by Rozenberg, G., Bäck,
T., and Kok, J.N. Berlin, Heidelberg: Springer Berlin Heidelberg, 637–671.
available from <https://doi.org/10.1007/978-3-540-92910-9_21>
Wong, E. (2011) ‘Abbreviated Injury Scale’. in Encyclopedia of Clinical
Neuropsychology [online] ed. by Kreutzer, J.S., DeLuca, J., and Caplan, B. New
York, NY: Springer New York, 5–6. available from <https://doi.org/10.1007/978-0-
387-79948-3_2>
World Health Organization (2022) Road Traffic Injuries [online] available from
<https://www.who.int/news-room/fact-sheets/detail/road-traffic-injuries> [17
February 2023]
Wu, H., Nie, C., Petke, J., Jia, Y., and Harman, M. (2020) ‘An Empirical Comparison of
Combinatorial Testing, Random Testing and Adaptive Random Testing’. IEEE
Transactions on Software Engineering 46 (3)
Xia, Q., Chai, Y., Lv, H., and Shu, H. (2021) Research on Accelerated Testing of Cut-In
Condition of Electric Automated Vehicles Based on Monte Carlo Simulation. 1–11

171
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Xu, D. and Tian, Y. (2015) ‘A Comprehensive Survey of Clustering Algorithms’. Annals

of Data Science [online] 2 (2), 165–193. available from
<https://doi.org/10.1007/s40745-015-0040-1>
Yang, X.-S. (2021) ‘Chapter 8 - Particle Swarm Optimization’. in Nature-Inspired
Optimization Algorithms (Second Edition) [online] Second Edi. ed. by Yang, X.-S.
Academic Press, 111–121. available from
<https://www.sciencedirect.com/science/article/pii/B9780128219867000159>
Yang, X.S. (2014) Nature-Inspired Optimization Algorithms.
Yannis, G., Dragomanovits, A., Laiou, A., and Basta, O. (2020) ‘Road Safety Modelling :
Macroscopic and Microscopic Approach’. Proceedings of 8th Transport Research
Arena TRA 2020 [online] 8–11. available from
<https://www.nrso.ntua.gr/geyannis/wp-content/uploads/geyannis-pc370.pdf>
Yeong, D.J., Velasco-hernandez, G., Barry, J., and Walsh, J. (2021) ‘Sensor and Sensor
Fusion Technology in Autonomous Vehicles: A Review’. Sensors 21 (6), 1–37
Yoneda, K., Suganuma, N., Yanase, R., and Aldibaja, M. (2019) ‘Automated Driving
Recognition Technologies for Adverse Weather Conditions’. IATSS Research
[online] 43 (4), 253–262. available from
<https://doi.org/10.1016/j.iatssr.2019.11.005>
Zang, S., Ding, M., Smith, D., Tyler, P., and Rakotoarivelo, T. (2019) ‘The Impact of
Adverse Weather Conditions on Autonomous Vehicles’. IEEE (March), 103–111
Zhan, L. and Northrop, W.F. (2021) ‘Impact of Fog Particles on 1.55 Μm Automotive
LiDAR Sensor Performance: An Experimental Study in an Enclosed Chamber’. in
SAE Technical Papers. held 2021
Zhang, S., Peng, H., Zhao, D., and Tseng, H.E. (2018) Accelerated Evaluation of
Autonomous Vehicles in the Lane Change Scenario Based on Subset Simulation
Technique. 3935–3940
Zhang, X., Tao, J., Törngren, M., and Sanchez, J. (2021) Finding Critical Scenarios for
Automated Driving Systems : A Systematic Literature Review. 1–37
Zhang, Y., Carballo, A., Yang, H., and Takeda, K. (2023) ‘Perception and Sensing for
Autonomous Vehicles under Adverse Weather Conditions: A Survey’. ISPRS
Journal of Photogrammetry and Remote Sensing [online] 196, 146–177. available
from <https://www.sciencedirect.com/science/article/pii/S0924271622003367>
Zhang, Y., Carballo, A., Yang, H., and Takeda, K. (2021) Perception and Sensing for
Autonomous Vehicles Under Adverse Weather Conditions : A Survey.

172
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

APPENDICES
APPENDIX A: PEGASUS framework

Figure_APDX A shows the entire PEGASUS method for assessing highly automated
driving functions (SAE Level 3 ≥). It shows the importance of collecting and generating
the right and critical scenarios to accelerate V&V activities. It has been retrieved from:
https://www.pegasusprojekt.de/files/tmpl/Pegasus-Abschlussveranstaltung/PEGASUS-
Gesamtmethode.pdf.

173
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving systems in a mixed traffic environment

Figure_APDX A: Schematic PEGASUS workflow for the assessment of SAE Level 3 or greater systems.

174
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

APPENDIX B: STATS 19 data preparation code

Appendix B shows the data preparation code used to prepare the STATS 19 data.

175
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving systems in a mixed traffic environment

APPENDIX C: Layer 1 high frequency scenarios

Appendix C presents the clustering results in tables (Table_APDX1 – Table APDX4) for the first layer, using the k-medoids algorithm on
the STATS 19 data. For each table, the cluster centres have been highlighted in yellow. For example, in Table_APDX 1 for cluster 1, the 1st
road class is likely to be unclassified and hence, highlighted in yellow as the cluster centre. The cluster centres can then be used to build the
functional scenarios.

Table_APDX 1: Clustering results for layer 1 for cluster 1 to cluster 7

Attributes Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Cluster 6 Cluster 7

Total observations: 5200 439 2817 6656 2105 137 1486

1st road class:

Motorway 0 0 0 0 4 0 0

A(M) 0 0 0 0 0 0 0

A 1664 200 1209 2777 1186 63 668

B 732 48 440 1028 414 19 186

C 411 44 231 563 203 13 102

Unclassified 2393 147 937 2288 298 42 530

2nd road class:

Motorway 3 0 1 7 7 0 0

A(M) 0 0 1 0 0 0 0

176
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving systems in a mixed traffic environment

A 284 37 152 292 161 10 138

B 177 18 104 238 251 6 61

C 241 24 154 334 331 8 78

Unclassified 4495 360 2405 5785 1355 113 1209

Road type:

Roundabout 6 0 3 0 4 0 0

One-way street 91 10 30 64 4 4 43

Dual carriageway 276 41 109 269 2088 8 110

Single carriageway 4810 386 2673 6313 9 125 1327

Slip road 17 2 2 10 0 0 6

Speed limit:

20-30 5200 439 2817 6656 0 137 1486

40-50 0 0 0 0 0 0 0

60-70 0 0 0 0 2105 0 0

Area:

Urban 4309 402 2544 5736 58 127 1329

Rural 891 37 273 920 2047 10 157

Junction control:

Auto traffic signal 444 40 278 437 27 11 115

Stop sign 37 4 17 48 24 0 8

177
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving systems in a mixed traffic environment

Give way or uncontrolled 4719 395 2522 6171 2054 126 1363

Light condition:

Daylight 5200 0 0 6656 2105 0 1486

Darkness 0 439 2817 0 0 137 0

Weather condition:

Fog or mist 7 4 18 14 20 1 3

Fine 4683 358 2251 6093 1856 111 1370

Raining 482 73 535 535 218 24 110

Snow 28 4 13 14 11 1 3

Road surface condition:

Dry 4209 281 1692 87

5468 1565 1251
Wet or damp 942 151 1096 49
1169 504 229
Snow
20 2 4 3 9 0 3
Frost or ice
28 5 24 15 27 1 3
Flood
1 0 1 1 0 0 0
Oil or diesel
0 0 0 0 0 0 0
Mud
0 0 0 0 0 0 0

Vehicle manoeuvre:

Turning left 0 439 0 0 0 0 1486

178
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving systems in a mixed traffic environment

Reversing 0 0 0 0 0 31 0

U-turn 0 0 0 0 0 106 0

Turning right 0 0 2817 6656 0 0 0

Going ahead 5200 0 0 0 2105 0 0

1st point of impact:

Front 3608 227 1365 2990 1706 24 583

Back 181 19 111 336 38 28 119

Offside 626 95 762 2009 172 69 276

Nearside 785 98 579 1321 189 16 508

179
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving systems in a mixed traffic environment

Table_APDX 2: Clustering results for layer 1 for cluster 8 to cluster 14

Attributes Cluster 8 Cluster 9 Cluster 10 Cluster 11 Cluster 12 Cluster 13 Cluster 14

Total of observations: 1301 2153 1892 347 1442 151 48

1st Road class:

Motorway 0 2 2 0 1 0 0

A(M) 0 0 0 0 0 0 0

A 830 841 1200 143 959 86 34

B 251 348 348 37 283 24 1

C 53 188 171 23 60 14 5

Unclassified 167 774 171 144 139 27 8

2nd Road class:

Motorway 5 4 4 0 6 0 0

A(M) 1 1 0 0 1 0 0

A 75 122 145 22 82 23 6

B 138 92 237 11 130 20 12

C 126 100 330 11 145 23 3

Unclassified 956 1834 1176 303 1078 85 27

Road type:

Roundabout 0 4 1 0 1 0 0

180
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving systems in a mixed traffic environment

One-way street 1 33 1 5 4 1 0

Dual carriageway 247 145 157 29 199 16 12

Single carriageway 1051 1969 1726 312 1236 130 36

Slip road 2 2 7 1 2 4 0

Speed limit:

20-30 0 2153 0 347 0 0 0

40-50 1301 0 0 0 1442 0 0

60-70 0 0 1892 0 0 151 48

Area:

Urban 442 1885 73 313 525 7 6

Rural 859 268 1819 34 917 144 42

Junction control:

Auto traffic signal 149 239 20 25 97 0 0

Stop sign 24 21 24 1 17 2 0

Give way or uncontrolled 1128 1893 1848 321 1328 149 48

Light condition:

Daylight 1301 0 1892 347 1442 151 0

Darkness 0 2153 0 0 0 0 48

Weather condition:
7 2
Fog or mist 7 15 15 1 1

181
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving systems in a mixed traffic environment

Fine 1161 1743 1732 323 1333 136 33

Raining 127 383 143 21 98 11 14

Snow 6 12 2 2 4 2 0

Road surface condition:

Dry 20
993 1303 1520 296 1171 126
Wet or damp 25
292 811 361 49 264 21
Snow 0
6 8 2 2 2 2
Frost or ice 3
10 27 8 0 5 2
Flood 0
0 4 1 0 0 0
Oil or diesel 0
0 0 0 0 0 0

Mud 0
0 0 0 0 0 0

Vehicle manoeuvre:
0 0 0 0 0 151 48
Turning left
0 0 0 99 0 0 0
Reversing
0 0 0 248 0 0 0
U- turn
0 0 1892 0 1442 0 0
Turning right
1301 2153 0 0 0 0 0
Going ahead

1ST point of impact:

Front 1035 1588 734 67 576 67 19

Back 42 67 183 97 130 24 10

182
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving systems in a mixed traffic environment

Offside 112 245 738 137 505 45 17

Nearside 112 253 237 46 231 15 2

183
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving systems in a mixed traffic environment

Table_APDX 3: Clustering results for layer 1 for cluster 15 to cluster 21

Attributes Cluster 15 Cluster 16 Cluster 17 Cluster 18 Cluster 19 Cluster 20 Cluster 21

Total of observations: 41 629 476 439 5 492 56

1st Road class:

Motorway 0 3 0 0 0 0 0

A(M) 0 0 0 0 0 0 0

A 28 382 316 297 4 340 41

B 6 112 95 69 0 81 6

C 3 54 21 41 1 23 4

Unclassified 4 78 44 32 0 48 5

2nd Road class:

Motorway 0 2 1 1 0 0 0

A(M) 0 0 0 0 0 0 0

A 1 53 21 41 0 29 3

B 1 86 48 52 1 46 2

C 1 108 49 91 1 47 9

Unclassified 38 380 357 254 3 370 42

Road type:

Roundabout 0 0 1 0 0 0 0

184
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving systems in a mixed traffic environment

One-way street 0 2 1 0 0 4 0

Dual carriageway 6 71 112 45 1 99 19

Single carriageway 35 550 361 390 4 388 37

Slip road 0 6 1 4 0 1 0

Speed limit:

20-30 0 0 0 0 0 0 0

40-50 0 0 476 0 0 492 56

60-70 41 629 0 439 5 0 0

Area:

Urban 3 27 212 30 0 241 31

Rural 38 602 264 409 5 251 25

Junction control:

Auto traffic signal 2 15 81 12 0 74 8

Stop sign 0 6 8 3 0 9 0

Give way or uncontrolled 39 608 387 424 5 409 48

Light condition:

Daylight 41 0 0 0 0 0 0

Darkness 0 629 476 439 5 492 56

Weather condition:

Fog or mist 1 11 9 5 1 6 1

185
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving systems in a mixed traffic environment

Fine 36 475 377 355 3 399 44

Raining 3 134 87 79 1 85 10

Snow 1 9 3 0 0 2 1

Road surface condition:

36 299 260 238 281 33
Dry 2
5 309 211 197 207 22
Wet or damp 3
0 9 2 0 2 1
Snow 0
0 10 3 4 2 0
Frost or ice 0
0 2 0 0 0 0
Flood 0
0 0 0 0 0 0
Oil or diesel 0

Mud 0 0 0 0 0 0 0

Vehicle manoeuvre:

Turning left 0 0 0 0 0 0 56

Reversing 4 0 0 0 1 0 0

U- turn 37 0 0 0 4 0 0

Turning right 0 0 0 439 0 492 0

Going ahead 0 629 476 0 0 0 0

1st point of impact:

Front 7 514 375 192 2 219 23

Back 3 10 14 24 0 20 7

Offside 28 54 53 167 3 172 10

186
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving systems in a mixed traffic environment

Nearside 3 51 34 56 0 81 16

187
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Table_APDX 4: Clustering results for layer 1 for cluster 22 to cluster 24

Attributes Cluster 22 Cluster 23 Cluster 24

Total of observations 43 185 15

1st Road class:

Motorway 0 0 0

A(M) 0 0 0

A 30 126 12

B 4 28 2

C 2 8 0

Unclassified 7 23 1

2nd Road class:

Motorway 0 0 0

A(M) 0 0 0

A 1 20 0

B 3 11 0

C 3 14 0

Unclassified 36 140 15

Road type:

Roundabout 0 0 0

One-way street 0 1 0

Dual carriageway 11 52 8

Single carriageway 32 129 7

Slip road 0 3 0

Speed limit:

20-30 0 0 0

40-50 43 185 15

60-70 0 0 0

Area:

Urban 18 90 11

Rural 25 95 4

188
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Junction control:

Auto traffic signal 0 18 5

Stop sign 0 1 0

Give way or uncontrolled 43 166 10

Light condition:

Daylight 43 185 0

Darkness 0 0 15

Weather condition:

Fog or mist 0 2 0

Fine 41 167 12

Raining 1 16 3

Snow 1 0 0

Road surface condition:

Dry 38 145 8

Wet or damp 4 40 7

Snow 1 0 0

Frost or Ice 0 0 0

Flood 0 0 0

Oil or diesel 0 0 0

Mud 0 0 0

Vehicle manoeuvre:

Turning left 0 185 0

Reversing 3 0 2

U- turn 40 0 13

Turning right 0 0 0

Going ahead 0 0 0

1st point of impact:

Front 4 59 3

Back 5 29 3

Offside 25 55 8

Nearside 9 42 1

189
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving systems in a mixed traffic environment

APPENDIX D: Layer 2 low frequency scenarios and rare event scenarios

Appendix D presents the clustering results for layer 2 in Table_ADPX 5 to Table_APDX 6. The clusters in the second layer have been derived
by re-clustering Cluster 3 obtained from the first layer.

Table_APDX 5: Re-clustering results of cluster 3 showing clusters 3.1 to 3.6.

Attributes Cluster 3 Cluster 3.1 Cluster 3.2 Cluster 3.3 Cluster 3.4 Cluster 3.5 Cluster 3.6

Total of observations 2817 634 535 475 13 601 113

1st Road class:

Motorway 0 0 0 0 0 0 0

A(M) 0 0 0 0 0 0 0

A 1209 477 217 0 6 412 89

B 440 157 84 0 2 189 5

C 231 0 43 105 0 0 8

Unclassified 937 0 191 370 5 0 11

2nd Road class:

Motorway 1 0 0 0 0 1 0

A(M) 1 0 0 0 0 0 1

A 152 30 32 14 1 39 15

B 104 30 21 7 0 39 5

C 154 42 29 13 0 50 8

190
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving systems in a mixed traffic environment

Unclassified 2405 532 453 441 12 472 84

Road type:

Roundabout 3 0 0 0 0 0 3

One-way street 30 0 1 0 0 0 29

Dual carriageway 109 0 17 5 0 0 81

Single carriageway 2673 634 516 470 13 600 0

Slip road 2 0 1 0 0 1 0

Speed limit:

20-30 2817 634 535 475 13 601 113

40-50 0 0 0 0 0 0 0

60-70 0 0 0 0 0 0 0

Area:

Urban 2544 571 481 431 11 536 110

Rural 273 63 54 44 2 65 3

Junction control:

Auto traffic signal 278 72 49 10 2 81 41

Stop sign 17 4 5 2 0 3 0

Give way or uncontrolled 2522 558 481 463 11 517 72

Light condition:

Daylight 0 0 0 0 0 0 0

191
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving systems in a mixed traffic environment

Darkness 2817 634 535 475 13 601 113

Weather condition:

Fog or mist 18 0 0 0 0 0 0

Fine 2251 634 0 475 0 601 113

Raining 535 0 535 0 0 0 0

Snow 13 0 0 0 13 0 0

Road surface condition:

Dry 1692 467 4 345 2 454 90

Wet or damp 1096 161 529 122 7 142 23

Snow 4 0 1 0 2 0 0

Frost or Ice 24 6 1 7 2 5 0

Flood 1 0 0 1 0 0 0

Oil or diesel 0 0 0 0 0 0 0

Mud 0 0 0 0 0 0 0

Vehicle Manoeuvre:

Turning left 0 0 0 0 0 0 0

Reversing 0 0 0 0 0 0 0

U- turn 0 0 0 0 0 0 0

Turning right 2817 634 535 475 13 601 113

Going ahead 0 0 0 0 0 0 0

1st point of impact:

192
Ing. Simon Perveen
 An accident data-driven scenario generation methodology for testing automated driving systems in a mixed traffic environment

Front 1365 0 275 0 5 601 45

Back 111 44 20 38 2 0 7

Offside 762 304 144 268 4 0 38

Nearside 579 286 96 169 2 0 23

193
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Table_APDX 6: Re-clustering results of cluster 3 showing clusters 3.7 and 3.8.

Attributes Cluster 3.7 Cluster 3.8

Total of observations 18 428

1st Road class:

Motorway 0 0

A(M) 0 0

A 8 0

B 3 0

C 2 73

Unclassified 5 355

2nd Road class:

Motorway 0 0

A(M) 0 0

A 1 20

B 0 2

C 2 10

Unclassified 15 396

Road type:

Roundabout 0 0

One-way street 0 0

Dual carriageway 0 6

Single carriageway 18 422

Slip road 0 0

Speed limit:

20-30 18 428

40-50 0 0

60-70 0 0

Area:

Urban 15 389

Rural 3 39

Junction control:

Auto traffic signal 2 21

194
Ing. Simon Perveen
An accident data-driven scenario generation methodology for testing automated driving
systems in a mixed traffic environment

Stop sign 0 3

Give way or uncontrolled 16 404

Light condition:

Daylight 0 0

Darkness 18 428

Weather condition:

Fog or mist 18 0

Fine 0 428

Raining 0 0

Snow 0 0

Road surface condition:

Dry 3 327

Wet or damp 14 98

Snow 0 1

Frost or ice 1 2

Flood 0 0

Oil or diesel 0 0

Mud 0 0

Vehicle manoeuvre:

Turning left 0 0

Reversing 0 0

U- turn 0 0

Turning right 18 428

Going ahead 0 0

1st point of impact:

Front 11 428

Back 0 0

Offside 4 0

Nearside 3 0

195
Ing. Simon Perveen