This article has been accepted for publication in a future issue of this journal, but has not been

fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/OJITS.2022.3140493, IEEE Open
Journal of Intelligent Transportation Systems
OJ-ITS-2021-10-0081.R1 1

A Credibility Assessment Approach for Scenario-Based

Virtual Testing of Automated Driving Functions
Christoph Stadler1 , Francesco Montanari1 , Wojciech Baron1 , Christoph Sippl and Anatoli Djanatliev
An immense test space is pushing the development and testing of automated driving functions from real to virtual environments.
The virtual world is provided by interconnected simulation models representing sensors, vehicle dynamics, and both static and
dynamic environment. For the virtual validation of automated driving, special attention must be paid to the simulation’s credibility,
which can be impaired by inappropriate or inaccurate simulation models and tools. Therefore, in this work a method is proposed to
assess the credibility of simulation-based testing for automated driving. The approach allows a qualitative and relatively quantitative
comparisons between scenarios as well as between different simulation setups. Therefore, several uni- and multivariate metrics
are applied towards a scoring of similarity of the behavior between simulation and real test drive. This is achieved by using
ground truth data in form of simulation scenarios from real world measurement data. In this way, the virtual automated vehicle
encounters the same conditions and surroundings than its counterpart in the real world for evaluating their similarity. The practical
applicability of the proposed credibility assessment approach is demonstrated in a case study, in which the credibility of an exemplary
simulation-based test bench is inferred.

Index Terms—automated driving; software-in-the-loop; scenario-based approach; virtual testing; virtual development; virtual
validation; computer simulation; automotive engineering; intelligent vehicles; automated vehicles

I. I NTRODUCTION credibility

T ESTING of automated driving functions (ADFs) can be

performed on public roads, on proving grounds, or in a
virtual environment. Testing on public roads is associated with gap
an immense monetary effort, due to the required prototype
vehicles and the training of safety drivers and co-drivers. On
proving grounds, the entire environment can be controlled, but
of course this resource has a limited capacity. Taking into ac-
count that several billion test kilometers have to be completed
?
to validate ADFs [1], [2], one comes to the conclusion that
this huge test space is actually impossible to manage. The idea Open Road Simulation
of using the ADF to control a virtual vehicle in a simulated
world emerged [3]. However, this raises the question on the Fig. 1: The credibility of testing approaches.
credibility of the tests performed in the simulation. It cannot
be denied that a gap exists in relation to real world test drives
[4], [5], [6], [7], cf. Fig. 1, which is caused by the fact that pedestrians, cyclists or car drivers, are assigned as fixed, but
simulation models can only represent the reality to a limited configurable behavior within the setting. A virtual test drive
extent. The challenge is to quantify this gap and to answer the is limited to one challenge to be accomplished by the ADF,
question whether this gap can be kept sufficiently narrow to i.e., one scenario. Test scenarios can be generated in different
allow credible virtual testing. ways [10]: manually by experts, virtually by simulations,
In scenario-based testing [8], [9], the virtual world is lim- systematically by models or extracted from real test drive data.
ited to a concrete setting. Other traffic participants such as Finally, the entire set of scenarios is collected in a scenario
Manuscript created October 21, 2021; revised December 3, 2021. This
catalog. The basic validation idea consists of the successful
work is supported by the AUDI AG and partially by German Federal completion of the entire scenario catalog by the ADF. By
Ministry of Transport and Digital Infrastructure (BMVI) within the Automated testing directly specific concrete traffic situations, the scenario-
and Connected Driving funding program under Grant No. 01MM20012A
(SAVeNoW).
based approach makes it possible to be independent of the
1 These authors contribute equally to the work. likelihood of occurrence of scenarios. As a consequence, the
C. Stadler is with the Automated Driving Department of the AUDI AG, main motivation behind the scenario-based approach is to
Ingolstadt, Germany (e-mail: [email protected]).
F. Montanari is with the Automated Driving Department of the AUDI AG,
reduce the testing amount which results in less testing time and
Ingolstadt, Germany (e-mail: [email protected]). costs. Furthermore, by working with a scenario catalog it is
W. Baron is with the Computer Networks Chair of the Computer Science possible to check which scenarios have been covered and with
Department of the Friedrich-Alexander University of Erlangen-Nuremberg,
Erlangen, Germany (e-mail: [email protected]).
which rating. This eases the process for type approval. Hence,
C. Sippl is with the Pre-Development for Automated Driving Department in this work the focus is on the scenario-based approach.
of the AUDI AG, Ingolstadt, Germany (e-mail: [email protected]). Nowadays, virtual scenario testing is common practice
A. Djanatliev is with the Computer Networks Chair of the Computer
Science Department of the Friedrich-Alexander University of Erlangen- among major car manufacturers [11], [12], [13]. However, it
Nuremberg, Erlangen, Germany (e-mail: [email protected]). is difficult to prove that the virtually tested scenarios are valid

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/OJITS.2022.3140493, IEEE Open
Journal of Intelligent Transportation Systems
OJ-ITS-2021-10-0081.R1 2

and can at least partly substitute testing in the real world [14]. is described. It does not only contain the consecutive frames,
For this reason the key question is, how to prove that simulated but also the actions, events and goals executed by all the
scenarios can be used for the validation and assessment of actors and the events that are happening. In [16] the authors
real automated driving vehicles. The ADF is designed and characterize a scenario by five layers in order to simplify
developed for functioning in the real world, but the testing is the structure and the categories of information in a scenario:
intended to be done mostly virtually in the simulation. If the the static environment is defined by three layers. The first
same ADF behaves differently in the simulation than in the layer describes the road geometry and topology, like street
real world, the results in the simulation have no significance. dimensions with its number of driving lanes or the radius
Hence, it is necessary to show that an ADF behaves in a similar of a curve. The second layer adds traffic infrastructure, like
way to its counterpart in the real world when the same software traffic signs or traffic lights an finally the third static layer
versions are used and confronted with the same conditions in adds temporary manipulations of the first two layers like
its surrounding. In this way, it is possible to reproduce the construction sites. The dynamic environment is defined in the
results of a physically tested scenario and to show that errors fourth layer where all objects, e.g. vehicles, pedestrians, etc.,
and failures can be avoided. are included with their respective interactions and maneuvers.
In this work the authors propose a new method for assessing The fifth layer describes all environmental conditions, like
the credibility for virtual scenario-based testing of ADFs. The weather or daytime, with their effects on the first four layers.
idea is to perform a real test drive with an ADF and to
log the bus data of the vehicle. Afterwards, the scenarios B. ASAM Standards
are identified and extracted from the real driving data and In recent years, the Association for Standardization of
resimulated including the static environment and all traffic Automation and Measuring Systems (ASAM) has set many
participants with the ADF connected to the simulation. Finally, standards in the field of automotive simulation [17]. As ASAM
the behavior of the driving function in the virtual world is standards have a broad basis in research and industry, these
analyzed with respect to the one in the real world. Depending standards are solely used in this work in order to ensure the
on whether the function can reproduce its real behavior the applicability of the method to the broad research community.
credibility of the virtual testing can be deduced. The proposed The ASAM OpenDRIVE (ODR)1 simulation standard is
method is a necessary preliminary work to show how repre- used for the description of static elements of a road network.
sentative the results of the virtual scenario-based testing are. The standard specifies the geometry of the road as well
The contributions of this work are: as infrastructural elements that influence its logic, such as
• A method for assessing the credibility of a ADF simula- roadmarks, traffic signs and traffic lights.
tion. ASAM OpenSCENARIO (OSC)2 serves as a simulation
• A set of adapted metrics to quantify and qualify a similar standard for describing the dynamic entities in a traffic simu-
behavior between simulation and real drive. lation. In an OSC file the actions and states of each individual
• A normalized relative credibility assessment for the com- vehicle, pedestrian, further road user and entity can be defined.
parison of different simulation setups. OSC allows a scenario to be described in two ways, (1) based
This article is organized as follows: Section II and Sec- on driving actions (individual maneuvers) which consist of
tion III present background information and a literature review. actions, events, goals and values or (2) based on trajectories
Section IV introduces the new approach of this work subdi- which consist of coordinates and timestamps.
vided into five sections: real test drive, scenario extraction,
simulation environment, simulation execution and credibility C. Definition of Credibility in Simulation
assessment. Section V presents the experimental setup with
There are many scales that try to measure the quality in
its case studies and results. The discussion and interpretation
modeling and simulation. However, the main questions is
of the results follows in Section VI including its limitations.
always if the simulation is credible and therefore can be used
Section VII concludes and presents an outlook on future
in the verification and validation process to ensure whether a
topics.
real test can be replaced by a virtual one or not.
Credibility in modeling and simulation results is defined as
II. BACKGROUND
“the quality to elicit belief or trust” according to the NASA
In this section relevant background information on the standard 7009a [18]. Liu et al. [19] add that credibility has
method presented in Section V is given, beginning from the in always be defined for a predetermined purpose. This also
depth understanding about a scenario used within the scenario- includes the data used and the results obtained from simulation
based simulation approach and the standards applied in the models. Credibility is not directly linked to the level of
case study. Furthermore, details about credibility in simulation, quality in modeling and simulation as common verification
scenario extraction as well as the necessity of determinism are and validation processes are, but it is strongly related to it
provided. [20]. However, metrics are needed to quantitatively evaluate
the credibility assessment which define a threshold for the
A. Definition of Scenario different levels.
According to Ulbrich et al. [15] a scenario is a temporal 1 https://www.asam.net/standards/detail/opendrive/

sequence where a development from an initial to the last scene 2 https://www.asam.net/standards/detail/openscenario/

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/OJITS.2022.3140493, IEEE Open
Journal of Intelligent Transportation Systems
OJ-ITS-2021-10-0081.R1 3

D. Scenario Generation known dynamic and static objects and their properties is built.
Scenarios can be generated using different approaches. A The planning step is responsible for the decision-making and
widespread practice in the automotive industry is the gener- trajectory generation. Finally, in the actuation step set values
ation of scenarios by experts based on their existing domain are passed on to actuators or on to a vehicle dynamics model
knowledge [21]. However, these scenarios are generated only like in this case. The various steps typically consist of several
by existing subjective and restricted knowledge. Thus it is compact software modules, which are connected to each other
challenging to prove the validity and completeness of scenarios via diverging and merging data flow.
for validation of the whole system. Therefore, it is a necessity The distributed nature of the SIL poses some threats with
to rely on scenarios that may and have occurred on public respect to determinism, which might affect the repeatability of
roads when taking into consideration the credibility of the the experiments that are performed. The repeatability property
scenarios used for validation. is particularly important, because it forms the foundation for
Automotive companies drive a huge number of testing regression testing and test automation. The high computational
kilometers every day with vehicles at different stages of effort and the sheer number of interconnected software mod-
development [22]. Such testing vehicles record the real driving ules require the ADF to be executed on multiple machines and
data while performing their endurance and performance runs. parts of it even on heterogeneous platforms, such as graphics
This results into thousands of hours of existing real driving cards. Not only it must be ensured that all software modules
data that can be exploited and used to understand what is meet their deadlines, but attention must also be paid to data
happening in real traffic by identifying and extracting traffic determinism. The same applies to the virtual environment.
knowledge, driving scenarios and parameter distributions. The Multiple models (e.g. submicroscopic traffic simulation, sensor
resulting scenarios from real test drives have in fact occurred models, vehicle dynamics models) need to be coupled and
on public roads and therefore are valid and traceable test cases as the simulation is computationally expensive, there might
for the validation of ADFs. be a need to be run on multiple simulation machines. The
However, the data is unlabeled, i.e. it is unknown what kind simulation models and tools must be synchronized with respect
of scenarios happened during the test drive and is associated to the simulation time. Hazards regarding non-deterministic
with the raw data. A scenario is not directly measurable. message loss, message order and message timing need to be
Nonetheless, it can be extracted by an interpretation of mul- prohibited. In [29], [30] these phenomena are explained in
tiple signal patterns and their context. There exist different more detail and synchronization mechanisms are presented,
approaches for the identification of scenarios [23], [24], which that are also used for the scope of this work. Consequently
can be subdivided into different categories: the rule-based the conducted simulation runs in this work produce repeatable
approach starting from a scenario catalog, the supervised results.
learning approach based on ground truth reference data and
the unsupervised approach based on pattern recognition in the III. R ELATED W ORK
data. In this work a simple rule-based algorithm is presented In recent years different approaches for the validation and
and used in the method. verification of ADFs have been published.
In the research project PEGASUS [31] 17 partners from
scientific institutions and industry propose a scenario-based
E. Software-in-the-Loop
verification and validation approach for ADFs. Their central
The development and testing of ADFs in virtual envi- elements are based on the definition of requirements for the
ronments (VEs) is carried out in so-called software-in-the- ADF, data processing methods, a joint and public database, the
loop (SIL) simulation setups. It requires the coupling of assessment of the ADF and finally the safety argumentation.
several simulation tools and models with the ADF, which This project can be seen as a first open step from a distance-
itself also consists of several software components. Fig. 2 based validation approach to a systematic scenario-based one.
illustrates the SIL configuration for automated driving with A focus of the project is the testing of scenarios on a real test
typical components. In general, the core of the simulation is track with the ability to repeat those scenarios by controlling
formed by a submicroscopic traffic [25] simulator, that often the vehicles remotely. In this way it is possible to improve or
contains more than just pure traffic simulation (e.g. vehicle update the ADF and analyze its behavior when confronted with
dynamics, behavior, and sensor models). The submicroscopic the same circumstances. One realization of the project was that
traffic simulator is at times not sufficient for a reasonably simulation will play a big role in the verification and validation
realistic representation of the virtual environment and the process. Therefore, two further research projects SET Level3
coupling of further simulation models might be required [26], and VVM4 arose with a stronger focus on the development of
[27]. The models provide features that the standalone simulator testing methods and simulation based testing.
do not yet include (e.g. network simulation) or they are In the research project SAVe5 and its successor SAVeNoW6
custom-designed models that fulfill their function particularly partners from the industry and scientific institutions develop a
well (e.g. models of specific sensors).
3 https://setlevel.de/projekt
The operation of the ADF can be described by means of the 4 https://www.vvm-projekt.de/en/
sense-model-plan-act (SMPA) [28] methodology. The world is 5 https://save-in.digital/
perceived through sensors, or in this case, sensor models. In 6 https://www.bmvi.de/SharedDocs/DE/Artikel/DG/AVF-
the model step an internal accumulated world model with all projekte/savenow.html

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/OJITS.2022.3140493, IEEE Open
Journal of Intelligent Transportation Systems
OJ-ITS-2021-10-0081.R1 4

Simulation Software
Automated Driving Function
Behavior Models
(e.g. Bicycles, Vehicle
Sense
Pedestrians, Dynamics noisy and
Drivers) delayed snapshot
Submicroscopic Traffic Simulation of the world Model
(e.g. road network, traffic control, weather
control)
Plan
Network Sensor Models
Simulation (e.g. Camera, Radar)
Act

longitudinal and lateral control of the ego vehicle

Fig. 2: The SIL simulation setup for automated driving.

methodology for a holistic simulation model of a city and its Integrity Level (ASIL) decomposition, criteria for coexistence
traffic. The simulation model is used for the optimization of of elements of different ASIL classifications in a system, and
the traffic flow and the virtual testing of mobility services and requirements for safety analysis.
ADF. The vision is to ensure a secure release of automated In addition to specific processes for the validation and
vehicles with the help of simulation by having a digital twin verification of ADFs, there are also approaches focused on
of a real digital test field in order to compare real test drive the simulation-based validation in general.
results with its virtual twins. Sargent [35],[36] describes the challenge of validation and
In [32] the authors propose different aspects to consider in verification of simulation models. There are different ways
order to improve validation effectiveness compared to a brute- to evaluate a model and different validation approaches and
force approach: they mention that behavioral requirements techniques. However, the author states that for each single
must be identified before testing the correctness in order to problem, which should be solved with the simulation, a
be able to provide pass and fail criteria. Even if the testing separate and appropriate model is needed. Therefore, there
on the vehicle level finds no failures at all, that does not is no fixed set of methods or practices for the verification
mean the ADF is necessarily safe. They state also potential of simulation models because every model poses its own
solutions in order to improve the safety without the need to unique challenge. Some of the presented techniques, e.g.
validate any hypothetical scenario: for instance by providing animation, comparison to other models, event validity or
the external guarantee that the vehicle will not encounter a parameter variability-sensitivity analysis are also considered
scenario it cannot handle. Alternatively, the ADF may contain in this work.
the capability to detect that it is in a situation outside of its In [37] the authors propose an approach for appropriate
operational domain and put the vehicle to a safe state. scenario selection and model validation for the virtual testing
As far as official regulations are concerned, requirements of ADFs. At first they present a coverage-based and data-
for the approval of ADF are not completely defined yet. driven approach for scenario generation for their simulation
However, as a first step towards an official document in this pipeline. Secondly, they show how to evaluate the behavior of
field, there exists a proposal of the United Nations Economic the simulation model with real data with the help of statistical
Commission for Europe (UNECE) for the approval of vehicles validation metrics and regression techniques. They state that
with an automated lane keeping system (ALKS) [33]. In there are deviations between simulation and reality. In addition
this document the commission presents the requirements the they just use simulation as a tool, i.e. they do not evaluate or
manufacturer must demonstrate to the technical service during attempt to prove and improve the validity of the simulation or
the inspection for approval. It includes all the test specifica- reduce modeling errors.
tions and their pass criteria, parameters and parameter ranges, The work in [38] is based on a collected catalog of real
operator information, data recordings during the drive, cyber high-severity collision scenarios. The authors reproduce those
security aspects and more. Simulation for the verification of scenarios in the simulation and perform a so called ”what-
the safety concept is also mentioned in particular for scenarios if” analysis by replacing one of the human-driven crash
that are difficult to test on a test track or on public roads. participants with an ADF. The ADF can prevent a collision in
However, the manufacturers have to demonstrate the scope of some cases. Respectively, it can at least mitigate the severity of
the specific virtually tested scenarios as well as the validity of the crash and an improvement of traffic safety can be shown.
the simulation tool chain itself. None of the published approaches related to automated
Further framework conditions are provided by ISO26262 driving covers the whole simulation chain and compares the
[34], which defines a process model together with required impact of the ADF in the real world with its impact in the
activities and work products as well as methods to be applied virtual world. They are either methods on how to improve and
in development and production. Part 9 of ISO26262 adds test the ADF in the simulation or are focused on the evaluation
special safety-oriented methods such as Automotive Safety of subsegments of the whole simulation setup. The research

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/OJITS.2022.3140493, IEEE Open
Journal of Intelligent Transportation Systems
OJ-ITS-2021-10-0081.R1 5

question in this work addresses the credibility whether real Table I lists the minimum required signal set that must be
driven scenarios can be replaced by virtual driven scenarios. incorporated in the logging system. Timestamp and ego vehicle
For this reason, the whole simulation chain is taken into signals, i.e. global positioning system (GPS) or electronic
account with the focus described at the end of Section I. stability control (ESC), are needed for positioning the ego
vehicle and understanding its driving behavior. The infrastruc-
IV. M ETHODOLOGY ture signals result from the camera and are required for the
detection of the lines of the driving lane. Later they are used
When testing ADFs in simulation two important questions for the identification of lane-change and cut-in maneuvers. The
arise: How valid or beneficial are the tests in the virtual entity signal, which results from typical sensing sensor, are
environment? And how can their validity or benefit be proved? used to calculate the positioning of the surrounding vehicles
This work focuses on a methodology for a valid virtual relative to the ego vehicle and their respective speed and class.
testing procedure for ADFs and indicate how to quantify and
strengthen the credibility of the simulation connected to an
ADF. In the following sections the single steps are presented
TABLE I: Signal logging requirements during the real test
and their importance and necessity are explained in detail.
drive.
Fig. 3 summarizes the conducted steps.
Category Sensor Signal Units
A. Real Test Drive General - timestamp [ms]
The initial step of the proposed method is to perform a real latitude
test drive with an automated vehicle. During the driving the Ego vehicle
GPS longitude [deg]
ADF is activated without any interruptions and is supervised heading
by a safety driver and an additional monitoring system. The ESC speed [m/s]
target parameters, e.g. target velocity, driving style or des- X lane distance left
tination, of the function are settled. Defined data is logged Infrastructure Camera
Y lane distance left
[m]
during the entire trip. These measures ensure that all the X lane distance right
required data for subsequent reproduction in the simulation is Y lane distance right
available. In addition to the target parameters of the function, X distance
[m]
the test vehicle needs to log the bus communication data in Y distance
order to gather all the information from the control units and Entities Camera/Radar/Lidar X speed
[m/s]
environmental sensors, e.g. cameras, radars and lidars. By this Y speed
it is possible to comprehend and replicate the behavior of the class -
test vehicle itself and of the surrounding static and dynamic
entities.

categories Virtual World Real World

Maneuver
drivers Model Driver Automated Driving Function
Driver

OSC
environments Simulation Scenario Scenario Real Test Drive
Execution Format Extraction

Maneuver Model Function

Real Driving
datasets Simulation Simulation Simulation
Data
Replay Data Data Data

assessment Validation & Comparison

CONFIDENTIAL
Fig. 3: Graph of the proposed method which shows the single steps with its components.

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/OJITS.2022.3140493, IEEE Open
Journal of Intelligent Transportation Systems
OJ-ITS-2021-10-0081.R1 6

B. Scenario Extraction distant objects that are oscillating between being in and out
The goal of this step is to take the recorded data and extract of a sensor’s detection range.
scenarios for the resimulation. As described in Section II-D After having preprocessed the real test drive data, the next
the identification of scenarios from real test drive data can be step is to convert the information into a simulation format. In
performed in different ways. The aim of this work is not to this case it is recommended to convert the information in to
establish a novel way for identifying scenarios or the creation the provided OSC format in order to be able to compare and
of a comprehensive scenario catalog, but rather find sample exchange scenarios between different use-cases, simulation
scenarios for the validation. For the presented purpose, it is tools and development teams.
sufficient to use a rule-based method in order to find a variety For the transformation to OSC the scenario is divided into a
of scenarios. Two different state machines help to identify sequence of lateral and longitudinal maneuvers, e.g. accelera-
lane-change and car-following scenarios on the highway. Fig. 4 tion, deceleration or lane-change. The simulation is maneuver-
shows two state machines used in this work. In Fig. 4a once based, that means the virtual vehicles are assigned simulation
a vehicle starts approaching a line the data is labeled as time or position-based actions and not fixed trajectories. The
lane-change scenario between S2 and the ending state S4. advantage is, that the scenario is described in a configurable
In Fig. 4b as soon as the requirements between S1 and S2 manner and can be altered in an intuitive and comprehensive
are fulfilled once the distance is kept the labeling starts at S3 way by test engineers. However, while abstracting the real test
and ends with S4. drive data into a sequence of maneuvers information is lost due
to simplifications e.g. linear acceleration.
approach line gain distance
approach
line
cross stop
C. Simulation Environment
S1 S2 line S3 gaining S4
Within a closed-loop simulation setup, a valid virtual en-
gain
distance
vironment model is a prerequisite for testing and validating
ADFs, despite valid further components like sufficient sensor
(a) State machine for lane-change detection. models and a vehicle dynamics model. The quality of the
virtual environment model, also referred to as the digital twin,
keep distance
used for the simulation of automated driving is often not
not lane-change keep change clearly specified and the generation of highly accurate and
S1 & object in front S2 distance S3 distance S4 detailed models is very costly on a large scale. Nevertheless,
comparability with the real world has to be ensured. In the
(b) State machine for car-following detection. context of driving simulation a digital twin of the surrounding
environment is a virtual model with physical objects and
Fig. 4: State machines used for identifying sample scenarios. geometries in the street space. Therefore, as a minimum re-
quirement for driving simulation a virtual environment model
After identifying scenarios, the signal data has to be of a street space is needed, which is comparable to the
converted into simulation-tool-readable format. Thereby, the one in real world where the scenario actually happened.
authors build on previous work that is described in [39] in As a consequence, so called high-definition (HD) maps are
detail. available, which are based on the surveying of traffic networks
At the beginning a preprocessing consisting of three steps is with cameras and/or laser scanners in combination with precise
performed. In the first step a unique identification number (ID) positioning systems. These raw data are transformed in a
is assigned to each traffic participant. Even if an ID is already machine readable format like ODR.
assigned by the multi-sensor system, the ID can be repeated From the perspective of automated driving, the real world
after a certain amount of time and is ambiguous. Therefore, is a complex system that could have an infinite number of dif-
especially for longer scenarios it is essential to clarify which ferent characteristics and interactions. Additionally, simulation
detected object is related to which ID in order to avoid vehicle fidelity is not clearly defined within the generated maps. Con-
hopping. In the second step, an up- and downscaling of the sequently, rebuilding the digital twin by measuring the reality
multi-rate data from the different sensors is performed and is is almost impossible in terms of accuracy and completeness
synchronized to a single frequency. Thereby, it is important of appearing contents. For this reason, the virtual environment
to find a trade-off between upsampling, i.e. interpolation, of used for a specific application in driving simulation has to
low frequency signals and downsampling, i.e. information loss, be checked whether its modeled level-of-detail is sufficient or
of high frequency signals. In this case the lowest common not to assure validity. This can be achieved by identifying
denominator of the sensor periods is applied to ensure a valid the relevant parameters in the virtual environment of the
approach, because the generation of new information during road network including its near surrounding. The environment
the upsampling process, i.e. working with interpolated and model described by the ODR format is represented by a
thus not recorded data, can be seen more critical than loss finite number of parameters. Therefore, relevant parameters
in information. In the third step the algorithm detects and can be identified e.g. by sensitivity analysis by variation of
deletes implausible objects like, that are within the sensor’s the original ODR parameters and the investigation of the
upper detection limit only for a short period of time, i.e. influence on the virtual drive within these samples. With

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/OJITS.2022.3140493, IEEE Open
Journal of Intelligent Transportation Systems
OJ-ITS-2021-10-0081.R1 7

further evaluations with methods like the Optimal Subpattern

Assignment (OSPA) metric [40][41], a quantification of the
Measurement data Simulation data
impact and the relevance of parameters can be achieved. This
serves as a quality assessment for virtual environment models.
This information can be looped back to improve accuracy if
Preprocessing w.r.t. scenario time
necessary of influential parameters or respectively take the
known uncertainties into account for validating the whole
driving simulation [42].
Maneuver comparison
D. Simulation Execution
Criticality comparison
The SIL for automated driving (AD) consists of multiple
simulation units and the ADF. The key simulation unit is the Correlation of key signals
submicroscopic traffic simulator. The submicroscopic traffic
simulator is responsible for the representation of the virtual Quantitative metrics
world including the road network. The map is loaded via the Normalization
standardized ODR format. In the presented work, it is always of metrics
the same file, since the test drive was limited to a bounded
region. The dynamic behavior of the road users is described
via the standardized OSC format. Per scenario, the higher-level
simulation control takes over the loading of a new file. The Credibility assessment
dynamics of the ego vehicle are represented by an additional
Fig. 5: Procedure for validation of simulation data.
vehicle dynamics model. The configuration of the vehicle dy-
namics model was based on the characteristics of the prototype
vehicle with which the real test drives were carried out. The
taken into account as well as the completeness and fidelity of
perception of the ego vehicle is limited by several sensor
the virtual environment model. This background knowledge
models. Geometric sensor models are used for this purpose.
about the input data quality and a proper simulation execution
The parameters like mounting points and fields of view are
with regards to determinism is needed to exclude potential
based on the prototype vehicle. The granularity of simulated
side effects and to avoid misinterpretations in the following
perception is based on objects, not raw sensor data. There is
validation steps, when simulation data is evaluated against the
an interface mismatch between simulation units and function
data from the real test drive.
units. Conversion modules handle the mapping of simulation
The simulation data has to be pre-processed in terms of
signals into function signals and vice versa. The time progress
the timestamps considered for evaluation before quantitative
of the simulation is not bound to real-time. The data exchange
validation can be applied. Fig. 6 shows the graph of a plausible
between individual modules and the distribution of simulation
deviation error ∆ per timestamp against ground truth data. On
time to the modules is handled by a higher-level middleware
the one hand side, state of the art vehicle dynamics models
that features conservative synchronization mechanisms. For
need some time from the start of the simulation execution
the same scenario and the same parameters including the same
until they are calibrated. This is particularly the case when the
initial values, the ego vehicle always behaves in the same way.
simulation starts with a high initial velocity of the ego vehicle.
The virtual test drives are reproducible. Therefore, a worst-
Therefore, simulation should only be used for validation after
case behavior analysis over several simulation runs can be
omitted.

E. Credibility Assessment
The aim of the approach is to assess the credibility of the
simulation results with the artefacts generated described in
the steps from Section IV-A to Section IV-D so far. For this
purpose the generated simulation results are compared with
the real test drive data which serves as ground truth. In order t0 VLPXODWLRQWLPH tE
to validate whether the ADF behaves equivalently in the real
and virtual drive and to ensure the simulation is sufficiently Fig. 6: Interval for validation comparison.
detailed, a step-wise procedure is presented to ensure relative
credibility of the simulated scenarios which is summarized in the end of the initialization period at t0 . On the other side,
Fig. 5. the length of the extracted scenario must be strictly satisfied.
As one prerequisite the quality and accuracy of the test Immediately after the end of the last extracted maneuver
drive data, i.e. typical noise floor originated from the installed sequence at time tE the ADF still controls the ego vehicle,
sensors, must be known. Furthermore, the assumptions made but the executed maneuvers have no reference in the test drive
for extraction and abstraction of the scenarios has to be data and thus cannot be compared.

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/OJITS.2022.3140493, IEEE Open
Journal of Intelligent Transportation Systems
OJ-ITS-2021-10-0081.R1 8

Before determining the absolute deviations between the real is conducted within the simulations variants to distinguish
and virtual test drive, the maneuvers sequences are juxtaposed between the different sources of errors occurring.
for a scenario. If the same maneuvers are driven in simulation A large number of signals are available for evaluation in
and are triggered at approximately the same time, an equiva- each time step of the simulation. Therefore, a unique metric
lent behavior of the ADF can be implied. that in the best case represents as much information as possible
In order to test whether the resimulation is appropriate in in one index is desirable in order to achieve a meaningful
case of criticality, a criticality assessment of the investigated expression of the simulation quality. Further requirements are
scenario has to be conducted. As a generic metric the time- the general applicability for all kinds of driving scenarios,
to-collision (TTC) can be used to compute the time until a the ability to combine static environment information with
collision occurs, upon condition that the vehicles involved the vehicle movement as in a closed-loop simulation that
maintain their current speed according to amount and heading information is directly linked to the ego vehicle’s response.
and that they are on a collision course. TTC at a specific time These requirements can be fulfilled with the OSPA metric [40],
i is defined as [41]. The OSPA metric consists of multiple components.
di
T T Ci = (1) OSP A = (dplo + dpca + dpla ) p .
1
(4)
vreli
with the distance d and the relative velocity vrel between The first component dplo
contains the localization error, i.e. the
the vehicle under test and a surrounding object. In literature distance function between two assigned objects. The second
the TTC is defined as critical for values T T Ccrit ≤ 1.5s component dpca corresponds to the cardinality error, i.e. the
[43]. In addition to that a multidimensional criticality analysis unassigned objects compared to ground truth. The labeling
as presented in [44] can also be applied. A scenario can error component dpla penalizes incorrect assignments. In all
be critical (a) or not (b) in the real test drive. In case of components p refers to the order of the used Wasserstein met-
(a) and the simulation also detects critical maneuvers, the ric for calculation. A more detailed computation of the three
methodology can provide a valid virtual testing regarding the components of the OSPA metric can be found in literature
criticality assessment. That is also be valid for case (b) and [40], [41]. Accordingly, the OSPA allows the states of multiple
the resimulation being noncritical. If an opposing criticality of objects to be evaluated over time.
real and virtual behavior is observed, function testing regarding The calculated error values provide a rating about the
safety cannot be applied and resimulation is not credible. absolute difference between the real and virtual test drive
In a next step, the correlation of key signals, i.e position, and gives a first estimation on the quality of the simulation
velocity and acceleration, is computed to identify the signals results in terms of credibility. Additionally, a solely maneuver
that negatively affect the deviations calculated by the more de- simulation is conducted with an ideal model components as
tailed metrics in the following steps. Given two datasets x and well as with a reference model to be able to classify the ADF
y as paired {(x1 , y1 ), . . . , (xn , yn )} the Pearson correlation r test drive.
[45] is defined as So fare the calculated values for the metrics are based
Pn on absolute deviations for specific scenarios. Therefore, a
(xi − x)(yi − y)
r(x, y) = pPni=1 . (2) normalization of the metrics presented above is introduced
2 2
i=1 (xi − x) (yi − y) in order to compare scenario independently and to allow a
The correlation describes a linear dependence of the examined scoring against a threshold. For each metric its normalized
signals and is at its maximum (r = 1) when there is a perfectly value is defined as the probability P (zt ≤ Zth ) with zt as
positive relationship. Conversely, it is at a minimum (r = the value of a metric at timestamp t and a threshold value
−1) when there is an anti-correlation, that is, when there is a Zth of the corresponding metric. As thresholds are dependent
perfectly negative relationship. r = 0 means that there is no on the requirements defined for a specific test case an overall
linear relationship between the datasets. This helps to improve credibility assessment can only be derived relatively to the
potential uncertainties of the input models as well as only thresholds. For a final decision binding for approval a link
translational displacements of the driven maneuvers. to the requirements which have to be fulfilled has to be
In order to compare the credibility of simulation results to established.
real test drive data, quantitative metrics are needed. There is a
series of standard metrics for computer simulation validation V. R ESULTS
[46]. The most widely applied metric is root mean squared The test drive for the experiment of this work is carried out
error (RMSE) [45] in a pre-development prototype vehicle equipped with a data
r Pn logging system and an ADF. The test drive is recorded on the
2
i=1 kxi − yi k digital test field on the highway A9 with three lanes in the
RM SE = , (3)
n proximity of the city of Ingolstadt. For this region a pool of
which computes the average deviations in form of the eu- map data with road networks in the ODR format exists. As
clidean distances. As RM SE > 0 by definition, lower values submicroscopic simulation tool Virtual Test Drive (VTD) [47]
indicate a better fit between two datasets than higher ones. In is used. However, the proposed methodology is based on the
the evaluations the RMSE is not used to directly compare the ASAM simulation standards and is thus compatible with any
trajectories driven by the vehicles, instead a deeper analysis supporting simulation tool.

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/OJITS.2022.3140493, IEEE Open
Journal of Intelligent Transportation Systems
OJ-ITS-2021-10-0081.R1 9
4 AUDI AG I/XX Präsentationstitel Datum

Ego vehicle • The real test drive of the prototype automated vehicle
Dynamic Objects driving in the real world, which refers to as Real Function
Drive (RFD).
• The virtual test drive data of the virtual automated vehicle
driving in the virtual world, which refers to as Virtual
Function Drive (VFD).
• The virtual maneuver-based OSC drive, where the ego
vehicle retraces extracted maneuvers, which refers to as
Virtual Maneuver Drive (VMaD).
• The virtual model-based drive, where the ego vehicle is

𝑶𝑽𝑻 𝑨𝑷𝑹 𝑪𝑼𝑻

controlled by a tool-specific driver model, which refers
wait for passing, lanechange approach, follow cut-in to as Virtual Model Drive (VMoD).
Fig. 7: Maneuver definition for the scenarios of the conducted In the next sections the evaluation is divided into six
case study. steps categorized from abstract to detailed. It starts from a
comparison of the occurred maneuvers, secondly a criticality
analysis, thirdly a correlation analysis, followed by a detailed
RMSE and OSPA comparisons and ends with a normalization
A large number of scenarios from the real test drive can be
step.
generated. From the variety of scenarios, three representational
scenarios are selected with distinct characteristics for further A. Maneuver Comparison
investigation. With SOV T a lane-change scenario and with
First, the maneuvers performed for the four setups are deter-
SAP R a car-following scenario are chosen as representatives
mined and compared. The analysis is divided into lateral (lane-
for lateral, respectively longitudinal control actions. Addition-
changes) and longitudinal (acceleration processes) maneuvers.
ally a cut-in scenario is analyzed, as the reaction of the ego
The goal is that the maneuvers performed in real driving by
vehicle is solely triggered by the surrounding object and is
ego vehicle are also reproduced in the simulation. This builds
considered as safety-critical. Fig. 7 illustrates the actions of
the basis for a similar impact of the ADF in the real and in
the selected scenarios described in detail in the following:
the virtual world. A more in-depth analysis is considered to be
In SOV T a lateral maneuver of the ego vehicle occurs. meaningful only if there is strong correspondence concerning
In the beginning the ego vehicle is following a bus, but this matter.
has the intention to overtake because the velocity is too low For the determination of the longitudinal maneuvers accel-
compared to the target velocity. As there is right-hand traffic eration values of the ego vehicle are used and for the lateral
in Germany, in order to overtake a lane-change to the left maneuvers the position data of the ego vehicle in relation
need to be performed. However, the lane on its left side is to the position data of the lanes is applied. The results of
occupied. Therefore, the ego vehicle waits until the occupying this analysis are depicted in Fig. 8. In all scenarios, there
vehicles pass by in the middle lane. As soon as the middle is a strong correspondence between VMaD, VFD, and RFD
lane is unoccupied, the ADF releases a left lane-change and for both longitudinal and lateral maneuvers. As expected, the
acceleration to the target velocity is triggered. simulated maneuvers also show the strongest correspondence.
SAP R is a car-following scenario where the ego vehicle In SAP R , there is only a slight shift in the deceleration
approaches the car in front and follows it. In this scenario there process and in SOV T , there is a slight shift in the initiation of
are only longitudinal maneuvers. During the whole scenario the lane change process. The lane-change also takes slightly
the middle lane on its left side is occupied by various vehicles. longer. SCU T completely matches for the VMaD and the VFD.
Even if there is an intention to overtake, it is impossible to The VFD shows a comparably good quality for SOV T . Only
change the lane because the gaps are too short. Therefore, the the beginning and the length of the lane-change is slightly
ego vehicle remains on the right side and follows the vehicle different. In SAP R , the VFD accelerates initially and the
in front. deceleration process is shorter. A closer look at the data reveals
In the beginning of SCU T the ego vehicle is driving on a free that the acceleration and deceleration values in this scenario
lane with its target velocity and without any object in front. A are very low in terms of magnitude. In addition, the ego
vehicle overtakes the ego vehicle from the left side, performs a vehicle is very close to the threshold of the safety distance
cut-in maneuver and merges into the lane of the ego vehicle. to the vehicle in front. As a consequence, a minor difference
Since the safety distances are respected the ADF keeps its in the mapping of the position of the front vehicle already
velocity without any hesitation. Thus, in this scenario the ego has an effect on the maneuvers performed. In the RFD a short
vehicle is performing only one keep velocity maneuver during deceleration occurs in SOV T , which was not performed by
the whole time and should not be affected by the happening any of the other variants. This is caused by a measurement
in the surroundings. error. In contrast, the VMoD that is used as reference performs
After performing the test drive and executing the proposed significantly deviating maneuvers. In SOV T and SAP R , for
steps of Section IV, four different setups to evaluate the suc- example, additional lane changes are executed that did not
cess and the credibility of the test scenarios in the simulation occur in the RFD. There is no lane change in SCU T , but heavy
are introduced and compared: acceleration and deceleration can be detected.

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/OJITS.2022.3140493, IEEE Open
Journal of Intelligent Transportation
5)' Systems
OJ-ITS-2021-10-0081.R1 10
   
WLPH>V@
NHHSYHORFLW\ GHFHOHUDWH ODQHFKDQJHOHIW
DFFHOHUDWH IROORZODQH ODQHFKDQJHULJKW

90R' 90R' 90R'

9)' 9)' 9)'

90D' 90D' 90D'

5)' 5)' 5)'

                

WLPH>V@ WLPH>V@ WLPH>V@
(a) SOV T (b) SAP R (c) SCU T
Fig. 8: Comparison of the identified lateral and longitudinal maneuvers.

B. TTC equivalent criticality and no violations with regard to T T Ccrit ,

The second step of the credibility assessment is based on further analysis are conducted to quantify the credibility.
criticality in order to verify whether safety-relevant criteria
are met in a similar way for the real and virtual drives. The C. Correlation Analysis
TTC is applied because it is presumably the best-known and In the next step, the Pearson correlation coefficients are
understood safety-related parameter. The TTC is determined examined for the position si , velocity vi , and acceleration
from the perspective of the ego vehicle in every simulation ai data of the ego vehicle for i ∈ [x, y], where x denotes
time step. The evaluation is capped to a maximum TTC of the longitudinal and y the lateral direction. The correlation is
30s. Larger values are not considered to be relevant, as they performed in each case in reference to the RFD. A maximum
are very far from the critical threshold of T T Ccrit = 1.5s. positive correlation is desirable (r = 1), because then the
No TTC can be determined for time points without collision ego vehicle signals in simulation behave qualitatively identical
course. For these time points the TTC is set to the maximum to the ones in the RFD. The trajectories or the acceleration
value of 30s. Thus, the ratio of critical to non-critical time profile, for instance, are then similar.
points is not distorted. A cumulative distribution function At first, a check is done at which point the maximum of the
(CDF) is composed from the TTC time series. The CDFs for cross-correlation of the individual signals is located in order
all scenarios are illustrated in Fig. 9. For none of the three to identify potential temporal shifts. The result is that there
scenarios is any drive even close to the criticality threshold. In is no time lag for any of the variants or signals. The Pearson
addition, there are barely any time points with collision course correlation factors are thus determined between the respective
for scenarios SOV T and SCU T . In the scenario SAP R the ego unshifted signals, as illustrated in Fig. 10.
vehicle approaches a leading vehicle, but cannot overtake due The same pattern emerges for all three scenarios. Positions
to an occupied lane. In all cases, however, an anticipated speed are in each case, for all variants, better reflected than velocities,
adjustment of the ego vehicle takes place, so that the TTC and velocities are better reflected than accelerations. The
values are also in uncritical ranges above 10s. In this process, superior results in the position data are certainly linked to
VFD and VMaD replicate the criticality of RFD better than the map and road layout in the simulation. Since the virtual
VMoD. Consequently, as all of the scenarios and drives show ego vehicle does not drive off the road, the correlation values

  

TTCcrit 5)' TTCcrit 5)' TTCcrit 5)'
9)' 9)' 9)'
 90D'  90D'  90D'
90R' 90R' 90R'
3W 7>@

3W 7>@

3W 7>@

  

  

  

  
                    
77&>V@ 77&>V@ 77&>V@
(a) SOV T (b) SAP R (c) SCU T
Fig. 9: Criticality assessment of the scenarios.

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/OJITS.2022.3140493, IEEE Open
Journal of Intelligent Transportation Systems
OJ-ITS-2021-10-0081.R1 11



90R'       90R'       90R'      

90R'      

9)'       9)'       9)'   
9)'  
   

90D'      
90D'       90D'       90D'       

SRVB[

SRVB\

YHOB[

YHOB\

DFFB[

DFFB\

sx sy vx vy ax ay sx sy vx vy ax ay sx sy vx vy ax ay 
(a) SOV T (b) SAP R (c) SCU T
Fig. 10: The Pearson correlation coefficients of the variants.

are already very high. From the velocity correlation it can the magnitude of 10m. The VMaD and the VFD exhibited
be seen that in each case a similar goal is pursued by the very similar maneuvers, albeit somewhat temporally delayed.
virtual ego vehicle: namely, comfortably reaching the desired This translated into a low single-digit error in meters. No
speed without increasing criticality. However, how this goal statement can be made as to whether the absolute positioning
is reached is reflected in the acceleration correlations. The of the ego vehicle or the relative positioning to other traffic
vehicle dynamics model also has a major influence on this. The participants is better reproduced. The same applies to velocity
decreasing velocity correlations could be an indication that the and acceleration errors. The high acceleration errors in the
vehicle dynamics model could possibly be parameterized more VFD are remarkable. The cause lies in the oscillation of the
effectively to match the dynamics of the real prototype vehicle. acceleration of the ego vehicle. All scenarios start with a high
A comparison of the variants also shows a similar trend for initial speed of the ego vehicle on the highway. This poses
all three scenarios. The VMaD shows the highest correlation, a challenge for the ADF and vehicle dynamics model, since
closely followed by the VFD. The VMoD performs worst internal states are not initialized.
for all three scenarios. Partially, there is no or even negative
correlation in the velocity profiles. E. OSPA
In order to analyze the simulation data of the trajecto-
D. RMSE ries over time compared to a ground truth, in our case the
For the first time, a look at absolute error values expressed measurement data from the real test drive, the OSPA metric
by the RMSE is taken by calculating the RMSE for all simula- described in Section IV-E is used. On the one P hand, the
tion variants, all simulation scenarios and for each simulation overall deviations occurred during a scenario ( ∆step ) are
time step. In addition, it is distinguished the error of position, evaluated in Fig. 11, and on the other hand especially for
velocity and acceleration (1) of the ego vehicle itself and (2) maneuver-based simulations the error entries per timestamp
of all relative vectors to all other traffic participants located (∆step ) allow conclusions regarding the maneuvers driven over
in the simulation. The relative error vectors are normalized to time.
error per one observable traffic participant in the corresponding For the results of the accumulated deviations computed with
simulation time step. the OSPA metric similar trends can be observed qualitatively
The average RMSE and the standard deviation over all for each kind of simulation data independent of the scenario.
simulation time steps are illustrated in Table II. The general The VMaD performs best, while the VFD approximately
trend of lowest errors for the VMaD and the highest errors for doubles this error. A closer look at the error entry per time
the VMoD remains. The VFD is positioned in between. As it step shows more or less constant deviations over time but with
has already been shown, deviating maneuvers are performed sawtooth-shaped behavior. The sawtooth curve originates from
in the VMoD. This is reflected in a positioning error of the lower sampling rate of the GPS signal used in the real

TABLE II: Results of the RMSE evaluation.

VMaD VFD VMoD

Absolute Relative Absolute Relative Absolute Relative
SOV T ∆s [m] 0.99 ± 0.24 2.53 ± 0.74 1.44 ± 0.40 2.67 ± 1.23 9.28 ± 7.52 9.33 ± 5.58
∆v [ ms
] 0.23 ± 0.11 0.72 ± 0.42 0.54 ± 0.32 0.96 ± 0.35 2.77 ± 1.25 2.74 ± 1.16
∆a [ sm2 ] 0.20 ± 0.08 0.60 ± 0.51 0.59 ± 0.34 0.73 ± 0.54 0.83 ± 0.80 1.09 ± 0.78

SAP R ∆s [m] 0.91 ± 0.23 3.01 ± 1.79 3.52 ± 1.96 3.33 ± 1.88 11.37 ± 4.95 10.70 ± 3.95
∆v [ ms
] 0.24 ± 0.13 0.97 ± 0.52 1.07 ± 0.54 1.29 ± 0.44 3.37 ± 1.47 3.54 ± 1.77
∆a [ sm2 ] 0.22 ± 0.22 0.87 ± 0.77 0.94 ± 0.52 1.35 ± 0.78 1.31 ± 0.81 1.76 ± 1.01

SCU T ∆s [m] 1.66 ± 0.84 1.17 ± 0.46 1.38 ± 0.88 0.60 ± 0.33 11.07 ± 4.23 10.51 ± 3.46
∆v [ ms
] 0.39 ± 0.06 0.59 ± 0.53 0.89 ± 0.28 0.60 ± 0.42 1.75 ± 1.04 1.39 ± 1.48
∆a [ sm2 ] 0.09 ± 0.04 0.58 ± 0.69 1.05 ± 0.50 1.15 ± 0.72 0.46 ± 0.24 0.88 ± 0.85

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/OJITS.2022.3140493, IEEE Open
Journal of Intelligent Transportation Systems
OJ-ITS-2021-10-0081.R1 12

∆step of the case study. The comparison is calculated per timestamp.

20 In general, the mean as well as the SD reveal values in the
order of magnitude of 10−2 for the VFD and VMaD. In
P

0 contrast the VMoD show error deviations in the order of

magnitude of 10−1 . Consequently, the conducted approach
delivers similar results for each scenario of the case study,
∆step

0.25 what emphasizes the general applicability of the analyses.

0.00 TABLE III: Comparison of the mean and standard deviation

0 5 10 15 of ∆step of the OSPA metric for the case study.
time [s]
VMaD VFD VMoD
(a) SOV T Mean SD Mean SD Mean SD
SOV T 0.019 0.014 0.030 0.023 0.20 0.14
∆step

20
SAP R 0.023 0.024 0.085 0.051 0.23 0.11
P

SCU T 0.034 0.022 0.071 0.046 0.15 0.08

0
∆step

0.25 F. Normalization
In the last step the data is normalized to a range of values
0.00
0.0 2.5 5.0 7.5 10.0 12.5 between 0 and 1. The value 0 corresponds to a bad credibility
time [s] and the value 1 to the highest credibility. During normalization,
particular attention is paid to the comparability of the resulting
(b) SAP R metrics both between scenarios and between entire simulation
setups. Thus, it should be comprehensible for which scenarios
∆step

10
(e.g. urban vs. rural) the simulation setup still has weaknesses.
In addition, an improvement in individual models should also
P

0 be reflected in a higher overall score. An evaluation of different

simulation tools can also be conducted. The normalization of
the individual metrics is carried out as follows:
∆step

0.2 • Maneuvers: the time fraction, where the executed maneu-

ver matches the reference scenario
0.0 • TTC: the time fraction, where the criticality matches the
2 4 6 8
time [s] reference scenario
• Correlation: linear mapping of the interval [−1, 1] to [0, 1]
VFD VMaD VMoD
• RMSE: the time fraction, at which the physical parame-
ters do not exceed defined threshold values
(c) SCU T
• OSPA: the time fraction, at which the OSPA metric does
Fig. 11: Evaluation of OSPA metric per timestamp and accu- not exceed a defined threshold value
mulated over simulation time The results for the utilized simulation setup are depicted
in Fig. 12. A larger occupied area within the kiviat chart
is associated with higher credibility. There are no major
test drive compared to the high frequency driving simulation differences in quality between the three scenarios presented.
setup by a factor of ten. In contrast to quite similar and low The maneuver-based extraction of simulation scenarios from
deviations of VFD and VMaD, VMoD performs worst among the measurement data works very well. A perfect mapping is
the three simulation results. As a standard driving model not possible, because smoothing and linearization is performed
is used for VMoD higher deviations are reasonable. High during the classification into maneuvers. Short-term and fine-
deviations can especially be observed in SOV T from second granular deviations of the physical parameters are lost in
seven following when a lane-change is conducted according the process. In the VFD the coarse-granular quantities such
to the maneuver definitions or in SCU T when a cut-in occurs as the safety and the maneuvers are reproduced accurately.
in front of the ego vehicle until second five. The described However, it is noticeable in the physical quantities that the
behavior can be qualitatively observed for each scenario of quality decreases from position to velocity to acceleration. The
the case study. results suggest that better tuning may be possible between the
In order to evaluate the three scenarios irrespective of their ADF and the vehicle dynamics model. Afterwards, one could
duration and resimulated maneuver types, Table III compares re-trigger the evaluation chain and check if an improvement
the mean and standard deviation (SD) of ∆step shown in has occurred. As expected, VMoD performs worst in the
Fig. 11 for the three different variants and among each scenario credibility assessment.

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal,
0.2 but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/OJITS.2022.3140493, IEEE Open

∆ste
Journal of Intelligent Transportation Systems
OJ-ITS-2021-10-0081.R1 0.0 13
2 4 6 8
time [s]
VFD VMaD VMoD

XYHU 6DIH X YHU 6DIH X YHU 6DIH

0DQH W\ 0DQH W\ 0DQH W\
0DWFK  77& 0DWFK  77& 0DWFK  77&

  

  

506( 3& 506( 3& 506( 3&
$FFHOHUDWLRQ

$FFHOHUDWLRQ

$FFHOHUDWLRQ
  

3RVLWLRQ

3RVLWLRQ

3RVLWLRQ
  

3& 506( 3& 506( 3& 506(

506( 3& 506( 3& 506( 3&

9HORFLW\ 9HORFLW\ 9HORFLW\

(a) SOV T (b) SAP R (c) SCU T

Fig. 12: Multivariate credibility results after normalization

The normalized OSPA as combined metric of several input the maneuver-based one in terms of reproducing the traffic
data correlates with the areas in the kiviat diagrams for scenario. However, accuracy and flexibility must be weighed
VFD, VMaD as well as for VMoD (Table IV). The lower here. Only maneuver-based scenarios can be reasonably al-
values compared to the areas in the radar chart are due to tered by parameter variation, allowing for greater test space
criticality and maneuver measures, which are not an input coverage. Furthermore, it is only possible in maneuver-based
of the OSPA metric, but perform very well in each of the scenarios that interaction between the automated ego vehicle
three scenarios. Consequently, a credibility statement relative and other traffic participants can take place. In trajectory-based
to other simulation setups or thresholds is possible with the scenarios, the trajectory is rigorously followed by the traffic
introduction of normalized metrics. participant. This is considered to be an exclusion criterion for
tarjectory-based scenarios, since they have no relevance, at
TABLE IV: Normalized OSPA metric
least for the presented use case.
VMaD VFD VMoD Through the evaluation approach from coarse to fine, pro-
SOV T 0.932 0.747 0.123 found insights for the state of simulation for the safeguarding
of ADF can be gained, where until today it is unclear which
SAP R 0.884 0.322 0.075
simulation quality is sufficient. The visual evaluation of the
SCU T 0.967 0.550 0.132 video footage of the real and the virtual function drive shows
that the drives were very similar and it was difficult to detect
a difference. Lateral and longitudinal maneuver evaluation
VI. D ISCUSSION further confirm this result. In the RMSE, this has an effect
Thanks to measurement data from a prototype automated of a trajectory deviation of a few single-digit meters. The
vehicle, an end-to-end validation loop can be closed across threshold of a valid to an invalid simulation can be assumed
the entire co-simulation system. Therein, the response of the in approximately in this range as the velocities driven in the
ADF to the virtual events is contrasted with the response of scenarios are greater than 20[m/s]. A detailed determination
the ADF in the real test drive. The quality of the ADF is of these thresholds are depended on the requirements of the
not in the scope of the work, merely the demonstration of a test case. A distinction must also be made between lateral
comparable behavior in the simulation. offset, where a deviation is more severe, and longitudinal
An important aspect of reproducing the test drive in the offset, where it is less severe. The same is true for distant
simulation is the generation of the OSC file and the associated objects, for which relative positioning to the ego vehicle is less
maneuver drive. The ADF interacts with the traffic participants important, and for close objects, for which relative positioning
in the simulation. If these behave with a too big deviation from is central to the decision-making of the ADF.
the test drive data, this can also severely impair the reaction of As the results in Section V illustrate, the resimulation of the
the ADF in simulation, because there is an interplay between three scenarios considered yields fair results both qualitatively
ego and external vehicles. Particularly critical points in time and quantitatively. Nevertheless, the presented procedure for
are the threshold zones between deviating decision making, the methodology shows three limitations respectively uncer-
which is reflected in particular in the increased error in SAP R , tainties which came to the authors notice during the case study.
where the ego vehicle constantly faces a minimum distance As an assumption ideal sensor models are used, which
threshold to the vehicle it follows. A trajectory-based genera- exactly detect the vehicles environment as it is modelled within
tion of the scenarios could certainly produce better results than ODR for the static and and within OSC for dynamic entities.

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/OJITS.2022.3140493, IEEE Open
Journal of Intelligent Transportation Systems
OJ-ITS-2021-10-0081.R1 14

Therefore, possible uncertainties originated from noisy sensor can only be tested and calibrated with ground truth data, i.e.
signals of the perception system are not reproduced in simula- only against scenarios driven in reality.
tion as well as uncertainties of state, class and existence of the The method shows promising results. The authors are
objects and weather effects. This assumptions is necessary to able to show similar behavior in five different categories
limit the occurring deviations to the steps scenario extraction, and metrics: maneuvers, criticality, correlation, RMSE and
simulation environment and simulation execution. However, OSPA. In order to have a reference the simulation results are
high fidelity sensor models, like phenomenological or physical normalized and compared to threshold with is dependent on
ones can be integrated in the procedure as well. the requirements of the test case.
As the presented approach is data driven, the credibility
of the resimulated scenarios compared to the real test drive As future work, this analysis should be performed on a
strongly depends on the quality of the measurement data in larger scale in order to understand which type of scenarios
terms of accuracy and availability. As the information for the are more credible in the simulation compared to others. In this
modelling of the surrounding traffic is based on the perception way, it could be defined in which component the simulation
system of the ego vehicle and is measured relatively to ego. has to improve and what type of scenarios should be tested
Consequently, the worse the sensor equipment and the object more accurately in the real world. In this work the credibility is
recognition, the less accurate are the input data for extracting quantified with not further defined thresholds. However, as fu-
the behavior of the other traffic participants. This results in ture work it is intended to find thresholds for different quality
propagating errors and increases the uncertainty in modeling steps of the credibility, which are derived from the requirement
of the surrounding traffic within the OSC. Therefore, it is specifications. As described in Section VI the quality of the
reasonable that higher deviations are detected in the validation measurement data, especially of the environmental sensors,
step for these objects compared to the ego vehicle. This point is responsible for the extraction and reconstruction of the
is not the case during the conducted case study but should behavior of the traffic participants in the OSC. An idea for
generally be taken into account during validation comparison. improving and validating the data could be to perform drives
The presented simulations are conducted with VTD as tool with multiple vehicles that can see each other. In this way, non
for driving simulation. However, the approach from Section IV ideal sensors can also be added in the simulation and perform
is generally transferable and applicable with every state of the the credibility analysis also for the sensor set. Lastly, it would
art simulation tooling which supports ASAM standards and be interesting to use the proposed method on different state-
guarantees strong determinism. Nevertheless, some simulators of-the-art simulation tools in order to establish and understand
may interpret OSC and ODR files slightly different internally. the differences in quality.
This circumstance provokes deviations by using a different
tool set. Therefore, metadata are needed for the simulation R EFERENCES
tools which provide information about the interpretation of [1] P. Koopman and M. Wagner, “Challenges in autonomous vehicle testing
specific data as well as for the ASAM standards as simulation and validation,” SAE International Journal of Transportation Safety,
input in order to perform a tool independently reliable analysis vol. 4, no. 1, pp. 15–24, 2016.
[2] W. Wachenfeld and H. Winner, “The Release of Autonomous Vehicles,”
for validation purposes. in Autonomous driving. Springer, 2016, pp. 425–449.
[3] H. Winner, K. Lemmer, T. Form, and J. Mazzega, PEGASUS - First Steps
for the Safe Introduction of Automated Driving, ser. Lecture Notes in
VII. C ONCLUSION Mobility. Springer International Publishing, 2019, p. 185–195.
[4] A. Ngo, M. P. Bauer, and M. Resch, “A Multi-Layered Approach for
The approval of ADFs is a huge challenge in the auto- Measuring the Simulation-to-Reality Gap of Radar Perception for Au-
motive industry and research. In order to validate an ADF, tonomous Driving,” 2021 IEEE International Intelligent Transportation
simulation- and scenario-based approaches are rising in popu- Systems Conference (ITSC), pp. 4008–4014, 2021.
[5] K. Groh, S. Wagner, T. Kuehbeck, and A. Knoll, “Simulation and Its
larity. Because of the immense test space, it is known that the Contribution to Evaluate Highly Automated Driving Functions,” SAE
validation without the use of computer simulations might not International Journal of Advances and Current Practices in Mobility,
be feasible or practical. The question of the credibility of the vol. 1, no. 2019-01-0140, pp. 539–549, 2019.
[6] D. Notz, M. Sigl, T. Kühbeck, S. Wagner, K. Groh, C. Schütz, and
scenario-based SIL simulation arises. An ADF is designed and D. Watzenig, “Methods for Improving the Accuracy of the Virtual
developed for functioning in the real world, but the testing is Assessment of Autonomous Driving,” in 2019 IEEE International Con-
intended to be done mostly virtually. If the same ADF behaves ference on Connected Vehicles and Expo (ICCVE), 2019, pp. 1–6.
[7] S. Wagner, K. Groh, T. Kühbeck, and A. Knoll, “Towards Cross-
differently in the simulation than in the real world, the results Verification and Use of Simulation in the Assessment of Automated
in the simulation have no significance. Driving,” in 2019 IEEE Intelligent Vehicles Symposium (IV), 2019, pp.
In this work a methodology for assessing simulation sce- 1589–1596.
[8] A. Höfer and M. Herrmann, Scenario-based approach for developing
narios and quantifying their credibility is presented: A real ADAS and automated driving functions. Springer Fachmedien Wies-
test drive is performed with an automated vehicle and the baden, 2017, p. 215–225.
recorded data are elaborated in order to resimulate the occurred [9] C. Sippl, F. Bock, C. Lauer, A. Heinz, T. Neumayer, and R. German,
“Scenario-Based Systems Engineering: An Approach Towards Auto-
scenarios. In the simulation the ADF is confronted with mated Driving Function Development,” in 2019 IEEE International
the extracted scenario in a virtual environment of the road Systems Conference (SysCon), April 2019, pp. 1–8.
network. If the behavior of the ADF in the physical and in [10] G. Bagschik, T. Menzel, C. Körner, and M. Maurer, “Wissensbasierte
Szenariengenerierung für Betriebsszenarien auf deutschen Autobahnen,”
the virtual world is exactly or approximately the same, the in Workshop Fahrerassistenzsysteme und automatisiertes Fahren. Bd,
simulation is considered to be credible. The simulation setup vol. 12, 2018, p. 12.

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/OJITS.2022.3140493, IEEE Open
Journal of Intelligent Transportation Systems
OJ-ITS-2021-10-0081.R1 15

[11] L. Wang, T. Vogt, J. Dobberstein, J. Bakker, O. Jung, T. Helmer, [36] R. Sargent and D. Goldsman, “Use of the internal statistical procedure
and R. Kates, “Multi-functional open-source simulation platform for for simulation model validation,” in Proceedings of the 2015 Winter
development and functional validation of adas and automated driving,” Simulation Conference, 2015, pp. 60–72.
in Fahrerassistenzsysteme 2016, R. Isermann, Ed. Wiesbaden: Springer [37] S. Riedmaier, D. Schneider, D. Watzenig, F. Diermeyer, and B. Schick,
Fachmedien Wiesbaden, 2018, pp. 135–148. “Model Validation and Scenario Selection for Virtual-Based Homologa-
[12] K. Groh, S. Wagner, T. Kuehbeck, and A. Knoll, “Simulation and Its tion of Automated Vehicles,” Applied Sciences, vol. 11, no. 1, 2021.
Contribution to Evaluate Highly Automated Driving Functions,” 2019. [38] J. M. Scanlon, K. D. Kusano, T. Daniel, C. Alderson, A. Ogle, and
[13] F. A. Schiegg, J. Krost, S. Jesenski, and J. Frye, “A Novel Simulation T. Victor, “Waymo Simulated Driving Behavior in Reconstructed Fatal
Framework for the Design and Testing of Advanced Driver Assis- Crashes within an Autonomous Vehicle Operating Domain,” 2021.
tance Systems,” in 2019 IEEE 90th Vehicular Technology Conference [39] F. Montanari, C. Stadler, J. Sichermann, R. German, and A. Djanatliev,
(VTC2019-Fall), 2019, pp. 1–6. “Maneuver-based Resimulation of Driving Scenarios based on Real
[14] H. Abdellatif and C. Gnandt, “Use of Simulation for the Homologation Driving Data,” in 2021 IEEE Intelligent Vehicles Symposium (IV), 2021,
of Automated Driving Functions,” in ATZelectronics worldwide, vol. 14, pp. 1124–1131.
no. 12, 2019, pp. 68–71. [40] D. Schuhmacher, B. Vo, and B. Vo, “A Consistent Metric for Perfor-
[15] S. Ulbrich, T. Menzel, A. Reschka, F. Schuldt, and M. Maurer, “Defin- mance Evaluation of Multi-Object Filters,” IEEE Transactions on Signal
ing and Substantiating the Terms Scene, Situation, and Scenario for Processing, vol. 56, no. 8, pp. 996–1005, 2008.
Automated Driving,” in 2015 IEEE 18th International Conference on [41] B. Ristic, B. Vo, D. Clark, and B. Vo, “A Metric for Performance
Intelligent Transportation Systems. IEEE, 2015, pp. 982–988. Evaluation of Multi-Target Tracking Algorithms,” IEEE Transactions
[16] G. Bagschik, T. Menzel, and M. Maurer, “Ontology based Scene on Signal Processing, vol. 59, no. 7, pp. 3452–3457, 2011.
Creation for theDevelopment of Automated Vehicles,” in 2018 IEEE [42] C. Stadler, K. Rauner, R. German, and A. Djanatliev, “Simulation-Based
Intelligent Vehicles Symposium (IV). IEEE, 2018, pp. 1813–1820. Parameter Identification for Accuracy Defnitions in Virtual Environment
[17] ASAM e.V., “ASAM SIM:GUIDE - standardization for highly Models for Validation of Automated Driving,” in 2021 IEEE Intelligent
automated driving,” Tech. Rep., 2021. [Online]. Available: https: Vehicles Symposium (IV), 2021, pp. 1138–1143.
//www.asam.net/asam-guide-simulation/ [43] A. R. Mamdoohi, M. Fallah Zavareh, C. Hydén, and T. Nordfjærn,
[18] R. Roe, “Standard for models and simulations,” National Aeronautics “Comparative Analysis of Safety Performance Indicators Based on In-
and Space Administration, Tech. Rep. NASA-STD-7009A, 2016. ductive Loop Detector Data,” Promet - Traffic&Transportation, vol. 26,
[19] F. Liu, M. Yang, and Z. Wang, “Study on simulation credibility metrics,” no. 2, pp. 139–149, Apr. 2014.
in Proceedings of the 2005 Winter Simulation Conference, 2010, pp. [44] B. Huber, S. Herzog, C. Sippl, R. German, and A. Djanatliev, “Evalu-
166–183. ation of Virtual Traffic Situations for Testing Automated Driving Func-
[20] R. Rabeau, Credibility in Modeling and Simulation. John Wiley & tions based on Multidimensional Criticality Analysis,” in 2020 IEEE
Sons, Ltd, 2013, ch. 3, pp. 99–157. 23rd International Conference on Intelligent Transportation Systems
[21] A. Erdogan, B. Ugranli, E. Adali, A. Sentas, E. Mungan, E. Kaplan, and (ITSC), 2020, pp. 1–7.
A. Leitner, “Real-World Maneuver Extraction for Autonomous Vehicle [45] D. Freedman, R. Pisani, and R. Purves, Statistics (international student
Validation: A Comparative Study,” in 2019 IEEE Intelligent Vehicles edition), 4th ed. New York: Norton & Company, 2007.
Symposium (IV). IEEE, 2019, pp. 267–272. [46] C. Beisbart and N. J. Saam, Eds., Computer Simulation Validation. Fun-
[22] Daimler AG, “Mercedes-benz präsentiert in genf limousine und damental Concepts, Methodological Frameworks, Philosophical Per-
coupé der neuen e-klasse (2009).” [Online]. Available: https://www. spectives. Cham: Springer International Publishing, 2019.
presseportal.de/download/document/115362-pi-mb-genf-2009-d.pdf [47] K. von Neumann-Cosel, “Virtual test drive,” Dissertation, Technische
[23] F. Montanari, R. German, and A. Djanatliev, “Pattern Recognition Universität München, München, 2014.
for Driving Scenario Detection in Real Driving Data,” in 2020 IEEE
Intelligent Vehicles Symposium (IV), 2020, pp. 590–597.
[24] F. Montanari, H. Ren, and A. Djanatliev, “Scenario Detection in Unla-
beled Real Driving Data with a Rule-Based State Machine Supported by
a Recurrent Neural Network,” in 2021 IEEE 93rd Vehicular Technology
Conference (VTC2021-Spring), 2021, pp. 1–5.
[25] S. P. Hoogendoorn and P. H. L. Bovy, “State-of-the-art of Vehicular
Traffic Flow Modelling,” vol. 215, p. 21, 2001. Christoph Stadler received the B.Sc. (2015) degree
[26] R. L. Bucs, R. Leupers, and G. Ascheid, “Multi-Scale Multi-Domain in Engineering Science and the M.Sc. (2018) degree
Co-Simulation for Rapid ADAS Prototyping.” IEEE, Oct 2018, p. in Mechanical Engineering from the Technical Uni-
532–535. versity of Munich, Germany. Currently, he is pursu-
[27] M. Paulweber, Validation of Highly Automated Safe and Secure Systems. ing the Ph.D. degree in a collaboration between the
Springer International Publishing, 2017, p. 437–450. AUDI AG and the Chair of Computer Networks and
[28] J. Hertzberg, K. Lingemann, and A. Nüchter, “Roboterkontrollarchitek- Communication Systems of the Friedrich-Alexander
turen,” in Mobile Roboter. Springer, 2012, pp. 317–333. University Erlangen-Nuremberg, Germany. His re-
[29] W. Baron, C. Sippl, K.-S. Hielscher, and R. German, “Repeatable search interests are focused on quality requirements
Simulation for Highly Automated Driving Development and Testing,” for virtual test fields and credibility of simulation
in 2020 IEEE 91st Vehicular Technology Conference (VTC2020-Spring). results for automated driving.
IEEE, May 2020, p. 1–7.
[30] W. Baron, A. Arestova, C. Sippl, K.-S. Hielscher, and R. German,
“LETT: An Execution Model for Distributed Real-Time Systems,” in
2021 IEEE 94th Vehicular Technology Conference (VTC2021-Fall).
IEEE, September 2021, p. 1–7.
[31] Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR)
Institut für Verkehrssystemtechnik, “Pegasus method,” Tech. Rep., Francesco Montanari received his B.Sc. (2015) and
2019. [Online]. Available: https://www.pegasusprojekt.de/files/tmpl/ M.Sc. (2018) degree in Electrical and Computer En-
Pegasus-Abschlussveranstaltung/PEGASUS-Gesamtmethode.pdf gineering from the Technical University of Munich,
[32] P. Koopman and M. Wagner, “Toward a framework for highly automated Germany. He is currently pursuing the Ph.D. degree
vehicle safety validation,” SAE Technical Paper, Tech. Rep., 2018. in a collaboration between the Automated Driving
[33] United Nations Economic Commission for Europe (UNECE), “Proposal Department of the AUDI AG and the Computer
for a new UN Regulation on Uniform Provisions Concerning the Networks and Communication Systems Chair of
Approval of Vehicles with Regards to Automated Lane Keeping System the Computer Science Department of the University
(ECE/TRANS/WP.29/2020/81),” 2020. Erlangen-Nürnberg, Germany. His research interests
[34] “ISO 26262-2:2018 Road vehicles - Functional safety,” International are focused on driving data analysis, scenario iden-
Organization for Standardization, Geneva, CH, Tech. Rep., 2018. tification and resimulation of scenarios.
[35] R. Sargent, “Verification and Validation of Simulation Models,” in
Proceedings of the 2010 Winter Simulation Conference, 2010, pp. 166–
183.

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/OJITS.2022.3140493, IEEE Open
Journal of Intelligent Transportation Systems
OJ-ITS-2021-10-0081.R1 16

Wojciech Baron received his B.Sc. (2015) and

M.Sc. (2018) degree in Information and Communi-
cation Technology from the University of Erlangen-
Nuremberg, Germany. At the moment, he is working
as a Ph.D. student at the Department of Computer
Science at the University of Erlangen-Nuremberg.
He collaborates with AUDI AG in diverse projects
that evaluate synchronization mechanisms and real-
time capabilities in distributed simulation systems in
the context of automated driving.

Christoph Sippl received his Ph.D. degree (Dr.-

Ing.) and his M.Sc. in computer science from the
engineering faculty of the University of Erlangen-
Nuremberg in 2020 and 2014, respectively, and
his B.Sc. in Medical Information Technology from
Ostbayerische Technische Hochschule Regensburg
in 2012. His current research interests are focused on
scenario-based development and virtual validation
methods for automated driving.

Anatoli Djanatliev is the head of the Connected

Mobility Group at the Chair of Computer Networks
and Communication Systems (FAU). He received
his Ph.D. degree (Dr.-Ing.) and his M.Sc. in com-
puter science (Dipl.-Inf. Univ.) from the engineering
faculty of the University of Erlangen-Nuremberg in
2015 and 2008, respectively. His current research
interests include various topics on simulation and
modeling. Major application areas are simulation of
vehicular networks, innovative aspects of connected
mobility, and future mobility services.

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/