2311 JCAV AutomotiveSecurityTestingStandards

Downloaded from SAE International by Nan Kai University of Technology, Friday, November 17, 2023
ARTICLE INFO
Article ID: 12-07-02-0013
© 2024 International Alliance
for Mobility Testing and
Standardization (IAMTS)
doi:10.4271/12-07-02-0013
A Global Survey of Standardization

and Industry Practices of
Automotive Cybersecurity
Validation and Verification Testing
Processes and Tools
Andrew Roberts,1 Stefan Marksteiner,2,6 Mujdat Soyturk,3 Berkay Yaman,4 and Yi Yang5
1
Tallinn University of Technology, Estonia
2
AVL List GmbH, Austria
3
Marmara Üniversitesi, Turkey
4
BigTRI, Turkey
5
AVL China, China
6
Mälardalen University, Sweden
Abstract History
Received: 07 Mar 2023
The United Nation Economic Commission for Europe (UNECE) Regulation 155—Cybersecurity and Revised: 25 Aug 2023
Cybersecurity Management System (UN R155) mandates the development of cybersecurity manage- Accepted: 24 Oct 2023
ment systems (CSMS) as part of a vehicle’s lifecycle. An inherent component of the CSMS is cyber- e-Available: 16 Nov 2023
security risk management and assessment. Validation and verification testing is a key activity for
measuring the effectiveness of risk management, and it is mandated by UN R155 for type approval. Keywords
Due to the focus of R155 and its suggested implementation guideline, ISO/SAE 21434:2021—Road Cybersecurity standards,
Vehicle Cybersecurity Engineering, mainly centering on the alignment of cybersecurity risk manage- Validation and verification,
ment to the vehicle development lifecycle, there is a gap in knowledge of proscribed activities for Cybersecurity testing, Best
validation and verification testing. This research provides guidance on automotive cybersecurity Practices
testing and verification by providing an overview of the state-of-the-art in relevant automotive
standards, outlining their transposition into national regulation and the currently used processes Citation
and tools in the automotive industry. Through engagement with state-of-the-art literature and Roberts, A., Marksteiner, S.,
workshops and surveys with industry groups, our study found that national regulatory authorities Soyturk, M., Yaman, B. et al.,
are moving to enshrine UN R155 as part of their vehicle regulations, with differences of implementa- “A Global Survey of
tion based on regulatory culture and pre-existing approaches to vehicle regulation. Validation and Standardization and
verification testing is developing aligned to UN R155 and ISO21434:2021; however, the testing Industry Practices of
approaches currently used within industry utilize elements of traditional enterprise information Automotive Cybersecurity
technology methods for penetration testing and toolsets. Electrical/electronic (E/E) components Validation and Verification
Testing Processes and
such as embedded control units (ECUs) are considered the primary testing target; however, connected
Tools,” SAE Int. J. of CAV
and autonomous vehicle technologies are increasingly attracting more focus for testing. 7(2):2024,
doi:10.4271/12-07-02-0013.
© 2024 International Alliance for Mobility Testing and Standardization (IAMTS). Published by SAE International. This Open
Access article is published under the terms of the Creative Commons Attribution Non-Commercial, No Derivatives License ISSN: 2574-0741
(http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits use, distribution, and reproduction in any medium, e-ISSN: 2574-075X
provided that the use is non-commercial, that no modifications or adaptations are made, and that the original author(s) and
the source are credited.
1
2 Roberts et al. / SAE Int. J. of CAV / Volume 7, Issue 2, 2024
1. Introduction •• We conducted a survey of tools and practices commonly

used by manufacturers and admission bodies and
U
analyzed the development of cybersecurity test tools
NECE’s regulation 155 (UN R155) [1] requires a struc- and procedures.
tured approach to cybersecurity engineering of auto-
motive systems, using a cybersecurity management •• We discussed the findings of the state-of-the-art and
system. This regulation is mandatory for compliance with auto- survey and analyzed the progress of the adoption of
motive type approval within many of the most important auto- UN R155.
motive markets including Europe, Korea, and Japan. Without
compliance to the UN R155, original equipment manufacturers
(OEMs) may currently not commission new models, and, from
mid-2024, will be restricted from selling to these markets.
2. Methodology
Therefore, OEMs are motivated to comply with this regulation
The initial stage of the study focused on establishing the stan-
due to the financial risk of losing market access due to noncom-
dards and regulatory environment for V&V testing of key
pliance. Closely related to UN R155 is the ISO/SAE 21434:2021—
global automotive regions. The central questions used to guide
Road Vehicles—Cybersecurity engineering, which is commonly
the research were:
accepted as the guiding standard for automotive cybersecurity
[2]. UN R155 requires automotive manufacturers to have for •• RQ1 What is the state-of-the-art for automotive
their automotive product a Cybersecurity Management System cybersecurity V&V standards?
(CSMS). The ISO/SAE 21434:2021 provides, so far, the only •• RQ2 How have these standardization approaches been
global standardized approach for development of an automotive transposed to national regulation?
CSMS (however, it is not explicitly mandatory that a CSMS
follows that standard). UN R155 and ISO/SAE 21434:2021 •• RQ3 What are the V&V testing processes, procedures,
require the structured measures to be verified and documented and tools used by industry?
in a comprehensible and replicable manner using structured
These questions enable the extrapolation of key areas of
testing procedures. However, the details of how to conduct
interest for automotive cybersecurity V&V:
testing applicable to the requirements of UN R155 for type
approval and to the standard expected for automotive risk •• Are there variances between regions in the
management are mainly left to technical services, vendors, and implementation of regulation and national initiative
suppliers. The global standards (including ISO/SAE 21434:2021) developed to improve V&V testing, and if so, why?
only recommend testing methodologies at a very high level (i.e., •• What are the key trends for V&V testing adopted in
functional testing, vulnerability scanning, fuzz testing, penetra- industry? What can these trends tell us about the
tion testing), and provide suggestions for test targets (e.g., evolving nature of V&V testing to meet
checking for exposed debug interfaces, the presence of a secure technology innovation?
boot mechanism, usage of encryption in communications, etc.).
The complexity of vehicular systems, in conjunction with a To answer these research questions, analysis was conducted
diverse ecosystem of standards and procedures make it infea- on three data sources (see Table 1): (1) literature from govern-
sible to define a solid, standardized testing procedure that spans ment authorities, industry, and standardization groups, (2)
over the whole (in-homogeneous) system and over the whole expert knowledge derived from open-format workshops with
life cycle. The development of standardized processes is further regional representatives from a global mobility testing industry
challenged, as each large OEM has its own established proce- working group, and (3) an academic literature from key confer-
dures and guidelines, partially stemming from internal design ences in the automotive cybersecurity field. The purpose of the
and coding guidelines as well as from procedures from adjacent academic literature review is to provide a brief overview of the
domains such as functional safety testing. There also exists a key trends as they relate to ISO/SAE 21434:2021.
lack of literature that explores the state-of-the-art of automotive
cybersecurity testing and how the global standards are being
implemented regionally and how industry is developing its
2.1. Related Work
cybersecurity testing programs. To confront these challenges, There have been numerous reviews of automotive cybersecu-
the main idea of this research is to provide a starting point on rity standardization during and after the drafting of ISO/SAE
identifying test targets and testing methods from a global and 21434:2021 and the UNECE Regulation R155. Macher et al. [3]
regional perspective, as well as exploring the usage and appli- first review in 2019 found two predominant challenges of stan-
cability of such methods currently used in the automotive dardization of automotive cybersecurity testing. First, the
industry. To this end, the contributions of this research are cross-relations between standards, guidance, recommenda-
as follows: tion, and regulation created a complex environment that was
•• We conducted a state-of-the-art analysis of automotive difficult to interpret. Second, select automotive technologies
validation and verification testing (V&V) for global and were governed by diverse standards. An example was given of
regional automotive cybersecurity standards OBD-II interface, which is mentioned in hardware security
and regulations. and certificate standardization documents. However, the
Roberts et al. / SAE Int. J. of CAV / Volume 7, Issue 2, 2024 3
TABLE 1 Data sources for survey of standardization efforts for the standard also mentions cybersecurity testing by empha-
automotive cybersecurity V&V testing. sizing its importance and providing a high-level guidance.
Review Data source Worth noting, the document doesn’t provide a detailed
analysis for the testing methodologies, processes, and tools.
Literature review •• Official government documents
of national (legislation, govt. department Further, the standard brings description and distinguishes
between the verification and validation. Because of the lack
© International Alliance for Mobility Testing and
standards and documents)

regulations •• Automotive and transportation reports of test-related details in ISO/SAE 21434:2021, WG11
and standardization reports (Cybersecurity working group) under ISO/TC22/SC32
•• Academic literature (Committee of Electrical and Electronic Components and
General System Aspects) has proposed ISO PWI 8477, which
Industry survey •• Open format workshops with regional
representatives from EU, China, Japan, is a new standardization project for automotive cybersecurity
Standardization (IAMTS)
and North America verification and validation. This project is intertwined with a
•• Written survey with structured questions second project: “ISO/SAE PWI 8475: Road vehicles—
Cybersecurity Assurance Levels (CAL) and Target Attack
Academic survey •• Literature from automotive security
research in academia and standardization Feasibility (TAF),” which is targeted to define automotive
body journals cybersecurity assurance levels (CALs) and target attack feasi-
bility (TAF), whereby the CALs are focused on engineering
assurance and the TAFs are on the expected strength of tech-
certificate standardization was legacy and was written in 2006, nical controls. However, there is not yet (as of June 2022) an
at which, advances in hardware security were not apparent. official standards project, any results are therefore pending.
The second standardization review by Schmittner and Macher The standards document SAE J3061_202112 (Cybersecurity
[4] in 2020 focused on the draft [5] of the ISO/SAE 21434:2021 Guidebook for Cyber-Physical Vehicle Systems) [8] contains
standard. In addition to lauding the effort to contribute a an appendix regarding the existing security test tools. Another
common framework and language for automotive cybersecu- initiative from SAE International, which is in progress, is the
rity, shortcomings identified included ambiguity in descrip- J3061-2 (Security Testing Methods) [2]. The document has
tions of processes and approaches and the difficulty in been issued by the Vehicle Cybersecurity Systems Engineering
providing a standardized context for diverse methods, guide- Committee with the aim of providing a detailed analysis on
lines, and best practices. Schober and Griessnig [6] mapped the security testing methods on both hardware and software.
the cross-relations of automotive cybersecurity regulations A global regulation initiative on automotive cybersecurity
(UNECE No. 155 and 156) and standards (ISO/SAE 21434:2021, is brought recently within an addendum to UNECE 1958 treaty
ISO PAS 5112, ISO 24089). As this study was written at the (Regulations 141-160), namely UNECE R. 155 [1] and 156 [9]
initial release of ISO/SAE 21434:2021 and before UNECE R155 for automotive cybersecurity. These regulations have a direct
and 156, the national level initiatives to support innovations impact on OEMs and suppliers as the compliance to UNECE’s
for automotive cybersecurity testing were not captured. homologation regulations are fundamental for the automotive
type approval process and product development for the market.
UNECE Regulation No. 155 (–legally binding document ECE/
3. Global Regional TRANS/WP.29/2020/79 [10]) mandates the installment of a
CSMS as defined in ISO/SAE 21434:2021 [7] to ensure an
Perspectives on accompanying cybersecurity process to be executed during the
Automotive Product automotive system development lifecycle. In the document, the
OEM is required to verify the effectiveness of implemented
V&V Testing cybersecurity measures by testing and the approval authority
shall refuse the type approval if this cannot be demonstrated
Standardization and including the adequateness of the testing procedures them-
selves. Lastly, the authority by itself shall also verify the effec-
Regulation tiveness of security measures by testing, especially concen-
trating on the high-risk samples. With the increased threads
on cybersecurity of automotive systems due to increased
3.1. Attack Automotive complexity and connectivity; there are initiatives brought by
Product V&V Testing the governments for regulation and standardization. It is seen
Standardization that it is a general tendency by the governments to prepare the
industry for the regulations, with guideline documents on how
As a standard released in 2021, ISO/SAE 21434:2021 [7] brings to properly implement and test the cybersecurity mechanisms
a specification and a framework for cybersecurity risk manage- (e.g., [11] and [12]). While most of the regulation initiatives
ment in different phases of product lifecycle: concept, develop- across the governments regarding the V&V are still in similar
ment, production, operation, maintenance, and decommis- phases of preparation; there are some issuances of documents
sioning of electrical and electronic systems. While covering regarding the type approval by the ministries of Korea and
the whole engineering process of road vehicles’ cybersecurity, China (see subsections Republic of Korea and China under
Section 3.3.1). In the U.S., the government encourages the cybersecurity testing of critical infrastructure differ it is
industry to collaborate with the regulation activities by important to understand how national governments are trans-
commenting on the documents published by the agencies. posing UNECE R155 into their respective ecosystems and
Regarding the standardization, there are two remarkable initia- how they are supporting the introduction of regulations with
tives brought by the national standardization organizations of initiatives to assist industry and authorities. It is also observed
China and Japan, which brought some practical standards on that, despite China is not a contracting party of the UNECE
cybersecurity testing and verification (see Section 3.3.1). WP.29 1958 Agreement [21] (hence not obliged to follow
UNECE R155); the national government perform similar
activities referring to ISO/SAE 21434:2021. In North America,
3.2. Automotive Academic situation is different due to the performed system of self-
Survey of V&V assessment in that region. Despite this, there are national
activities with respect to ISO/SAE 21434:2021.
Current trends in academic literature focus on the To elucidate this, two components of national approaches
following areas: to automotive cybersecurity testing are analyzed: (1) gover-
•• Novel vulnerability testing of intelligent vehicular nance and implementation of regulation and ISO21434:2021
technologies and autonomous, self-driving and (2) national initiatives with regard to automotive
control algorithms. V&V testing.
•• Methods for automating cybersecurity testing.
3.3.1. Asia
Novel attacks on intelligent vehicular and self-driving
technologies focus on the advanced hardware technologies China
that support perception (LiDAR, camera, radar), localization
(LiDAR, GNSS), and vehicular communication [vehicle-to- Governance and Implementation of Regulation and
vehicle (V2V), vehicle-to-infrastructure (v2x)]. Testing is ISO/SAE 21434:2021 The Chinese market has seen an
predominantly conducted in high-fidelity digital twin simula- emergence of self-driving and interconnected technologies
tion environments and progressively, real-world environments for vehicles. Due to this, the Chinese government ministries
and proving grounds. Tools common in testing of these are focused on developing policies for cybersecurity and data
systems include adversarial neural networks that generate security of intelligent and connected vehicles (ICVs). To
malicious robust physical invariants to perturb object detec- support these policies, corresponding standards committees
tion and semantic segmentation, fuzzers for protocol vulner- are developing national standards, of which the majority still
ability assessment [13, 14], and, in intelligent vehicles, to send are in draft version. In particular, three ministries work in
malicious unsanitized sensor telemetry input to impact the field of cybersecurity and data security of ICV: the
LiDAR [1, 15], radar, and inertial measurement sensors [16, Ministry of Industry and Information Technology (MIIT)
17]. White-box testing tends to be more popular for testing of and Cyberspace Administration of China (also called Office
neural networks due to the complexity of understanding the of the Central Cyberspace Affairs Commission), and the
impact of attacks of black-box testing and to optimize testing Ministry of Natural Resources.
based on knowledge of the learning model. Automation of In late 2021, the MIIT has published two notices [22, 23]
cybersecurity testing has focused on aligning fuzz testing to address the security requirement of connected vehicles. In
techniques with contemporary software development these notices, it mandates that both cybersecurity and data
processes. Fuzzing approaches are being developed, which security of connected vehicle must be fully considered before
incorporate guidance of the ISO/SAE DIS 21434 to utilize going to market. Building a complete vehicular security
threat and risk assessment (TARA) and cybersecurity assur- standard system is also prescribed to all subdepartments,
ance levels (CALs) to systematically identify and prioritize organizations, and companies. Meanwhile, a mandatory
attack vectors [18]. Novel methods for testing are being standard for vehicle cybersecurity and technical requirements
explored on digital twin, digital replications of embedded for vehicle cybersecurity has been issued [24]. Furthermore,
systems, to understand attack vectors and resultant impacts ISO/SAE 21434:2021 is being converted to Chinese national
in a safe, and repeatable and agile test environment [19, 20]. standards as well.
National Initiatives With Regard to Automotive

3.3. N
ational Regulatory and Product V&V Testing For general technical security
requirements, the National Information Security
Standardization Standardization Technical Committee (NISSTC) released
Approaches for GB/T 40861-2021 [25] on October of 2021, which involved the
Automotive Product V&V security of software, electrical and electronic hardware, data,
Testing onboard communication, and V2X communication.
Furthermore, the authenticity, confidentiality, integrity, avail-
Each signatory of UNECE R155 is required to transpose this ability, access control, anti-repudiation, auditability, and
regulation into national legislation. As approaches to preventability should be considered to the corresponding
security system, if applicable. Compared to other standards, evaluation guide for ECU and hardware/software vulnerabili-
this standard provides a more complete technical requirement ties. JASO TP-15002 guideline is an evaluation guideline for
of in-vehicle security. Some standards released by NISSTC automotive information security analysis. Japan Automotive
focus on the technical requirements as well as test methods Software Platform and Architecture (JASPAR) is a collabora-
of specified system and component. Standard GB/T 41578- tion project of engineers from the automotive industry. The
2022 [26] addresses in-vehicle charging system and corre- aim of JASPAR is [32]: “identify common issues that will
sponding communication security. It further specifies detailed be faced in the future in the car electronics sector, and then
test methods at hardware, software, data, and communication undertake standardization initiatives aimed at resolving those
aspects. GB/T 40856-2021 [27] concerns the security test issues, creating common objectives across the entire
methods for hardware, communication, operation system, automotive industry.”
application, and data. GB/T 40857-2021 [28] addresses
hardware, software, communication, and data security for National Initiatives With Regard to Automotive
CAN gateway, ethernet gateway, and hybrid gateway. GB/T Product V&V Testing The JASPAR project provides refer-
40855-2021 [29] involves on-board terminals security, ence architectures for secure design of automotive compo-
communication security, and platform security in the scope. nents and verification testing. The standards are focused on
With regard to different kinds of security, standards also areas of cybersecurity of car electronics where there are gaps
provide a few general best practices for testing. For hardware, in other available standards and areas that are a priority for
this includes checking for exposed debug interfaces and their the Japanese automotive industry. These include software-
authentication mechanisms, the disclosure of the PCB wiring over-the-air updates, ECUs, CAN-FD, secure communication,
and design, and for backdoors. For software, checks for secure and vehicular messaging. JASPAR project details a list of stan-
boot and software integrity, access control, logging mecha- dards applicable to cybersecurity testing of automotive
nisms, as well as vulnerability scans are recommended. The products: TD-CST-4—ECU Penetration Testing Guide
data should be checked for susceptibility to tampering, confi- Version 1.0, ST-CST-1—ECU Vulnerability Test Requirements
dentiality on export, collecting after user approval, sensitive Ver.1.1, STOTA-09—OTA Software Update Compliance Test
information protection, effectiveness of its deletion, as well as Specification OTA Master Ver.1.0, ST-OTA-10—OTA Software
its security during transmission. Communication links should Update Compliance Test Specification—Target ECU
prove their authentication, integrity confidentiality avail- Ver.1.0 [32].
ability, and non-repudiation.
Republic of Korea
Japan
Governance and Implementation of Regulation and
Governance and Implementation of Regulation and ISO/SAE 21434:2021 There are two main actors in Korea
ISO/SAE 21434:2021 Japanese METI (Ministry and for type approval and certification of vehicles Ministry of
Economy, Trade and Industry) published a document about Land and Infrastructure, Transport (MOLIT) and Korea
cybersecurity measures for autonomous vehicles in 2018. This Automobile Testing & Research Institute (KATRI) [33]. There
document describes the schedule for implementing ISO/SAE are two regulations that pertain to the testing and evaluation
21434:2021. First, JASPAR (Japan Automotive Software of automotive:
Platform and Architecture) collaborates with other countries
to establish the standard while suggesting rules and policies •• Korea Motor Vehicle Safety Standard (KMVSS)—
that fit in Japanese automotive environment. While devel- Technical Regulation
oping ISO/SAE 21434:2021, METI and MLIT (Ministry of •• Korea Vehicle Management Act (Self-Certification
Land, Infrastructure, Transport and Tourism) create guide- system and Safety Standards for Motor Vehicles)
lines that describe requirements to develop and operate auto-
motive vehicles, with some governmental organizations such In June 2020, MOLIT established the UNECE R155 inter-
as JASPAR. Besides, METI creates a more concrete guideline national standards for automotive cybersecurity as the main
for testing and validation/certification of autonomous vehicles content for recommendations for ROK automotive manufac-
collaborating with organizations in industrial sector such as tures. The central component being that the automotive
IPA (Information Processing Agency). Until now, MLIT has manufacturer has a cybersecurity management system
published guidelines for requirements of autonomous vehicle (CSMS) and demonstrate that automotive cybersecurity is
development like [30] (Japanese). Also, IPA has published and managed accordingly. To integrate UNECE R155 local laws
revised more practical guidelines such as [31]. This guideline and regulations will be amended as appropriate [33]. MOLIT
includes threat analysis and possible measures in a develop- plans to issue the Automotive Cybersecurity law and safety/
ment cycle, namely management, planning, development, and security regulation in 2022. Until that time, they will have
operation. National standards are determined by organiza- published recommendations and guidelines to fill the gap
tions such as JAPSAR, based on the international standards. between the practice of automotive company and the require-
The national standards describe requirements that the ments imposed by the registration such as Korea Motor
industry must meet in the development process against Vehicle Safety Standard (KMVSS) and Korea Vehicle
assumed security threats. Especially, they have formulated Management Act. The approach taken by MOLIT is to ease
the new policy implementation and adoption recommenda- systematic standardization and regulation process, the agency
tions step-by-step. Currently, the differences of UN R155 and involves the industry in the regulation and standardization
the ROK implementation are that ROK extends the R155 to process by encouraging the formation [37] of Auto-ISAC [38]
their self-certification approval in addition to type approval, and receiving comments on the publications/reports that are
manufacturers obligation to report are focused on data published by the agency [39]. Currently, there are no standards
sharing between manufacturers, and administrative matters or regulations for automotive cybersecurity testing and veri-
(procedures, document, penalties) and matters relating to type fication, which is brought by the NHTSA. However, in 2016,
approval (CSMS certification, DETA data sharing) are yet to the agency published a non-binding document describing
be included in the implementation [33]. As one of the recom- guidelines and best practices for automotive cybersecurity
mendations, MOLIT announced guidelines for security of [40], which is revised in 2020 concerning the ISO/SAE DIS
autonomous vehicles on December 15, 2020. The guidelines 21434 draft standard and a draft version has been published
include (1) Ethical Guidelines for Self-Driving Vehicles, (2) (2020 draft) [11]. According to the comments brought on the
Automobile Cybersecurity Guidelines, and (3) Level 4 draft, a pre-final version has been released in 2022 [41]. The
Autonomous Vehicle Manufacturing/Safety Guidelines, document refers ISO/SAE 21434:2021 and NIST’s
which provide basic directions for ethics and safety [33]. Cybersecurity Framework for standardizing the cybersecurity
Among that, Automobile Cybersecurity guidelines introduced development, maintaining, and testing process.
recommendations for security policy directions so that auto-
mobile manufacturers can develop a cybersecurity system in National Initiatives With Regard to Automotive
preparation for the implementation of the security standards Product V&V Testing NHTSA conducts multifaceted
to be issued in 2022 [34]. The recommendations proposed in research on vehicle cybersecurity that leverages NIST’s cyber-
the guidelines are the following: security framework [42] and aims to collaborate with the
industry to address the challenges in vehicle cybersecurity.
•• Security management such as a process for identifying, NHTSA’s best practices documents include recommendations
evaluating, classifying, and managing security threats for automotive cybersecurity testing and documentation.
must be established within the manufacturer’s Those practices defined in [41] are as follows:
organization and share relevant information.
•• Vehicle security threat identification, evaluation, security •• Cybersecurity testing, including penetration testing
measures, and sufficient security-related pre-tests must should be implemented as a part of the
be performed. Note that security measures include development process.
cyberattack detection and prevention measures, risk •• Qualified testers who have not been a part of the
monitoring support measures, data forensics support development process should be included in the
measures for cyberattack analysis, and the like. testing phases.
To support the implementation of R155 as part of •• Identified vulnerabilities during cybersecurity testing
domestic regulations, MOLIT has planned to implement an should be analyzed; the vulnerability and how the
Automotive Cybersecurity Support and Response System. vulnerability is managed should be documented.
This system consists of an automotive cybersecurity committee •• All commercial-off-the-shelf and open-source software
to coordinate initiatives including the foundation of an auto- components used in vehicle ECUs should be evaluated
motive security center. The role of the Automotive by the manufacturers in order to identify
Cybersecurity Support and Response System is to provide the vulnerabilities.
cybersecurity test and evaluation and enforcement support,
For addressing the need for effective information sharing
support the private sector with the development of automotive
across the industry, NHTSA encouraged the formation of the
technologies, provide cybersecurity incident response, and
Auto ISAC, a community established by partners from the
support for the automotive sector [33].
various domains of the industry. In collaboration with the
3.3.2. North America Alliance of Automobile Manufacturers (Auto Alliance) and
the Association of Global Automakers (Global Automakers),
United States of America the community published a set of best practices documents
on automotive cybersecurity [43]. One of these documents,
Governance and Implementation of Regulation and “Security Development Lifecycle,” covers the security needs
ISO/SAE 21434:2021 In the U.S., National Highway Traffic for the development process and distributes the testing process
Safety Administration (NHTSA) is the responsible entity into the phases of development as follows [44]:
under the U.S. Department of Transportation (U.S.DOT),
which issues Federal Motor Vehicle Safety Standards (FMVSS) i. Design: This phase is where a high-level test plan can
to regulate and standardize the requirements for the safety of be constructed, which identifies:
motor vehicles [35]. The agency undertakes the responsibility •• The best security verification methods (e.g., design
of standardization and regulation of automotive cybersecurity review, manual code review, automated code
in the U.S. while conducting research in order to address the analysis, component/unit testing, bench and
challenges in the area [36]. To provide a comprehensive and vehicle penetration testing).
•• Needed testing tools including special build National Initiatives With Regard to Automotive
components and infrastructure support. Product V&V Testing The Canada Vehicle Cybersecurity
Guidance [45] provides technology-neutral and non-prescrip-
•• An evidence sheet with details of software,
hardware level, date, pass/fail status, notes on tive guiding principles for the incorporation of cybersecurity
failures or unexpected behavior person running throughout the vehicle lifecycle. The guidance promotes the
the test and approver, and others as necessary. importance of international standards such as ISO/SAE
21434:2021 and other related functional safety standards. The
ii. Implementation: Secure implementation requires guide provides a descriptive overview of the context of cyber-
testing and verification in both hardware and attacks to vehicular systems and in particular that more
software levels. The methods for ensuring at the advanced attacks tend to be associated with “white-hat” cyber-
hardware level: security research, while real-world, cyber-criminal threat
•• Confirmation reviews or assessments actors make use of the data-driven ecosystem of vehicular
technologies to comprise attacks on back-end systems and
•• Penetration tests systems that generate and store telemetry. To this end, the
At the software level: guide recommends the implementation of layered security
•• Code reviews controls (known as defense-in-depth), privacy protection, and
information protection procedures and testing of data security,
•• Automated code analysis secure external vehicle communications, identity manage-
•• Penetration testing ment and access control, secure software development, secure
updates, and the extended vehicle environment. Cybersecurity
iii. Testing and Validation: This part defines the whole testing is recommended to be conducted throughout the
process of testing through phases of the vehicle lifecycle. Penetration testing is mentioned as an essen-
development lifecycle: tial part of security auditing. Cybersecurity testing and valida-
tion methods are not explicit in the guidance provided by
1. Cybersecurity Testing: The actual testing process is
done during the implementation and post- Transport Canada. Transport Canada provides tier 1 and 2
implementation phase, which evaluates the proper automotive suppliers with a self-assessment tool: the Vehicle
working of safeguard mechanisms and identify Cybersecurity Assessment Tool (VCAT). The VCAT is a self-
potential vulnerabilities that leads to residual assessment questionnaire applicable for all vehicle types with
risk assessments. varying levels of connectivity and automated features. The
self-assessment questionnaire assists with evaluating the
2. Internal Cybersecurity Sign-off Process: The sign- cybersecurity performance and resilience of vehicles and
off process includes the testing process, which vehicular components. The VCAT will provide a score,
verifies the system is secure enough to withstand measuring cybersecurity posture, as well as recommendations
the previously assessed threats. This process for mitigations [45].
should include the overall test plan, performed
functional tests, penetration tests, source code
audits, and so forth. 3.3.3. Europe The European Union has a diverse range of
3. Residual Risk Assessments: Residual risk regulatory initiatives for cybersecurity of the digital market-
assessments can be done as a part of the place, which impact upon automotive product development.
development lifecycle on a periodic basis as the The EU Cybersecurity Act (CSA) is the predominant form of
known residual risks evolve over time by the regulation for cybersecurity in the EU market. Among the
discovery of new attack methods or cost reduction range of important initiatives, the CSA establishes a frame-
due to newer/cheaper tools. work for certification of ICT products for cybersecurity called
the Common Criteria-based European Candidate Cyberse-
curity Certification scheme (EUCC). The aim of the scheme is
Canada
to enable, for the consumer, transparency and awareness of
Governance and Implementation of Regulation and the level of assurance for cybersecurity of a digital product.
ISO/SAE 21434:2021 In Transport Canada’s Vehicle The EUCC is still in development and its impact on the auto-
Cybersecurity Strategy, the Canadian Department of motive sector is yet to be detailed [46].
Transport is responsible for monitoring the work of the The EU Cyber Resilience Act (CRA) [47] is currently
National Research Council Canada’s Automotive and Surface being developed. This regulation will focus on providing
Transportation Centre. The Automotive and Surface common cybersecurity rules for manufacturers and vendors
Transportation Centre engages in research and testing related of tangible and intangible digital products and ancillary
to advanced vehicle technologies. Examples include examina- services. The CRA regulation envisages a process for the
tion of cybersecurity vulnerabilities in connected features, digital product cybersecurity assurance where essential
mapping, and connectivity for automated driving. The testing baseline security requirements are defined, which can
and evaluation of cybersecurity is closely tied to applicable be applied selectively according to a risk management assess-
motor vehicle safety and data privacy legislation [45]. ment of a device’s intended use, considering the ecosystem or
“operational environment” in which the device will be placed. (QMC) of the German Association of the Automotive Industry
The products to be governed by the CRA include [47]: (Verband der Automobilindustrie—VDA) issued a supple-
ment to the process management specification Automotive
•• Connected product: A finished product that is intended
SPICE (Software Process Improvement and Capability
to communicate directly or indirectly over the internet.
Determination), which conforms with ISO 15504 [7]. This
•• Finished product: A product usable for its intended supplement, called Automotive SPICE for Cybersecurity
functions without being embedded or integrated into Engineering [49], defined a set of process steps dedicated to
any other product. Components of a device, such as a cybersecurity engineering that is to be used in conjunction
processor or a sensor, should be outside the scope as with the current Automotive SPICE process; namely:
security functionalities need to be assessed holistically.
•• SEC.1 Cybersecurity Requirements Elicitation
In the public submissions to the CRA regulation, automo-
tive industry bodies (European Automobile Manufacturers’ •• SEC.2 Cybersecurity Implementation
Association, European Association of Automotive Suppliers,
•• SEC.3 Risk Treatment Verification
TÜV Association) pointed to other existing legislation as
impacting automotive cybersecurity [47]: •• SEC.4 Risk Treatment Validation, and a new
management step
•• Type-approval: UN R155 and 156
•• MAN.7 Cybersecurity Risk Management, as well as
•• Radio Equipment Directive (2014/53/EU) and its expanding the acquisition step
delegated act (2022/30) (For Connected Vehicles)
•• ACQ.2 Supplier Request and Selection—In particular,
•• NIS 2 Directive (2020/0359(COD)) the risk treatment verification prescribes a specification
As the EU CSA is in policy implementation phase and that is suitable to provide evidence for compliance with
the EU CRA is in policy conception phase, there is a sparsity the security requirements and the design
implementation and component integration is to
of detail as to how automotive technologies will be validated
be tested using defined test cases (according to a
and verified for cybersecurity.
verification strategy that is derived from the
Germany requirements and implementation). The corresponding
best practices provides hints on what to test:
•• Requirements-based testing and interface testing on
ISO/SAE 21434:2021 In Germany, the Federal Motor
system and software level,
Transport Authority (Kraftfahrt-Bundesamt—KBA) is
responsible for bringing UNECE R155 into national legislation •• Check for any unspecified functionalities,
by issuing guidance and legally binding rules for application •• Resource consumption evaluation,
and review of the regulation [48]. This application document
specifies testing verification procedures by document review, •• Control flow and data flow verification, and
as well as functional security and penetration testing of a •• Static analysis; for software: static code analysis, e.g.,
technical service (e.g., TÜV) under witness/supervision of a industry-recognized security-focused coding
neutral party (KBA or an authorized body). standards. As well as some testing techniques (non-
exhaustive)
•• Network tests simulating attacks (non-authorized
Product V&V Testing As the EU CSA is in policy imple-
commands, signals with wrong hash key, flooding the
mentation phase and the EU CRA is in policy conception connection with messages, etc.), and
phase, there is a sparsity of detail as to how automotive tech-
nologies will be validated and verified for cybersecurity. •• Simulating brute force attacks,
•• Audits,
Germany Governance and Implementation of
Regulation and ISO/SAE 21434:2021 In Germany, the •• Inspections,
Federal Motor Transport Authority (Kraftfahrt–Bundesamt— •• Peer reviews,
KBA) is responsible for bringing UNECE R155 into national
legislation by issuing guidance and legally binding rules for •• Walkthroughs,
application and review of the regulation [48]. This application •• Code reviews.
document specifies testing verification procedures by
document review, as well as functional security and penetra-
tion testing of a technical service (e.g., TÜV) under witness/ Test cases could be derived by:
supervision of a neutral party (KBA or an authorized body). •• Requirements analysis,
•• Building equivalence classes,
Product V&V Testing The Quality Management Center •• Testing edge cases (boundary values),
•• Experience-based testing. The specification also proposes a part of Department for Business, Energy & Industrial
to establish bidirectional traceability between the Strategy. The center conducts research and publishes docu-
verification activities and the system design. ments regarding the safety and security of CAVs. The Centre
Analogously, the risk treatment has to be validated, for the Protection of National Infrastructure (CPNI) is another
which means the adequacy of the implemented measures entity that contributed on the research for security of CAVs
(whereas the verification assures the compliance of the [54]. In 2017, DfT, CPNI, and Centre for Connected and
measures with the requirements). The validation includes Autonomous Vehicles published a guidance document [54],
activities to also detect priorly unidentified which explained the cybersecurity needs of automotive
vulnerabilities (e.g., through penetration testing), while industry in eight principles. In 2021, BSI published a white-
the methodology is similar to the verification. paper [55], which defines the cybersecurity threat vectors for
connected vehicles and how to meet the compliance require-
France
ments defined by the ISO/SAE 21434:2021. The paper includes
Governance and Implementation of Regulation and an overview of ISO/SAE 21434:2021 and BSI’s E2E automotive
ISO/SAE 21434:2021 In 2021, the French legislature incor- cybersecurity model, which is compliant to a set of interna-
porated UNECE R155 & 156. The regulatory environment in tional standards including ISO/SAE 21434:2021.
France is conducive of close cooperation with the EU. The BSI PAS 1885:2018 [12] is a standards document that
Ministère de la Transition écologique et solidaire is the super- details the fundamental principles of cybersecurity across the
vising authority responsible for vehicle type approval. vehicle’s lifetime. The document provides principles that focus
Association Française de Normalisation (AFNOR) is respon- on organizational management of cybersecurity risks,
sible for automotive standardization, including cybersecurity management of the supply chain, third parties and subcon-
standards. The Agence nationale de la sécurité des systèmes tractors, and recommendations for cybersecurity design,
d’information (ANSSI) is the primary agency responsible for resilience, and response measures. Principle 6, “The security
cyber expertise and its role involves monitoring the cyber threat of all software is managed throughout its lifecycle,” prescribes
landscape, raising awareness of the necessary protections a list of recommendations for testing and evaluation of vehic-
required in the digital environment of France through best ular software. In summary, the recommendations are:
practices and standardization and providing technical advice •• Open source or third-party software should be reviewed
and assistance including cyber incident response through CERT for vulnerabilities using formal code inspection reviews.
France (CERT-FR) [50]. Among numerous measures contained Automated tools should be used to analyze the structure
in the Critical Information Infrastructure Law 2013, ANSSI and security of the code.
can impose technical and organizational requirements for
security and trigger audits. Recent domestic legislative updates •• Configuration and management control should include
in France reflect the widespread adoption in the EU of the EU evidence of testing, including test scenarios and results.
Cybersecurity Act and other related measures [51]. Also, unresolved test defects, deficiencies, and anomalies
should be documented.
National Initiatives With Regard to Automotive •• Updates shall be tested.
Product V&V Testing The French Ministry of the Interior
(Ministère de l’Intérieur) issued a position paper on auto- There is also an effort put by the British government
mated driving (L’automatisation des véhicules) [52] that toward the adaptation of CAVs. In 2019, the Centre for
contains an annex covering cybersecurity (Annexe 9: la Connected and Autonomous Vehicles has started a program,
Cybersécurité). Regarding testing, this annex contains the called CAVPASS, in order to implement standardization,
notion to use risk analyses, compliance audits, and penetra- testing, and monitoring processes to ensure the resilience of
tion tests. The ANSSI states in an analysis of contributions for CAVs against cyberattacks [56]. Zenzic is another organization
a—generic, but also including vehicles—cybersecurity certi- founded by the government and industry in order to embrace
fication scheme for the usage of static source code analysis the cybersecurity and safety challenges brought by the
tools, vulnerability scanners, automation of configuration Connected and Autonomous Mobility (CAM). The organiza-
audit, and protocol fuzzers for verification [53], which is, tion published a feasibility report in 2020 [57], which stated
however, a very high-level recommendation. the outcomes of several projects. The report included a part
regarding the measurement and monitoring the cyber resil-
United Kingdom ience, mentioning the digital twin technology for validation,
assurance, and certification of CAVs.
ISO/SAE 21434:2021 Department of Transportation (DfT)
and British Standardization Organization (BSI) are the main 4. Processes and Tools
entities in the United Kingdom toward the regulation and
standardization of the automotive cybersecurity, including Used in The Industry
the cybersecurity for connected and autonomous vehicles
(CAVs). DfT accommodates a center called “Centre for In order to examine which processes and tools are used in the
Connected and Autonomous Vehicles,” which serves also as industry, we issued questionnaires to experts in the field,
consisting of members of OEMs, suppliers, and automotive lifecycle. These are (1) fuzzing, (2) penetration testing, and (3)
engineering companies. The questions targeted in collecting functional testing. This section compiles these methods by
common practices on what is to be tested (test targets), how describing and referring to the phases of development that
to test (standards usage, test types, and test derivation), and each type of testing utilized. We also give further detail by
how to support the testing (test tools). adding other methods that are applicable for automotive
cybersecurity testing, which are found in the literature. These
are (3) model-based security testing, (4) risk-based security
4.1. Test Targets testing, and (6) vulnerability scanning.
E/E components remain the predominant areas of focus for Fuzzing: Fuzzing, or fuzz testing, refers to subjecting the
SUT due their importance for functionality of the vehicle. software system (or components individually) to a large
Due to the preponderance of connected vehicular technolo- volume of invalid, unexpected, or random inputs that are
gies, communication protocols are an area of concentric known as "fuzz.” By exposing the executable software to a
concern for cybersecurity testing. Emerging SUTs include the wide range of invalid data, vulnerabilities can be identified
end-to-end driving technology which supports autonomous- that are not known previously. To generate a variety of inputs
assisted and autonomous driving. Third-party service that can lead the program to failure, which is a difficult process
providers for verification and validation are popularly used to cover all cases, there are several techniques used. One of
due to their existing experience of testing and certification, them is to generate the input data based on the analysis of a
alignment with ISO/SAE 21434:2021 and other standards program’s coverage, behavior, and source code, another is to
which emphasize the use of third parties for independent implement mutation techniques on the generated data
verification and validation, and lack of available skills for according to the program’s feedback from the previously fed
cybersecurity testing of automotive products. A majority of data, or to randomly generate [58]. Fuzzing is conducted
respondents answered that they have an established interface during the development and testing phases of ECUs and info-
agreement for cybersecurity testing. Most OEMs follow a tainment systems to discover vulnerabilities, software bugs,
or unexpected behavior that may lead to failures.
document-based audition process in their verification and
Penetration Testing: Penetration testing is conducted to
validation agreement.
assess the security of the hardware, software, and communica-
tion systems, by mimicking real-world security attacks on the
4.2. Standards Utilization subject. It involves actively scanning and exploiting vulner-
abilities in the system with methods such as injection and
Overwhelmingly, ISO/SAE 21434:2021 is used for cybersecu- tampering to determine its susceptibility to unauthorized
rity verification and validation. Respondents also mentioned access, data breaches, or malicious activities. Penetration
well-established, complimentary standards such as ISO/IEC testing is performed during the entire development lifecycle
15408 (Common Criteria) and ISO/IEC 27034 (Application and before deployment to identify security flaws and mitigate
Security Standards). The testing process for SUTs are mainly them before they can be exploited by attackers.
conducted on a case-by-case basis. The limited use of test SAE J3061 and o ISO/SAE 21434 state the necessity of
matrix and standard test sets can be seen as due to a variety penetration testing and it is included as part of the best prac-
of reasons including repeatable test processes cannot be ubiq- tices document published by Auto ISAC [43]. Also, a recent
uitously applied to diverse range of automotive technologies, study [59] shows its wide usage among security testing types.
level of integration, and architecture requires testing to It is also seen that, among different knowledge levels, black-
be approached on a case-by-case basis, lack of development, box testing is the most preferred one for penetration testing.
and adoption of testing metrics and criteria, cybersecurity Functional Security Testing: Focuses on evaluating the
testing is still developing and there is a lack of adoption of security features and mechanisms of the system to ensure they
testing processes that support automation and repeatable function as intended. It involves subjecting security proper-
testing. OEMs conduct functional testing, vulnerability ties, such as authentication, authorization, encryption, and
scanning, penetration testing, and fuzz testing. All of these secure communication mechanisms to test and verify their
test procedures are recommendations of ISO/SAE 21434:2021 compliance with the security requirements and validate the
and are essential as part of an automotive cybersecurity testing behavior. This type of testing is applicable by both software-
program. Specifications coverage is the most popular method in-the-loop and hardware-in-the-loop testbeds, which may
to measure and maximize test coverage of the SUT. This aligns be utilized throughout the development [60]. Functional
with product development lifecycle and the focus on assurance security testing can be conducted throughout the development
for the intended functionality of the automotive component. lifecycle and pre-deployment stage to verify and validate the
Emerging methods include considerations for the require- security features.
ments from UN R155. Model-based Security Testing: Model-based security
testing involves creating formal or semi-formal models of a
feature, and using these models to perform security analysis
4.3. T
ypes of Testing and verification of conformity to requirements. Models can
Our survey results show that our respondents practice various be security properties (i.e., confidentiality, integrity, authen-
types of testing during different stages of their development tication, etc.), vulnerabilities, and security safeguards that are
being designed for the overall system, and also threats and equally proliferated to use white box (full access to informa-
attacks to the system [61]. This type of testing helps identify tion about the SUT), black box (just the SUT “as is,” with no
potential security weaknesses in the design and earlier phases additional information), and gray box (some information,
of the development lifecycle and enables engineers to mitigate mainly handbooks, API documentation, etc.) approaches.
the vulnerabilities by designing robust features. Only a minority (one-third) of the respondents claimed that
Risk-based Security Testing: Focuses on assessing the they use a baseline for testing. This means a minimum set of
security of a system based on the potential security risks and tests generically issued to all of their SUTs, regardless of their
their impact. This type of testing is based on threat analysis nature. The relative majority of those uses testing the require-
and risk assessment (TARA) techniques to prioritize the most ment specification followed by using prepared test plans, test
critical assets, threats, and vulnerabilities for allocating cases, and test data and, lastly, testing the design specification
testing resources accordingly [59]. Risk-based security testing and predefined generic tests for the source code itself. One
considers the likelihood of an attack, its potential impact on specific test set mentioned is testing all wireless and wired
the system, and the value of the assets at risk. It is performed interfaces (e.g., OBD) for their susceptibility to act as an entry
throughout the development lifecycle to ensure most critical vector into the vehicle.
security risks are addressed.
Vulnerability Scanning: This is a systematic process to
test the system for known vulnerabilities that can be exploited 4.5. Test Tool Categories
by known threats. This type of testing can target the source
code by conducting either static or dynamic analysis to under- Respondents use a diverse range of commercial-off-the-shelf
stand whether the software poses vulnerabilities due to (COTS), open-source (OS), customized, and in-house (inter-
memory usage or the interfaces for discovering unprotected nally developed) tools in their penetration testing activities.
entries (i.e. port scanning) [60]. Automated tools and scripts The results show a bias toward COTS and OS tools. The
are used in this approach so that they can be implemented as respondents also identified a number of tools that were used
part of the DevOps cycle to conduct regular and repeatable to test recent high-profile vulnerabilities such as Blueborne
tests with each increment, during development, and after (a well-known Bluetooth attack) and ROCA (cryptographic
deployment (i.e., for updated software). weakness). With the emphasis ISO/SAE 21434:2021 places on
Two-thirds of respondents confirmed that they utilize TARA, it is apparent that automotive cybersecurity testers are
functional testing and penetration testing within their veri- agile in developing and utilizing toolsets to keep pace with
fication and validation processes, which support the entire the dynamic threat environment. Table 2 categorizes specifi-
automotive development lifecycle. Validation activities were cally mentioned tools. When asked for specific tools during
conducted close to the end of the product development phase the phases of an attack test—pre-attack (scanning, CAN
and before release for post-development and consisted of analysis, etc.), attack (exploit frameworks, etc.), and post-
analysis and testing. Verification activities were conducted attack (reporting, life cycle management)—respondents
during the concept and product development phase and answered with a variety of tools.
consisted of review, analysis, and multiple rounds of penetra- Table 3 provides an overview of some commonly used
tion testing. One-third of respondents have not yet adopted tools, displaying the phase that are used in reconnaissance,
the cybersecurity verification and validation processes of the attack, or life cycle governance; the tool category (cf. Table 2);
ISO/SAE 21434:2021 standard. and the area of testing (IP/web, wireless, and in-vehicle
networks as well as reverse engineering). In that context, IP
Network/web testing tools refer to tools originally used in
4.4. Test Derivation traditional IT testing, targeting network, and web-based inter-
faces. Currently, they are ordinarily used mainly to perform
There is a couple of ways to derive test cases from a performed tests in automotive ethernet or on targets that have interfaces
asset/security analysis: based on derived requirements from similar to traditional IT systems, e.g., infotainment head units
a model (e.g., a TARA, cf. previous section) that could also running on an Android operating system. Wireless
be subject to model checking; based on specifications (both Automotive refers to tools to assess implementations of
standards and vendor specifications), based on the structure wireless protocol stacks that are popular in the automotive
(i.e., the architecture—e.g., tests that verify the correctness of industry, most prominently Bluetooth and WiFi. In-vehicle
a security gateway’s functioning), based on the experience of network (IVN) tools mainly refer to tools for testing CAN bus
the respective penetration tester (i.e., trusting the right test and Automotive Ethernet environments. Lastly, reverse engi-
cases to be designed to expert knowledge), or based on known neering tools are used to scrutinize binaries of automotive
faults. The respondents roughly evenly perform requirements, control systems and search for potential weaknesses inside
specification, and experience-based test derivation, while the code by following control flows. The other axis of the table
structure-based tests are significantly less (one-third) used, shows whether the tool is considered to be more in reconnais-
information is UNECE’s Regulation 155 (see above in the sance (information gathering) or attack (actual intrusion)
respective section) [1]. In its Annex 5 it defines a catalogue of phase of cracking a system, as well as life cycle management
countermeasures that can serve as requirements that might tools that support the security governance and help in
be verified by testing. Regarding the testing methods, it is planning tests throughout a system’s life cycle.
TABLE 2 Tool categories.

Tool category Description Automotive test usage
© International Alliance for Mobility Testing and Standardization

Vulnerability Enables performance of a scan of a device or information Nmap and Nessus could be used to find open
assessment system to discover vulnerability of the target system to communication ports on an infotainment head unit
known vulnerabilities. and its vulnerabilities.
Web application Enables analysis of the codebase of web applications and Predominantly used in the testing of infotainment
mobile device applications. systems and customer applications.
Reverse Used for analyzing the binary code of the software to
engineering identify vulnerabilities (due to memory usage, logic, etc.).
Tools such as IDA and Volatility (see Table 3) are used for
data extraction for analysis.
Protocol analysis Enables analysis of protocols to understand the Used for internal (CAN, LIN, MOST, FlexRay) and
architecture and identify vulnerabilities. external (Wireless, Radio, Bluetooth) networks.
Fuzzing Used to assess the security of a system to unsanitized Fuzzing is used ubiquitously from the embedded
data input. This can be either randomized or targeted hardware ECUs to the infotainment system, mostly
(IAMTS)
unsanitized data input. It is popularly used in software through customized or in-house tools aligned with the
engineering to identify bugs in the codebase. OEM software development processes.
5. Discussion leaving much room for interpretation open for implementers.

Regional standards are likewise high-level descriptive in
The most influential document is arguably UNECE R155 for general, focusing on engineering process topics. An exception
its normative and legally binding character. This document are some standards specifically from the Asian area that give
contains a list of requirements (in an annex) that could serve fine-grained descriptions for test procedures for single compo-
as test targets. Further details are specified on a national level, nents. The research of regional standards showed no clear bias
as is the details of the mandated CSMS. The specification of in testing procedures by region, although the underspecification
such a system is found in ISO 21434:2021. Both documents, leaves room for interpretation differences by both different
however, specify testing requirements at a very high level. regional authorities and implementers. What is missing globally
Therefore, the ISO maintains ongoing efforts to specify test is test implementation details for systems at vehicle level. The
classifications, as well as V&V procedures in more detail, giving reason, drawn out of expert interviews, is the early stage
guidance for testing. As the UNECE must be adopted into maturity of the topic. First, details for many of the components
national regulations, the concrete embodiments differ. have to emerge, before they can be tied to high-level test proce-
Nonetheless, it is common that the level of detail is coarse, dures at vehicle level. To perform testing and analysis, most
TABLE 3 Testing tools per attack phase, type, and category.

Wireless Reverse
Phase Tool IP network/web automotive IVN engineering Tool category
Reconnaissance Nessus ✓ Vulnerability assessment
Nmap ✓ Vulnerability assessment
Dirbuster ✓ Fuzzing
Bluescanner ✓ Vulnerability assessment © International Alliance for Mobility Testing and Standardization (IAMTS)
Wireshark ✓ Protocol analysis

GNU Radio Companion ✓ Protocol analysis
Universal Radio Hacker ✓ Protocol analysis
CANoe ✓ Protocol analysis
Attack tools Ghidra ✓ Reverse engineering
Android Studio ✓ Reverse engineering
Aircrack Suite ✓ Vulnerability assessment
URH ✓ Reverse engineering
Volatility ✓ Reverse engineering
Genymotion ✓ Protocol analysis
IDA ✓ Reverse engineering
Burpsuite ✓ Web application
American Fuzzy Lop ✓ Fuzzing
LCM PTC Integrity
players currently use general-purpose tools that are proliferated

in IT security testing (e.g., fuzzers, reverse engineering, and References
protocol analysis tools), as well as specialized hard- and software
for automotive systems (particularly CAN buses). These tools 1. United Nations Economic and Social Council—Economic
are used primarily in a manual testing process. There is few Commission for Europe, “Cyber Security and Cyber Security
automatic test generation and execution methodology for auto- Management System,” Regulation 155, Brussels, 2021.
motive security (such as [62]). Apart from systems for supporting 2. SAE International, “J3061-2 (WIP) Security Testing
the testing process by automating tasks (e.g., vulnerability Methods,” accessed June 28, 2023, https://www.sae.org/
scanning) and embedding this in an automated toolchain, one standards/content/j3061-2/; International Organization for
trend tends to be going toward model-based testing. Standardization and Society of Automotive Engineers, “Road
Vehicles—Cybersecurity Engineering,” ISO/SAE Standard
21434:2021, 2021.
3. Schmittner, C. and Macher, G., “Automotive Cybersecurity
6. Conclusion Standards—Relation and Overview,” in Computer Safety,
Reliability, and Security: SAFECOMP 2019 Workshops,
This research provided an overview of international and regional ASSURE, DECSoS, SASSUR, STRIVE, and WAISE, Turku,
standards and found that the current state-of-the-art lacks Finland, September 10, 2019, Proceedings, Romanovsky, A.,
Troubitsyna, E., Gashi, I., Schoitsch, E. et al. (Eds.) (Berlin,
proscribed detail of V&V procedures that would enable align-
Heidelberg: Springer-Verlag, 2019), 153-165, https://doi.
ment within different regions and industry. The developmental
org/10.1007/978-3-030-26250-1_12.
nature of V&V testing was further highlighted by the industry
working group responses, which demonstrated that traditional 4. Macher, G., Schmittner, C., Veledar, O., and Brenner, E.,
“ISO/SAE DIS 21434 Automotive Cybersecurity Standard—
enterprise information technology and processes were used.
In a Nutshell,” in Computer Safety, Reliability, and Security.
We see, however, that there is considerable development in this
SAFECOMP 2020 Workshops, Casimiro, A., Ortmeier, F.,
area with industry identifying connected and autonomous
Schoitsch, E., Bitsch, F. et al. (Eds.) (Cham: Springer
vehicle technologies as increasing in priority for testing and the International Publishing, 2020), 123-135.
focus on developing toolsets for automotive cybersecurity
5. International Organization for Standardization and Society
testing. Furthermore, we also see a concentration of effort by
of Automotive Engineers, “Road Vehicles—Cybersecurity
national authorities to enshrine UN R.155 into the national
Engineering,” ISO/SAE Draft International Standard DIS
regulatory frameworks for vehicle regulation and advocate for 21434, 2021.
best practice guidelines such as those in ISO/SAE 21434.
6. Schober, T. and Griessnig, G., “Cybersecurity Regulations
As the UNECE regulation and its accompanying traits
and Standards in the Automotive Domain,” in Systems,
are fairly new (first effective only in mid-2022), there is a
Software and Services Process Improvement (Communications
significant lack of experience on necessary test procedures. in Computer and Information Science), Yilmaz, M., Clarke,
Practical advice will emerge in greater detail when it could P., Messnarz, R., and Wöran, B. (Eds.) (Cham: Springer
be clarified how the legislation is actually handled. The same International Publishing, 2022), 530-539, https://doi.
applies for standards, as pivotal initiatives (e.g., from ISO) are org/10.1007/978-3-031-15559-8_38.
still in a very early project phase—with forthcoming of these 7. International Organization for Standardization,
endeavors more detailed specifications can be given. “Information Technology—Process Assessment—Part 5: An
Dedicated, automated toolchains will follow that trail, so far Exemplar Software Life Cycle Process Assessment Model,”
incipient stages are given. ISO/IEC Standard 15504-5, 2012.
8. Society of Automotive Engineers, “Cybersecurity Guidebook
Acknowledgments for Cyber-Physical Vehicle Systems,” SAE Standard
J3061_202112, 2021.
This work has been supported by the European Commission 9. United Nations Economic and Social Council—Economic
through the H2020 teaming project Finest Twins (grant No. Commission for Europe, “Software Update and Software
856602) and European Union’s Horizon 2020 Research and Update Management System,” Regulation 156,
Innovation Programme, under grant agreement No 883321 Brussels, 2021.
(CitySCAPE). The authors want to thank the International 10. United Nations Economic and Social Council—Economic
Alliance for Mobility Testing and Standardization (IAMTS) Commission for Europe, “UN Regulation on Uniform
for their support while writing this article. Provisions Concerning the Approval of Vehicles With
Regard to Cyber Security and of Their Cybersecurity
Management Systems,” Technical Report ECE/TRANS/
Contact Information WP.29/2020/79, Brussels, 2020.
11. National Highway Traffic Safety Administration,
Stefan Marksteiner “Cybersecurity Best Practices for the Safety of Modern
Corresponding author Vehicles (Draft Update 2020),” Draft Update of DOT HS 812
[email protected] 333, Washington, DC, 2020.
12. British Standards Institution, “The Fundamental Principles 23. The Ministry of Industry and Information Technology of
of Automotive Cyber Security—Specification,” BSI PAS China (MIIT), “Suggestions on Strengthening the Type
1885:2018, 2018. Approval Management of Intelligent & Connected Vehicle
13. Hu, S., Chen, Q.A., Sun, J., Feng, Y. et al., “Automated Manufacturers and Products,” MIIT 5, 2021.
Discovery of Denial-of-Service Vulnerabilities in Connected 24. National Technical Committee of Auto Standardization,
Vehicle Protocols,” in 30th USENIX Security Symposium “General Technical Requirements for Vehicle Cybersecurity,”
(USENIX Security 21), USENIX Association, Vancouver, GB/T 40861–2021, 2021.
Canada, 2021, 3219-3236, https://www.usenix.org/ 25. The Ministry of Industry and Information Technology of
conference/usenixsecurity21/presentation/hu-shengtuo. China (MIIT), “Notice of the Ministry of Industry and
14. Shen, J., Won, J.Y., Chen, Z., and Chen, Q.A., “Drift with Information Technology on Strengthening the Cyber
Devil: Security of Multi-Sensor Fusion Based Localization in Security and Data Security of Internet of Vehicles,” MIIT
High-Level Autonomous Driving under GPS Spoofing,” in Cybersecurity 134, 2021.
Proceedings of the 29th USENIX Security Symposium (2020), 26. The Ministry of Industry and Information Technology of
Boston, MA, 2020, 931-948. China (MIIT), “Security Technical Requirements for
15. Sun, J., Cao, Y., Chen, Q.A., and Morley Mao, Z., “Towards Connected Vehicle Based on Public Telecommunication
Robust LiDAR-Based Perception in Autonomous Driving: Network,” YD/T 3737-2020, 2020.
General Black-Box Adversarial Sensor Attack and 27. Chinese National Information Security Standardization
Countermeasures,” in Proceedings of the 29th USENIX Technical Committee, “Information Security Technology—
Security Symposium (2020), Boston, MA, 2020, 877-894, Cybersecurity Technical Requirements for In-Vehicle
arXiv:2006.16974. Network Equipment,” Technical Report, 2020.
16. Kim, H., Ozgur Ozmen, M., Bianchi, A., Berkay Celik, Z. et 28. Chinese National Information Security Standardization
al., “PGFUZZ: Policy-Guided Fuzzing for Robotic Vehicles,” Technical Committee, “Technical Requirements for
in Network and Distributed System Security Symposium Cybersecurity of Electric Vehicles Charging System (Draft
(NDSS), Virtual, 2021, 1-18, https://beerkay.github.io/papers/ for Comments),” GB/T, 2020.
Berkay2021PGFuzzNDSS.pdf. 29. Chinese National Automotive Standardization Technical
17. Kim, T., Kim, C.H., Rhee, J., Fei, F. et al., “RVFuzzer: Committee, “Technical Requirements and Test Methods for
Finding Input Validation Bugs in Robotic Vehicles through Cybersecurity of Remote Service and Management System
Control-Guided Testing,” in 28th USENIX Security for Electric Vehicles,” GB/T, 2021.
Symposium (USENIX Security 19), USENIX Association, 30. Japanese Ministry of Land, Infrastructure, Transport and
Santa Clara, CA, 2019, 425-442, https://www.usenix.org/ Tourism Automobile Bureau, “Safety Technical Guidelines
conference/usenixsecurity19/presentation/kim. for Self-Driving Vehicles,” Technical Report, 2018.
18. Vinzenz, N. and Oka, D.K., “Integrating Fuzz Testing into the 31. Information-Technology Promotion Agency, Japan,
Cybersecurity Validation Strategy,” SAE Technical Paper “Approaches for Vehicle Information Security,” Technical
2021-01-0139 (2021), doi:https://doi.org/10.4271/2021-01-0139. Report, 2013.
19. Ebrahimi, M. et al., “A Systematic Approach to Automotive 32. Japan Automotive Software Platform and Architecture
Security,” in Formal Methods, Lecture Notes in Computer (JASPAR), “About Us,” accessed November 10, 2023, https://
Science, Chechik, M., Katoen, J.-P., and Leucker, M. (Eds.) www.jaspar.jp/en/about_us.
(Cham: Springer International Publishing, 2023), 598-609, 33. Ministry of Land, Infrastructure and Transportation,
doi:10.1007/978-3-031-27481-7_34. “Approach of Republic of Korea Harmonizing the UN
20. Oka, D., “Fuzz Testing Virtual ECUs as Part of the Regulation No. 155,” Technical Report, 2021.
Continuous Security Testing Process,” SAE Int. J. Transp. 34. ATIC, “Brief Analysis of the July 2022 Korean Regulatory
Cyber. & Privacy 2, no. 2 (2020): 159-168, doi:https://doi. Updates,” Technical Report, 2022.
org/10.4271/11-02-02-0014. 35. National Highway Traffic Safety Administration,
21. United Nations Economic and Social Council—Economic “Understanding NHTSA’s Regulatory Tools,” Report,
Commission for Europe, “Agreement Concerning the Washington, DC, 2017.
Adoption of Harmonized Technical United Nations 36. NHTSA, “Vehicle Cybersecurity,” accessed June 28, 2023,
Regulations for Wheeled Vehicles, Equipment and Parts https://www.nhtsa.gov/technology-innovation/vehicle-
which can be Fitted and/or be Used on Wheeled Vehicles and cybersecurity.
the Conditions for Reciprocal Recognition of Approvals
37. National Highway Traffic Safety Administration, “Report to
Granted on the Basis of these United Nations Regulations,”
Congress: ‘Electronic Systems Performance in Passenger
ECE/TRANS/WP.29/343/Rev. 30, 2022, 43.
Motor Vehicles’,” Technical Report, 2015.
22. The Ministry of Industry and Information Technology of 38. McCarthy, C., Harnett, K., Carter, A., and Hatipoglu, C.,
China (MIIT), “Opinions of the Ministry of Industry and “Assessment of the Information Sharing and Analysis Center
Information Technology on Strengthening the Management Model,” Technical Report DOT HS 812 076, National
of Smart Connected Automobile Manufactures and Product Highway Traffic Safety Administration, Washington,
Permit,” MIIT Equipment Industry 103, 2021. DC, 2014.
39. NHTSA, “NHTSA Seeks Comment on Cybersecurity Best Sécurité. Inspection Generale de l’administration 16040-R,”
Practices for the Safety of Modern Vehicles,” accessed June 28, Inspection generale de l’administration and Conseil general
2023, https://www.nhtsa.gov/press-releases/nhtsa-seeks- de l’environnement et du developpement durable, 2017.
comment-cybersecurity-best-practices-safety-modern-vehicles.
53. Agence nationale de la sécurité des systèmes d’information,
40. National Highway Traffic Safety Administration, “Analyse Des Contributions Reçues Suite à l’appel à
“Cybersecurity Best Practices for Modern Vehicles,” Manifestation d’intérêt Sur La Certification de Sécurité de
Technical Report DOT HS 812 333, Washington, DC, 2016. Niveaux Substantiel et Élémentaire,” Technical Report, 2019.
41. National Highway Traffic Safety Administration, 54. United Kingdom Department for Transport, “The Key
“Cybersecurity Best Practices for the Safety of Modern Principles of Cyber Security for Connected and Automated
Vehicles,” Pre-Final, Washington, DC, 2022. Vehicles,” Technical Report, 2017.
42. National Institute of Standards and Technology, “Framework
55. British Standards Institution, “Automotive Cybersecurity
for Improving Critical Infrastructure Cybersecurity,”
Insights Paper,” BSI PAS, 2021.
Technical Report, Gaithersburg, MD, 2018.
56. “Centre for Connected and Autonomous Vehicles,
43. Automotive Information Sharing and Analysis Center, “Best
“Connected and Automated Vehicles: Process for Assuring
Practices,” Technical Report, 2016.
Safety and Security (CAVPASS),” accessed June 28, 2023,
44. Automotive Information Sharing and Analysis Center, “Best https://www.gov.uk/guidance/connected-and-automated-
Practices—Security Development Lifecycle,” Technical vehicles-process-for-assuring-safety-and-security-cavpass.
Report, 2020.
57. Zenzic, “Cyber Resilience in Connected and Automated
45. Transport Canada, “Canada’s Vehicle Cyber Security Mobility (CAM)—Cyber Feasibility Report,” 2020.
Guidance,” Technical Report T46-61/2020E, 2020.
58. Li, J., Zhao, B., and Zhang, C., “Fuzzing: A Survey,”
46. European Union, “The EU Cybersecurity Act,” Technical Cybersecurity 1, no. 1 (2018): 6, doi:https://doi.org/10.1186/
Report, 2020. s42400-018-0002-y.
47. European Union, “Cybersecurity Resilience Act,” Technical 59. Luo, F., Zhang, X., Yang, Z., Jiang, Y. et al., “Cybersecurity
Report, 2022. Testing for Automotive Domain: A Survey,” Sensors 22, no.
48. Kraftfahrt-Bundesamt, “Application of the Rules for 23 (2022): 9211.
Designation/Recognition for Technical Services (Categories 60. Mahmood, S., Nguyen, H.N., and Shaikh, S.A., “Automotive
A, B, D),” Technical Report, 2021. Cybersecurity Testing: Survey of Testbeds and Methods,” in:
49. VDA QMC Project Group 13, “Automotive SPICE—Process Digital Transformation, Cyber Security and Resilience of
Reference and Assessment Model for Cybersecurity Modern Societies, Studies in Big Data, vol. 84, Tagarev, T.,
Engineering,” Core Specification 1.0, Quality Management Atanassov, K.T., Kharchenko, V., and Kacprzyk, J. (Eds.) (2021),
Center of the German Association of the Automotive Springer, Cham, https://doi.org/10.1007/978-3-030-65722-2_14.
Industry, 2021. 61. Felderer, M., Zech, P., Breu, R., Büchler, M. et al., “Model-
50. Ministère de la Transition écologique et solidaire, Based Security Testing: A Taxonomy and Systematic
“Cybersecurity in France for Civil Aviation,” Technical Classification,” Software Testing Verification and Reliability
Report, Direction générale de l’Aviation civile, 2018. 26, no. 2 (2015): 119-148, doi:10.1002/stvr.1580.
51. Agence nationale de la sécurité des systèmes d’ information, 62. Marksteiner, S., Bronfman, S., Wolf, M., and Lazebnik, E.,
“Cybersecurity Act,” accessed November 10, 2023, https:// “Using Cyber Digital Twins for Automated Automotive
www.ssi.gouv.fr/administration/reglementation/ Cybersecurity Testing,” in 2021 IEEE European Symposium
cybersecurity-act/. on Security and Privacy Workshops (EuroS PW), Vienna,
52. Rocchi, J.-F., Bodino, P., De Tréglodé, H., Flury-Hérard, B. et Austria, 2021, 123-128, doi:https://doi.org/10.1109/
al., “L’automatisation Des Véhicules; Annexe No. 9: La Cyber EuroSPW54576.2021.00020.
© 2024 International Alliance for Mobility Testing and Standardization (IAMTS). Published by SAE International. This Open Access article is published under the terms
of the Creative Commons Attribution Non-Commercial, No Derivatives License (http://creativecommons.org/licenses/by-nc-nd/4.0/), which permits use, distribution,
and reproduction in any medium, provided that the use is non-commercial, that no modifications or adaptations are made, and that the original author(s) and the
source are credited.
Positions and opinions advanced in this work are those of the author(s) and not necessarily those of SAE International. Responsibility for the content of the work lies
solely with the author(s).

2311 JCAV AutomotiveSecurityTestingStandards

Uploaded by

Copyright:

Available Formats

2311 JCAV AutomotiveSecurityTestingStandards

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

2311 JCAV AutomotiveSecurityTestingStandards

Uploaded by

Copyright:

Available Formats

Downloaded from SAE International by Nan Kai University of Technology, Friday, November 17, 2023

A Global Survey of Standardization

2 Roberts et al. / SAE Int. J. of CAV / Volume 7, Issue 2, 2024

1. Introduction •• We conducted a survey of tools and practices commonly

Roberts et al. / SAE Int. J. of CAV / Volume 7, Issue 2, 2024 3

standards and documents)

4 Roberts et al. / SAE Int. J. of CAV / Volume 7, Issue 2, 2024

National Initiatives With Regard to Automotive

Roberts et al. / SAE Int. J. of CAV / Volume 7, Issue 2, 2024 5

6 Roberts et al. / SAE Int. J. of CAV / Volume 7, Issue 2, 2024

Roberts et al. / SAE Int. J. of CAV / Volume 7, Issue 2, 2024 7

8 Roberts et al. / SAE Int. J. of CAV / Volume 7, Issue 2, 2024

Roberts et al. / SAE Int. J. of CAV / Volume 7, Issue 2, 2024 9

10 Roberts et al. / SAE Int. J. of CAV / Volume 7, Issue 2, 2024

Roberts et al. / SAE Int. J. of CAV / Volume 7, Issue 2, 2024 11

12 Roberts et al. / SAE Int. J. of CAV / Volume 7, Issue 2, 2024

TABLE 2 Tool categories.

© International Alliance for Mobility Testing and Standardization

5. Discussion leaving much room for interpretation open for implementers.

TABLE 3 Testing tools per attack phase, type, and category.

Wireshark ✓ Protocol analysis

Roberts et al. / SAE Int. J. of CAV / Volume 7, Issue 2, 2024 13

players currently use general-purpose tools that are proliferated

14 Roberts et al. / SAE Int. J. of CAV / Volume 7, Issue 2, 2024

Roberts et al. / SAE Int. J. of CAV / Volume 7, Issue 2, 2024 15

You might also like

1. Introduction •• We conducted a survey of tools and practices commonly

5. Discussion leaving much room for interpretation open for implementers.