1

Safety and Security Co-Analyses:

A Systematic Literature Review
Elena Lisova, Irfan Šljivo, and Aida Čaušević

Abstract—Latest technological trends lead towards systems Safety engineering and security engineering as a way of
connected to public networks even in critical domains. Bringing addressing safety/security challenges have developed sepa-
together safety and security work is becoming imperative, as a rately. While the malfunctioning behaviour addressed by safety
connected safety-critical system is not safe if it is not secure.
The main objective of this study is to investigate the current engineering was the primary concern in such systems, the
status of safety and security co-analysis in system engineering increased risk of intentionally caused harm required additional
by conducting a Systematic Literature Review. The steps of the focus on security engineering. Nowadays, there is a need to
review are the following: the research questions identification; integrate safety and security engineering in such a way that
agreement upon a search string; applying the search string the unreasonable risk of harm due to either malfunctioning or
to chosen databases; a selection criterion formulation for the
relevant publications filtering; selected papers categorization and malicious intent is adequately addressed. This is particularly
analysis. We focused on the early system development stages important for highly connected modern safety-critical systems
and identified 33 relevant publications categorized as: combined that cannot be considered safe unless they are secure at the
safety and security approaches that consider the mutual influence same time. The way in which this integration is performed
of safety and security; safety informed security approaches that significantly influences the efforts needed to design a safe
consider influence of safety on security; security informed safety
approaches that consider influence of security on safety. The and secure system. For example, safety and security solutions
results showed that a number of identified approaches are driven do not always support each other, e.g., encrypting a message
by needs in fast developing application areas, e.g., automotive, needed for security reasons increases the time needed to
while works focusing on combined analysis are mostly application deliver the message, which may increase the delivery time
area independent. Overall, the study shows that safety and over the required safety threshold. If safety and security are
security co-analysis is still a developing domain.
being treated separately and their integration takes place at
Index Terms—Functional safety, security, co-analysis, require- later development stages, it implies greater effort to harmonise
ments engineering, hazards, vulnerabilities, threats. different solutions. As with requirements engineering, the later
the inconsistencies are detected, the more work needs to
I. I NTRODUCTION be performed due to repetition. The earlier the integration
of safety and security can be achieved, the fewer iterations
ITH ubiquitous presence of technology and our in-
W creased reliance on it, the risk of harm we face due to
such technology increases as well. The harm we are exposed
are needed to harmonise them. We have identified the early
system development stages where safety and security analyses
are performed as the most critical stage for their harmonisa-
to is not just direct physical harm due to for example car acci- tion. Significant amount of academic effort is being invested
dents, but it includes e.g., financial, environmental, emotional into researching harmonisation at early system development
harm, which can also lead to physical harm. Traditionally, stages [4]–[6]. At the same time, state of the practice is lagging
different causes that may lead to harm have been treated behind due to the strict certification and standardization re-
separately in safety-critical system engineering. For example, quirements that take longer time to adapt to new developments.
unreasonable risk of harm due to malfunctioning behaviour In this paper we investigate the existing research that
of technological systems is addressed under the umbrella addresses the analysis of both safety and security aspects.
of functional safety, where functional safety is described For this purpose we present a structured map of the available
as “a freedom from unacceptable risk” [1]. With increased research literature, focusing on the holistic safety and security
connectivity of these systems, the risk of undesirable conse- analysis by conducting a systematic literature review (SLR)
quences has increased due to the possibility of an adversary method as described in Section II. The goal of the study
intentionally causing the undesirable consequences. The risk is to get better comprehension of the available safety and
of such intentionally caused harm through the technological security analysis approaches. In particular, we explore what
systems has been generally addressed by security solutions, kind of integration the available approaches promote. This
which were traditionally analysed and proposed separately information can tell us if the research is converging towards a
from safety solutions [2]. Security is often defined as a system particular kind of integration, and what are the causes for such
property that allows the system “to perform its mission or convergence. The insights from this study might be useful for
critical functions despite risks posed by threats” [3]. both academia and industry, as the first might get a better view
of the directions and possible gaps in state-of-the-art, while the
E. Lisova, I. Šljivo, and A. Čaušević are with Mälardalen University,
Västerås, Sweden. latter can use the study as a source to find suitable co-analysis
E-mail: {elena.lisova, irfan.sljivo, aida.causevic}@mdh.se methods relevant for their domains. We present the results and
 2

The main reason for limiting the scope of the work to

the early system development stages is the importance of the
harmonisation of safety and security at those stages for the
overall cost and effort needed to design a safe and secure
system. Considering the whole development lifecycle could
introduce a larger amount of publications that we would not
be able to asses in the right way, possibly losing the focus from
the main goal. Hence, in this review, we focus on analyses that
take part during the hazards and threats identification and risk
assessment, and requirements elicitation/analysis. Since both
safety and security engineering promote to design safety and
Fig. 1: Steps of the SLR process security in the system from the beginning, the selected stages
become critical as there we identify the problems and choose
ways to address them. The earlier we discover inconsistencies
their analysis in Section III. The classification of the relevant between the chosen solutions, the fewer repetitions are needed.
papers is done with respect to type of the approach, where The efforts needed to achieve an adequate design depends
we have identified three categories: security informed safety, on whether the safety and security analyses are performed
safety informed security and combined safety and security separately and then the results are evaluated jointly, or if a
approaches. Additionally, we have investigated if the analysis unified analysis is performed. Hence, we also investigate the
is performed on the hazard/threat identification level or during nature of those analyses in more depth.
requirements engineering. Furthermore, we have investigated
whether the analyses of both safety and security are uni-
B. Scope of the Search
fied or they are parallel and need additional harmonisation.
Finally, we have identified different characteristics of each In the next step of the study we have specified the search
publication regarding the relation with industry, application string that is used to find relevant publications in selected
domain, relevant standards and the type of validation used. We databases. The search string is based on the keywords that are
discuss validity of presented results in Section IV, followed in line with the purpose of the paper, as discussed in Section I.
by Section V where we present the related work. The final We specify the following Boolean string to search the relevant
conclusions are described in Section VI. databases:
(“safety” AND “security” AND “analysis”)
II. R ESEARCH M ETHOD The following on-line databases have been part of the
The work in this paper is based on a SLR, an empirical investigation: IEEE Xplore digital library 1 , ACM digital li-
study with the purpose to evaluate and interpret all available brary 2 , Web of Science 3 , SpringerLink 4 . We have performed
research relevant to a particular question, topic of interest or our search to find suitable publications from year 2012 until
phenomenon. We have adopted an approach and guidelines the end of 2017. We have restricted the search to journal,
to conduct a SLR proposed by Kitchenham [7]. The aim of conference and workshop papers as well as peer-reviewed
a SLR is to present an impartial evaluation of a research book chapters, while excluding non-peer reviewed abstracts
topic using reliable and rigorous methodology. In the review and publications from the search. We have used the Mendeley
process, we have adopted a process that defined all steps application to record the search results.
needed to be taken for a review to account as a SLR. The
process includes description of the research questions, search
C. The Selection Criterion
string specification, publication sources identification as well
as preliminary study selection criterion and data mapping. The Our initial search, i.e., Stage 1 in Table I, resulted in
process is depicted in Fig. 1 and the details of individual steps 13711 papers in total. For a paper to be included in the
are provided in the following sections. next phase, the following criterion must be met, Criterion 1:
the publication must propose or discuss safety and security
A. Research Questions analysis approaches in system engineering. Thus, at Stage 1
at least one person has read all the titles and excluded papers
SLRs are driven by a specific purpose translated into a set that have not been related to the inclusion criterion. After this
of research questions that is the initial step of the study. We step we have had 351 papers for Stage 2 as given in Table I,
formulate the following research questions focusing on the where we have read both the titles and abstracts. This step has
research area of “Functional safety and security co-analysis been performed by all reviewers, i.e., the review authors, with
at the early system development stages”: a possibility to grade every paper with one out of three choices:
RQ-1: What are the analysis methods adopted to address in- X - paper is not relevant,  - paper is borderline, X- paper
terplay of safety and security at the early system development
1 http://ieeexplore.ieee.org/Xplore/home.jsp
stages? 2 https://dl.acm.org/
RQ-2: How do the identified analysis methods address the 3 http://webofknowledge.com/

interplay of safety and security? 4 https://link.springer.com/

 3

TABLE I: Study selection stages performed by the authors TABLE III: Stage 3 paper distribution
Stage Activity Papers Paper group Number of papers Relevant papers
1 Applied the search query to all the sources and 13711 X X X 41 22
gathered the results X X  8 2
2 Applied inclusion/exclusion criterion to the papers 351 X  X 7 3
titles and abstracts  X X 6 3
3 Applied inclusion/exclusion criterion to the full texts 69 X X X 3 0
4 Finalised the set of included papers 33 X X X 1 1
X X X 3 2
TABLE II: Stage 2 paper distribution
Paper group
Reviewer 1 Reviewer 2 Reviewer 3
Number of papers identified as relevant at the Stage 4 out of the the initial search
X X X 154 result at Stage 1 within a particular search source, varied from
X X X 41 0.1% to 2.4% and recall, i.e., how many papers within a par-
X  X 19
X X  21
ticular search source are identified as relevant out of relevant
 X X 19 papers collected from all sources, from 7% to 39%. IEEE
X   13 Explore and SpringerLink have been the sources with the most
   10 selected studies (13) and (14), and with an average precision
X  X 9
  X 8 of 0.6% and 0.2% respectively. SpringerLink also has had
X X  8 the highest number of items returned by the query (8479).
X  X 7 When it comes to the coverage level, SpringerLink has had
X   5 the highest coverage (42%), the next is IEEE Explore (39%),
 X X 6
X X X 4 then ACM digital library (12%) and Web of Science (WoS)
  X 4 (7%). IEEE Explore, ACM digital library and SpringerLink
X X X 3 have been chosen as prime sources for the search, while WoS
 X  5
 X  3
has been considered as a secondary source since it covers
X X  3 publications from multiple publishers. Since papers from WoS
X X X 2 have been overlapping with the papers identified from the
X X X 3 prime sources, only 15 papers from WoS, not already in other
 X X 2
X  X 1 databases, have been included in Stage 2 (see Table IV). We
X X X 1 have analysed what kind of papers have been published in this
X X X 0 area (Table Vb) and how many studies have been published
X X  0
per year (Table Va). Searches in all sources are covering the
 X X 0
range 2012 - 2017.

is relevant. As a result, the groups presented in Table II have E. Data Mapping

been derived. Stage 3 in Table III implies reading of complete
In this phase the 33 selected papers have been categorised
papers and includes papers for which at least two reviewers
into five groups. The classification has been based on titles,
have marked it as relevant (X). At stage 3, we have evaluated
abstracts, and full-text reading. A brief description of each
the papers to see whether they include a complete description
group is provided below, while detailed discussion can be
about an analysis method (i.e., position or work in progress
found in Section III.
papers have been discarded) that incorporates both safety and Five groups have been derived based on the following two
security, as well as evaluation on a case study. This approach criteria: (i) what is the overall reason for considering both
has provided us with 69 papers to be included at Stage 3 (see safety and security: to achieve a safe system, to achieve a
Table III). While reading the full papers, we have had cases secure system, or to achieve both safe and secure system; (ii)
when not all reviewers agreed upon whether the paper should how is the process of performing both safety and security
be included or not. Such cases have been discussed to make analyses done: jointly — both safety and security analysis
a consensus, and some of these papers have been included in are part of the same activity, parallel — safety and security
the final list. Within Stage 4 we have identified 33 papers as analyses are performed separately and an additional activity is
relevant for addressing our RQ-1 (Section II-A). needed to integrate the results. Based on the first criterion we
identify the following three groups: combined approaches —
D. Characterization of the Selected Papers safety and security are both the overall target of the analysis,
Our selection process resulted in 33 papers on safety and se- security informed safety approaches – safety is considered
curity analysis. Table IV shows the number of papers returned as an overall goal, safety informed security approaches —
by each source, as well as the number of papers that we have methods where performing both safety and security analyses
selected after applying the inclusion/exclusion criterion. The is done for the sake of achieving a secure system. We identify
table also includes the differences between sources in terms two more groups based on the second criterion: unified ap-
of included studies (precision) and coverage level (recall). The proaches — safety and security are analysed jointly, parallel
precision of the selected sources, i.e., how many papers are approaches — additional harmonization of separate safety and
 4

TABLE IV: Number of retrieved papers per source

Source Stage 1 Stage 2 Stage 3 Stage 4 Precision (%) Recall (%)
IEEE Explore 2264 167 40 13 0.6 39
ACM digital library 166 116 7 4 2.4 12
SpringerLink 8479 53 18 14 0.2 42
Web of Science 2802 15 4 2 0.1 7

TABLE V: Paper distributions

Publication year Quantity Percentage (%) Paper type Quantity Percentage (%)
2012 2 6 Conference paper 23 70
2013 2 6 Workshop paper 3 9
2014 6 18 Symposium paper 5 15
2015 5 15 Journal 1 3
2016 7 22 Book chapter 1 3
2017 11 33 Total 33 100
Total 33 100
(a) Paper distribution per year (b) Paper type distribution

security analyses outcomes is required to address the possible development phase, and provides as an outcome security and
dependencies between safety and security. safety requirements specification.
Beside this classification we also take into account informa- 2) Reichenbach et al. (2012) [11] propose an approach on
tion regarding application area, existence of validation within combined safety and security risk analysis by extending Threat
the approach, source of publication (i.e., research or industrial Vulnerability and Risk Assessment (TVRA) technique with
community), as well as whether the approach is associated in Safety Integrity Levels (SILs) from the generic functional
any way with existing standards (see Table VI). More detailed safety standard IEC 61508 [1]. The risk associated with a
classification of retrieved results have been done with respect function in this extended TVRA is calculated based on both
to which part of the lifecycle the approach is applicable to. security factors as well as SILs of the considered function.
We have considered Hazard Analysis and Risk Assessment The approach aims at identifying which security vulnerabilities
(HARA) [8], approaches that provide hazard identification, as are safety-relevant. The technique does not depend on safety
well as hazard analysis including identification and assessment analysis, but provides means to identify the influence of
of environmental conditions along with exposure or duration. security vulnerabilities on safety.
Additionally, Threat Assessment and Remediation Analysis 3) Silva and Lopes (2013) [12] present activities that have
(TARA) that has been defined in SAE J3061 [9] has been been performed in order to certify a safety-critical system in
considered. It is an engineering methodology to identify, the railway domain and describe how security can be taken
prioritize, and respond to cyber threats by introducing coun- care of without endangering reliability or safety. In this work
termeasures that reduce sensitivity to cyber attack. Finally, we they use Failure Modes, Vulnerabilities and Effect Analysis
have also considered analysis at the Requirement Engineering (FMVEA) and fault tree analysis where for every safety failure
(RE) stage, which is the process of requirements elicitation, event they derive possible security failure events.
analysis and conflict resolving (see Table VI). 4) Young and Leveson (2013) [6] propose a STPA-Sec
method, which is based on already existing top-down safety
III. R ESULTS AND A NALYSIS hazard analysis method System-Theoretic Process Analysis
In this section we first briefly describe papers that are (STPA). The method requires a multidisciplinary team con-
identified as relevant to RQ-1, and further present analysis sisting of security, operations, and domain experts to identify
results of our findings, relevant for answering RQ-2. and constrain the system from entering vulnerable states that
lead to losses and is useful at the concept phase. In the
A. Papers Identified as Relevant approach hazards are presented as control problems. Each
control action is reviewed under a set of a different conditions
This subsection presents a brief overview of the 33 papers
and guidewords to identify loss scenarios. The approach allows
ordered in the chronological and alphabetical order that we
to focus on vulnerable states in order to avoid threats to exploit
have identified as relevant.
them and create disruptions, and eventual losses.
1) Raspotning et al. (2012) [10] present Combined Harm 5) Chen et al. (2014) [13] build upon extending the NIST
Assessment of Safety and Security for Information Systems 800-30 [14] methodology to consider safety aspects contribut-
(CHASSIS) that is a high level approach combining safety ing to risk assessment by establishing a functional relationship
and security methods in order to provide a joint assessments between vulnerabilities, threats and hazards. Hazards occur-
approach, suitable for early phases of system development. rence levels are assigned depending on a value of a hazard-
The approach is based on modelling misuse cases and misuse threat conditional probability. The assets impact is assigned
sequence diagrams within a UML behaviour diagram, which based on a critical digit asset characterization. These values
might imply some additional modelling expenses for the early
 5

along with control risk reflecting safety and security design 11) Gu et al. (2015) [24] present an approach for treating
assessment define safety-security risk of an incident. safety and security requirements together with a focus on
6) Ito (2014) [15] proposes an analysis for threats and hazards resolving their conflicts. The analysis is based on identification
identification as an extension of the hazard identification of safety and security goals, their corresponding requirements
approach CARDION. The approach is iterative and includes and a connection between them, i.e, checking whether they
four phases: system sketching; top goal identification and its undermine or improve each other. A conflict resolutionis
decomposition; applying HAZOP guidewords to each goal; done based on weighting of resolutions values for conflicted
threat and hazards identification. System sketching can be requirements.
performed with UML, SysML or CATALYSIS [16]. 12) Kriaa et al. (2015) [25] present an approach for joint
7) Kriaa et al. (2014) [17] present a case study on an risk assessment that can be applied for both design and
industrial control system in which the previously developed operational phases of the system development. The S-cube
BDMP formalism is used to model safety and security interde- (SCADA Safety and Security modelling) approach takes as
pendencies. The approach allows reasoning about antagonism input the system architecture and provides attacks and failures
between safety and security, as well as conditional dependency scenarios that may lead to given hazards. The analysis relies
and mutual reinforcement between the two. The case study on a knowledge base of safety and security risks and uses
illustrates the ability of BDMP not only to evaluate risks, Figaro language to model different system components, each
but also to optimise the choice of countermeasures against of which is associated with related failure modes and attacks.
attacks. The analysis is performed as a single joint activity to 13) Macher et al. (2015) [26] describe Security-Aware Haz-
address both safety and security, but it may depend on other ard and Risk Analysis (SAHARA). The method combines two
safety/security activities for input. well known approaches HARA [27] coming from automotive
8) Schmittner et al. (2014) [18] propose the FMVEA, domain and STRIDE [20] that focuses on threat modelling
method based on already existing approach from the safety to review system design in a methodical way. The result
domain FMEA, described in IEC 60812 [19]. The method of the method is quantified security impact on the safety-
incorporates both failure mode and failure effect model for critical system development. Initially, the safety analysis is
safety and security cause-effect analysis. It is a high level done with respect to ISO 26262 and using HARA analysis,
approach suitable for design and verification phase in a system while the security analysis is done based on STRIDE method
development and for an analysis of only single causes of independently. The results form security analysis are further
an effect. In the approach threats are quantified using threat used in ASIL quantification concept providing the resulting
agents that represent attackers, threat modes are extracted security level.
using STRIDE model [20] that result in threat effects and 14) Popov (2015) [28] presents an approach for stochastic
attack probabilities. Since the analysis depends on the accuracy modelling of safety-critical systems considering both random
of a system model, one of the benefits of the approach is a failures and malicious attacks. In particular, the approach
possibility to reuse previously acquired results and redo the considers only those attacks that may lead to elimination of the
analysis in case a new threat or vulnerability is identified [4]. safe state of the device. By considering probabilistic modelling
9) Apvrille and Roudier (2015) [21] propose to use SysML- of both failures and attacks it is possible to quantify the risk
Sec to investigate possible impact of introducing security from cyber attacks.
solutions on safety-related functions for embedded and Cyber- 15) Steiner and Liggesmeyer (2015) [29] propose a Secu-
Physical Systems (CPSs). SysML-Sec adapts a goal-oriented rity Enhanced Component Fault Trees (SECFTs) analysis. In
approach for capturing requirements and a model-oriented order to assign probabilities to security related causes, i.e., to
approach for specifying architecture and threats. Within the conduct a quantitative analysis, basic events are grouped into
analysis resources to be protected and their connection to minimal cut sets (MCSs), and probabilities are assigned to sets
safety and security requirements are identified. The analysis instead of events. The probabilities values are picked from the
methodology is based on Y-chart approach [22] and follows discreet set aligned with classification from IEC 61025 [30].
V-cycle. The analysis is supported by an open-source soft- The qualitative analysis within the approach is based on
ware TTool for model specification and verification, and by identification of all MCSs and their handling depending on
AVATAR for analysing requirements and attacks. SysML-Sec included events nature, i.e., security, safety or mixed.
assesses compatibility of security requirements with regards 16) Wei et al. (2015) [31] describe an approach based on
to system safety at partitioning and design stages. HAZOP in which they strive towards including security related
10) Cimatti et al. (2015) [23] present an overview of the information into the hazard analysis, and apply it to an
D-MILS approach for verification of safety and security re- open source immobiliser protocol stack.The authors focus on
quirements. Both types of requirements are allocated to the the design phase in the system development and extend the
system components and formalised via component contracts. guidewords by reusing the attack taxonomy of the Computer
The verification of the requirements in the given system can Emergency Response Team (CERT). The approach provides
be performed by checking contract refinement between the a detailed information on a set of primary and secondary
contracts of components comprising the system. The result of guidewords and their combinations.
the refinement analysis can be previewed as fault trees showing 17) Islam et al. (2016) [32] propose a framework for threat
the dependencies of the system and components failures. analysis and risk assessment inspired by ISO 26262 [27].
 6

Due to the tight coupling with the automotive safety standard on security guidewords, useful when identifying possible at-
and inspired by the industry, the paper aims at providing a tack scenarios, similar to HAZOP from safety domain. The
framework readily applicable in the automotive domain. The approach provides unified safety and security constructs that
framework addresses security risks and aligns the proposed se- minimise safety and security integration effort in automotive
curity analysis with the ISO 26262 development process. The domain, and enable non-security engineers to identify infor-
work aims to ease co-certification of safety and security for a mation assets and protection goals.
given system. By proposing a security analysis aligned to the 24) Friedberg et al. (2017) [39] present a combined analysis
existing safety analysis, the approach addresses identification method for safety and security called STPA-SafeSec based
of all properties relevant for safety or security. on STPA [40] and STPA-Sec [6], and used to choose the
18) Nicklas et al. (2016) [33] propose a system engineering- most effective mitigation strategies to ensure system safety
based approach that consist of a SySML-based model ac- and security. The benefits of the approach is unified safety
companied with a procedure in order to establish safe and and security consideration while choosing suitable mitigation
secure design of cyber physical systems. Initially a system strategies, a possibility to prioritize the most critical system
definition is provided via the Generic Systems Engineering components for an in-depth security analysis (e.g. penetration
analysis and a safety case is described using SySML notation. testing). The analysis identifies potential system losses, caused
The combination of these two enables identification of possible by a specific security or safety vulnerability, and better miti-
attacks scenarios. A qualitative assessment of probabilities of gation strategies.
occurrence and goal achievement of the attack scenarios is 25) Howard et al. (2017) [41] propose a method to iden-
used to derive security structures containing the limitation tify and formally analyse safety and security requirements.
of communication and encryption. In the final step possible This approach is based on the STPA [40] methodology and
safety-security goal conflicts related to the analysed safety use combined with modelling, traceability and formal verification
case are harmonized into a sequence diagram to achieve an through use of the Event-B formal method. The aim is to
adequate safety and security level. generate critical requirements to be able to prevent undesirable
19) Ponsard et al. (2016) [34] present a methodology that system states. Using Event-B language and the Rodin toolset
utilises existing techniques such as Goal-oriented requirements they demonstrate and verify that these critical requirements
Engineering (GORE), to co-engineer safety and security. The fully mitigate against the undesirable system states.
approach takes results from safety and security analysis to 26) Kumar and Stoelinga (2017) [42] propose an approach
build a goal tree connecting requirements with the related handling attack-fault trees (AFT) with dynamic gates allow-
hazards/vulnerabilities where each object can be marked as ing to consider more complex multiple step scenarios. The
safety or security relevant. The analysis of safety and security authors present possible transformation of dynamic gates into
requirements is performed jointly, although the input to this stochastic times automata that allows to use UPPAAL model
technique from hazard/threat identification activities may come checker for statical model checking. The approach includes
from different sources. quantitative analysis of AFTs and consideration of several
20) Schmittner et al. (2016) [35] focus on improving existing safety-security scenarios, e.g., as-is scenario and what-if sce-
approach STPA-Sec [6] and concept phase in the lifecycle. nario, leading to identification of the most risky scenarios and
They have identified the guidance for the identification of selection of the most effective countermeasure.
intentional casual scenarios not being clear enough and pro- 27) Pereira et al. (2017) [40] present an analysis built on
posed some modifications, as well as a need to include security a combination of STPA and guidelines from NIST SP800-30.
relevant elements into control loop model. The rationale behind the analysis is merging of a system based
21) Shapiro (2016) [36] proposes a modification of STPA- approach addressing safety and a component-based approach
SEC [6] to support a technical risk analysis for privacy focused on threats and vulnerabilities. The authors demonstrate
engineering, namely STPA-Priv. The approach is based on the how to align safety and security workflows and where they
already existing one while introducing the systematic analysis need to overlap.
of system controls and their ability to constrain behaviours 28) Plósz et al. (2017) [43] propose a method combining parts
that might compromise privacy. of existing methodologies, STRIDE [20] and FMEA [19].
22) Troubitsyna (2016) [37] proposes an approach for inte- These safety and security analyses are divided in two parts
grated derivation and analysis of safety and security constraints with an integration stage after the first parallel activities that
built on top of the systems thinking paradigm presented by provides a combined safety and security threat catalogue.
STAMP, and the assurance case structuring via Goal Struc- Integration results are further fed into the second part of both
turing Notation (GSN). The proposed approach consists of a methods for impact assessment on the security side and likeli-
GSN pattern inspired by STAMP. The work proposes a joint hood assessment on the safety side. The approach advantages
treatment of safety and security requirements by using the are saving effort by taking care of commonalities of separate
described GSN pattern for their structuring. assessments at once, utilizing the combined catalogue to raise
23) Dürrwang et al. (2017) [38] describe a Security Guide- awareness on issues that has high impact or likelihood on both
word Method (SGM) approach used to identify information areas, and supporting multi-dimensional decisions made by
assets and protection goals relevant for safety where artefacts tackling security and safety together.
from the ISO 26262 hazard analysis are reused. SGM is based 29) Procter et al. (2017) [44] extend the Systematic Analysis
 7

of Faults and Errors (SAFE) to provide better integration of correlation of the used case study or example with a particular
security reasoning within safety. In this paper the authors company. We have also examined the connection of the
advocate that the Dolev-Yao model provides better integration proposed approaches with existing safety or security standards.
of security into safety, the model is extended with guidewords One can notice that the association with a standard is almost in
to accommodate both safety and security. all cases directly related to the targeted application area of the
30) Ruijters et al. (2017) [45] present an uniform meta- approaches. A large number of papers is aiming at addressing
model allowing to merge attack tree analysis (ATA) and fault safety/security concerns in the automotive domain, thus using
tree analysis (FTA) in AFT. The developed tool provides a ISO 26262 standard, an international standard for functional
bidirectional transformation between joined AFT model and safety of electrical and/or electronic systems in automotive
independent models The AFT model can be transferred to domain, followed by generic approaches applicable to any
UPPAAL for quantitative analysis purposes, e.g., reliability. domain, and industrial control systems domain. Furthermore,
31) Sabaliauskaite and Adepu (2017) [46] extend the six- we have identified which early system development stages do
step model for design of safe and secure CPSs with support the papers cover with their proposed contributions, identifying
for identification of possible failures and cyber attacks. In the whether they cover only one of the two stages, or both.
first two steps of the approach, the functions/requirements are In Table VII, we have grouped each paper in two cate-
defined together with the system architecture. In the next two gories: one considering the focus of the work; and the other
steps, failures and corresponding safety measures are added to identifying the way interdependencies are managed. In the first
the model. In the final two steps, attacks and the corresponding category (columns in Table VII), we have mapped papers into
security countermeasures are added to the model. The paper three groups (i) safety informed security; (ii) security informed
extends this model by introducing Information Flow Diagrams safety; and (iii) combined safety and security approaches.
(IFDs) that are used to support the safety and security steps. In the second category (rows in Table VII), we investigated
The approach captures different information flows related to whether the work proposes a unified way of analysing in-
different safety and security aspects in IFDs, and aims at terdependencies between safety and security or a parallel
analysing their interdependency. approach where additional harmonisation of interdependencies
32) Temple et al. (2017) [47] propose an approach combining is required. As shown in Table VII, we have not identified
STPA-Sec [6] and FMVEA [48], and integrating them into a works that focus on exploring only the influence of safety
unified analytical process called Systems-Theoretic Likelihood on security, i.e., safety informed security approaches. All
and Severity Analysis (STLSA). STLSA focuses on system publications focus on either exploring the influence of security
functional control actions, includes humans-in-the-loop and on safety or exploring the interdependencies between safety
incorporates semi-quantitative risk assessment aligned with and security. This two step categorization resulted in 4 groups
EN 50126. of papers. In the reminder of the section we discuss the typical
limitations of papers from each group.
33) Vistbakka et al. (2017) [49] describe a unified approach
Combined safety and security approaches that perform
that enables safety and security co-engineering. The main goal
safety and security analyses in parallel are located in the upper
of the approach is to demonstrate the benefit of formal methods
left cell of Table VII. Generally, approaches in this group
when analysing impact of security to safety and other way
require an integration activity to harmonise the results of the
around by using Event-B [50]. The initial model is based
separate safety and security analyses. While such approaches
on the abstract specification, further refined to include system
to analysing the interplay of safety and security may be the
nominal and failure behaviours. The authors consider the effect
easiest to implement in practice, they may also incur too many
of security vulnerabilities on system safety.
iterations needed for harmonising the conflicting safety and
B. Results security requirements. For example, Gu et al. (2015) [24]
The main information extracted from the papers has been require safety and security mechanisms already in place,
summarized in Table VI. It presents the summary of the fol- while Islam et al. (2016) [32] do not include formulation of
lowing characteristics of the identified papers: (i) whether the technical security requirements for the system nor assumptions
approach is associated with any of the relevant safety/security regarding hardware and software level based on the security
standards; (ii) a type of an approach validation presented level. The most important activity for approaches in this group
in the paper; (iii) whether the approach is proposed by is the integration activity for harmonising safety and security
industry or academia; (iv) which is the application area of analyses results. In this respect, we have identified the need
the approach demonstrated in the paper; and (v) which early for further improvement of the proposed integration activities
system development stages does the work cover, HARA/TARA in these types of approaches.
and/or RE. The mapping of the relevant papers as described Security informed safety approaches that take safety and
in Section II-E, is presented in Table VII. security analyses results performed in parallel and analyse the
In Table VI, we consider the following types of validation influence of security on safety are presented in the right upper
based on the paper text: case study, example, empirical study cell of Table VII. What we can say for all parallel approaches,
or conceptual validation. The latter implies only a sketch just as for the previous group, the post safety and security
of the approach without a concrete example. Moreover, we analyses integration activity is the most important aspect.
distinguish between academia and industry driven publications While in the previous group that activity included analysis
based on the origin of the authors as well as the explicit of dependencies of both safety on security and vice versa, in
 8

TABLE VI: Relevant papers characterization

Lifecycle stages coverage
Associated with Contribution Application HARA
Paper Validation RE
a standard origin area and TARA
1 Raspotnig et al. (2012) [10] No Example Academic Air traffic X
IEC 61508, ETSI TS 102
2 Reichenbach et al. (2012) [11] Example Industrial Control Systems X
165-1
3 Silva et al. (2013) [12] EN 5012x, IEEE 1474 Case Study Industrial Railway X
Young and Leveson
4 No Conceptual Academic Generic X X
(2013) [6]
5 Chen et al. (2014) [13] NIST 800-30 Case Study Academic Nuclear X
ISO 26262, ISO/IEC
6 Ito (2014) [15] Conceptual Industrial Automotive X
27000
7 Kriaa et al. (2014) [17] No Case Study Industrial Control Systems X X
IEC 61508, ISO/IEC
8 Schmittner et al. (2014) [18] Example Academic Automotive X
27000
Apvrille and Roudier
9 No Example Academic Automotive X
(2015) [21]
10 Cimatti et al. (2015) [23] No Example Academic Generic X
11 Gu et al. (2015) [24] No Example Academic Control Systems X X
12 Kriaa et al. (2015) [25] No Case Study Academic Control Systems X
13 Macher et al. (2015) [26] ISO 26262 Example Academic Automotive X
14 Popov (2015) [28] ISO 26262 Case Study Academic Automotive X
Steiner and Liggesmeyer IEC 61025, IEC
15 Conceptual Academic Generic X
(2015) [29] 60300-3-1
16 Wei et al. (2015) [31] No Case Study Academic Automotive X
17 Islam et al. (2016) [32] ISO 26262, SAE J3061 Example Industrial Automotive X
18 Nicklas et al. (2016) [33] No Case Study Academic Smart home X
19 Ponsard et al. (2016) [34] IEC61508, SAE J3061 Case Study Academic Automotive X
20 Schmittner et al. (2016) [35] ISO 26262, SAE J3061 Case Study Academic Automotive X X
21 Shapiro (2016) [36] No Example Academic Generic X X
22 Troubitsyna(2016) [37] No Conceptual Academic Generic X
Empirical
23 Dürrwang et al. (2017) [38] ISO 26262 Academic Automotive X X
Study
24 Friedberg et al. (2017) [39] No Case Study Academic Generic X
25 Howard et al. (2017) [41] No Conceptual Academic Generic X X
Kumar and Stoelinga
26 No Case Study Academic Generic X
(2017) [42]
27 Pereira et al. (2017) [40] NIST 800-30 Example Academic Generic X X
28 Plósz et al. (2017) [43] No Case Study Academic Generic X
29 Procter et al. (2017) [44] No Example Academic Medical X
30 Ruijters et al. (2017) [45] No Case Study Academic Generic X
Sabaliauskaite and
31 ISA-99 Example Academic Generic X
Adepu(2017) [46]
32 Temple et al. (2017) [47] EN 50126-1 Case Study Academic Railway X
33 Vistbakka et al. (2017) [49] No Case Study Academic Control Systems X

this group only influence of security on safety is considered. reducing the number of iterations for harmonising safety and
This is appropriate for those systems where security is relevant security is an important goal, the limitation of these methods
only if it influences safety. But if the intention is to also have a is that they are generally more complex and would require
secure system beyond the safety relevant security issues, then more time to perform than perhaps two separate activities for
these approaches are not appropriate for such systems as they analysis of safety and security. Furthermore, these approaches
do not cover analysing the influence of safety on security. For may be more challenging to implement in practice since they
example, one of the possible limitations of the work presented require more change to the state of practice for safety and
by Nicklas et al. (2016) [33], is the lack of information security processes used in companies. A general concern with
regarding the approach suitability in larger systems where both approaches from this group is the extent to which they support
safety and security may be equally important. safety and security, i.e., whether they succeed in identifying
hazards and vulnerabilities at least as good as the independent
Combined safety and security approaches that propose joint methods. For example, Young and Leveson (2013) [6] focus on
analysis of safety and security and their interdependencies are losses that are results from violations of integrity and availabil-
located in the bottom left cell of Table VII. In general, this is ity, while confidentiality is not tackled. Also, the ability of the
the group of approaches that support building both safe and approach to assist analysts in examining security constraints
secure systems. To reduce the amount of possible iterations degradation over time is not addressed. Kriaa et al. (2014) [17]
that may be incurred by the conflicting safety and security present an approach where it might be difficult to evaluate
requirements in parallel approaches, this group of approaches the parameters associated to the security part of the model.
proposes new ways of joint safety and security analyses To tackle this they address robustness of the decisions that
that treat their interdependencies during the analysis. While
 9

can be taken, trying to determine decisions that remain valid TABLE VII: Paper distribution based on their focus
for a wide range of values of the most uncertain parameters. Combined safety and Security informed safety
The approach presented by Cimatti et al. (2015) [23] that security approaches approaches
Parallel [24], [32], [40], [45], [46] [11], [33]
relies on MILS architecture and contract-based method can be Unified [6], [13], [15], [17], [23], [10], [12], [18], [21], [26],
seen as a promising approach given that it provides support [25], [34], [36], [39], [41], [28], [29], [31], [37], [38],
for modelling the system architecture, contract-based analysis [42], [43], [35], [49] [44], [47]
of the architecture, automatic configuration of the platform,
and assurance case generation from patterns. However, the
approach is very specific and lack of knowledge in this domain failure model to address failure dependencies and trade-offs
might provide incomplete results and there is no support for between safety and security. The approach proposed by Wei et
finer-grained information flow properties handling. Frieberg et al. (2015) [31] has a limitation in terms of failures connections.
al. (2017) [39] consider methods such as traditional failure As the future work, the authors plan to address more complex
modes and effects analysis (FMEA), more focused on com- dependencies between failures and guidewords used for the
ponent failure, while STPA-Sec is regarded as systems-based analysis, e.g., to consider multi-stages attacks. Dürrwang et al.
hazard analysis. This might question the scalability of the (2017) [38] aim to add item attributes in their approach and
approach as for systems with complex interactions or emergent consider guidewords, to cover more complex failure scenarios.
behaviour, becomes questionable whether lower level failures In general, we have noticed that the identified approaches
and threats are sufficient for system-level analysis [47]. do not focus on the fact that security is dynamic in its
Approaches proposing a unified way of analysing safety and nature [52]. This dynamic nature implies frequent system
security with safety as an overall goal, i.e., unified security updates as a response to a new attack being developed or
informed safety approaches, are grouped in the bottom right a new vulnerability being exploited. Such an update requires
cell of the Table VII. As this group of approaches is focused change impact analysis to the safety of the system, potentially
on safety as an overall goal, many of them are application leading to increase in time and cost. The challenge of efficient
specific due to alignment with a specific standard, however incorporation of a system update may limit the applicability
considered approaches are quite mature as limitations are of the proposed approaches. Addressing this challenge may be
already going into consideration of failures connections and needed for bringing safety and security co-analysis into safety
complex attacks. Since the overall focus of this group of and security-critical systems engineering state-of-the-practice.
approaches is safety, the potential limitation is the application
of these approaches in systems where also non-safety related C. Results Analysis
security issues are important. In such case there would be We analyse the information from Tables VI and VII to
duplication of work as a part of the security analysis would identify the trends in addressing the dependencies between
be performed in the unified security-informed safety activity, safety and security.
and the full security analysis would still have to be performed In Fig. 2, we present the correlation between the categories
separately. While this could reduce the amount of possible from Table VII and the early system development stages the
iterations for harmonising safety and security, it would still papers focus on. We group the approaches with respect to
mean duplication of work compared to the combined uni- the early system development stages on those addressing only
fied approaches. Furthermore, some of the approaches are RE or HARA/TARA, and those addressing both. We can
domain specific and may require further work to be applied notice that in general for all groups we have more unified
in other areas. For example, since the approach presented by than parallel approaches. This is in particular visible, when
Raspotnig et al. (2012) [10], specifies requirements based on considering RE where all approaches focus on unified analysis
ISO 26262 [27] and Hazard and Operability Study (HAZOP) of both safety and security while exploring the influence
tables combined with Boolean logic Driven Markov Processes of safety on security and vice versa. Furthermore, when it
(BDMP) [51] technique, thus a high level of details and comes to the distribution between security informed safety and
good expert knowledge are required. As it depends on the combined safety and security analyses, we can notice from
expert knowledge the reusability in repeated analysis is not Fig. 2 that approaches addressing only RE or HARA/TARA
applicable since the level of experiences might be different in have approximately equal focus on both. Conversely, the ap-
different teams, potentially affecting results [4]. The approach proaches addressing both activities focus on combined safety
presented by Silva et al. (2013) [12] is also aligned with a and security analysis.
standard from the railway domain, and in general depends on In Fig. 3, we examine trends of addressing the combined
the expert knowledge. Given this the authors have not been analysis on one side, and security informed safety analyses
completely convinced that the approach would be suitable on the other side, over the years. Over the years the focus
for other domains without tailoring it to the specific needs. is steadily increasing on the combined safety and security
Procter et al. (2017) [44] also aim to extend the SAFE analysis analyses side, while the research on security informed safety
proposed by them to other domains using guidewords. The has been in focus for some time already, with increased focus
analysis proposed by Schmittner et al. (2014) [18] is based in 2015. The trend of increased focus on combined safety and
on FMEA that considers only single causes of an effect, security analyses is continuing in 2017 as well.
which excludes multi-stage attacks consideration. The method In Fig. 4 we consider the three most active domains
presented by Popov (2015) [28] may require a more complex (automotive, generic, and control systems) and explore their
 10

10
14 9
8
12 7
6
10 5
4
8 Security-informed 3
safety 2
6 1 Unified
Combined safety
0

Generic

Generic
Automo1ve

Control Systems

Automo1ve

Control Systems
4 and security Parallel
2

0
Parallel Unified Parallel Unified Parallel Unified
RE only . H&TARA only . Both H&TARA
and RE Combined safety . Security
and security informed safety
Fig. 2: The paper distribution based on the stage they address
and their safety/security focus Fig. 4: The paper distribution based on the domain and their
safety/security focus

5
positive results are more likely to be published than negative
4 ones. Meaning, solutions that do not work might not get
published. In our search we have focused on three independent
3 Combined safety and publishers and a WoS as a common source. We have focused
security only on peer-reviewed publications in English, leaving out
2 Security informed grey literature such as PhD theses, reports and papers that
safety
have not been peer reviewed. Furthermore, we have seen fewer
1 results from the industry on this topic, which may be due
to the fact that an industrial funder chooses not to publish
0 certain results. This may be due to commercial opportunities,
2012 2013 2014 2015 2016 2017
but also not to reveal ways in which security vulnerabilities
Fig. 3: The yearly paper distribution based on their are handled, which may in itself be a security vulnerability if it
safety/security focus becomes known that a certain analysis misses certain types of
security vulnerabilities. Based on our contacts with industrial
focus on the interplay of safety and security. We can notice practitioners, we believe the risk of this threat is minimal.
that most works on combined safety and security analyses Nevertheless, we plan to investigate this threat in the future
address the problem in a generic manner, while the security by validating its results with the practitioners.
informed safety analyses are mostly associated with the auto-
motive domain. In fact, both generic and approaches from the B. Bias in Data Selection
industrial control systems domain put more focus on combined One of the steps that have been taken in order to identify
safety and security approaches, while the automotive domain relevant studies for this review has been discussion on research
is the only one that focuses on the security informed safety. questions, the inclusion/exclusion criterion, as well as search
Furthermore, we can notice that unified analyses dominate strategy. We have been able to agree upon research questions
both automotive and generic domains in security informed and derive from research questions a suitable search string.
safety. Although, unified analyses also dominate combined We have made sure that all involved researches had the same
safety and security approaches, there is quite some works that definitions of terms related to this study. Also, our selection
rely on harmonisation of parallel safety and security analyses process has been divided in several stages in order to further
in this category. reduce the risk of excluding relevant studies. Furthermore, all
IV. VALIDITY OF R ESULTS authors have been involved in the study selection process based
As with all empirical studies, there are many threats to on the inclusion/exclusion criterion. The collected publications
validity that may impair the generalisability of the results. have been reviewed first based on their titles and abstracts
In this section we address the most prominent threats to and in cases when no decision could be made based on the
validity [7] namely publication bias as well as bias in data abstracts and titles, a full-text reading was performed to decide
selection, extraction and classification. about the relevance of the paper for our study.
The decision results from Stage 2 (see Table II), where the
A. Publication Bias review has been conducted by all three reviewers, i.e., authors
A threat that the examined research literature does not repre- of this paper, have been analysed by means of Cohen’s kappa
sent all the available knowledge on the topic is always present, coefficient extended for a case with more than two reviewers
i.e., due to exclusion of on-line databases that might have and multiple grading scale [53]. The kappa coefficient for
relevant publications, in our case Science Direct. Publication Stage 2 is 0.48, which falls into Moderate Strength Agreement
bias is one of the reasons that contribute to that threat since group [54]. A possible cause for the level of agreement being
 11

only moderate is the fact that reviewers are coming from and evaluated approaches, and how safety and security overlap
three different domains, namely safety, security, and formal is addressed within them.
methods. To minimise the risk of excluding relevant studies,
we have discussed and taken to the next stage all papers that VI. C ONCLUSIONS
have been marked as relevant by at least two reviewers.
We have witnessed an increased need of safety and security
C. Bias in Data Extraction and Misclassification co-analysis in the recent years. In this paper we have presented
To reduce the risk of wrong data extraction and classifica- a systematic literature review exploring ways and trends in
tion, all authors have agreed upon the set of information to addressing safety and security co-analysis in system engineer-
be extracted from the selected papers. In many cases we had ing. Since safety and security can negatively influence each
to interpret information ourselves. For example, whether an other, analysing their interplay in an efficient manner means
approach focuses more on safety or security or both equally, reducing the effort that needs to be invested in achieving a safe
but even simpler information such as validation type could not and secure system. The results of our review indicate that the
be simply extracted, e.g., the type of validation used in a paper most works focus on unified safety and security analysis that
could not be simply taken as stated in the paper since different aims at exploring the influence of both security on safety and
papers consider the same type of validation differently. Case vice versa. This is the absolute case for approaches considering
study in one paper is an application example in another, so both threats/hazards analyses and requirements engineering.
we chose to interpret ourselves the type of validation so we Concerning the influence of security on safety within the
could have comparable values. To ensure the agreement over safety analysis, also referred to as security informed safety,
the extracted data and classification, first, each author extracted the automotive domain is the main driver in that direction.
data from a subset of papers. Then authors verified each others Considering that combined safety and security analysis can
data by reviewing the papers themselves. All differences were be used for both achieving safe and secure systems, we have
discussed amongst all the authors. noticed increase in published research of such analyses for the
reviewed period. The results also indicate that there is no work
V. R ELATED W ORK addressing safety within existing security analyses, i.e., safety
Safety and security interplay can be considered from many informed security analyses. Furthermore, we have identified
perspectives, e.g., one of the aspects is their joint consideration that many works lack extensive evaluation of the proposed
from a process point of view. Sabaliauskaite et al. [55] approaches and methodologies. We have also noticed that
consider domain lifecycle alignment on an example of ISA84 the identified approaches lack evaluation of their support for
(IEC 61511) and ISA99 (IEC 62443) standards. An overview efficient system update handling that characterises the security-
of lifecycles provided by standards from both domains is critical systems. The lack of focus on such an important issue
presented by Schmittner et al. [56], where authors have regarding the dynamic nature of security and its influence on
identified the main phases of safety and security processes and safety may impair the applicability of the approaches in safety
proposed a combined version. However, in this work we focus and security–critical systems. It is evident that more efforts are
only on analyses related to early system development stages. needed in proposing new and evaluating existing proposals for
Chockalingam et al. [57] present a survey on integrated safety co-analysis of safety and security in all application areas.
and security risk assessments methods and their application
domains. An overview of approaches based on attack and ACKNOWLEDGEMENT
fault trees has been presented by V. Nagaraju et al. [58]. In This work is performed within the following projects:
our review, we consider system analyses without a limitation RAASS project (the Vinnova PiiA program); SafeCOP project
to a particular approach form, moreover identified methods (ECSEL JU, grant agreement n692529 and National funding);
have been analysed depending on more general categories, the FiC project (SSF) and the SAFSEC-CPS project (KKS).
e.g., association to existing standards, approach validation,
etc. In 2013 Piètre-Cambacédès et al. [59] provided a survey
on differences and similarities with respect to security and R EFERENCES
safety approaches, along with their interdependencies and [1] CENELEC, IEC 61508: Functional Safety of Electri-
possible adaptation of approaches from one domain into the cal/Electronic/Programmable Electronic Safety-Related Systems.
Parts 1-7. International Electrotechnical Comission, 2010.
another. The authors have presented a comprehensive analysis [2] W. Young and N. G. Leveson, “An integrated approach to safety and
of both domains including operational principles, assessment security based on systems theory,” Commun. ACM, vol. 57, no. 2, 2014.
methods, architectural concepts and approaches suitable for [3] R. Kissel, Glossary of key information security terms. U.S. Dept. of
Commerce, National Institute of Standards and Technology, 2006.
adaptation in the other domain. S. Kriaa et al. [5] present a [4] C. Schmittner, Z. Ma, E. Schoitsch, and T. Gruber, “A Case Study of
survey on combined safety and security approaches with focus FMVEA and CHASSIS As Safety and Security Co-Analysis Method for
on industrial control applications. The main criteria for the Automotive Cyber-physical Systems,” in 1st ACM Workshop on Cyber-
Physical System Security, 2015.
analysis has been lifecycle phases for an approach application, [5] S. Kriaa, L. Piètre-Cambacédès, M. Bouissou, and Y. Halgand, “A
whether integration or unification of an approach is a base survey of approaches combining safety and security for industrial control
for a joint consideration of two domains, and whether it is systems,” Reliability Engineering and System Safety, 2015.
[6] W. Young and N. Leveson, “Systems thinking for safety and security,”
qualitative or quantitative method. In contrast to both above in Proceedings of the 29th Annual Computer Security Applications
mentioned works, our study is focused on already developed Conference, ser. ACSAC. ACM, 2013.
 12

[7] B. Kitchenham and S. Charters, “Guidelines for performing systematic [35] C. Schmittner, Z. Ma, and P. Puschner, Limitation and Improvement of
literature reviews in software engineering version 2.3,” EBSE Technical STPA-Sec for Safety and Security Co-analysis. Springer, 2016.
Report, Keele University and University of Durham, 2007. [36] S. S. Shapiro, “Privacy risk analysis based on system control structures:
[8] N. G. Leveson, Safeware: System Safety and Computers. ACM, 1995. Adapting system-theoretic process analysis for privacy engineering,” in
[9] SAE J3061, “Cybersecurity Guidebook for Cyber-Physical Vehicle Sys- IEEE Security and Privacy Workshops, 2016.
tems.” SAE International, 2016. [37] E. Troubitsyna, “An integrated approach to deriving safety and security
[10] C. Raspotnig, P. Karpati, and V. Katta, A Combined Process for requirements from safety cases,” in 40th Annual Computer Software and
Elicitation and Analysis of Safety and Security Requirements. Springer, Applications Conference. IEEE, 2016.
2012. [38] J. Dürrwang, K. Beckers, and R. Kriesten, “A lightweight threat analysis
[11] F. Reichenbach, J. Endresen, M. M. R. Chowdhury, and J. Rossebø, “A approach intertwining safety and security for the automotive domain,” in
pragmatic approach on combined safety and security risk analysis,” in International Conference on Computer Safety, Reliability, and Security.
23rd IEEE International Symposium on Software Reliability Engineer- Springer, 2017.
ing, 2012. [39] I. Friedberg, K. McLaughlin, P. Smith, D. Laverty, and S. Sezer, “STPA-
[12] N. Silva and R. Lopes, “Practical experiences with real-world systems: SafeSec: Safety and security analysis for cyber-physical systems,” Jour-
Security in the world of reliable and safe systems,” in 43rd Annual nal of Information Security and Applications, 2017.
IEEE/IFIP Conference on Dependable Systems and Networks Workshop [40] D. Pereira, C. Hirata, R. Pagliares, and S. Nadjm-Tehrani, “Towards
(DSN-W), 2013. combined safety and security constraints analysis,” in Int. Conference
[13] Y.-R. Chen, S.-J. Chen, P.-A. Hsiung, and I.-H. Chou, “Unified security on Computer Safety, Reliability, and Security. Springer, 2017.
and safety risk assessment - A case study on nuclear power plant,” in [41] G. Howard, M. Butler, J. Colley, and V. Sassone, “Formal Analysis of
TSA. IEEE, 2014. Safety and Security Requirements of Critical Systems Supported by an
[14] NIST, “NIST SP 800-30 Revision 1, Guide for Conducting Risk Extended STPA Methodology,” in 2017 IEEE European Symposium on
Assessments,” 2012. Security and Privacy Workshops, 2017.
[15] M. Ito, Finding Threats with Hazards in the Concept Phase of Product [42] R. Kumar and M. Stoelinga, “Quantitative security and safety analysis
Development, 2014. with attack-fault trees,” in 18th IEEE International Symposium on High
[16] D. F. D’Souza and A. C. Wills, Objects, Components, and Frameworks Assurance Systems Engineering, 2017.
with UML: The Catalysis Approach. Boston, MA, USA: Addison- [43] S. Plósz, C. Schmittner, and P. Varga, “Combining safety and secu-
Wesley Longman Publishing Co., Inc., 1999. rity analysis for industrial collaborative automation systems,” in In-
[17] S. Kriaa, M. Bouissou, F. Colin, Y. Halgand, and L. Pietre-Cambacedes, ternational Conference on Computer Safety, Reliability, and Security.
Safety and Security Interactions Modeling Using the BDMP Formalism: Springer, 2017.
Case Study of a Pipeline. Springer, 2014. [44] S. Procter, E. Y. Vasserman, and J. Hatcliff, “SAFE and Secure:
[18] C. Schmittner, T. Gruber, P. Puschner, and E. Schoitsch, Security Deeply Integrating Security in a New Hazard Analysis,” in 12th ACM
Application of Failure Mode and Effect Analysis (FMEA). Springer, International Conference on Availability, Reliability and Security, 2017.
2014. [45] E. Ruijters, S. Schivo, M. Stoelinga, and A. Rensink, “Uniform analysis
[19] International Electrotechnical Commission, “IEC 60812: Analysis tech- of fault trees through model transformations,” in 2017 Annual Reliability
niques for system reliability - procedure for failure mode and effects and Maintainability Symposium, 2017.
analysis (FMEA),” 2006. [46] G. Sabaliauskaite and S. Adepu, “Integrating six-step model with
[20] Microsoft Corporation, “The STRIDE threat model,” 2005. information flow diagrams for comprehensive analysis of cyber-physical
[21] L. Apvrille and Y. Roudier, Designing Safe and Secure Embedded and system safety and security,” in 18th IEEE International Symposium on
Cyber-Physical Systems with SysML-Sec. Springer, 2015. High Assurance Systems Engineering, 2017.
[22] F. Balarin, Y. Watanabe, H. Hsieh, L. Lavagno, C. Passerone, and [47] W. G. Temple, Y. Wu, B. Chen, and Z. Kalbarczyk, “Systems-theoretic
A. Sangiovanni-Vincentelli, “Metropolis: an integrated electronic system likelihood and severity analysis for safety and security co-engineering,”
design environment,” Computer journal, 2003. in Reliability, Safety, and Security of Railway Systems. Modelling,
[23] A. Cimatti, R. DeLong, D. Marcantonio, and S. Tonetta, Combining Analysis, Verification, and Certification. Springer, 2017.
MILS with Contract-Based Design for Safety and Security Requirements. [48] C. Schmittner, Z. Ma, and P. Smith, FMVEA for Safety and Security
Springer, 2015. Analysis of Intelligent and Cooperative Vehicles. Springer, 2014.
[24] T. Gu, M. Lu, and L. Li, “Extracting interdependent requirements and [49] I. Vistbakka, E. Troubitsyna, T. Kuismin, and T. Latvala, “Co-
resolving conflicted requirements of safety and security for industrial engineering safety and security in industrial control systems: A formal
control systems,” in 1st International Conference on Reliability Systems outlook,” in Software Engineering for Resilient Systems. Springer, 2017.
Engineering, 2015. [50] J.-R. Abrial, Modeling in Event-B: System and Software Engineering,
[25] S. Kriaa, M. Bouissou, and Y. Laarouchi, “A model based approach for 1st ed. Cambridge University Press, 2010.
scada safety and security joint modelling: S-cube,” in 10th IET System [51] L. Piètre-Cambacédès and M. Bouissou, “Modeling safety and se-
Safety and Cyber-Security Conference, 2015. curity interdependencies with BDMP (Boolean logic Driven Markov
[26] G. Macher, A. Höller, H. Sporer, E. Armengaud, and C. Kreiner, Processes),” in IEEE International Conference on Systems, Man and
A Combined Safety-Hazards and Security-Threat Analysis Method for Cybernetics, 2010.
Automotive Systems. Springer, 2015. [52] P. Johnson, D. Gorton, R. Lagerström, and M. Ekstedt, “Time between
[27] International Organization for Standardization (ISO), ISO 26262: Road vulnerability disclosures: A measure of software product vulnerability,”
vehicles — Functional safety. ISO, 2011. Computers & Security, 2016.
[28] P. T. Popov, Stochastic Modeling of Safety and Security of the e-Motor, [53] J. Fleiss, “Measuring nominal scale agreement among many raters,”
an ASIL-D Device. Springer, 2015. Psychological Bulletin, 1971.
[29] M. Steiner and P. Liggesmeyer, Qualitative and Quantitative Analysis [54] J. R. Landis and G. G. Koch, “The measurement of observer agreement
of CFTs Taking Security Causes into Account. Springer, 2015. for categorical data,” Biometrics, 1977.
[30] International Electrotechnical Commission, “IEC 61025: Fault Tree [55] G. Sabaliauskaite and A. P. Mathur, Aligning Cyber-Physical System
Analysis (FTA), year = 2006.” Safety and Security. Springer, 2015.
[31] J. Wei, Y. Matsubara, and H. Takada, “HAZOP-based security analysis [56] C. Schmittner, Z. Ma, and E. Schoitsch, “Combined safety and security
for embedded systems: Case study of open source immobilizer protocol development lifecylce,” in 13th IEEE International Conference on
stack,” in 7th International Conference on Electronics, Computers and Industrial Informatics, 2015.
Artificial Intelligence, 2015. [57] S. Chockalingam, D. Hadziosmanovic, W. Pieters, A. Texeira, and P. van
[32] M. M. Islam, A. Lautenbach, C. Sandberg, and T. Olovsson, “A risk Gelder, Integrated Safety and Security Risk Assessment Methods: A
assessment framework for automotive embedded systems,” in 2nd ACM Survey of Key Characteristics and Applications. Springer, 2016.
International Workshop on Cyber-Physical System Security, 2016. [58] V. Nagaraju, L. Fiondella, and T. Wandji, “A survey of fault and
[33] J. P. Nicklas, M. Mamrot, P. Winzer, D. Lichte, S. Marchlewitz, and attack tree modeling and analysis for cyber risk management,” in IEEE
K. D. Wolf, “Use case based approach for an integrated consideration International Symposium on Technologies for Homeland Security, 2017.
of safety and security aspects for smart home applications,” in 11th [59] L. Piètre-Cambacédès and M. Bouissou, “Cross-fertilization between
System of Systems Engineering Conference, 2016. safety and security engineering,” Reliability Engineering and System
[34] C. Ponsard, G. Dallons, and P. Massonet, Goal-Oriented Co-Engineering Safety, 2013.
of Security and Safety Requirements in Cyber-Physical Systems.
Springer, 2016.