View metadata, citation and similar papers at core.ac.

uk brought to you by CORE

provided by Greenwich Academic Literature Archive

ARMET: Behavior-Based
Secure and Resilient
Industrial Control Systems
By M u h a M M a d T a i Mo or K h a n , d i M i T r ios s e r pa nos , and h owa r d s h robe

ABSTRACT | In this paper, we introduce a design methodology I. I N T RODUC T ION

to develop reliable and secure industrial control systems
Industrial control systems (ICSs) are a significant class
(ICSs) based on the behavior of their computational resources
of cyber–physical systems (CPSs) that are increasingly
(i.e., process/application) and underlying physical resources
employed in the control and management of critical
(e.g., the controlled plant). The methodology has three
infrastructure around the world, from transport systems
independent, but complementary, components that employ
to water management systems and from the manufactur-
novel approaches and techniques in the design of reliable and
ing floor to smartgrids. They constitute typical hybrid
secure ICSs. First, we introduce reliable-and-secure-by-design
systems and their control process models are adopted
development of secure industrial control applications through
to several other application domains, especially health
stepwise sound refinement of an executable specification,
systems and the emerging Industrial Internet of Things
employing deductive synthesis to enforce functional and
(IIoT). Importantly, from the computational point of
nonfunctional (e.g., security and safety) properties of ICS
view, ICSs constitute special purpose computing systems
applications. Second, we present a runtime security monitor at
with different characteristics from typical information
the middleware level of ICSs that protects ICS operation in the
technology (IT) systems, in terms of purpose, ownership,
field through comparison of the application execution and the
interfaces, functional and nonfunctional requirements,
application specification execution in real time; the runtime
etc. ICSs are considered a different form of computation
security monitor can be synthesized from the executable
from IT system and as such are referred to as operational
specification. Finally, based on the specification, we perform
technology (OT) systems [1]. As special purpose systems,
a vulnerability analysis for false data injection (FDI) attacks,
ICSs (OT systems) are embedded, cyber–physical systems
which leads to ICS application designs that are resilient
with restricted functionality, typically, to the application
to this type of attacks. We demonstrate the methodology
domain targeted by the system.
through its application to a basic and typical ICS example
A typical ICS is shown in Fig. 1. In this typical con-
application, describing all the tools used and ARMET, the
figuration, measurements of a process’ data are collected
middleware monitor that constitutes the core component of
by the sensors and delivered to the programmable logic
the methodology.
controllers/remote terminal units (PLCs/RTUs) through
KEYWORDS | Computational attacks; cyber�physical systems a network, while commands from PLCs/RTUs travel back
(CPSs); efficient; false data injection (FDI) attacks; industrial through the network to actuators. PLCs/RTUs typically
control systems (ICSs); reliable-and-secure-by-design; resilient; implement simple operations and control sensors and
runtime monitoring actuators, while they are managed and coordinated by the
hierarchically higher layer supervisory control and data
acquisition (SCADA) system, which implements a con-
trol application or plant.
(Corresponding author: Muhammad Taimoor Khan.)
M. T. Khan is with the Institute of Informatics Systems, Alpen-Adria University,
The reliability and security of ICSs is of major impor-
9020 Klagenfurt, Austria (e-mail: [email protected]). tance considering the effects of their compromise or
D. Serpanos is with the Industrial Systems Institute/ATHENA RC, GR-26504 Patras,
Greece, and also with the ECE Department, University of Patras, GR-26504 Patras,
failure. A failed ICS can cause serious damage beyond the
Greece (e-mail: [email protected]). ICS itself, for example, through release of toxic chemi-
H. Shrobe is with the MIT Computer Science and Artificial Intelligence Laboratory
(MIT CSAIL), Cambridge, MA 02139 USA (e-mail: [email protected]).
cals. An attack on the power grid may have major sec-
ondary effects, disabling transportation, medical systems,
and pipelines. The ability to cause such effects through
 interconnections, adding security methods and moderniz-
ing systems that have been in the field, unprotected for long
periods. Operating systems are being increasingly adopted,
mostly specialized real-time operating systems (RTOSs)
or variations of workstation operating systems such as
Windows and Linux, providing access control mechanisms.
Intrusion detection mechanisms for the systems them-
selves or for the networks are becoming common, but most
of them are adopting tools and methods from the IT envi-
ronment, i.e., malware detection and intrusion detection
methods and tools that have been developed for general IT
environments [6], [7]. Importantly, little effort has been
made to include in the security methods the processes/
applications that get implemented by the ICSs, although
the requirements on the ICSs are placed by them for both
Fig. 1. Overview of ICS design.
functional and nonfunctional ones. Recently, the OT char-
acteristics and differentiators from classical IT systems have
been introduced in the efforts to develop effective security
cyber means alone was first demonstrated in the well-known and safety solutions, especially for defensing against false
Aurora experiment where a pure cyber attack successfully data injection (FDI) attacks, which constitute a special class
destroyed a diesel generator [2]; Stuxnet [3] and the recent of attacks that has been emerging against CPSs. In these
Ukrainian smartgrid attack [4] as well as several additional attacks, attackers do not compromise the computational
less known incidents have demonstrated the potential of systems or their networks, but introduce false data at points
cyber attacks to inflict serious societal damage. Finally, ICSs of measurements, i.e., at sensors, introducing false data to
may be compromised to serve as elements of a Botnet as the the sensors, who, in turn, send them to control centers and
recent Mirai attack indicates [5]. lead them to wrong actions [8], [9].
To address reliability and security in ICSs, one needs to We introduce a design methodology for developing
take into account the characteristics of OT technology that reliable and secure ICSs based on the process (applica-
differentiates ICSs from typical IT systems, because this tion) behavior rather than the classical intrusion detection
influences the security methods and solutions that are appro- mechanisms. We define as “behavior” of an ICS process,
priate for ICSs. ICSs typically have real-time requirements, in terms of software engineering, the set of functional and
often hard ones, whose violation may have significant, or nonfunctional (e.g., security, performance) characteristics
even catastrophic, results. It is important to note that func- of both the cyber and the physical resources of the process.
tional and nonfunctional requirements of ICSs, and CPSs Typically, the cyber resources have a discrete behavior,
in general, are different from the requirements of typical while the physical resources have a continuous one. The
IT systems and, thus, need to be carefully considered in the complexity of modeling such a process increases when
design and development of security and safety solutions. In functional and nonfunctional characteristics are a mix of
fact, requirements for ICSs involve domains for cyber and discrete and continuous ones. We manage the modeling
physical resources that have discrete and continuous opera- complexity by describing the “behavior” of an ICS process
tions based on linear and nonlinear constraints with strict
time bounds.
Until recently, security of the ICS infrastructure has not
been a significant requirement although safety and depend-
ability have been quite strong requirements. The reason is
that ICSs and networks have been typically considered as
vulnerable mainly to failures and not to attackers who tar-
get them on purpose. So, methodologies for dependability
and safety have been developed assuming stochastic mod-
els of failures of components and subsystems. Although
these methodologies have served control processes well
until recently, the growth of cyber attacks and the inter-
est of strong actors to damage critical infrastructure lead
to the need of developing new methods that protect ICSs
and networks against purposeful cyber attacks. The first
efforts in this direction have been to harden systems and Fig. 2. Gravity-draining water tank�an example scenario.
with an executable specification based on an abstract data secure-by-design application development, the real-time
type (in Coq). We derive reliable and secure implemen- security monitor, and the vulnerability analysis, our ongo-
tations of the ICS process by stepwise refinement of the ing work focuses on the integration of the three components
specification applying deductive synthesis (in Coq); each in a unified environment that will enable synthesis of the
refinement formally preserves the behavioral semantics. application code, the real-time monitor, and the FDI attack
Furthermore, we assure that the derived implementation is monitor automatically.
reliable in respect to characteristics of functional behavior The paper is organized as follows. Section II presents our
and is secure and efficient conforming to characteristics of threat model. Section III introduces an approach to develop
nonfunctional behavior. Importantly, the proof obtained reliable and secure ICS applications by design. Section IV
through deductive synthesis assures that only the derived demonstrates our approach for runtime security monitor-
implementation is reliable and secure and not the execu- ing, including a proof for system defense against known and
tion of the implementation. Thus, from the given execut- unknown computational attacks. Section V presents our
able specification, we generate a runtime monitor that approach for vulnerability analysis of FDI attacks in ICSs.
assures that the behavior of the implementation execution We demonstrate each of the aforementioned approaches
is also reliable, secure and efficient at runtime by check- with the help of an example of a water tank, as the con-
ing consistency between the specification produced predic- trolled plant, in each corresponding section. In Section VI,
tions and the runtime execution produced observations. we present prior art for each component of our approach
The monitor is bound to detect any arising inconsistency separately.
under the assumption that the execution of the specifica-
tion is always correct, i.e., that it cannot be attacked; this
II. T H R E AT MODEL
can be achieved by executing the specification in a strongly
protected environment, such as SGX (Intel) or Trust Zone A program for an ICS, e.g., a typical controller (PLC) pro-
(ARM). Hence, the runtime monitor detects any compu- gram or a supervisory (SCADA) program, is composed of
tational attack but may not be able to detect FDI attacks. instructions (also known as source code) operating on some
Therefore, we synthesize the executable specification after external-input data values. Based on this view, we consider
vulnerability analysis that identifies potential FDI attacks a threat model that includes computational attacks for
through the application of an SMT solver for nonlinear, real instructions as well as FDI attacks for external-input values.
functions. Based on the results of the solver, we either syn- The program implementation P, its execution PE, or the pro-
thesise the specification so that the implementation is not gram external-input values are considered to have a threat,
vulnerable to false data attacks, or consider the values of the when some insider or external adversary (e.g., user or other
identified FDI attacks as attack vectors to be monitored and program) with legitimate or illegitimate access attempts to
detected at runtime. affect their behavior.
Our threat model includes both computational attacks In a computational attack, on one hand, the adver-
and FDI attacks. As computational, we consider all attacks sary attempts to modify the behavior of the program by:
that lead to the active alteration (injection, deletion, or 1) modifying the instructions or internal-data values of the
replacement) of application code and/or system code, or implementation or execution of P; 2) exploiting any vul-
of variable values (data or control) that lead to the incor- nerability or a bug in them; or 3) injecting an additional
rect execution of the application code on the ICSs; such code (e.g., command) into P or PE. On the other hand, the
attacks include viruses, worms, etc. As FDI attacks, we con- adversary may attempt to affect the nonfunctional behav-
sider the ones that input (insert) into the ICSs values that ior of PE by running some other program, for instance,
are different from the intended ones, i.e., sensor measure- running a malware that may affect the performance of PE.
ments that have been tampered with, for example, between Note that P refers to the program of the application imple-
the environment and sensors, sensors and PLC/RTUs, etc. mentation and not the program of the specification imple-
Importantly, our approach leads to systems that defend mentation. The latter always runs correctly as discussed in
against the attacks, both computational and data ones, of Section I.
our threat model because the attacks constitute violations of In an FDI attack , the adversary attempts to change
the security requirements. external data values that are input for computation at the
In this paper, we demonstrate our approach to secure controller by attacking sensors or the network between
and safe ICSs by describing the design methodology using sensors and the computing system. However, as the values
a simple example of a control application for a water tank. belong to data-type “real,” therefore, small variation in the
We describe the methods and the tools that we use as parts values may not violate the system constraints and thus goes
of the methodology, focusing on ARMET, the real-time undetected leading to some advanced persistent threat.
security monitor that constitutes our main research con- The specification of functional and nonfunctional behav-
tribution. Although our work is currently progressing the ior of a program allows us to detect computational threats
three components of the methodology separately, i.e., the by comparing the behavior of P or PE with the specified
behavior. Such specification also enables identification of the specification is refined into a fully deterministic imple-
legal data values that lead to FDI attacks by the application mentation that is not only correct with respect to its specifi-
of the nonlinear verification method. cation but is also obviously secure and efficient, respecting
security constraints and employing efficient representations
and algorithms, respectively. In detail, first the initial pro-
III. R EL I A BL E-A N D -SEC U R E-BY-
gram is realized by its high-level declarative specification of
DE SIGN A PPL IC AT IONS
its functionality and security. Then, through an iteration of
Developing reliable and secure ICS applications by design semantically preserved optimizations, an efficient and cor-
through derivation of the application implementation from rect executable implementation is generated that also con-
a given declarative specification formally assures that the forms specified security constraints. The optimizations can
application meets its requirements on one hand and is free be modified through sound refinements to meet nonfunc-
from various classes of vulnerabilities and attacks (e.g., FDI, tional requirements, e.g., security and performance.
cross-site scripting) on the other hand. Furthermore, runtime In the reliable-and-secure-by-construction approach,
monitoring of such an application based on the declarative we use the Coq proof assistant to encode an abstract-data-
specification assures that the application execution is reli- type-based declarative specification of an ICS behavior that
able and secure at runtime. However, formally deriving such includes both functional and nonfunctional properties as its
reliable and secure implementations for ICS applications is first class elements. The encoding is based on abstraction
challenging because it involves reasoning about both dis- relation as specified by Hoare [16]. Then, we employ deduc-
crete and continuous system behavioral models. tive synthesis of the specification through sound refine-
Most of the existing approaches have focused on either ments. Importantly, all refinement steps are encoded in
deriving reliable or secure implementations of ICS applica- the proof assistant and thus provide high assurance of reli-
tions. Many of these approaches only allow modeling of timed able resulting implementation that is correct, secure, and
security properties or security policies of ICS applications, efficient. Every refinement corresponds to an optimization
and then deriving application implementations that assure script that resolves nondeterminism of the program in a
enforcing the security properties and policies at runtime (functional and nonfunctional) behavior-preserving way as
[10]–[13]. Quite a few have attempted to derive reliable ICS shown in Fig. 3.
application implementations, for instance, Soulat [14] derives
correct implementation of schedulers of hybrid systems.
Based on Fiat [15], we introduce a different approach
A. Example
that employs deductive synthesis to develop reliable-and- Based on the water tank example (see Fig. 2), we dem-
secure-by-design ICS applications through interactively onstrate our approach in a familiar notation, i.e., Java-like
stepwise refinement of declarative specifications, where syntax as shown in Listing 1.
description of both cyber and physical resources of ICSs are For our example specification (Listing 1), in each
first class models on one hand and nonfunctional proper- clock tick, either the pump FILLs the water tank, or does
ties (i.e., security, performance) are modeled integral to NOTHING. The water tank is continuously DRAINing as
functional properties on the other hand. In this approach, enumerated by “Action” [see L(ine).1]. The specification
the user starts with an initial nondeterministic program/ says that the water tank is initially empty (see L.4). When a
specification with obscure nonfunctional characteristics, for user issues a command to fill the water tank up to a certain
example, security and efficiency. Then, the specification is level, if the command (i.e., reading) is in the range of sensor
synthesized through stepwise refinements; each refinement accuracy (see L.7), then we either accept the value or any
replaces some statements in the specification with other at
least equally deterministic statements such that no extra
behavior is introduced that is beyond that of the replaced
statements on one hand, and none of the security properties
are violated on the other hand, as depicted in Fig. 3. Finally,

Fig. 3. Deductive program synthesis through sound refinements. Listing 1. Water tank specification.
 sensor data to be lower than it should be will result in the
controller issuing a “FILL” command when it should not).
Thus, while it is certainly an advantage to have constructed
a provably correct and secure controller, one must also
actively monitor the runtime behavior of the system to guar-
antee that the controller continues to behave correctly.

I V. RU N T I M E SEC U R I T Y MON I T OR
The workflow of our runtime security monitor (RSM)
is shown in Fig. 4. The RSM requires both the specifica-
tion (AppSpec) and implementation (AppImpl) of an ICS
application for monitoring [17]. Since the specification and
Listing 2. Water tank code. implementation of the application operate at different levels
of abstraction, the “Wrapper” wraps the implementation in
order to share the observed data of interest with the RSM
value (see L.8). At each time stamp (see L.10), the water in a way that is comparable to the executable specification
tank is either in of the application. During monitoring, the RSM checks for
the consistency of the runtime behavior (observations gen-
• FILLing state, i.e., the pump is pumping in the water
erated by the “Wrapper”) of the application and the appli-
such that water tank does not overflow (see L.13-14)
cation’s expected behavior (predictions generated by the
and the water tank attains a new water level (see
“AppSpec”). The RSM raises an alarm if an inconsistency is
L.19); or
detected. To support the real-time constraints of ICSs, the
• DRAINing state, i.e., the water is draining at a con- alarm can be used by AWDRAT [18], which may first sus-
stant GRAVITY_RATE assuring that the tank 1) does pend the execution and later resume the execution in a safe
not underflow (see L.15–16) and 2) attains a new state, after diagnosis.
water level (see L.20). In detail, the “AppSpec” allows us to model the behavior
Based on design decisions and synthesising the speci- of cyber and physical resources of ICSs as first class models
fication (Listing 1), we derive the Java implementation through description of
(Listing 2). • the normal behavior (“good behavior”) of the (cyber
Analogous to specification description, water tank and physical) resources by decomposing their behav-
implementation (Listing 2) says that initially water tank is ior into various submodules, by encoding precondi-
empty (see l.4). We accept any value from a user command tion and postconditions and invariant(s) for each
(see L.7). At each time stamp, we submodule;
• FILL the water tank to attain new height of the water • the flow and control model of values as data-flow and
(see L.14), if the tank does not overflow (see L.11-13); or control-flow links connecting the submodules;
• DRAIN the water tank to attain new height of the • the exceptional behavior of the resources, known
water (see L.17), if the tank does not underflow (see attacks, and suspected attack plans to rigorously
L.16); or characterize the misbehavior (“bad behavior”) of a
• do NOTHING (see L.20), if all the water has been module/submodule.
drained.
Although deductive synthesis produces a correct-and-
secure-by-construction controller, this by itself is inad-
equate. The proof is constructed under the assumption that
the operating system and runtime environment in which
the controller executes are correct. These assumptions are
not generally correct; the runtime libraries supporting C
(and other language) code are known to contain vulner-
abilities and these vulnerabilities can be exploited to change
the code that executes in the controller. Furthermore, FDI
attacks (i.e., corrupting the sensor data before it reaches the
controller) can cause the control algorithm to issue com-
mands that will have disastrous effects, even if the control-
ler is unmodified (for example, corrupting the water level Fig. 4. Runtime security monitor (RSM).
Fig. 5. Top level syntactic domains.

Based on the model (AppSpec) of the application

implementation (AppImpl), the RSM runs the AppSpec
and AppImpl in parallel and checks their consistency by
comparing predictions (generated from the model, i.e., Listing 3. Example specification of code-attack.
“AppSpec”) and runtime observations (produced by the
implementation, i.e., “AppImpl”). The level of granularity
Based on the assumption that the specification runs
at which the monitoring application operates determines
safely, our security monitor is sound and complete, i.e., there
the performance–diagnosis tradeoff. Coarse-grained moni-
are no false alarms or undetected computational attacks by
toring lessens the execution overhead, but limits the result-
ing diagnostic information. On the other hand, fine-grained the monitor. Specifically, we have proved the soundness and
monitoring incurs higher computational overhead, but is completeness of our monitor for some specific constructs,
able to produce quick and thorough diagnoses. as shown in [17] and [20] to show that our proof method
The novelty of the RSM arises from the specification works, in principle.
language (its elements, e.g., attack plans, formalism, and
encoding cyber and physical resources as first class models A. Example
based on their both functional and nonfunctional proper-
In order to demonstrate our approach, we have defined
ties) of the application it monitors, whose high level domains
a specification language that allows to describe the behavior
are shown in Fig. 5. In principle, our specification language
allows to describe 1) discrete behavior of cyber–physical of a simple PID controller (see Listing 4) that manages the
ICS resources; and 2) continuous behavior of physical ICS level of a water tank through a pump. We have also imple-
resources side by side as first class specifications. Monadic mented a working prototype for our security monitor and
have applied it to monitor the PID controller. As a start-
second-order logic and event-calculus-based rich formalism
of the specification language enables us to describe system ing point, we have modeled the cyber resources (i.e., PID
controller) and physical resources (i.e., feed-water subsys-
behavior at various levels of abstraction, with higher degree
of modularity. Semantically, such logical formalism-based tem) of a typical ICS. In detail, the application specifica-
executable specification language can be directly compiled tion (“AppSpec”) includes a cyber model that specifies the
into a machine code, and is thus, inherently efficient for
our runtime behavioral comparison. In detail, the formal-
ism of our specification translates into a finite automaton
that recognizes only the words that satisfy the specification
[19]. “AppSpec” is an active model of normal behavior [18]
and consists of a decomposition into submodules as well as
in precondition and postconditions and invariant(s) for each
submodule. Furthermore, data-flow and control-flow links
connect the submodules, specifying the intended control
and flow of values. The preconditions and postconditions
and invariant(s) are first-order statements about the set of
data values (that flow into and out of the submodules) and
arbitrary constraints, respectively. Optionally, the model
can also specify suspected/known undesired behavior of
a resource and associated potential attack plans, allowing
diagnostic reasoning to characterize the component’s mis-
behavior. In principle, attack plans are hypothetical attacks
based on rules that describe different ways of compromising
a component by specifying “bad” behavior of the systems.
The monitor exploits the attack plans at runtime to detect
any such misbehavior, thus making the monitor more robust.
For an example attack plan for our controller, see Listing 3. Listing 4. Example specification of controller-step.
computations performed by the PID controller, and a physi- e.g., using ladder logic, while the RSM will be part of the
cal model that specifies physical characteristics and dynam- operating system, considering the current trend of PLC
ics of the water tank subsystem as shown in Fig. 2. The lat- environments where real-time operating systems (RTOSs)
ter is employed to detect computational and FDI attacks, by are continuously adopted, or the middleware that runs on
observing the systematic deviation of the sensor’s behavior the PLC, if it is not an operating system. We assume that the
from the state of the physical plant, as predicted by the PLC runs a control cycle on the water tank subsystem every
model. Moreover, based on “AppSpec,” bugs in the imple- 0.1 s, which is a very low rate of control cycles for such an
mentation (AppImpl) can also be detected. ICS. Our experiment simulates the subsystem for 100 s, i.e.,
The focus of our prototype development of RSM is to 1000 control cycles, and we observe that, when we enable
detect “computational” attacks, where an attacker success- fine-grained monitoring, the RSM consumes 8.93 × 10 −4 s
fully alters the operation or parameter values of the PID of CPU time and 9 . 07 × 10 −4 s of real time for each cycle of
controller implementation (AppImpl). For demonstration, the PID algorithm, which is very small (less than 2%) com-
we have modeled the PID controller with four parameters: pared to the length of the control cycle (0.1 s).
the set point, and three weighting factors Kp, Ki and Kd. It In the evaluation of our prototype implementation,
performs the following steps (see :components in Listing 4). the RSM quickly detects arbitrary modifications (i.e., arti-
ficial attack), because of the low abstraction gap between
1) Use the sensor values to estimate the state of the
the RSM model (AppSpec) and the actual computation
system.
(AppImpl). For instance, if the parameter Ki is altered,
2) Compute the difference between the estimated system then the aforementioned step 5(c) produces a different out-
state and the set point of the controller (error term). put that is instantaneously detected as being inconsistent
3) Compute the local derivative of the error. with the predicted value.
4) Integrate the error. Our fine-grained modeling allows us to classify the nature
5) Compute the corresponding correction: of the attack immediately through fine-grained diagnostic
a) multiply the error by Kp (see Listing 5); resolution. For instance, in the above case, we can easily
b) multiply the integral of the error by Ki; deduce that either Ki or the implementation of step 5(c) was
c) multiply the derivative of the error by Kd. altered. However, fine-grained behavioral monitoring suffers
6) Compute and output the sum of the above three from a computational overhead. Therefore, to achieve high
terms as the correction term. performance, we can simply monitor the input and output
Each component of the PID controller implementation of the algorithm and ignore its intermediate steps. Clearly,
(AppImpl) corresponds to each of the above six algorithmic this reduction in communication overhead would result in a
steps. Similarly, each component has an AppSpec model higher diagnostic resolution overhead, because the only infor-
including preconditions and postconditions. The actual mation we have is that the algorithm is somehow corrupted.
implementation of each of these steps is wrapped, and their Nevertheless, we can bypass other intermediate monitoring
input and output are presented to the RSM for their consist- checks, such as the procedure execution order, and instead
ency checking. monitor the correctness of the data and control flows. All the
We have implemented our prototype monitor in Allegro aforementioned variations are evaluated as shown in Fig. 6.
Common Lisp on a MacBook Pro with a 2.8-GHz Intel Core A security monitor is necessary to detect cyber com-
i7 processor. In a real operational environment, the PID putational attacks at runtime. However, it is important to
algorithm will run as the application in a programmable understand what attacks it is capable of detecting and what
logic controller (PLC), while our RSM will run as part of the remaining vulnerabilities remain. Particularly, in the case of
PLC’s middleware. Thus, the PID application will be devel- FDI attacks it has been shown that it is possible to construct
oped through any application development environment, a “stealthy” attack [21] that evades detection by the runtime
monitor. Thus, we also have constructed methods to analyze

Listing 5. Example specification of the compute�proportional�term. Fig. 6. Application execution performance.

the behavior of the runtime monitor thereby either proving combinations that constitute FDI attacks. We consider that
that the monitor provides complete coverage or identifying a CPS implements a control loop, as shown in Fig. 7, for
types of attacks that can evade the monitor. a process P. At every instant t, the state of the system is
described with a function P(xt), where xt is the set of input
variables to the process at time t. In the implementation of
V. V U L N ER A BIL I T Y A NA LYSIS FOR F DI
the system, the variables x are measured with sensors, input
AT TACK S
to the controller, and used to calculate the state of the sys-
FDI attacks against CPSs are becoming increasingly popu- tem as well as the necessary actions, denoted with at. We
lar among attackers, because they avoid attacking comput- denote with zt the measurements of variables xt, i.e., the
ing systems and their networks and just compromise the measurements of variables x at time t. We also consider that,
input data to control systems. The goal of an FDI attack is as in typical control systems, the system is monitored by a
to lead the control system to make wrong decisions and take monitor mon(x, z), which, at time t, inputs the measure-
wrong actions based on manipulated input data, rather than ments zt and evaluates them for acceptance due to potential
compromising the systems themselves; in these attacks, sys- sensor failures.
tems operate correctly, but on the wrong input data. A sim- Considering a fault-free operation, mon(x, z ) accepts all
ple example, referring to our water tank case, would be to measurements z and calculates the state P(x, z ) of the system
input to the controller a sensor value for the water level that using them. A successful FDI can be launched against the
is close to zero, leading the controller to open the pump, system using a set of measurements z′, if 1) mon(x,z′) accepts
introduce more water in the tank (to reach the targeted z′; and 2) z′ are compromised values of z, i.e., z′ ≠ z. Thus,
water level) and thus, overflow the tank. In industrial envi- we can identify the existence of one or more FDI attacks, if
ronments, cars, medical systems, and similar application we can answer positively the question: Does there exist a z′
domains, such actions can be catastrophic, even fatal. for the system P(x, z) and mon(x, z)?
Recently, FDI attacks have drawn research attention, Our approach identifies the existence of FDI attacks
focusing mostly on power systems [8], [22]–[25]. In these by answering the above question. To achieve this, we rep-
efforts, power systems have been analyzed to identify condi- resent the state of the system with a real function f( ) and
tions under which FDI attacks remain undetected and to pro- we express the above question as an input to an SMT solver
pose defenses. Most efforts have been following traditional for real functions, namely dReal [26]. If the expression that
approaches to attack and failure detection, based on network represents the FDI existential question is satisfiable, then
topology characteristics and statistical analysis of data. our system is vulnerable to FDI attacks; importantly, the
We follow a different approach to defend against FDI SMT solver provides us with one set z′ that constitutes such
attacks. We have developed a method for vulnerability an attack.
analysis that identifies FDI vulnerabilities of a system at the When FDI attacks are identified, a method to defend
design phase [8]. Based on the identified vulnerabilities, we against them is to place constraints on the values of the
redesign the system, introducing constraints that lead to measured parameters, so that the set of values that satisfy
elimination of identified FDI attacks. This redesign process the state function is reduced. As we introduce more con-
is repeated until either FDI vulnerabilities cannot be identi- straints, the space of solutions to the state function becomes
fied, or the expected FDI attacks have been considered. smaller and thus, reduces—if it does not eliminate—the
The basic concept of our method is to describe the sys- potential FDI attacks. Based on this, when we identify an
tem through a state function, which is analyzed for input FDI vulnerability, we can redesign the system introducing
constraints and thus, reducing the attack surface. The new
design can be subsequently analyzed for new FDI attacks
with the new constraints. If FDI vulnerabilities are identi-
fied again, the system can be further constrained and rede-
signed and so on. The process can be repeated until the real-
istic FDI attacks are eliminated or reduced to meet the set
system specifications.

A. Example
In order to demonstrate our approach, we provide an
example using a variation of the water tank that we have
analyzed through the paper, as shown in Fig. 2. In this vari-
ation, the water tank has two pumps, one with incoming
water and one with outgoing water, each pump with one
Fig. 7. ICS control loop. sensor measuring its water rate (rin and rout for the incoming
and outgoing flow, respectively) and one actuator, opening
it to a specified flow rate. Furthermore, we assume that the
system has one sensor measuring the water level. With this
model, we analyze a system with three sensors rather than
the trivial case of two sensors, as the gravity-draining water
tank. For convenience, we consider that the rates rin and
rout are real numbers in the range { 0, 1} and the horizontal
cross-section area of the tank is 1. These normalized values
lead to the property that the volume of water in the tank has Listing 6. Example water tank specification.
the same value as the height of the water in it, making the
state function expression clearer.
For the example water tank, we consider that it operates attack surface. For example, one can decide that rin should be
in discrete time and at the beginning of every time unit the an integer, i.e., 0 or 1, rather than a real number, reducing
state of the pumps may change. The state of system, which the attack surface significantly. A similar constraint on the
is monitored by the monitor mon(x, z ), is expressed with value of rout will make the potential attacks quite few, lead-
the function H(t + 1) = H(t) + rin (t + 1) − rout (t + 1) that ing to a new design and a significantly more robust system. It
represents the height H( ) of the water at the end of the time should be noted that the gravity-draining water tank that we
unit t + 1. have analyzed in this paper is a constrained system (rout is not
Let us consider the following scenario: at time t, a parameter), which provides a smaller attack surface than
(t) = 5, and at time (t + 1) the system is configured with the two-pump tank we analyze here; this is the reason why it
rin (t + 1 ) = 0.5 and rout (t + 1) = 0. Clearly, H(t + 1 ) = 5.5. constitutes a trivial example for this analysis.
If the sensors of the system provide to the monitor the correct Clearly, this methodology can be applied to any system
values rin (t + 1 ) = 0.5, rout (t + 1 ) = 0, and H(t + 1 ) = 5.5, with a monitoring function that can be input to an SMT solver
the monitor will accept the state of the system, based on like dReal. We have applied our method to more complex
H(t) = 5. However, an attacker could compromise the systems, specifically alternating current (ac) state estimators
values for rin (t + 1), rout (t + 1), and H(t + 1) and provide for smartgrids [27], with successful results. We have ana-
fake values to the monitor. The monitor will accept them, lyzed benchmark power distribution networks, based on the
if they are consistent with the function for H(t + 1), and IEEE 14-Bus, 30-Bus, 157-Bus, and 300-Bus benchmark suite,
reject them, i.e., detect the attack, if the fake values are demonstrating not only the effectiveness of the approach,
not consistent with the function H(t + 1). For example, but also its feasibility for realistic and practical systems. As
if an attacker provides as input the values rin (t + 1 ) = 1, we have demonstrated in our power state analysis, FDI vul-
rout (t + 1) = 0, and H(t + 1) = 8, then the monitor nerabilities are identified successfully with analyses that are
will detect an inconsistency since H(t + 1) ≠ H(t) + efficient, independently of the size of realistic networks. Fig.
rin − rout ⇔ 8 ≠ 5 + 1 − 0. However, if the attacker 2 in [27] shows that the time required to analyze benchmark
provides the values rin (t + 1 ) = 1, rout (t + 1 ) = 0, and power networks for FDI on state estimation is in the order
H(t + 1 ) = 6, the monitor will accept them because they of seconds or minutes, depending on the exact network con-
satisfy H(t + 1) = H(t) + rin − rout ⇔ 6 = 5 + 1 − 0 figuration and the constraints on node branches and buses.
although they are not the real values; this demonstrates However, for a design cycle of a practical power distribution
that the system is vulnerable to an FDI attack when (sen- network, this performance of the analysis makes the approach
sors) values rin, rout, and H are compromised. We can iden- an indispensable tool for the development of networks that
tify such an FDI attack with our method through the use of are robust against FDI attacks.
the SMT solver, providing the monitoring function H( ) as
an input and asking if there is a solution to this equation V I. R EL AT ED WOR K
with parameter values rin, rout, and H in the defined ranges,
which is different from the real action. Listing 6 shows the This section includes state of the art for each component of
input to dReal for our described example scenario as well as our design methodology, i.e., secure-by-design ICS applica-
the result, which is a successful FDI attack to the system as tions, runtime security monitoring, and vulnerability analy-
depicted by results in Listing 7. Analyzing the system, one sis for FDI attacks, respectively.
can easily realize that the space of existent FDI attacks is
large, considering all the combinations of values that sat-
isfy H( ) and are different from the specific real values of
the example.
Based on the above and considering the large space of
potential attacks, a designer can introduce constraints on the
vulnerable parameters (rin, rout and H), in order to reduce the Listing 7. Example FDI detection.
A. Reliable-and-Secure-by-Design ICS Applications In contrast to the aforementioned approaches, our
Based on the principle of correct-by-construction approach derives correct and secure implementation from
approach [28], we have devised an approach to develop a specification that treats models of cyber and physical
secure-by-design ICS applications. Here, the challenge resources as first class models with complex behavioral
is to model behavior of both cyber and physical as first characteristics. The derivation is based on stepwise refine-
class models, though they possess fundamentally different ments of the hybrid model using Coq. We encode such com-
semantics (i.e., discrete and continuous) and characteristics plex models (including security and reliability properties)
(i.e., linear and nonlinear). Yet another challenge here is to and characteristics (including discrete and continuous) as
adequately model uncertain physical environmental varia- abstract data types (ADTs) in Coq and then derive imple-
tions that may influence behavior of physical resources of mentations through deductive synthesis of ADTs using Coq
an ICS on one hand, and operate on variable levels of infor- that is secure and correct by construction with a formal
mation abstraction about physical environment on the other proof. As far as we are aware, no other design methodology
hand. Based on Fiat [15], secure-by-design approach allows allows the derivation of secure, correct, and efficient imple-
to derive correct and secure implementation from specifica- mentations of ICSs from a specification treating cyber and
tion through stepwise refinements using Coq. Thus, allow- physical models as first class models.
ing to construct reliable and secure ICS applications by 2) ADT-Based CPS Modeling: There have not been more
design, the approach has advanced existing approaches in results in the modeling of cyber–physical resources using
at least one of the following ways: 1) reliability and security ADTs. Recently, in [33], based on VDM, the authors have
by design; 2) ADT-based CPS modeling; and 3) deductive developed a design methodology that allows comodeling of
synthesis of security properties. cyber and physical resources using operations defined over
1) Reliability and Security by Design: For the last decade, ADTs. The methodology allows to interpret the model as
there have been various efforts to develop approaches that a bond graph, which is a directed graph that compares the
allow reliable and secure implementations by design. For comodel at the end.
instance, Ur/Web [29] is a language for developing reliable 3) Deductive Synthesis of Security Properties: As far as we
and secure web applications by construction/design. For are aware, deriving correct and secure implementations
reliability, the language assures that the application will from a given ADT-based CPS model through deductive syn-
1) not crash during generating web pages; 2) not return inva- thesis has not been addressed so far. However, there have
lid HTML; and 3) not produce dead intra-application links, been efforts to derive correct implementations for simple
to name a a few. Furthermore, Ur/Web assures that the domains from a given specification through a deductive
application 1) does not suffer from code-injection attacks; synthesis. For instance, Paige and Henglein [34] show the
and 2) does not attempt invalid SQL queries, to name a few. derivation of initial implementations of the ADT-based
The language supports a rich-type system based on depend- specification employing fix-point iterations, and later opti-
ent types that guarantees that the developed application mizing the implementation using finite differencing. Lately,
respects the aforementioned features. However, the goal in [35], Hawkins et al. have applied the deductive synthe-
of Ur/Web was to provide a unified web model, where a sis to derive ADT-based implementations based on abstract
programmer develops web application in a single program- relational descriptions for various database operations, e.g.,
ming language that can be compiled to other web standards query and update.
supporting encapsulation of state and concurrency of mul- Our approach extends the above works by allowing to
tithreaded applications. In [10], Yang et al. have developed encode the specification in a general purpose theorem
a language Jeeves whose runtime enforces security policies prover, Coq, which enables us to perform sound synthesis
and guarantees that the programs respects security prop- by checking the proof for consistency. However, our goal
erties by construction. However, the goal of Jeeves was to is to apply the same method to the CPS domain, by algo-
enforce security policies at runtime. Recently, there has rithmically generating secure-and-reliable implementations
been some efforts that focused on applying the aforemen- from ADT-based specifications through a deductive synthe-
tioned alike approaches in ICS domain. For instance, the sis using Coq.
ROSCoq framework [30] has been developed in Coq to
model cyber and physical resources of robots. The frame-
work has extended logic of events to model the resources of B. Runtime Security Monitoring
CPS involving CoRN theory of constructive real analysis and Runtime security monitoring of ICSs is a complex and
then to assure various properties of the model. In [31] and challenging task, as it involves physical processes and criti-
[32], the authors have developed a Coq library “VeriDrone” cal infrastructures. In order to address these challenges, we
as a reasoning framework to ensure security of CPS mod- have developed a runtime security monitor that advances
els at different but independent levels, i.e., from high level the existing techniques in at least one of the following ways:
models to CPS implementation in C. 1) formalism of our specification language; 2) modeling
security properties; 3) monitoring based on executable properties of ICSs. For instance, a language ASLan++ [45]
specification; 4) modular and abstract specification; and allows to model security properties of ICSs for a watar treat-
5) performance efficiency. ment plant in discrete time. Here, our approach is similar
to APEX [50] which employs recent results in reachability
1) Formalism: Formalism of our specification language
analysis [51] to verify hybrid systems. In detail, APEX allows
consists of monadic second-order logic (MSL) [36] and
to model discrete and continuous system constraints and
event calculus [37] operating over algebraic data structures.
then checks whether the system reaches unsafe regions. The
This formalism is the most appropriate to model cyber and
tool suffers from scalability issues due to the state explosion
physical resources of ICSs and their security properties, with
problem in model checking. However, the goal of our work
strict real-time requirements. Based on equivalence results
is to check the consistency between hybrid system con-
of finite automaton and MSL [38]–[41], any MSL formula
straints and a “run” of the system at runtime. In fact, we
can be efficiently translated into a machine code [19], ena-
deal with a single instance of execution at a time and thus
bling our monitor to perform efficient comparison of pre-
avoid scalability concerns.
dictions and observations respecting real-time constraints
of ICSs. In order to handle potential unknown attacks, our 3) Monitoring Based on Executable Specifications: Building
language allows to specify various attack plans that are later runtime security monitors from executable specifications
exploited by the monitor for timely detection of such attacks for monitoring real-time systems has recently started get-
and threats including advanced persistent threats and zero- ting attention [52]. However, executable specifications are
day attacks. Crash Hoare-logic [42] is similar to our formal- powerful in detecting any violation of real-time constraints
ism, as it also allows to specify the unsafe behavior of a file while executing the specification in parallel to the applica-
system. In order to specify fundamentally different charac- tion are more suitable for the runtime security monitoring
teristics (e.g., hybrid systems with a mix of discrete and con- of ICSs. In [53], an executable specification ASML for runt-
tinuous behaviors) and semantics of physical and cyber mod- ime monitoring has been developed at Microsoft. ASML is
els, our formalism allows composition and construction of developed based on state transition systems whose states
high-level system behavior (i.e., discrete) from semantically are first-order algebras [54]. An executable specification
different low-level behavior(s) (i.e., continuous) by employ- language for runtime monitoring of timed systems has been
ing a method analogous to classical set builder together with proposed by Chupilko and Kamkin [55] who use extended
closure property of MSL formulas under function composi- time interval as a pair of a time event and a time interval to
tion [43], [44]. The formalism of most of the existing ICS model properties, which is used by the monitor to check the
security monitors is based on temporal logic [45], [46], rule conformance of an implementation word and the specifica-
systems [47], regular expressions and grammars [48], to tion trace. Also in [56], Ghezzi et al. developed executable
name a few. However, the expressive power of these formal- specification specification and environment (TRIO/TRIO+)
isms does not comprehend ICS requirements for modeling based on events and their relationship, interpreted in first-
physical processes and security properties as first class spec- order temporal logic. However, the language is not suitable
ifications, integral with functional specification. for modeling real-time systems, as it does not support mod-
eling hierarchical and modular specifications [57]. Recently,
2) Modeling Security Properties: Adequate modeling of an
the authors have developed a method for attack detection
ICS application is a complex task, as it involves modeling of
in ICSs (a water treatment plant) by deriving physical pro-
both cyber and physical resources that have different char-
cess invariants for each stage of the CPS from its design and
acteristics and real-time constraints [49]. Therefore, as a
then monitoring the invariants at runtime. Like [58], we
prerequisite of a security model of an ICS application, our
also specify physical process invariants to detect any secu-
language allows to describe physical processes/resources
rity threat to ICSs. Additionally, we also specify other func-
as first class models along with cyber resources. However,
tional and nonfunctional constraints, e.g., performance. In
with cyber and physical models as first class models, mod-
contrast to aforementioned executable specification lan-
eling and monitoring security properties become challeng-
guages, the set theoretic formalism of our specification lan-
ing because security properties need to combine the discrete
guage directly supports classification of observed behaviors
and continuous behaviors of cyber and physical processes.
that belong to different sets of specification. Furthermore,
Therefore, we model a security property as a relationship
our language enables modeling hierarchical and modular
between a set of discrete behavior of cyber processes and a
specifications.
set of continuous behavior of physical processes based on
event calculus. In fact, in a typical CPS, cyber processes 4) Modular and Abstract Specification: The ICS physical
are realized as discrete controllers that are responsible for processes are dynamic, as they depend on physical environ-
making decisions, while the physical processes are realized mental conditions, and thus, are abstract in nature, subject
as continuous controllers that determine physical dynam- to evolution. For instance, the filtering membrane of a water
ics and execute the decisions of the discrete controllers. desalination plant may start filtering at a different rate with
Recently, there has been some effort to model security respect to the amount of humidity in the environment.
Thus, in order to support modeling of such systems and below the real-time constraints of ICSs for various applica-
their evolving constraints, we have introduced model-based tion domains. For example, water or power management
abstract [59] and modular specification, whose syntax (i.e., systems have a desired response delay of a few milliseconds.
constructs) and operational semantics (i.e., execution) are Additionally, we will develop a mechanized proof show-
not directly dependent on the structure of the ICS applica- ing that the monitor only alarms if there is an attack, and
tion implementation. In contrast to our language, classical- vice versa, using some proof assistant, e.g., Coq. We have
model-based specification languages, for instance [60], already shown in [17] and [20] that our proof method works
allow only a contract-based model, whose execution flow in principle.
operation (i.e., structure and semantics) representation
is dependent on the syntactic structure of the application C. Vulnerability Analysis for FDI Attacks
implementation. Such models 1) do not support the mod-
eling of information that is independent of the applica- FDI attack modifies the measurement values that are
tion implementation, e.g., nonfunctional constraints; and exchanged among various ICS subsystems. Such values
2) supports limited modularity, only in the case when the eventually mislead the controller application to conclude
application implementation is modular. Furthermore, our undesired results [8]. Most of the existing approaches have
language is highly modular and abstract, e.g., in our case, attempted to model linear state estimates of power grids
application specification (AppSpec) is independent of the [23], [64]–[67]. In fact, analysis of realistic nonlinear state
application’s implementation (AppImpl). Hence, our lan- estimation models is much harder [22], [24], [66]. Many of
guages allow modeling those behavioral details that operate the existing nonlinear models bypass solving complete non-
on top of the implementation, for instance, component spe- linear constraints involved in state estimation, for instance,
cific workflow execution under additional (dynamic) con- flow equations in power grid. Thus, such solutions only
straints or (evolving) security policies/requirements. The offer analysis of system topology or data based on statistical
syntax of our language is similar to that of the executable techniques.
specification language OBJ [61]. Since our language is devel- In contrast to the aforementioned approaches, we aim to
oped in Lisp, the efficiency and the strength of the abstrac- develop FDI free models by design based on recent results
tion and modularity of our language lie in Lisp’s underlying in delta-decision procedures [68]. Our approach allows to
expert system Joshua [62]; Joshua provides deductive facili- specify nonlinear ICS models in dReal [26], which then
ties to our language, based on forward and backward chain- searches those input values of the variables for which the
ing rules that are realized as generic functions to support monitor does not alarm by reasoning about nonlinear logical
arbitrary abstraction. Furthermore, Joshua has builtin sup- constraints over real numbers.
port for modularity that is identifiable and reusable. Joshua
enables the selection of arbitrary data structures to achieve V II. CONC LUSION A N D F U T U R E WOR K
the desired efficiency.
We have introduced a complete behavior-based approach to
5) Efficiency: Developing runtime security monitors to the design of secure ICSs. Our method targets to produce
meet real-time ICS constraints is a tradeoff between secu- an industrial control application implementation that satis-
rity and efficiency of the ICS application. Thus, in order to fies desired security properties (secure-by-design) as well as
meet the strict real-time ICS constraints, we have devel- a runtime security monitor that identifies runtime attacks
oped a tunable runtime security monitor that monitors by comparing the expected application behavior with the
adequate behavior (i.e., all preconditions, postconditions, behavior of the executed code. The method is feasible and
and invariant) at the time of high threat, and monitors par- practical because it originates from an executable specifica-
tial (i.e., any combination of preconditions, postconditions, tion of the application. Importantly, our approach includes
and invariant) behavior otherwise. There are no fixed per- a method to develop systems that are resistant not only to
formance metrics for runtime security monitoring of ICS computational attacks but also FDI attacks, through the use
applications [63]. However, an evaluation of such moni- of a vulnerability analysis technique that leads to secure
tors is required against the real-time constraints of ICSs. application designs or identification of FDI attack values
In principle, the real-time constraints are periodic, i.e., that can be monitored at runtime. We have demonstrated
the response of a certain component or a certain decision the effectiveness and practicality of our method through
is expected to be completed within a certain time period, detailed descriptions of the development of a secure appli-
say T. Therefore, we ensure that the longest execution of cation for a simple water tank management ICS.
our monitor’s implementation completes in time T, thus Our approach leads to a proposed design methodology
respecting the real-time constraints of the associated com- that is composed of three components: secure-by-design
ponent of the monitored application. Furthermore, our application development, production of a security runtime
results (see Section IV-A) show that, even in highly threat- monitor, and production of an FDI attack monitor through
ened scenarios, our monitor executes in less than 1 ms, vulnerability analysis. In our experiments, up to date, we
well
have been using Coq and dReal for the secure-by-design com- developments will allow our RSM and other associated com-
ponent and the vulnerability analysis component, while we ponents (e.g., control application, operating system) to run at
have developed the security runtime monitor, ARMET. the same level of abstraction (i.e., specification language), thus
In future work, we will be working on the automation of ensuring computation overhead to be negligible on one hand,
all required procedures to develop a unified design method- and no scalability issues on the other hand. Furthermore, to
ology, envisioning the ability to produce reliable and secure achieve even higher performance, our RSM allows to monitor
application code, runtime security monitors, and runtime desired ICS behavior on demand, e.g., to monitor, any or all,
FDI attack value monitors automatically from a single exe- among preconditions, postconditions, and invariant.
cutable specification of the ICS application.
Furthermore, with the recent great developments in for- Acknowledgment
mal verification of software applications and operating sys- The authors would like to thank Prof. A. Chlipala, Prof. A.
tems [69], [70], our monitoring component will run without Solar-Lezama, and Dr. S. Gao for their indispensable col-
any performance and scalability issues in real-time ICSs. Such laboration and support of this project.

REFERENCES [13] I. Matteucci, “Automated synthesis of information against smart power grids,” in
[1] W. A. Conklin, “IT vs. OT security: A time enforcing mechanisms for security Proc. IEEE Global Commun. Conf.
to consider a change in CIA to include properties in a timed setting,” Electron. Notes (GLOBECOM), Dec. 2012, pp. 3153–3158.
resilienc,” in Proc. 49th Hawaii Int. Conf. Syst. Theor. Comput. Sci., vol. 186, pp. 101–120, [25] M. A. Rahman, E. Al-Shaer, and M. A.
Sci. (HICSS), Jan. 2016, pp. 2642–2647. Jul. 2007. Rahman, “A formal model for verifying
[2] M. Zeller, “Myth or reality—Does the aurora [14] R. Soulat, “Synthesis of correct-by-design stealthy attacks on state estimation in power
vulnerability pose a risk to my generator?” in schedulers for hybrid systems,” Ph.D. grids,” in Proc. IEEE 4th Int. Conf. Smart Grid
Proc. 64th Annu. Conf. Protective Relay Eng., dissertation, École Normale Supérieure Commun. (SmartGridComm), Vancouver, BC,
Apr. 2011, pp. 130–136. Paris-Saclay—ENS Cachan, Cachan, France, Canada, Oct. 2013, pp. 414–419.
Feb. 2014. [26] S. Gao, S. Kong, and E. M. Clarke, “dReal:
[3] R. Langner, “Stuxnet: Dissecting a
cyberwarfare weapon,” IEEE Security Privacy, [15] B. Delaware, C. Pit-Claudel, J. Gross, and An SMT solver for nonlinear theories over
vol. 9, no. 3, pp. 49–51, May 2011. A. Chlipala, “Fiat: Deductive synthesis of the reals,” in Proc. 24th Int. Conf. Autom.
abstract data types in a proof assistant,” in Deduction, 2013, pp. 208–214.
[4] B. Kang, K. McLaughlin, and S. Sezer, Proc. 42nd Annu. ACM SIGPLAN-SIGACT
“Towards a stateful analysis framework for [27] S. Gao, L. Xie, A. Solar-Lezama,
Symp. Principles Programm. Lang. (POPL), D. N. Serpanos, and H. E. Shrobe,
smart grid network intrusion detection,” in Mumbai, India, Jan. 2015, pp. 689–700.
Proc. 4th Int. Symp. ICS SCADA Cyber Secur. “Automated vulnerability analysis of AC
Res., 2016, pp. 1–8. [16] C. A. R. Hoare, “Proof of correctness of data state estimation under constrained false
representations,” Acta Inf., vol. 1, no. 4, data injection in electric power systems,” in
[5] U. Lindqvist and P. G. Neumann, “The pp. 271–281, 1972. Proc. 54th IEEE Conf. Decision Control (CDC),
future of the Internet of Things,” Commun. Dec. 2015, pp. 2613–2620.
ACM, vol. 60, no. 2, pp. 26–30, Jan. 2017. [17] M. T. Khan, D. Serpanos, and H. Shrobe, “A
rigorous and efficient run-time security [28] E. W. Dijkstra, “A constructive approach to
[6] R. Mitchell and I.-R. Chen, “A survey of monitor for real-time critical embedded the problem of program correctness,”
intrusion detection techniques for cyber- system applications,” in Proc. IEEE 3rd World Circulated Privately, Tech. Rep., Aug. 1967.
physical systems,” ACM Comput. Surv., vol. Forum Internet Things (WF-IoT), Dec. 2016,
46, no. 4, pp. 55-1–55-29, Mar. 2014. [29] A. Chlipala, “Ur/Web: A simple model for
pp. 100–105. programming the Web,” Commun. ACM,
[7] S. Cheung, B. Dutertre, M. Fong, [18] H. Shrobe, “AWDRAT: A cognitive vol. 59, no. 8, pp. 153–165, Aug. 2016.
U. Lindqvist, K. Skinner, and A. Valdes, middleware system for information
“Using model-based intrusion detection for [30] A. Anand and R. Knepper, ROSCoq: Robots
survivability,” in Proc. 18th Conf. Innovative Powered by Constructive Reals. Berlin,
scada networks,” in Proc. SCADA Secur. Sci.
Appl. Artif. Intell., vol. 2. 2006, pp. 1836–1843. Germany: Springer-Verlag, 2015, pp. 34–50.
Symp., Jan. 2007, pp. 1–12.
[19] B. Courcelle and J. Engelfriet, Graph [31] G. Malecha, D. Ricketts, M. M. Alvarez, and
[8] Y. Liu, P. Ning, and M. K. Reiter, “False data
Structure and Monadic Second-Order Logic: A S. Lerner, “Towards foundational
injection attacks against state estimation in
Language-Theoretic Approach. Cambridge, verification of cyber-physical systems,” in
electric power grids,” ACM Trans. Inf. Syst.
MA, USA: Cambridge Univ. Press, 2012. Proc. Sci. Secur. Cyber-Phys. Syst. Workshop
Secur., vol. 14, no. 1, pp. 13-1–13-33, Jun.
2011. [20] M. T. Khan, D. Serpanos, and H. Shrobe, (SOSCYPS), Apr. 2016, pp. 1–5.
“Sound and complete runtime security [32] M. Chan, D. Ricketts, S. Lerner, and
[9] A. A. Cárdenas, S. Amin, Z.-S. Lin, Y.-
monitor for application software,” Dept. G. Malecha, “Formal verification of stability
L. Huang, C.-Y. Huang, and S. Sastry,
“Attacks against process control systems: Comput. Sci. Artif. Intell. Lab., MIT, properties of cyber-physical systems,” in
Risk assessment, detection, and response,” in Cambridge, MA, USA, Tech. Rep. MIT- Proc. CoqPL, Jan. 2016.
Proc. 6th ACM Symp. Inf., Comput. Commun. CSAIL-TR-2016-017, Dec. 2016. [33] J. Fitzgerald, K. Pierce, and C. Gamble,
Secur., 2011, pp. 355–366. [21] C. Kwon, W. Liu, and I. Hwang, “Analysis “A rigorous approach to the design of
[10] J. Yang, K. Yessenov, and A. Solar-Lezama, “A and design of stealthy cyber attacks on resilient cyber-physical systems through
language for automatically enforcing privacy unmanned aerial systems,” J. Aerosp. Inf. co-simulation,” in Proc. IEEE/IFIP 42nd Int.
policies,” in Proc. 39th Annu. ACM SIGPLAN- Syst., vol. 11, no. 8, pp. 525–539, 2014. Conf. Depend. Syst. Netw. Workshops (DSN-W),
SIGACT Symp. Principles Programm. Lang., [22] G. Hug and J. A. Giampapa, “Vulnerability Jun. 2012, pp. 1–6.
New York, NY, USA, 2012, pp. 85–96. assessment of ac state estimation with [34] R. Paige and F. Henglein, “Mechanical
[11] M. Zhang, Y. Duan, Q. Feng, and H. Yin, respect to false data injection cyber-attacks,” translation of set theoretic problem
“Towards automatic generation of security- IEEE Trans. Smart Grid, vol. 3, no. 3, specifications into efficient RAM code—A
centric descriptions for android apps,” in pp. 1362–1370, Sep. 2012. case study,” J. Symbolic Comput., vol. 4, no. 2,
Proc. 22nd ACM SIGSAC Conf. Comput. [23] A. Teixeira, S. Amin, H. Sandberg, pp. 207–232, Aug. 1987.
Commun. Secur., New York, NY, USA, 2015, K. H. Johansson, and S. S. Sastry, “Cyber [35] P. Hawkins, M. Rinard, A. Aiken, M. Sagiv,
pp. 518–529. security analysis of state estimators in and K. Fisher, “An introduction to data
[12] F. Martinelli and I. Matteucci, “An approach electric power systems,” in Proc. 49th IEEE representation synthesis,” Commun. ACM ,
for the specification, verification and Conf. Decision Control (CDC), Dec. 2010, vol. 55, no. 12, pp. 91–99, Dec. 2012.
synthesis of secure systems,” Electron. Notes pp. 5991–5998. [36] J. Henriksen, “Mona: Monadic second-order
Theor. Comput. Sci., vol. 168, pp. 29–43, [24] M. A. Rahman and H. Mohsenian-Rad, logic in practice,” in Tools and Algorithms for
Feb. 2007. “False data injection attacks with incomplete the Construction and Analysis of Systems
 (Lecture Notes in Computer Science), [48] F. Chen and G. Rosu, “MOP: An efficient [59] J. M. Wing, “A specifier’s introduction to
vol. 1019. 1995, pp. 89–110. and generic runtime verification formal methods,” Computer, vol. 23, no. 9,
[37] G. C. Borchardt, “Event calculus,” in Proc. framework,” ACM Sigplan Notices, vol. 42, pp. 8–23, Sep. 1990.
9th Int. Joint Conf. Artif. Intell., vol. 1. 1985, no. 10, pp. 569–588, 2007. [60] T. Gary Leavens and Y. Cheon (2006). Design
pp. 524–527. [49] I. Ruchkin, S. Selva, S. Bradley, R. Amanda, by Contract With JML. A Tutorial. [Online].
[38] R. J. Büchi, “Weak second-order arithmetic and G. David, “Challenges in physical Available: ftp://ftp.cs.iastate.edu/pub/
and finite automata,” Zeitschrift Mathe. modeling for adaptation of cyber-physical leavens/JML/jmldbc.pdf
Logik Grundlagen Math., vol. 6, nos. 1–6, systems,” in Proc. IEEE World Forum [61] J. A. Goguen, T. Winkler, J. Meseguer,
pp. 66–92, 1960. Internet Things MARTCPS, Dec. 2016, K. Futatsugi, and J.-P. Jouannaud, Introducing
pp. 210–215. OBJ. Boston, MA, USA: Springer-Verlag,
[39] J. R. Buechi, “On a decision method in
restricted second-order arithmetic,” in Proc. [50] M. O’Kelly, H. Abbas, S. Gao, S. Shiraishi, 2000, pp. 3–167.
Int. Congr. Logic, Methodol. Philos. Sci., 1962, S. Kato, and R. Mangharam, “APEX: [62] S. Rowley, H. Shrobe, R. Cassels, and
pp. 1–11. Autonomous vehicle plan verification and W. Hamscher, “Joshua: Uniform access to
execution,” in Proc. SAE World Congr., vol. 1. heterogeneous knowledge structures or why
[40] M. O. Rabin, “Decidability of second-order Apr. 2016, pp. 1–13.
theories and automata on infinite trees,” joshing is better than conniving or
Trans. Amer. Math. Soc., vol. 141, nos. 1–35, [51] S. Kong, S. Gao, W. Chen, and E. Clarke, planning,” in Proc. AAAI, Seattle, WA, USA,
p. 4, 1969. dReach: δ-Reachability Analysis for Hybrid 1987, pp. 48–52.
Systems. Berlin, Germany: Springer-Verlag, [63] A. Kane, “Runtime monitoring for safety-
[41] C. C. Elgot, “Decision problems of finite
2015, pp. 200–205. critical embedded systems,” Ph.D.
automata design and related arithmetics,”
Trans. Amer. Math. Soc., vol. 98, no. 1, [52] M. Blum and H. Wasserman, “Software dissertation, Dept. Electr. Comput. Eng.,
pp. 21–51, 1961. reliability via run-time result-checking,” Carnegie Mellon Univ., Pittsburgh, PA,
J. ACM, vol. 44, pp. 826–849, 1994. USA, 2015.
[42] H. Chen, D. Ziegler, T. Chajed, A. Chlipala,
M. F. Kaashoek, and N. Zeldovich, “Using [53] M. Barnett and W. Schulte, “Runtime [64] G. Dán and H. Sandberg, “Stealth attacks
crash hoare logic for certifying the FSCQ file verification of.NET contracts,” J. Syst. Softw., and protection schemes for state estimators
system,” in Proc. 25th Symp. Oper. Syst. vol. 65, no. 3, pp. 199–208, 2003. in power systems,” in Proc. 1st IEEE Int. Conf.
Principles, 2015, pp. 18–37. Smart Grid Commun. (SmartGridComm), Oct.
[54] E. Borger and R. F. Stark, Abstract State
2010, pp. 214–219.
[43] R. Alur, A. Durand-Gasselin, and A. Trivedi, Machines: A Method for High-Level System
“From monadic second-order definable Design and Analysis. New York, NY, USA: [65] O. Kosut, L. Jia, R. J. Thomas, and L. Tong,
string transformations to transducers,” Springer-Verlag, 2003. “Malicious data attacks on the smart grid,”
in Proc. 28th Annu. ACM/IEEE Symp. IEEE Trans. Smart Grid, vol. 2, no. 4,
[55] M. M. Chupilko and A. S. Kamkin, “Runtime
Logic Comput. Sci. (LICS), Mar. 2013, pp. 645–658, Dec. 2011.
verification based on executable models:
pp. 458–467. On-the-fly matching of timed traces,” in [66] A. Teixeira, I. Shames, H. Sandberg, and
[44] B. Courcelle, “Monadic second-order Proc. 8th Workshop Model-Based Test., 2013, K. H. Johansson, “Revealing stealthy attacks
definable graph transductions: A survey,” pp. 67–81. in control systems,” in Proc. 50th Annu.
Theor. Comput. Sci., vol. 126, no. 1, Allerton Conf. Commun., Control, Comput.
[56] C. Ghezzi, D. Mandrioli, and A. Morzenti, (Allerton), Oct. 2012, pp. 1806–1813.
pp. 53–75, 1994. “Trio: A logic language for executable
[45] M. Rocchetto and N. O. Tippenhauer, specifications of real-time systems,” [67] L. Xie, Y. Mo, and B. Sinopoli, “Integrity
“Towards formal security analysis of J. Syst. Softw., vol. 12, no. 2, pp. 107–123, data attacks in power market operations,”
industrial control systems,” in Proc. ACM May 1990. IEEE Trans. Smart Grid, vol. 2, no. 4,
Asia Conf. Comput. Commun. Secur., 2017, pp. 659–666, Dec. 2011.
[57] S. Gérard, H. Espinoza, F. Terrier, and
pp. 114–126. B. Selic, “Modeling languages for real-time [68] S. Gao, J. Avigad, and E. M. Clarke, “δ-Complete
[46] A. Bauer, M. Leucker, and C. Schallhart, and embedded systems: Requirements and decision procedures for satisfiability over the
“Runtime verification for LTL and TLTL,” standards-based solutions,” in Proc. Int. reals,” in Proc. 6th Int. Joint Conf. Autom.
ACM Trans. Softw. Eng. Methodol., vol. 20, Dagstuhl Conf. Model-Based Eng. Embedded Reasoning-(IJCAR), Manchester, U.K., Jun.
no. 4, pp. 1–68, 2011. Real-Time Syst., 2010, pp. 129–154. 2012, pp. 286–300.
[47] H. Barringer, D. E. Rydeheard, and K. [58] S. Adepu and A. Mathur, Using Process [69] Deepspec. [Online]. Available: https://
Havelund, “Rule systems for run-time Invariants to Detect Cyber Attacks on a Water deepspec.org/page/Research/
monitoring: From eagle to ruler,” J. Logic Treatment System. Cham, Switzerland: [70] Certikos. [Online]. Available: http://flint.
Comput., vol. 20, no. 3, pp. 675–706, 2010. Springer-Verlag, 2016, pp. 91–104. cs.yale.edu/certikos/

ABOUT THE AUTHORS

Muhammad Taimoor Khan graduated in com- Dimitrios Serpanos (Senior Member, IEEE) received
puter science from the Islamia University Baha- a Diploma in computer engineering and informat-
walpur, Bahawalpur, Punjab, Pakistan, in 2001, ics from the University of Patras, Patras, Greece, in
and received the M.S. degree in advanced dis- 1985 and the Ph.D. degree in computer science from
tributed systems from the University of Leicester, Princeton University, Princeton, NJ, USA, in 1990.
Leicester, U.K., in 2008 and the Ph.D. degree from He is the Director of the Industrial Systems
the Research Institute for Symbolic Computa- Institute/ATHENA RC and a Professor at the
tion (RISC), Johannes Kepler University, Austria, Department of Electrical and Computer Engi-
in 2014. neering, University of Patras. Before joining the
He is a Postdoctoral Researcher with the Institute of Informatics, University of Patras, he was a Research Staff Member at IBM, T.J. Wat-
Alpen-Adria University, Klagenfurt, Austria. In the last decade, he has son Research Center, Yorktown Heights, NY, USA (1990�1996) and faculty
been applying formal methods as a powerful tool to assure reliability member at the Department of Computer Science, University of Crete,
and security of various software systems, for instance, industrial control Crete, Greece (1996�2000), where he also worked at the Institute of
systems, computer mathematics-based systems, to name a few. He has Computer Science of the Foundation for Research and Technology-Hel-
extensive experience in both software industry and research institutes. He las (ICS-FORTH). He has been Principal Scientist at the Qatar Computing
has been working as a scientist at various premier international research Research Institute (2013�2016). He has served as President of the Univer-
institutes, including INRIA, France and MIT CSAIL, Cambridge, MA, USA; he sity of Western Greece (2010�2013) and another term as Director of the
is jointly working with these institutes now. Industrial Systems Institute/ATHENA RC (2008�2013), where he has been
Dr. Khan has won various research and academic awards including conducting research since 2000. His research interests include embedded
best paper award(s). systems, security systems, industrial systems, and computer architecture.
 Prof. Serpanos is currently a member of the Board of Governors of the initiative. He has served twice as a program manager at DARPA. From
IEEE Computer Society (2017). He is a member of the New York Academy 1994 to 1997, he served as Chief Scientist of the Information Technology
of Sciences, the Association for Computing Machinery (ACM), the Ameri- Office and led the Information Security Initiative; from 2010 to 2013, he
can Association for the Advancement of Science (AAAS), and the Technical served as a Program Manager in TCTO and then I2O, leading the CRASH
Chamber of Greece. and MRC programs. While at DARPA, he initiated and led the Evolutionary
Design of Complex Software program as well as the Information Surviv-
Howard Shrobe received the M.S. and Ph.D. ability program. While at MIT, he has participated in several Cyber Security
degrees from the MIT's Artificial Intelligence research projects such as DARPA's Intrusion Tolerant Systems program,
Laboratory, Cambridge, MA, USA, in 1975 and the OASIS program, the Self-Regenerative Systems and IARPA's NICECAP
1978, respectively. program.
He is a Principal Research Scientist at MIT Dr. Shrobe was selected Fellow of the American Association for the
Computer Science and Artificial Intelligence Lab- Advancement of Science (AAAS) in November 2016.
oratory (MIT CSAIL), Cambridge, MA, USA. He is
a former Associate Director of CSAIL and is the
current Director of CSAIL's CyberSecurity@CSAIL