This article has been accepted for publication in IEEE Transactions on Industrial Informatics.

This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2023.3244329

ECS-Grid: Data-Oriented Real-Time Simulation

Platform for Cyber-Physical Power Systems
Tianshi Cheng, Graduate Student Member, IEEE, Tong Duan, Member, IEEE, and Venkata
Dinavahi, Fellow, IEEE

Abstract—ECS-Grid is the first data-oriented real-time electro- grid and power system automation [6]. Thus, detailed and
magnetic transient simulation platform for cyber-physical power accurate real-time simulation of cyber-physical power systems
systems (CPPS). Traditional simulation tools are constrained (CPPS) [7] is necessary for future power system research [8]–

LY
by object-oriented programming (OOP) architecture, which is
now a significant obstruction to creating a comprehensive cyber- [10]. However, the scalability, flexibility, and performance of
physical simulation. Therefore, the proposed ECS-Grid platform the traditional power system analysis tools and communica-
follows a new data-oriented paradigm based on an Entity- tion analysis tools are inadequate. For example, the existing
Component-System (ECS) framework, which delivers higher software-based simulation approaches such as EPOCHS [11],
flexibility, extensibility, scalability, and performance to support GECO [12], INSPIRE [13] and the simulators proposed in
cyber-physical system research. ECS-Grid proposes a layer of
virtual intelligent electronic devices (vIEDs) to model IEDs in [14]–[16] aim to create a network interface for existing power
CPPSs. The vIEDs directly talk to physical components and system simulation tools and glue the two domains in power

N
communicate asynchronously with cyber services via the pro- grid simulator and communication network simulators (NS-
posed high-performance JSON-like binary protocol. Tests with 2, NS-3 and OPNET et al.), which cannot reflect the be-
the islanding and the man-in-the-middle cyber attack scenarios haviors of IEDs in real-time environments and must handle
on a 711-node AC-DC microgrid cluster based on a modified
CIGRE 15-Bus system are performed and give accurate results. the complicated synchronization between two different sim-
A faster-than-real-time performance is achieved on the 10th Gen
Intel® CoreTM i7 computer, and real-time performance is achieved
on distributed embedded NVIDIA® Jetson platform. The ECS-
O ulation domains. Some works such as [17]–[20] bridged the
commercial real-time power system electromagnetic transient
(EMT) digital simulators to the communication simulation
Grid design and test results demonstrate the potential of the systems, which can achieve real-time performance and more
ECS data-oriented paradigm and may inspire the renovation of
industrial simulation software. realistic behaviors. However, the high-cost commercial EMT
simulators were designed for industrial verification purposes
Index Terms—Cyber-physical power systems, data-oriented
D
and still lack the scalability and flexibility for CPPS-related
programming, digital twin, entity-component-system, electromag-
netic transients, faster-than-real-time, field programmable gate academic research.
arrays, intelligent electronic devices, real-time systems, micro- The essential problem is that both power grid and commu-
grids, parallel processing nication network simulation tools are initially designed for
their single domain. The traditional software mainly based
EA

on object-oriented programming (OOP) has become a huge

I. I NTRODUCTION obstruction to building a native and comprehensive cyber-
The transition to clean and renewable energy in the power physical simulation platform. The industrial programs are deal-
industry is playing a significant role in reducing the emis- ing with various forms of data and their combinations, while
sion of greenhouse gases and fighting climate change [1]. the OOP paradigm emphasizes predetermined inner structure
However, the traditional power systems that rely on cen- and relationships of objects. Plain data such as an array of
tralized control networks and controllable generators become float numbers can represent many things in the computer
insufficient to meet the challenges of future power systems world. On the opposite, a class and its object can only be
R

such as microgrids with the high penetration of uncertain used for one purpose predefined by abstract templates, which
and unstable renewable energy [2]. Therefore, new intelligent brings significant difficulties to repurposing existing designs
decentralized control solutions based on modern information and thus cannot elegantly describe and solve the problems in
and communication technologies are emerging to face the the complex interdisciplinary CPPS.
new challenges [3]–[5]. The new research works heavily Therefore, this paper proposes for the first time, the ECS-
involve communication between control centers and intelligent Grid: a novel real-time cyber-physical EMT simulation plat-
electronic devices (IEDs), which are the foundation of smart form with virtual IEDs (vIEDs) based on the cutting-edge
entity-component-system (ECS) software framework. The pro-
This work is supported by the Natural Science and Engineering Research posed ECS-Grid simulation platform has the following major
Council of Canada (NSERC).
Tianshi Cheng and Venkata Dinavahi are with the Department of Electrical advantages:
and Computer Engineering, University of Alberta, Edmonton, Alberta T6G 1) High Flexibility: Compared to the traditional dominating
2V4, Canada. Email: [email protected], [email protected]. object-oriented paradigm which is based on Polymor-
Tong Duan is with the National Digital Switching System Engineer-
ing and Technological Research Center, Zhengzhou, China. Email: duan- phism, Abstraction, Inheritance, and Encapsulation, the
[email protected]. ECS framework is based on a data-oriented paradigm:

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF ALBERTA. Downloaded on February 13,2023 at 22:51:06 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2023.3244329

Entity (usually an integer), Component (pure data struc- as IEC-60870-5-104 is also discussed. Section IV presented
ture), and System (plain functions to perform algo- the microgrid cluster study case with results from one steady-
rithms on components), where entities are defined by state scenario and one cyber-attack scenario. Section V is the
the combination of data components, and component conclusion.
functionalities are defined by systems. This data-oriented
paradigm avoids dependency complexities caused by II. P ROPOSED DATA -O RIENTED A RCHITECTURE OF
OOP inheritance and brings flexible model description ECS-G RID
ability. Any data component and the system can be
replaced not only at the compiled time but also at the A. Data-Oriented ECS Architecture
run time. Such a feature is highly desired for cyber- The information exchange between physical and cyber
physical simulation since the various form of data and systems is the major concern in a cyber-physical simulation.
data flows are the major concerns. For example, real- Information is carried by data, and generally, cyber systems are
world IEDs are composed by multi-functional circuit built to transport and process data that carry useful informa-
boards and these replacable boards can be represented tion. However, while data can represent almost anything in the

LY
by data components on a vIED entity. digital world, an object which contains both data structure and
2) High Extensibility: with the advanced data-oriented de- behaviors can only carry limited information whose pattern
sign, components and systems are grouped into plugins is pre-defined by its abstract templates: the Class, without
in ECS-Grid, and a simulation application is composed the ability to mutate its structure. The Inheritance makes it
of a set of plugins. In contrast to traditional software even worse due to the extra dependencies between Classes. As
which often provides a huge library as an undividable shown in Fig. 1 (a), the OOP paradigm creates abstract base
whole, ECS-Grid allows users to only pay for what Classes for different domains while inherited implementations

N
they need. Although it is initially designed for CPPS are realized in sub Classes. This adds difficulties in refactoring
simulation, it can run pure physical simulation similar to and optimization. Since cyber-physical systems in the big
simulators without cyber layers, or run the cyber features data age are transporting enormous unstructured data, a data-
for other purposes without the physical EMT simulation. oriented solution that focuses on data processing and data
Moreover, the users can create plugins easily even in
dynamic libraries with their customized components
and systems, and add or override core functionalities
O combinations is highly preferred to OOP solutions.
Data-oriented programming means data combinations deter-
mine functionalities, which is also the core concept of ECS-
such as the matrix solvers or additional communication Grid. The ECS framework starts to play a significant role in the
protocols. game industry and modern software engineering, which is now
3) High Scalability: With the benefits from the ECS frame- the backbone of MinecraftTM [23], Data-Oriented Technology
Stack (DOTS) in Unity® [24], Call of Duty® : Vanguard, Ar-
D
work, a vIED layer is proposed which mainly uti-
lizes scalability protocols from the message-oriented cGIS Runtime SDKs by EsriTM , and many modern large-scale
asynchronous ZeroMQ [21]. A MessagePack-based [22] commercial software projects. However, it is still not utilized
JSON-like simulator protocol is proposed for the simula- for cyber-physical simulation in power industries which are
tor to bridge various industrial protocols. The utilization full of data-intensive applications. Currently, there are three
EA

of the middleware makes it easy to scale ECS-Grid major types of ECS frameworks: bitset, archetype, and sparse-
from a single CPU node to multi-thread applications or set, where sparse-set is the most popular one due to its high
even distributed networks which resembles real-world flexibility and archetype has the best theoretical performance.
automation systems. The performance test on a single- The specific types of components are managed by an entity
thread ZeroMQ vIED with MessagePack-based protocol registry to provide database-like access to the data objects.
shows a minimal latency of 6µs and an average latency In this paper, the sparse-set-based EnTT [23] is used as the
of 20µs with an effective 60Mbit/s bandwidth, which is entity registry, which is also used in MinecraftTM . Everything
quite enough for a wide range of application scenarios. under the ECS framework belongs to an Entity, Component, or
R

System. The inheritance is replaced by an entity’s composition

A 711-node AC/DC microgrid cluster based on the mod- of data components in the ECS framework. An Entity is an
ified CIGRE-15 Bus microgrid system with a man-in-the- integer identifier that is linked to multiple components in an
middle cyber-attack scenario is set up for demonstration and entity data registry which can be seen as a data table in Fig. 1
performance evaluation. The simulation results show that the (b). A Component is a structure with data to process. As shown
proposed solution can achieve faster-than-real-time (FTRT) in Fig. 1 (b), entities are rows of the data table. For example, a
performance on 10th Gen Intel® CoreTM i7 CPUs and real- voltage-source converter (VSC) entity is composed of the four
time performance on NVIDIA Jetsons with dual-core ARM circles in a row of the table view; the EMT model object which
v8 CPUs. is the same circuit object of the traditional OOP design such
The paper is organized as the following: Section II in- as an averaged-value model VSC; the IO module which holds
troduces the fundamental architecture and methodologies in measured signals and controller signals; the VSC controller
ECS-Grid. Section III introduces the simulator MessagePack holds data for control logics and the IED communication
protocol and the performance test of vIEDs with different module holds the information of network sockets and other
transportations; the way to implement industrial protocols such parameters such as latencies.

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF ALBERTA. Downloaded on February 13,2023 at 22:51:06 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2023.3244329

Physical Comm & Control Network Interface

Implement Abstract Implement Abstract Implement Abstract
and still needs some exploration. Details about EMTModel
RLC Bloat IO data Modbus TCP/IP
are discussed in Section II.B. IED components and systems
AC IEC-104
Comm are discussed in Section II.C.
UDP Net
Source Physical ... Interface
Device
Multi- IEC-
Battery RS485
inherance 61850
VSC
IO VSC
Controller MAC B. Physical EMT Simulation
Interface Control Layer
... Virtual functions
(a)
Currently, only EMT physical simulation is implemented in
IED
Entities Component Data Registry ECS-Grid. In EMT simulation, dynamic physical components
composed
by row
such as capacitors and inductors are represented by differential
EMT IO VSC Battery IED
Model Module Control Control Module equations. Small and simple circuits such as RC, LC, and RLC
ESS 1
can be solved with a form of ordinary differential equations or
VSC1 state-space format. However, a power grid usually consists of a
Line 1

Lines
pure physical
component
large amount of different physical components, which is more
RLC
suitable to solve as differential-algebraic equations (DAEs)

LY
AC
AC 1 with nodal analysis. The nodal analysis is based on solving
Source

(b) a circuit equation system with nodal voltages as primary

unknown variables. Previous research works often emphasize
Fig. 1: Traditional OOP and data-oriented ECS design for
elementwise derivation, while in this paper the simulation is
CPPS simulation: (a) Inheritance and abstract interfaces create
purely expressed by the language of linear algebra. Using
complex object relationships to represent physical objects with
Kirchhoff’s Current Law (KCL) any RLC circuits with k nodes
IED under the OOP paradigm. (b) An entity is defined by data
(ground node excluded) can be represented by:
component combinations similar to a row in a data table under

N
X
the ECS framework, while data are processed by columns. iout = iL + iC + iG
Z
dv
= WL vdt + WC + WG v = s,
dt
A specific combination of data components will be pro-
cessed by relevant Systems which contain all program logic.
A System is a plain function that can process columns of com-
O 1
WL = BLT [ ]BL , WC = BC
L
T
[C]BC , WG = BG T
[G]BG ,
(1)
ponents in the table of Fig. 1 (b). It usually takes the registry where B is the oriented incidence matrix whose rows are
as the input parameter and creates a query view of components corresponding to the physical components and columns are
from the registry just like a database query. The queries are corresponding to nodes, L is the inductance, C is the capac-
optimized by the ECS framework depending on its storage type itance and G is the admittance, s is the vector of current
D
and usually are blazing fast. Because the system function is injections by sources. B is a transformation to gather the port
only called once on specific types of components, it eliminates voltages from global nodal voltages v, while B T can scatter
the bloating issue caused by intermediate interfaces or CPU the branch currents into the nodal injection vector. The W
overhead in dynamic virtual function calls per object; since matrices are the weighted Laplacian matrices of different types
EA

the components are stored in compact arrays, it is also more of components, which are also called admittance matrices and
cache-friendly and easier to take advantages of modern single- play important roles in solving power grid equation systems.
instruction-multiple-data (SIMD) hardware such as graphical [X] means diagonalized matrix of 1-D vector X.
processing units (GPUs). In other words, it can fully avoid the To solve (1) with the Trapezoidal Rule, the following
usage of inheritance and polymorphism to build more complex equations can be obtained:
and efficient software with an ECS framework. Also, ECS X
ioutn+1 = iLn+1 + iCn+1 + iGn+1 = sn+1 , (2)
brings impressive flexibility to modify an Entity. For example,
one can replace the VSC control component without breaking
R

other VSCs with the same physical model, or replace a system 2 ∆t n+1 n+1
at the runtime to change the functionality. (WG + WC + WL )vn+1 = sn+1 + ILeq + ICeq ,
∆t 2
The proposed ECS-Grid currently uses a hybrid ECS solu- n+1 ∆t
tion to fully reuse the traditional OOP EMT simulation code. ILeq = − WL vn − iLn ,
2
The EMT simulation loop is untouched, and no modification is n+1 ∆t
added to any physical component class. The only difference is ICeq = WC vn + iCn ,
2
the traditional physical components are now managed as a part (3)
n+1 n+1
of an entity in an ECS registry instead of an all-in-one object. where ILeq and ICeq are synthetic current sources created by
The IED feature is added by introducing new components and discretization. A similar rule is applied to all other dynamic
systems to the physical software. This hybrid solution can be or time-varying physical components in the power system for
very useful for industrial developers to transfer from traditional EMT simulation, which gives the general form:
OOP to data-oriented design under the ECS framework. The X
n+1
Y vn+1 = sn+1 + Ieq ,
full transition to an ECS data-oriented simulation framework X (4)
requires many critical changes to traditional design patterns Y = Wdif f ,

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF ALBERTA. Downloaded on February 13,2023 at 22:51:06 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2023.3244329

where Y indicates the final admittance matrix for the system General Module Structure of an IED in Power System
solution, which can be inversed to solve the primary unknown
n+1 1 2 3 4 5 6 7 8
variable vn+1 ; Ieq indicates all equivalent current sources board:
generated by physical components; Wdif f is the admittance
matrix derived from differential equations such as ∆t 2 WL for RJ-45 for Ethernet TX AO 0 AI 0 AI 5 DI 0 DO 0 PWR+
AO 1 AI 1 AI 6 DI 1
inductors. RX AO 2 AI 2 AI 7 DI 2
DO 1
DO 2
PWR-
GND
For nonlinear components such as diodes that have nonlin- …… AI 3 …… …… …… ……
CPU CPU TX AO
CPUn AI 4CPUAI n DI
CPUn DO n
CPU CPU
ear voltage-current characteristics. (4) is extended to:
RX
X Time PPS
n+1
iN + Y vn+1 = sn+1 + Ieq , synchronization
IRIG-B
(5)
T CPU GOOSE/SV AO AI DI DO PWR
iN = BN [f (BN vn+1 )]
where f is an elementwise nonlinear function of BN vn+1 .
Newton’s method is utilized to linearize the system and solve DSP CPU
Network
DSP Analog Analog Binary Binary Power

LY
Controll (Etherne
it, which converts Equation (5) into the following: er t)
(IEC Output Input Output Input Supply
61850)
X
n+1
F (vn+1 ) = iN + Y vn+1 − (sn+1 + Ieq ) = 0, Controller
Communication
Power supply
(not useful in
Module IO Modules simulation)
Modules
δiN
F ′ (vn+1
m
)= |vm + Y = J m , (6)
δv n+1 Fig. 2: A real-world IED consists of the controller, communi-
m
F (v ) m−1 cations, IO modules, and power supply.
m+1
vn+1 m
= vn+1 − ′ n+1 m
m ) = vn+1 − J
m
F (vn+1 ),
F (vn+1

N
VSC_IO VSC_Control VSC_IED
where m denotes the iteration index of the Newton method,
query query query
J m is the system Jacobian matrix at mth iteration. Equation
VSC IO IO MMCControl IO VSCControl IED
(6) can be reorganized into:
for_each (vsc,io) for_each (io, ctrl) for_each (io,ctrl, ied)

m+1
J m vn+1 = (J m − Y )vn+1
m m
m

m
− im
N + (sn+1 +

= Wnl vn+1 − iN + (sn+1 +

X
n+1
Ieq ),
n+1
Ieq )
(7)
O X vsc.Iabc

vsc.Vabc

vsc
io.AI

io.AI

io.DO
io.AI ctrl.refs

ctrl.control_func
io.all

ied.out_msg
ied.rx

ied.in_msg

.Gates
io.DO io.AO ied.tx ctrl.refs
where
m δiN T m VSC Entities New Components New Systems (functions)
Wnl = |vm = BN [∇f (BN vn+1 )]BN . (8) Tsample T control Tcomm
δv n+1
D
VSC IO VSCControl IED VSC_IED
VSC_IO VSC_Control
Therefore, the nonlinear components have the harmonized New Stage For
Old Physical Simulation Loop
format of admittance matrices and artificial current injections, Cyber Features
which gives the following recursion formula: PreUpdate Update PostUpdate IED stage
EA

X X
m+1 m+1
J m vn+1 = sn+1 + n+1
Ieq + Inleq ,
(9) Fig. 3: IED sampling, control, and communication systems
m+1 m m m
Inleq = Wnl vn+1 − iN . execution in the proposed data-oriented framework of ECS-
Grid.
The Jacobian matrix needs to be assembled and inverted
serval times in each simulation time-step. Therefore, the f of
many nonlinear components may be converted into piece-wise
CPU, Network DSP (only for optical IEC-61850 GOOSE/SV),
linear function or use vn to approximate vn+1 to speed up
AI, AO, DI, DO for analog or digital inputs/outputs (IOs), and
the computation. (1-9) cover the fundamentals of the physical
the power source, which are circuit boards is responsible for
R

power system EMTModel components and systems in ECS-

specific tasks. Different configurations of the board modules
Grid.
and internal firmware will define the functionalities of the IED.
It can be beneficial to model the IEDs inside EMT simulation
C. Data-Oriented IED Simulation to reflect real-world communication behaviors such as network
In most CPPS simulations, there were only two layers: latencies, time synchronizations, and protocol analysis, and
the physical layer and the cyber layer, which often ignored also bring many new possibilities to cyber-physical research.
first-class citizens in real-world CPPSs: intelligent electronic The structure of real-world IEDs is a perfect match for the
devices (IEDs). The IEDs are generally protection, control, proposed data-oriented architecture.
and monitoring devices in power grids, which are cornerstones ECS-Grid proposed the layer of vIEDs, which is an in-
for the modern power system automation and smart grid [25]. dependent set of components and systems to model IEDs
Therefore a comprehensive cyber-physical simulation should in power grids to conduct control or communication tasks.
model these IEDs to be as realistic as possible. These digital twins of real-world IEDs bring more realistic
As shown in Fig. 2, a commonly used IED in power and more consistent experiences from real-world CPPS. The
systems is composed of multiple modules: DSP controller, vIEDs consist of IO, control, and communication components

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF ALBERTA. Downloaded on February 13,2023 at 22:51:06 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2023.3244329

Simulation Extensions Plugin Interfaces ECS Simulation Core MsgPack IEC 104 APDU
.cpp,.hpp Customized Pipelines: Combine the Plugins Accordingly Topic: VSC_1_TX COT: periodic
Physical Plugin
MicroGrid Config HVDC Config
Communication { IOA
Control Center
Test Config Interoperability 101
Source Codes "Type": "MEAS",
Microgrid EMT MMC + Fake Data "P": 1.0, 2048 Data
Solver Plugin Models Bergeron Lines Generator
.a "Quality": "valid", 10.0.0.1:2404
Valid Quality
KLU Solver KLU Solver "ts": 1653951927
Static Libraries Control Plugin CP56 Virtual IEC-104 RTU
(unix timestamp) Time tag
Dummy time2a
Droop Control VDC Control } Protocol Conversion
Control
.so,.dll IED Plugin ZeroMQ vIED ZeroMQ vIED Various vIED
Dynamic
Libraries Measurement Failure Recorder Failure Recorder
Plugin Signal IOA Scale Protocol Conversion
VSC_1.P 101 2048
Fig. 4: Example plugins and their configurations in the pro- Data Mapping
Configuration VSC_1.Q 102 2048
posed ECS-Grid. Publish control Subscribe
cmds measurements

IED Communication

LY
Configuration
in the ECS-Grid. which covers the fundamentals of a real- Socket Address Topic ZMQ TCP broker
world IED in Fig. 2. PUB 127.0.0.1:9001 VSC_1_TX XSUB XPUB

As shown in Fig. 3, the components of VSC entities in SUB 127.0.0.1:9002 VSC_1_RX

Fig. 3 are similar to the modular boards in Fig. 2; The
systems:VSC_IO, VSC_Control and VSC_IED are similar to Publish
measurements
the software programmed into the physical IED; the systems
Subscrible
are grouped and called in the IED Stage which is considered as control cmds

N
a new sensing layer compared to old physical simulation loop.
Similar to the replaceable boards and upgradable programs in vIEDs
real-world IEDs, the components and systems of vIEDs can
be replaced or reorganized to serve different purposes both at
O VSC 1 VSC 2 VSC 3
compile-time and runtime. This is realized elegantly within
Fig. 5: The communication between vIEDs and real-world IEC
a data-oriented ECS framework while OOP cannot compose
60870-5-104 clients.
it nicely due to its fixed pre-defined structures. The vIED
extension is implemented by a very simple plugin interface
introduced in Section II-D.
III. P ROPOSED DATA -O RIENTED P ROTOCOL FOR
R EAL -T IME C YBER -P HYSICAL S IMULATION
D
D. Plugins Made Easy Traditional CPPS simulations are usually based on available
commercial simulators, where the signals of the physical
The extensibility is important for a cyber-physical simu- power grid are grouped, converted to industrial protocols, and
lation platform and that should be a significant advantage sent to cyber simulation machines. However, the industrial
EA

of a data-oriented design. The functionalities of ECS-Grid protocols are designed for production environments which
are defined by a combination of plugins, which is similar should consider security issues, standard requirements, guide-
to many popular ECS frameworks such as Flecs and Bevy. lines of power system operations, and the limits of existing
Plugins can have various inner structures and definitions as industrial communication routes and devices. However, the
long as they provide a plain function with a declaration of void simulation environments should provide a more generic pro-
build(World &world); as the entry point. In this way, a plugin tocol to simulate various scenarios which cannot be covered
with functionalities in Fig. 3 can be loaded from a header-only by a single industrial protocol. Also, the simulator itself
library, a static library, and even a dynamic library loaded should provide exclusive remote control and management
R

at run-time. The implementations are quite straightforward functionalities for simulation-only purposes which are not
and an example C++ header of the ZeroMQ vIED plugin is considered by industrial protocols and IEDs.
included in Appendix A. Although some platforms [14] use Open Platform Commu-
As shown in Fig. 4, a simulation based on the ECS frame- nication (OPC) or CORBA (Common Object Request Bro-
work is composed of various plugins, which is flexible and ker Architecture) DIM (Distributed Information Management)
bloat-free. For example, although the solver and vIED plugins protocol to unify the protocols within the simulator, these tra-
are the same, the simulation for microgrids uses exclusive ditional OOP protocols are based on late-1990s standards and
microgrid systems such as renewable energy sources and stor- technologies which cannot meet the data-oriented demands of
age units along with the droop controllers, while the HVDC modern cyber-physical simulation. The OOP protocols often
simulation configuration only uses the MMC and Bergeron need a cumbersome object library to decode the messages
line model plugins. To test the vIED only, the physical plugins and many functionalities are fixed. Therefore, a data-oriented
are removed and replaced by dummy data sources. These protocol and a local simulation network are proposed for the
configurations are practically applied to produce the results vIEDs as a unified middleware interface to the outer systems.
presented in Section III and IV. The data-oriented protocol should be:

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF ALBERTA. Downloaded on February 13,2023 at 22:51:06 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2023.3244329

Cyber System Academic Researchers platform for developing future distributed multi-layer control
Simulated MGCC Python schemes and other cyber components as shown in Fig. 6.
Control Data Web
Dash APP EMS Database
ZMQ TCP PUB/SUB
Strategy Analysis Frontend Real-time communication performance is easy to achieve
C++ since simulation environments have much better computing
Modbus power, bandwidth, and reliability than field devices. Modern
SCADA
Industrial Users

Rust
ZeroMQ TCP
Rust CPUs have quite a large memory bandwidth that is larger
IEC-104 PUB/SUB Broker Cyber than high-end optical networks. For example, Intel® Core™
Real IED

Attacker
Virutal
Network
Physical
Network
X-Series Processors can achieve a bandwidth of 94GB/s with
Man-in-the-middle
Protocol attack 4-channel DDR4 2933Mhz memories [27], which is nearly
Converter
8 times faster than high-end 100Gbit/s Ethernet. The current
10/100Mbps industrial Ethernet bandwidth is no match to the
Power Grid

C++ C++
Simulation

Vitis® HLS Virtual ZeroMQ RTU

DSP
IEDs
vIED vIED
CPU’s internal bandwidth.
Controller VSC MMC BMS Group 1 Group n Although the customizable protocols are useful for simula-
tion environments, the industrial protocols cannot be ignored.

LY
Analog IO

Device-Level Physical Physical

Physical Simulation As shown in Fig. 5, protocol converters are the solutions,
Transient System 1 System n
FPGA Embedded Hardware
(Xilinx® VCU-118) ( Nvidia® Jetson Arm v8 CPU) PC/HPC (8 cores x86) which map the MessagePack protocol to a specific protocol
such as IEC 60870-5-104. Protocol converters are common
Fig. 6: The distributed network architecture of the ECS-Grid in real-world power automation systems and many IEDs can
platform. do protocol conversions internally according to the firmware
or hardware configurations. Thanks to the multi-transportation
ZeroMQ, the protocol conversion can have multiple choices to

N
(1) generic: it should be able to represent different messag-
meet users’ demands and interoperability can be ensured by
ing patterns used in microgrids and not be restricted to specific
customizing MessagePack messages. If users want industrial
transport media;
protocols directly built into the IED, they can follow the same
(2) high-performance: it should not add heavy overhead to
plugin development principles to integrate their protocols.
the simulation systems and can handle a large volume of data;
it should have distributed and concurrent features to make full
use of modern hardware;
O B. Comparsion of Various Middleware Protocols
Currently, there are three communication plugins avail-
able for vIEDs in ECS-Grid: ZeroMQ [21], eProsima Fast
(3) customizable: unlike industrial protocols where all are
defined by standards, the CPPS simulation should enable more DDS Real-Time Publish-Subscribe protocol (RTPS) [28], and
possibilities for research explorations of future power systems Eclipse Paho MQTT [29]. Fast DDS is the middleware used
by allowing users to customize the protocol. in Robot Operation System 2 (ROS2). The MQTT is used for
D
Internet-of-Thing (IoT) applications and partly in microgrid
applications with IoT devices. ZeroMQ is a widely used
A. MessagePack Format for User Applications message-oriented middleware. The latency test results of dif-
The ZeroMQ mainly abstracts the sockets for higher-level ferent protocols under the one-publisher-one-subscriber vIED
applications, the payloads being transported depends on the scenario are listed in Table I-V.
EA

user’s decision. The default vIED plugin uses a simple solution Table I shows the results from the Fast DDS RTPS protocol.
based on javaScript object notation (JSON) and MessagePack There is a spike in maximum latency when the message
is proposed for a generic and customizable application-layer number increases, which is normally due to unreliable User
simulation protocol. JSON is the first-class data format inside Datagram Protocol (UDP) transportation. In summary, RTPS’s
ECS-Grid for configurations and data exchanging shown in performance is high and stable, and it has advanced features
Fig. 5. It is faster and smaller than the current XML format which can be very useful for vIED applications. However,
used in industrial applications [26]. The JSON format is self- it requires many dependencies, and the provided advanced
describing, so there are no complex data models predefined features are not used in power systems. Moreover, it is not easy
R

by an Interface Definition Language (IDL) to decode the to use and the support documents should be greatly improved
messages. MessagePack is an efficient binary serialization compared to other solutions.
format and it can exchange JSON data cross multiple lan- Table II shows the results from Eclipse MQTT Paho clients.
guages more efficiently [22]. The receivers can easily decode The MQTT is not designed for microsecond-level latency, and
the messages to JSON objects like dictionaries in Python, it must have a broker, which is an Eclipse Mosquitto broker
ECMAScript, and Rust and handle them in their program [30]. The default configuration also enables message persis-
logic. The utilization of MessagePack can provide a faster tence on the broker server. Therefore, the latency is 1-100ms
serialization and deserialization speed without a pre-defined level which is good for most IoT applications but not good for
schema and reduce the size by more than 40% compared low-latency communications. However, the bandwidth reaches
to a plain-text JSON message. The example for the JSON- the top of all protocols when the published message number
like protocol and the protocol conversion. The MessagePack is 1000. In all, the MQTT solution can be useful for some IoT
design enables users to simulate specific scenarios and make scenarios since not all devices need microsecond-level latency.
custom virtual cyber services such as microgrid control center Table III- V shows the ZeroMQ vIED performance under
(MGCC) upon vIEDs, which can provide a very convenient different configurations. The in-memory inter-thread commu-

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF ALBERTA. Downloaded on February 13,2023 at 22:51:06 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2023.3244329

TABLE I: vIED Latency and Bandwidth Using eProsima Fast TABLE III: vIED Point-to-Point Latency and Bandwidth Us-
DDS (RTPS) ing ZeroMQ (Inter-Thread)
Messages Max (µs) Min (µs) Mean (µs) Pub Bandwidth (Mbit/s) Messages Max (µs) Min (µs) Mean (µs) Pub Bandwidth (Mbit/s)
100 98.70 14.42 16.99 34.28 100 203.21 6.63 25.23 18.28
1000 211.44 12.77 15.80 36.96 1000 210.52 6.69 10.26 47.91
10000 293.81 13.11 14.77 38.15 10000 191.43 6.37 7.88 60.40
1000000 7483.24 11.88 15.06 38.63 1000000 230.29 6.05 7.08 67.03

TABLE II: vIED Latency and Bandwidth Using Eclipse Paho

TABLE IV: vIED Point-to-Point Latency and Bandwidth Us-
MQTT
ing ZeroMQ (TCP)
Messages Max (ms) Min (ms) Mean (ms) Pub Bandwidth (Mbit/s)
Messages Max (µs) Min (µs) Mean (µs) Bandwidth (Mbit/s)
100 99.81 95.75 96.86 22.21
1000 99.87 75.60 92.67 71.41 100 366.13 20.76 84.35 17.50
10000 140.22 27.90 80.04 18.34 1000 396.42 16.44 24.87 48.28

LY
1000000 145.71 0.92 78.44 15.20 10000 347.94 15.49 20.70 60.59
1000000 332.94 15.04 18.48 62.83

nication reaches the lowest latency of 6µs, which is quite TABLE V: vIED Latency and Bandwidth Using ZeroMQ
enough for IEC-104 applications since the protocol time- (TCP with Broker)
stamp has a resolution of milliseconds. The ZeroMQ point-
to-point TCP pub/sub latency is around 20µs and the TCP Messages Max (µs) Min (µs) Mean (µs) Pub Bandwidth (Mbit/s)

N
pub/sub with a broker test is just a doubled point-to-point TCP 100 343.06 34.46 64.24 54.05
latency. The single-thread publisher’s bandwidth is high and 1000 266.82 34.81 52.09 61.29
10000 357.40 33.43 43.15 55.50
stable without throughput optimization and well suited for a 1000000 399.11 26.02 40.98 61.54
real-world IED which mainly has a 100Mbit/s Ethernet port.
Besides, ZeroMQ is quite flexible and easy to use in every
major programming language. The only problem is it requires
more user decisions to establish an in-production network,
O RLC components for convenience. Three modified CIGRE
however, it is an advantage for a simulator that can give users 15-Bus microgrids are connected to the three-terminal high-
the maximum freedom to establish customized scenarios. voltage direct current (HVDC) system. The ±50kV HVDC
For the generic and high-performance goals, ZeroMQ is rec- system consists of three 51-level 50MW modular multilevel
ommended to be the message bus between vIEDs. ZeroMQ is converters (MMCs) and MMC-1 is designated to control the
D
a high-performance asynchronous messaging library, aimed at DC voltages. The other 2 MMCs are set to drain 1MW from
use in distributed or concurrent applications. ZeroMQ supports the HVDC system. The MMCs are modeled by detailed-
scalability protocols (pub/sub, request/reply, client/server, and switching models which means voltage balancing of submod-
others) over a variety of transports (TCP, in-process, inter- ules is needed. In this paper, the nearest-level modulation
EA

process, multicast, WebSocket, and more). This keeps the code is used for MMC’s lower-level controller. The upper-level
clear, modular, and scalabe from very low-latency in-memory controllers for MMCs are similar to VSCs in microgrids which
communication to the large-scale cloud computing scenario. control the DC voltages or the power generations.
Fig. 8 shows two scenarios for the test results. Droop_0
IV. C ASE S TUDY, R ESULTS AND P ERFORMANCE IED in MG-1 is the main research target. Scenario 1 is used
Fig. 7 shows the microgrid cluster connected by a multi- to evaluate the islanded microgrid clusters and produce the
terminal DC system, which forms a 711-node power system steady state for Scenario 2. Scenario 2 conducts a man-in-
with 60 vIEDs to evaluate the proposed simulation platform’s the-middle cyber attack to manipulate secondary frequency
R

functionalities and performance. The microgrid is a 15-Bus regulation command and cause catastrophe across the cluster.
power distribution system derived from the CIGRE report [31] Scenarios 2 is similar to the real-world industroyer cyber
and pandapower [32] case files; loads are reduced to 10% and attack in 2016 and industroyer2 attack in 2022 on Ukraine
a 5MW Li-ion battery storage is added to Bus-1 to ensure power grids [33], which hijacked supervisory control and data
the ability of islanded operation. The microgrid has 16 VSC acquisition (SCADA) systems and sent dangerous commands
stations and they are all modeled by the average-value model to IEC-104 RTUs and IEDs.
to reduce the complexities of control and simulation. Each Fig. 9 shows the setup of the real-time hardware platform
VSC station has a VSC controller and a vIED acting as a introduced in Fig. 6. The three NVIDIA® Jetson AGX Xavier
remote terminal unit (RTU) to the VSC station. Each battery embedded computers with real-time Linux installed are used to
storage has an extra vIED to control the battery charging. The simulate physical microgrids, the corresponding MMC station,
distributed power sources are controlled as PQ nodes which and vIEDs. The Xilinx® VCU118 board is used to handle fast
have fixed power generations, while the storage stations are signal IO to support hardware-in-the-loop functions. The PC
controlled by a droop controller to auto-balance the system server runs cyber services such as virtual MGCC, IEC 60870-
and provide a stable frequency. The loads are modeled by fixed 5-104 clients, and cyber simulation tools.

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF ALBERTA. Downloaded on February 13,2023 at 22:51:06 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2023.3244329

External Grid (a) Scenario 1: Islanding (b) Scenario 2: Man-in-The-Middle Attack

2) cyber Attack at t=147s 1) Disconnected at t=15s
20kV MG1_Droop_0 Primary Frequency Regulation Secondary Frequency Regulation
hacker AC +-50kV
MMC1 MMC2 F (Hz) F (Hz)
MGCC DC MGCC Islanded t>15s Islanded At t=147s the operator
MG1 MG2 set Pref=0.8
1 100km 2 Expected frequency , t>15s
But hijacked
Multi-level 50.02 Initial t = 0s 50.02 by hacker!
control system 100km 50 0.02 50
100km Kp =0.1 0.2 Pref = 1.0 Kp =0.1
Power System 0.8 1.0 P (p.u.) 0.8 1.0 P (p.u.)
HVDC MGCC
Control Center Control MG3
3
MMC3 {
4 Wrong { Hijack and repack cmd
Type:”Set”,
reaction 3 Type:”Set”, data 2 1 “Pref”: 0.8
External 100k
VSC 1 “Fref”: 0.5
Eat this! } MGCC 9 What is
} happening!!!
V RX TX
1 Hacker
20kV RX
Additional TX { {
19 IEDs Storage Type:”REP”, Type:”REP”,
8
Virtual “Fref”: 0.5, 7 Virtual Op System is
3 Droop VSC 2 13 IED 5 “Pref”: 0.8,
Station
Fabricate false doomed

LY
Status: OK 6 Response Status: OK
3 Li-ion Batteries 5MW }
20kW Droop_0 }
13 PQ VSC BAT_0
3
Fig. 8: Configurations and expected results of test scenarios:
PQControl
20kW
4
30kW 14 (a) Scenario 1: islanding operation; (b) Scenario 2: Man-in-
the-middle cyber attack.
5 10 9
BAT_2 10kW
310kW
Droop_2 15 MGCC &Operator
S3
CHP
IEC104 Client

N
Diesel
212kW S1
600kW 30kW 12 Fuel
Cell PC Server
40kW 30kW 8
S2
7

Fuel
Cell
6 11

Fuel
30kW
VSCDroop_1
O1500kW

VCU118
33kW Cell
BAT_1
Modified CIGRE FPGA
14kW 200kW 15-Bus Microgrid
PCIE Board
Fig. 7: System topology and the detailed configuration of the
D
711-node microgrid cluster. DAC
MG-1 MG-2 MG-3
NVIDIA Jetson Cluster
A. Results and Performance
EA

Fig. 9: Distributed hardware setup of proposed real-time ECS-

The results of Scenario 1 are shown in Fig. 10 (a)-(f).
Grid platform.
Fig. 10 (a)-(c) are the frequency, real power, and bus voltage
waveforms of VSCDroop_0 storage station in each microgrid,
respectively; when t < 15s, the frequency in MG-1 FDroop1 is
voltage fluctuation in MG-1. The results are verified against
50Hz since the ideal three-phase AC source is attached to MG-
theoretical analysis and commercial PSCAD/EMTDC® ; all
1 Bus-0, while the other microgrids have different frequencies;
data are measured from vIEDs with the interval of 100ms
at t = 15s the ideal source is removed so that a large deviation
and recorded by remote self-made Supervisory Control and
occurred to FDroop1 , while MG-2 and MG-3 have no obvious
R

Data Acquisition (SCADA) system.

changes because the DC system can allow asynchronous
frequencies; the steady-state value of FDroop1 is 50.02 and The results of Scenario 2 are shown in Fig. 10 (g)-(i).
PDroop1 is 4.0MW (0.8 p.u.), which meets the droop control
equation F −Fref = Kp (Pref −P ) = 0.02. This is considered TABLE VI: FTRT Performance with Various Communication
a primary frequency regulation process. Fig. 10 (d)-(f) are Intervals
the DC voltage, real power, and reactive waveforms of MMC
stations, respectively; the rated DC voltage is 100kV so that
Interval Tcom tcps (s) rf trt Efficiency
all MMC stations maintained a nominal voltage according to
2 100µs 17.88 2.24 0.44
Fig. 10 (d), while MMC-1 was impacted by the disconnecting 20 1ms 11.02 3.63 0.71
event at t = 15s; Fig. 10 (e) shows the real power balancing 200 10ms 10.24 3.91 0.77
between three MMCs, where MMC-1 provides 2MW and 1000 50ms 8.08 4.95 0.97
2000 100ms 8.07 4.96 0.98
other MMCs drain the planned 1MW from the DC system;
the reactive power was set to 0MVar, however, it seems to System Scale: 711 nodes, 60 vIEDs in total; Simulation duration: 40s, ∆t =
have large deviation at MMC-1 which may cause the larger 50µs, tphy = 7.87s.

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF ALBERTA. Downloaded on February 13,2023 at 22:51:06 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2023.3244329

Scenario 1: Droop_0 Waveform Scenario 1: MMCs Waveform Scenario 2: Droop_0 Waveform

F (Hz) V (kV) F (Hz)
100.8

100.6 MIM attack

MM DC Voltages
MG frequency

MG frequency
islanded
t=15s MG-2 ,MG-3 100.4
50.06Hz t=15s
100.2 Secondary
Primary 100kV frequency
frequency MG-1 100.0 t=147s
regulation regulation
50.02Hz t=15s
50.02 99.8
t(s) 99.6 t(s)
t(s)
(d) (g)
(a) V (p.u.)
P (MW) P (MW)

islanded 4.1MW
Storage Power

MG Voltages
MM DC Power

LY
islanded
t=15s
1.8MW t=147s

t(s) t(s)
t(s)
(e) (h)
(b)

Bus Voltages From Fault Recoder

V (p.u.) Q (MVar) V (kV)
30 t=147.99s
MMC Reactive Power

MG Nodal Voltages (kV)

20

N
Bus Voltages

10
islanded
0
islanded
-10

-20

(c)
t(s)
O (f)
t(s)

(i)
t(s)

Fig. 10: Simulation results: Scenario 1: islanding operation:(a) Frequency of three Droop_0 stations in each microgrid . (b)
Real power of each Droop_0 station. (c) Bus Voltages of each Droop_0 station. (d) MMC DC voltages during the islanding
operation. (e) Real power of each MMC station. (f) Reactive Power of each MMC station. Scenario 2: simulated cyber
attack: (g) Frequency comparison between normal operation and cyber attack situation of Droop_0 in MG-1. (h) Droop_0
D
Bus voltages comparison. (i) Droop_0 EMT bus voltage waveforms captured by virtual fault recorder.

the dashed line indicates the waveforms under cyber attack stability. The simulation of Scenario 2 shows the catastrophic
EA

while the solid line indicates the normal reactions; since the consequence of cyber attacks in a vulnerable power cyber
frequency of MG-1 was 50.02Hz after islanded, the operator network.
sent a command at t = 147s to reset Pref to 0.8 since the Besides the real-time Jetson platform, Table VI shows
current real power is 0.8 p.u., which is a secondary frequency the performance of the proposed cyber-physical simulation
regulation process to restore rated operation point. Fig. 10 platform on an x86 machine (Intel® CoreTM i7 10700k
(g) shows the frequency of VSCDroop_0 in MG-1; under [email protected], 32GB DDR4 3000MHz, Ubuntu 20.04,
cyber attack situation, the frequency regulation command was GCC 11.1). The pure physical parallel simulation consumes
intercepted and replaced to Fref = 0.5p.u., which generates 7.87s which is 5.08 times faster than real-time, and all
R

very drastic deviations in all measurements from IEDs after cyber-physical simulations also achieved faster-than-real-time
t = 148s such as the voltages in Fig. 10 (h). However, (FTRT) performance even with 100µs ticking interval. For the
these drastic deviations are measured from PLLs and may millisecond-level communication intervals, the cyber-physical
not reflect the real situation in the physical systems under co-simulation can achieve high efficiency and the overhead
faulty conditions and that is why real-world power systems is almost deflectable. The overhead can be further reduced
also have digital fault recorders to record the EMT waveforms with more concurrency for socket polling, data encoding, and
when faults occurred. Fig. 10 (i) shows the EMT voltage decoding since the communication systems currently execute
waveforms of Bus-1 captured by virtual digital fault recorders, in series inside the main simulation loop. The FTRT function-
which are triggered by the fault detection mechanism in the ality can enable predictive and preventative control actions in
proposed simulation platform. The EMT waveforms revealed energy control centers.
the physical details that happened after t = 147s; the system
started to react about 1s later than receiving the hacked V. C ONCLUSION
message, and the drastic deviations in measurements may
ECS-Grid is a novel data-oriented cyber-physical simulation
be caused by the high-frequency oscillation which can affect
platform for microgrids under the ECS framework proposed

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF ALBERTA. Downloaded on February 13,2023 at 22:51:06 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2023.3244329

10

to model the IEDs in a power system with flexible data 61 world.add_stage_after(PostUpdate,

62 vIEDStage::Communicate);
components and extensible plugin architecture. Furthermore, 63

a modern JSON-like MessagePack-based protocol is proposed 64 world.add_system_to_stage(stage->name(), sender);

65 world.add_system_to_stage(stage->name(), receiver);
for the vIEDs and is capable of completing various tasks 66 }
needed for cyber-physical transient simulation. The results 67 };
68 } // namespace zmq
from the scenarios in the microgrid cluster study case show 69 } // namespace vIED
the accurate system behaviors and real-time or FTRT per-
formance of ECS-Grid. The IED systems and components
R EFERENCES
can be extended to cyber-physical power dynamic or steady-
state simulations thanks to the data-oriented design. The data- [1] H. J. Schellnhuber, S. Rahmstorf, and R. Winkelmann, “Why the right
climate target was agreed in paris,” Nature Climate Change, vol. 6, no. 7,
oriented ECS-Grid can inspire the renovation of industrial pp. 649–653, 2016.
software tools and boost further research of the future CPPS. [2] S. Bolognani, R. Carli, G. Cavraro et al., “On the need for communi-
cation for voltage regulation of power distribution grids,” IEEE Trans.
Control. Netw. Syst., vol. 6, no. 3, pp. 1111–1123, Sept. 2019.

LY
A PPENDIX [3] N. Patari, V. Venkataramanan, A. Srivastava et al., “Distributed opti-
A mization in distribution systems: Use cases, limitations, and research
needs,” IEEE Trans. Power Syst., p. [Early Access], 2021.
ZeroMQ C++ Plugin Header for ECS-Grid [4] D. K. Molzahn, F. Dörfler, H. Sandberg et al., “A survey of distributed
optimization and control algorithms for electric power systems,” IEEE
1 #pragma once
2 // details of components
Trans. Smart Grid, vol. 8, no. 6, pp. 2941–2962, Nov. 2017.
3 #include "components.hpp" [5] D. Gutierrez-Rojas, P. H. J. Nardelli, G. Mendes et al., “Review of
4 #include "entt/entity/fwd.hpp" the state of the art on adaptive protection for microgrids based on
5 // details of systems communications,” IEEE Trans. Ind. Informat., vol. 17, no. 3, pp. 1539–
#include "vIED/zmq/IEDSystems.hpp" 1552, 2021.

N
6
7 [6] I. Ali and S. M. Suhail Hussain, “Communication design for energy
8 // ECS container management automation in microgrid,” IEEE Trans. Smart Grid, vol. 9,
9 #include "world/world.hpp"
no. 3, pp. 2055–2064, May 2018.
10
11 /// This is your module namespace [7] R. V. Yohanandhan, R. M. Elavarasan, P. Manoharan et al., “Cyber-
12 namespace vIED physical power system (CPPS): A review on modeling, simulation, and
13
14
15
16
{
namespace zmq
{
/// declare your components
O analysis with cyber security applications,” IEEE Access, vol. 8, pp.
151 019–151 064, 2020.
[8] K. Mets, J. A. Ojea, and C. Develder, “Combining power and communi-
cation network simulation for cost-effective smart grid analysis,” IEEE
17 namespace Components Commun. Surveys Tuts., vol. 16, no. 3, pp. 1771–1796, 2014.
18 {
19 struct ZMQIEDContext;
[9] IEEE Task Force on Interfacing Techniques for Simulation Tools, S. C.
20 struct IEDMessageBuffer; Müller, H. Georg et al., “Interfacing power system and ICT simulators:
21 struct IED; Challenges, state-of-the-art, and case studies,” IEEE Trans. Smart Grid,
vol. 9, no. 1, pp. 14–24, Jan. 2018.
D
22 struct IEDSocket;
23 [10] A. A. Jahromi, A. Kemmeugne, D. Kundur et al., “Cyber-physical
24 /// a set of IED components attacks targeting communication-assisted protection schemes,” IEEE
25 /// can be grouped into a bundle Trans. Power Syst., vol. 35, no. 1, pp. 440–450, Jan. 2020.
26 using IEDBundle = basic::Bundle<IED, [11] K. Hopkinson, X. Wang, R. Giovanini et al., “EPOCHS: a platform
27 IEDMessageBuffer, IEDSocket, Name>;
28
for agent-based electric power and communication simulation built from
EA

29 } // namespace Components commercial off-the-shelf components,” IEEE Trans. Power Syst., vol. 21,
30 no. 2, pp. 548–558, May 2006.
31 /// declare your customized stage [12] H. Lin, S. S. Veda, S. S. Shukla et al., “GECO: Global event-driven
32 enum class vIEDStage co-simulation framework for interconnected power system and commu-
33 { nication network,” IEEE Trans. Smart Grid, vol. 3, no. 3, pp. 1444–1456,
34 Communicate 2012.
35
[13] H. Georg, S. C. Müller, C. Rehtanz et al., “Analyzing cyber-physical
36 };
37
energy systems:the INSPIRE cosimulation of power and ICT systems
38 /// declare your systems using hla,” IEEE Trans. Ind. Informat., vol. 10, no. 4, pp. 2364–2373,
39 namespace vIEDSystem 2014.
40 { [14] M. H. Cintuglu, O. A. Mohammed, K. Akkaya et al., “A survey on
R

41 /// this initialize context and sockets smart grid cyber-physical system testbeds,” IEEE Commun. Surveys
42 void initialize(entt::registry& reg); Tuts., vol. 19, no. 1, pp. 446–464, Jan. 2017.
43 /// this send messages using tx socket [15] E. Moradi-Pari, N. Nasiriani, Y. P. Fallah et al., “Design, modeling, and
44 /// including serialization
simulation of on-demand communication mechanisms for cyber-physical
45 void sender(entt::registry& reg);
46 /// this receive messages using rx socket energy systems,” IEEE Trans. Ind. Informat., vol. 10, no. 4, pp. 2330–
47 /// including deserialization 2339, 2014.
48 void receiver(entt::registry& reg); [16] M. Thornton, M. Motalleb, H. Smidt et al., “Internet-of-things hardware-
49 }; // namespace vIEDSystem in-the-loop simulation architecture for providing frequency regulation
50 with demand response,” IEEE Trans. Ind. Informat., vol. 14, no. 11, pp.
51 struct Plugin 5020–5028, 2018.
52 { [17] B. Chen, K. L. Butler-Purry, A. Goulart et al., “Implementing a real-time
53 static void build(ecs::World& world)
cyber-physical system test bed in RTDS and OPNET,” in 2014 North
54 {
55 // register your systems to the simulator American Power Symposium (NAPS), November 2014, pp. 1–6.
56 using namespace vIEDSystem; [18] M. Ni, H. Tong, L. Zhao et al., “A flexible hardware-in-the-loop testbed
57 for cyber physical power system simulation,” IET Cyber-Phys. Syst.
58 world.add_startup_system_to_stage(PostStartUp, Theory Appl., vol. 4, July 2019.
59 initialize); [19] H. Tong, M. Ni, L. Zhao et al., “A flexible hardware-in-the-loop testbed
60 auto&& stage = for cyber physical power system simulation,” IET Cyber-Phys. Syst.,
Theory Appl., vol. 4, 12 2019.

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF ALBERTA. Downloaded on February 13,2023 at 22:51:06 UTC from IEEE Xplore. Restrictions apply.
 This article has been accepted for publication in IEEE Transactions on Industrial Informatics. This is the author's version which has not been fully edited and
content may change prior to final publication. Citation information: DOI 10.1109/TII.2023.3244329

11

[20] G. Cao, W. Gu, C. Gu et al., “Real-time cyber physical system co-

simulation testbed for microgrids control,” IET Cyber-Phys. Syst. Theory
Appl., vol. 4, February 2019.
[21] “An open-source universal messaging library,” The ZeroMQ authors,
July 22, 2022. [Online]. Available: https://zeromq.org/
[22] “messagepack,” GitHub, July 22, 2022. [Online]. Available: https:
//github.com/msgpack/msgpack
[23] “Entt: Gaming meets modern c++,” GitHub, July 22, 2022. [Online].
Available: https://github.com/skypjack/entt
[24] “Data-oriented technology stack (DOTS),” Unity, July 22, 2022.
[Online]. Available: https://unity.com/dots
[25] I. Jahn, F. Hohn, G. Chaffey et al., “An open-source protection ied for
research and education in multiterminal hvdc grids,” IEEE Trans. Power
Syst., vol. 35, no. 4, pp. 2949–2958, 2020.
[26] “FlatBuffer serialization C++ benchmarks,” Google FlatBuffer, Nov. 12,
2022. [Online]. Available: https://google.github.io/flatbuffers/flatbuffers_
benchmarks.html
[27] “Theoretical maximum memory bandwidth for intel® core™

LY
x-series processors,” Intel, July 12, 2021. [Online].
Available: https://www.intel.com/content/www/us/en/support/articles/
000056722/processors/intel-core-processors.html
[28] “eProsima Fast DDS,” eProsima, Nov. 12, 2022. [Online]. Available:
https://www.eprosima.com/index.php/products-all/eprosima-fast-dds
[29] “Paho is an IoT project,” Eclipse Foundation, Nov 22, 2022. [Online].
Available: https://www.eclipse.org/paho/
[30] “Eclipse Mosquitto™,” Eclipse Foundation, Nov. 12, 2022. [Online].
Available: https://mosquitto.org/

N
[31] C. Marnay, C. Abbey, G. Joos et al., “Microgrids 1 engineering,
economics, & experience,” Electra; CIGRE, 2015.
[32] L. Thurner, A. Scheidler, F. Schäfer et al., “pandapower - an open
source Python tool for convenient modeling, analysis and optimization
of electric power systems,” IEEE Trans. Power Syst., vol. 33, no. 6, pp.
6510–6521, Nov. 2018.
[33] “Industroyer2: Industroyer reloaded,” welivesecurity, Nov. 22,
O
2022. [Online]. Available: https://www.welivesecurity.com/2022/04/
12/industroyer2-industroyer-reloaded/

Tianshi Cheng (S’19) received the B.Eng. degree in

electrical engineering and automation from South-
east University, China, in 2017. From 2017 to 2018,
he was a substation automation engineer of NARI
D
Group Corporation (State Grid Electric Power Re-
search Institute), China. He is currently pursuing the
Ph.D. degree in electrical and computer engineering
with the University of Alberta, Canada. His research
interests include electromagnetic transient simula-
tion, transient stability analysis, real-time simulation,
EA

parallel processing, microgrid and power electronics.

Tong Duan (S’18-M’22) received the B.Eng. degree

in electronic engineering from Tsinghua University,
Beijing, China, in 2013, and the Ph.D. degree in
energy systems from University of Alberta, Ed-
monton, Alberta, Canada, in 2021. He is currently
an assistant professor at National Digital Switch-
ing System Engineering & Technological Research
Center, Zhengzhou, China. His research interests
include smart grid, industrial network, power system
R

simulation and communication networking.

Venkata Dinavahi (Fellow, IEEE) received the

B.Eng. degree in electrical engineering from Visves-
varaya National Institute of Technology (VNIT),
Nagpur, India, in 1993, the M.Tech. degree in
electrical engineering from the Indian Institute of
Technology (IIT) Kanpur, India, in 1996, and the
Ph.D. degree in electrical and computer engineering
from the University of Toronto, Ontario, Canada,
in 2000. He is currently a Professor with the De-
partment of Electrical and Computer Engineering,
University of Alberta, Edmonton, Alberta, Canada.
He is a Fellow of the Engineering Institute of Canada. His research interests
include real-time simulation of power systems and power electronic systems,
electromagnetic transients, device-level modeling, large-scale systems, and
parallel and distributed computing.

© 2023 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See https://www.ieee.org/publications/rights/index.html for more information.
Authorized licensed use limited to: UNIVERSITY OF ALBERTA. Downloaded on February 13,2023 at 22:51:06 UTC from IEEE Xplore. Restrictions apply.