See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/332057582

PV Cyber Security Research - Final Report

Technical Report · March 2019

DOI: 10.13140/RG.2.2.27993.98405

CITATIONS READS
0 408

1 author:

Jay Johnson
Sandia National Laboratories
139 PUBLICATIONS   1,204 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Smart Grid International Research Facility Network (SIRFN) View project

PV Arc-Fault and Ground Fault Detection and Mitigation View project

All content following this page was uploaded by Jay Johnson on 28 March 2019.

The user has requested enhancement of the downloaded file.

 PV Cyber Security Research
Sandia National Laboratories

Final Technical Report

Project Title: PV Cyber Security Research

Project Period: 12/1/16 – 11/30/18

Submission Date: 1/15/19

Recipient: Sandia National Laboratories

Address: Sandia National Laboratories

1515 Eubank SE
Albuquerque, NM 87123

Website (if available) www.sandia.gov

Project Team: Sandia National Laboratories

Principal Investigator: Jay Johnson

Principal Member of Technical Staff
Phone: 505-284-9586
Email: [email protected]

Business Contact: Adriana Manriquez

Phone: (505) 284-4080
Email: [email protected]

Technology Manager: M. Kemal Celik, Ph.D.

Project Officer: Thomas Rueckert

Sandia National Laboratories is a multimission laboratory managed and operated by National

Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell
International Inc., for the U.S. Department of Energy's National Nuclear Security Administration
under contract DE-NA0003525.

Page 1 of 24
 PV Cyber Security Research
Sandia National Laboratories

Executive Summary
Extensive deployment of interoperable distributed energy resources (DER) on power systems is
increasing the power system cyber security attack surface. National and jurisdictional
interconnection standards require DER to include a range of autonomous and commanded grid
support functions which can drastically influence power quality, voltage, and bulk system
frequency. This project was split into two phases. The first provided a survey and roadmap of the
cybersecurity for the solar industry. The second investigated multiple PV cybersecurity research
and development (R&D) concepts identified in the first phase.

In the first year, the team created a roadmap for improving cybersecurity for distributed solar
energy resources. This roadmap was intended to provide direction for the nation over the next five
years and focused on the intersection of industry and government and recommends activities in
four related areas: stakeholder engagement, cyber security research and development, standards
development, and industry best practices. At the same time, the team produced a primer for DER
vendors, aggregators, and grid operators to establish a common taxonomy and describe basic
principles of cyber security, encryption, communication protocols, DER cyber security
recommendations and requirements, and device-, aggregator-, and utility-level security best
practices to ensure data confidentiality, integrity, and availability. This material was motivated by
the need to assist the broader PV industry with cybersecurity resilience and describe the state-of-
the-art for securing DER communications. Lastly, an adversary-based assessment of multiple PV
devices was completed at the Distributed Energy Technologies Laboratory at Sandia National
Laboratories to determine the status of industry cybersecurity practices. The team found multiple
deficiencies in the security features of the assessed devices.

In the second year, a set of recommendations was created for DER communication protocols—
especially with respect to the state-of-the-art requirements in IEEE 2030.5. Additionally, several
cybersecurity R&D technologies related to communications-enabled photovoltaic systems were
studied to harden DER communication networks. Specifically, the team investigated (a) using
software defined networking to create a moving target defense system for DER communications,
and (b) engineering controls that prevent misprogramming or adversary action on DER
devices/networks by disallowing setpoints that will generate unstable power system operations.

Page 2 of 24
 PV Cyber Security Research
Sandia National Laboratories

Contents
Executive Summary ........................................................................................................................ 2
Background ..................................................................................................................................... 4
Project Objectives ........................................................................................................................... 6
Project Results and Discussion ....................................................................................................... 6
PV Cybersecurity Roadmap ........................................................................................................ 7
DER Cybersecurity Primer ......................................................................................................... 8
Adversary-Based Assessments ................................................................................................. 10
Trust Recommendations ........................................................................................................... 11
Moving Target Defense ............................................................................................................ 13
Engineering Controls ................................................................................................................ 18
Significant Accomplishments and Conclusions............................................................................ 20
Inventions, Patents, Publications, and Other Results .................................................................... 21
Path Forward ................................................................................................................................. 22
References ..................................................................................................................................... 23

Page 3 of 24
 PV Cyber Security Research
Sandia National Laboratories

Background
The large-scale deployment of distributed energy resources (DER), principally PV, energy storage
systems and demand response, is transforming today’s power grid. Increasingly, communications-
enabled functionality is being incorporated into DER to enable price response and configurable
grid-support functionality in coordination with markets, utility control systems, DER aggregators.
Communications also enables DER owners, utility system operators, and equipment manufacturers
to interact with and possibly reconfigure DER devices. As significant centralized generation
capacity is displaced, DER will be required to provide critical reliability services such as frequency
and voltage regulation. Because many of these interactions will occur through open
communication channels including the Internet, where additional cyber vulnerabilities come into
play, there is a concern about DER cybersecurity and information protection. A key question is
the extent to which these increase vulnerabilities can compromise the ability of DER to provide
critical reliability services and system response and recovery in case threat events occur.

Interconnection of power electronics-interfaced DER has been increasing worldwide for the last
two decades due to renewable portfolio standards, environmental standards, and customer
preference [1]. A sizable portion of renewable energy installations are residential-scale systems
which have traditionally been designed to avoid participation with grid operations and to
disconnect from the grid when there are voltage or frequency disturbances, per the U.S.
interconnection standard IEEE 1547-2003 [2]. However, renewable energy penetrations have
reached a point where challenges with voltage regulation, protection, and bulk system control are
emerging because DER are displacing thermal plants with inertialess, non-dispatchable, variable
sources of generation [3-5]. To mitigate these challenges, utilities, independent system operators
(ISOs), distribution system operators (DSOs), and transmission system operators (TSOs) are
pushing for updated DER interconnection standards which include DER grid-support functions.
For instance, the California Public Utilities Commission (CPUC) updated Electric Rule 21 in early
2015 [6] to include seven autonomous grid-support functions and Hawaii Rule 14 was updated to
include new grid-support functionality for DER [7]. Similar changes to interconnection standards
and grid codes have been occurring in Italy, Germany, Austria, Australia and New Zealand, and
other places around the world [8-10].

In most of these regions, grid-support functions are programmed before installation and operate
with fixed settings for their lifetimes. However, CPUC Electric Rule 21 [11] and a IEEE 1547-
2018 [12] require DER communications. Interoperability allows commanded functions to be
employed and autonomous function parameters can be changed. The cybersecurity risk to the
power system increases significantly when extending communications to DER devices because
the utility supervisory control and data acquisition (SCADA) controls are now issued over public
internet channels as opposed to using the traditional dedicated telecommunications lines. Some
larger DER plant controllers connect to grid operators through fiber-optic cables, copper telephone
lines, cellular modems, and microwave or other radio relays [13], so there are several access points
to these systems.

Page 4 of 24
 PV Cyber Security Research
Sandia National Laboratories

Many utility-scale systems use DNP3, IEC, or RTU communications to the PV plant [14-16].
Residential and commercial DER manufacturers typically communicate to their equipment using
a gateway device which in turn communicates Modbus to the power equipment. The Modbus maps
are often proprietary but, more recently, there has been heavier adoption of SunSpec Alliance de
facto standard information models [17]. The Modbus protocol does not provide confidentiality
(e.g., native encryption), authentication, or authorization capabilities. For this reason, it is difficult
to ensure data integrity between aggregators or utilities and the DERs. Other protocols—such as
IEEE 2030.5 Smart Energy Profile 2.0 (SEP2) [18], IEEE 1815 (DNP3) [19], or OpenADR 2.0
[20]—include encryption capabilities, but DER vendors designed their equipment with limited
computing power to minimize costs, so these devices may not be capable of encryption.

However, inverter manufacturers are regularly using these communications to modify DER
operations. Enphase Energy made headlines worldwide when it remotely updated 800,000
Inverters (154 MW of capacity) on the Hawaiian Islands of O’ahu, Hawai’i, Moloka’i and Lana’i
in 2015 [21], [22]. While many praised the achievement as a breakthrough for reducing the costs
of retrofitting power systems, others warned of the cyber security implications. If one company
could remotely update the settings of 100s of megawatts of power equipment, anyone with access
to that control network would be able to make malicious changes to those devices as well. Certain
settings could damage equipment, cause distribution overvoltages, or initiate a blackout if the
contingency reserve was not sufficient. For example, on the island of O’ahu, there will be an
estimated 400 MW of installed PV capacity in 2017 but only 180 MW of contingency reserves
[23]. Therefore, disconnecting or curtailing a sizable portion of the solar generation on a sunny
day could cause a blackout because backup power is sized for N-1 contingencies, not a cyber-
attack.

As DER enter the Internet of Things (IoT) environment, there have been some early cyber security
warning signs. Upon gaining access to a VPN tunnel established for a DC optimizer data manager,
Fred Bret-Mounet discovered 1,000 other PV devices on the same subnet. Had he desired, he could
have also remotely disconnected these devices [24]-[25]. A large Distributed Denial-of-Service
(DDoS) attack using a botnet of IoT devices affected many websites including Amazon, Twitter,
and Netflix in Oct. 2016 [26].

Consequently, it is imperative to secure DER communications to provide grid reliability and

resiliency. Many DER devices communicate via unsecured serial protocols (e.g., Modbus), so
there has been an effort to develop translators that integrate with DER to take encrypted protocols
such as IEEE 2030.5 and only unencrypt the communications within the DER housing [27]. This
approach mitigates the security risk because the adversary needs physical access to the devices to
subvert them using cleartext messages. As part of a California Solar Initiative grant, Sandia
National Laboratories led a team to generate cyber security recommendations for PV Inverters
using SunSpec Modbus removable communications modules [28]. The team presented several
threats, vulnerabilities and high-level recommendations for residential inverter-based DER
systems covering physical security, access control, integrity, confidentiality, encryption, and
policy.
Page 5 of 24
 PV Cyber Security Research
Sandia National Laboratories

Novel methods for detecting, mitigating, and recovering from cyber-attacks must also be
developed to counteract rapidly evolving threats and vulnerabilities. Techniques of identifying and
removing compromised/unauthorized DERs, segmenting DERs into resource pools to minimize
damage in the event of successful compromise and safeguarding the DER from mass compromise
are being developed by Sandia and many other research institutions.

Project Objectives
The objective of the project was to lay a foundation for the emerging R&D in solar cybersecurity,
assess the state-of-the-art security of DER, and conduct exploratory research into a few promising
cybersecurity technologies. In the first year, Sandia produced the “Roadmap for Photovoltaic
Cyber Security” which outlined a 5-year plan for cybersecurity activities in the private and public
sectors. Based on that report, three primary areas were identified for DOE and government
agencies to contribute. These areas were stakeholder engagement, standards development, and
cybersecurity R&D. The remainder of the project touched on each of these areas:
• Stakeholder Engagement: A “Cyber Security Primer for DER Vendors, Aggregators, and
Grid Operators” Sandia report was published to orient the broader community to the state-
of-the-art DER communications and associated security mechanisms.
• Standards Development: Adversary-based cybersecurity assessments of DER products
were conducted, which provided the basis for several of the certification tests developed in
the SunSpec/Sandia DER Cybersecurity Workgroup. This draft standard will be used as
input to a new UL certification testing standard for DER security features. Additionally,
the team produced a report offering future cybersecure options for PV trust and encryption
as input to communication protocol standards development processes.
• Cybersecurity R&D: This project included extensive discussion of R&D topic areas that
would improve the solar communities’ cyber posture. Ultimately, the team investigated
Moving Target Defense and DER protocol engineering control technologies as novel
security options for DER communications.
This three-pronged approach for improving PV cybersecurity is believed to be a good strategy and
appropriate role for DOE and the national laboratory complex to provide significant improvements
to the security of the nation’s power system.

Project Results and Discussion

The project results will be presented in the following sections:
1. PV cybersecurity roadmap
2. DER cybersecurity primer
3. Adversary-based DER assessments
4. Trust recommendations
5. Moving target defense
6. Cybersecure engineering controls

Page 6 of 24
 PV Cyber Security Research
Sandia National Laboratories

PV Cybersecurity Roadmap
The roadmap described the process for improving cyber security for PV systems over the next 5
years, represented in Figure 1. At the top of the figure, PV system cyber security is nested into the
larger context of the ICS/Energy cybersecurity landscape, whereby best practices from a range of
communities are being directed into two primary thrusts: stakeholder engagement and cyber
security research and development (R&D). Within the stakeholder engagement thrust, public-
private partnerships establish workshops, working groups, educational opportunities, and reach
out to other cyber security working groups. Within the R&D thrust, cyber security and solar
researchers design and evaluate new technologies for securing photovoltaic systems. Both the
stakeholder engagement and R&D efforts feed into the creation of cyber security requirements for
PV systems. With the adoption of these standards, industry will integrate new cyber security
features into PV communication networks and commercialize concepts from R&D activities.

Figure 1: Process for achieving cyber security of PV systems.

One of the major contributions of the roadmap was to identify and categorize research areas. As
shown in Figure 2, these R&D activities were compartmentalized into Identify and Protect,
Detect, and Respond and Recover research areas to thwart attacks. Many of these areas have
Page 7 of 24
 PV Cyber Security Research
Sandia National Laboratories

since seen additional investment through this project and others from DOE EERE and DOE
CESER. In this project, the moving target defense and engineering controls were studied.

Figure 2: Thwarting malicious cyber activities by strengthening defensive elements through R&D,
adapted from the National Science and Technology Council (NSTC) strategic plan. The roadmap R&D
topics are shown in the boxes at the bottom of the figure.

DER Cybersecurity Primer

The primer was designed to inform DER vendors, aggregators, and utilities about the basics of
DER cybersecurity. This work was broken into sections to provide an encompassing primer of
cybersecurity for DER. Basic tenets of cybersecurity including confidentiality (encryption),
integrity, availability, authentication, authorization, and non‐repudiation were provided. Common
types of cyber security attacks, e.g., eavesdropping, masquerading, man-in-the-middle, replay
attacks, Trojan horses, denial-of-service, etc. were described; and the current U.S. requirements
for DER communications were enumerated. The primer also included a review of cybersecurity
recommendations, guidelines, and reports by:
• Department of Energy (DOE)
• Department of Homeland Security Industrial Control Systems Cyber Emergency Response
Team (DHS ICS-CERT)
• Electric Power Research Institute (EPRI)
• Federal Information Processing Standard (FIPS)
• Institute of Electrical and Electronics Engineers (IEEE)
Page 8 of 24
 PV Cyber Security Research
Sandia National Laboratories

• International Council on Large Electric Systems (CIGRE)

• International Electrotechnical Committee (IEC) and International Organization for
Standardization (ISO)
• Internet Engineering Task Force (IETF)
• National Institute of Standards and Technology (NIST)
• North American Electric Reliability Corporation (NERC) and Federal Energy Regulatory
Commission (FERC)

The primer also covered cybersecurity requirements for DER standards, as shown in Table 1.

Table 1: Cybersecurity features of DER communication standards and information models.

Protocol: IEEE
Protocol: IEC Protocol: IEEE
1815 Protocol: Modbus
61850 2030.5
Information Information
Information Information
Model: DNP3 Model: SunSpec
DER Protocol Cyber Model: Model:
Application or MESA Models
Security Features IEC 61850-90-7 CSIP
Note Security
Security Security
Security Requirements:
Requirements: Requirements:
Requirements: None
IEC 62351 Series IEEE 2030.5 + CSIP
IEEE 1815
DER, Power DER, Smart Grid Utility, Grid Utility, Grid, ICS
Device Support
Systems Devices devices Devices devices
Encryption
Non-Native Yes BITW BITW
Capability
Encryption Required No Yes No No
Authentication
Non-Native Yes Optional Non-Native
Support
IEC 61850-90-7 Communication
Communication Communication
Type of contains functions protocol for
protocol for device protocol for real-
Communication for power real-time
integration with time monitoring
Protocol converter-based monitoring and
the Smart Grid and control
DER systems control
No security
Type of Security requirements for
IEC 62351 Series IEEE 2030.5 + CSIP IEEE 1815
Requirements Modbus
communications

Lastly, the primer provided recommendations for DER interoperability standards, cyber security
requirements, and risk management procedures for DER vendors, aggregators, and grid operators.
Specific recommendations for use of each of the communications standards were provided.

Page 9 of 24
 PV Cyber Security Research
Sandia National Laboratories

Adversary-Based Assessments
A team of cyber security researchers at Sandia National Laboratories conducted red team
assessments of multiple DER devices at the Distributed Energy Technologies Laboratory (DETL).
DER communication channels are designed to allow utilities, aggregators, and other grid operators
the ability to enable and configure various grid-support functions. However, these capabilities
expand the power system cyber security attack surface and pose a significant risk to the resilience
of the electric grid if controlled in aggregate. To advise the solar industry of the current risks and
provide evidence-based recommendations to the community, Sandia performed cyber security
assessments of a communications-enabled PV inverter and remote grid-monitoring gateway. The
team found several well-designed security features but also some weaknesses. Based on these
findings, recommendations are provided to improve the security features of DER devices.

To conduct the assessments, an isolated, controlled network was created for selected DER devices.
All the experiments were performed at the Distributed Energy Technologies Laboratory (DETL)
at Sandia National Laboratories. The following security tests and experiments were performed:
• Network Reconnaissance
• Packet Replay
• Man in the Middle (MITM)
• Denial of Service (DoS)
• Modified Firmware Upload
• Maintained Logs per Device
• Password Handling

Scanning the DER network, host device ports, and services, and performing red team analyses
identified cyber security vulnerabilities in the interoperable DER. The anonymized findings 1 of
this assessment are summarized in Table 2.

Table 2: DER cyber assessment comparison.

Device A Device B
Protocol UDP/IP TCP/IP
Analyzed Interface Ethernet Ethernet
Reconnaissance ✓ ✓
Packet Replay x o
MITM x x
DoS x x
Mod Firmware o o
Prevalent Logs x x
Password Handling x x
x = Exploits Exist, ✓= Successful, o = Incomplete

1
C. Carter, I. Onunkwo, P. Cordeiro, J. Johnson, “Cyber Security Assessments of Distributed Energy Resources,”
IEEE PVSC, Washington, DC, 25-30 Jun 2017.
Page 10 of 24
 PV Cyber Security Research
Sandia National Laboratories

As part of the assessment, multiple recommendations were collated to assist the PV and DER
community mitigate cyber weaknesses in the control networks. The recommendations included
the following:
• Encrypt data exchanges and do not pass information in plaintext.
▪ Use of telnet for remote logins should be discontinued or upgraded to the latest
version, sTELNET.
▪ Applications such as FTP should be replaced with another secure file transfer
system, FTP-SSL.
• Secure password strategies and policies should be implemented and enforced for all system
users. Require these credentials to be different for privileged access.
• Utilize practical firewall rules to mitigate the effects associated with Denial of Service
attacks and unauthorized access to DER network.
• Lock MAC address on the network devices and on each port of a switch to prevent receiving
unauthorized traffic.
• Implement the AAA framework: Authentication, Authorization, and Accounting.
▪ Authentication: Ensure users, devices, and applications attempting to access system
resources are valid and trusted.
▪ Accounting: Ensure all devices and systems are accounted for cyber security best
practices.
▪ Authorization: Ensure users, devices, and applications attempting to access system
resources are authorized for access.
• Practice Principle of Least Privilege.
▪ Every module (such as a process, a user, or a program, depending on the subject)
must be able to access only the information and resources that are necessary for its
legitimate purpose.
▪ If a user or resource no longer needs access to perform a legitimate task, disable their
access.
▪ Disable all ports that are not being used for normal operation.

Ultimately, the team found multiple security weaknesses that could be exploited to gain access or
control of DER devices. These findings were shared with the device vendors to take corrective
actions. The findings were also shared with the solar industry in anonymized form to provide
example concerns, best practices, and recommendations for improving the cyber security posture
of the DER devices and US power system.

Trust Recommendations
Unfortunately, Internet of Things connectivity often outpaces security implementations, resulting
in regular news of data breaches, distributed denial of service attacks, and other malicious activities
from millions of relatively weakly protected, internet-connected devices. These attacks may be
motivated by fraud, influence, profit, or pure disruption. Power system cyber resilience must be
improved by increasing the difficulty of launching these types of attacks.

Page 11 of 24
 PV Cyber Security Research
Sandia National Laboratories

Most of the solar industry’s efforts in securing DER has focused on protecting data-in-transport,
i.e., creating a secure data path between communicating devices. Table 3 shows the cryptographic
and trust requirements of several DER communication protocols. Since California is an early
adopter of communications-enabled DER equipment, Sandia conducted a deep dive into the
benefits and challenges derived from their IEEE 2030.5 and the California Smart Inverter Profile
(CSIP) implementation. Based on this analysis, recommendations for improvements to trust and
encryption in DER communication networks were provided. The primary recommendations
include (a) creating an ecosystem that supports certificate revocation lists, (b) supporting updates
for stronger encryption algorithms, (c) clarify/standardize the interface between the DER network,
utility DMZ, and utility IEEE 2030.5 server, (d) improved physical and data-at-rest security, and
(e) adding security requirements to the IEEE 2030.5 standard pertaining to aggregation servers.2

Looking toward the future, there are multiple novel technologies that could be employed to
improve the common approaches in DER communication standards. Some of these trust
alternatives are:
• Mobile Trusted Module (MTM) - A security element and a newly approved Trusted
Computing Group (TCG) specification for use in mobile and embedded devices. A secure
boot sequence measures the boot process and aborts any non-approved state transition.
• Mobile Device Management (MDM) Software - A suite of software, that is used by an
enterprise to manage its mobile devices. MDM software performs functions such as
hardware inventory creation and tracking, security policy enforcement, software
distribution, and more.
• Per-Application VPN – A per-app VPN prevents mixing various classes of usage where
data from network communicating processes must be kept separate, i.e., each application
is on its own per-app VPN tunnel.
• ARM TrustZone - ARM TrustZone partitions all the system-on-chip (SoC) hardware and
software resources so they exist in either the secure world or normal-world. Hardware logic
present in the TrustZone-enabled bus fabric ensures that no secure-world resources can be
accessed by the normal world components.
• Post-Quantum Crypto - Post-quantum crypto systems based on lattices, error correcting
codes, hash functions and multivariate equations.
• Blockchain PKI – A decentralized public key infrastructure (DPKI) capable of preventing
man-in-the-middle attacks from stolen certificates.

Each of these technology options is discussed in the “Recommendations for Trust and Encryption
in DER Interoperability Standards” Sandia technical report.

2
J. Obert, P. Cordeiro, J. Johnson, G. Lum, T. Tansy, M. Pala, R. Ih, “Recommendations for Trust and Encryption
in DER Interoperability Standards,” Sandia Technical Report, SAND2019-1490, Feb 2019.
Page 12 of 24
 PV Cyber Security Research
Sandia National Laboratories

Table 3: Trust and Cryptography Features in Common DER Communication Protocols.

Node
Protocol Encryption Key Management
Authentication
IEC 62351-9 covers generating,
distributing, revoking, and handling
IEC 61850, X.509 Digital
IEC 62351-3 requires TLS public-key and symmetric keys for
IEC 62351 Certificates
groups (GDOI) but does not define
the type of keys or cryptography
VPNs and IPSec are
recommended. TLS is
optional. Multiple TLS IEEE 1815-2012 allows pre-shared
IEEE 1815, cipher suites are permitted, X.509 Digital keys but also includes methods for
DNP3-SA but Certificates symmetric and asymmetric
TLS_RSA_WITH_AES_128 cryptography.
_SHA shall be supported at
minimum.
SunSpec
None None None
Modbus
IEEE 2030.5 requires key
IEEE 2030.5 requires TLS.
management by a public key
AES-128 in the Counter with
infrastructure which shall use
IEEE 2030.5, Cipher Block Chaining – X.509 Digital
Ephemeral Elliptic Curve Diffie–
CSIP Message Authentication Certificates
Hellman key exchange with Elliptic
Code Mode shall be
Curve Digital Signature Algorithm
supported
signatures (ECDHE_ECDSA)

Moving Target Defense

Historically, control systems have primarily depended upon their isolation from the Internet and
traditional Information Technology (IT) networks as a means of maintaining secure operation in
the face of potential remote attacks over computer networks. However, these networks are
incrementally being upgraded and interconnected with external networks. PV communications
routed over the public internet is an excellent example of this situation. The static nature of these
systems eases the process of gathering reconnaissance information that can be used to design,
develop, and launch attacks by adversaries.

Moving Target Defense (MTD) is a class of technologies that dynamically modify a system
environment to create uncertainty for adversaries by overlaying another control network on the
publicly addressable one. MTD leverages software defined networking (SDN) to randomize
network parameters (IP addresses and ports) and communication paths. It is possible to randomize
IP addresses and port numbers at fixed intervals or in response to detected network activity—i.e.,
dynamic defense. Randomizing IP addresses at a configurable frequency supports evading
adversarial discovery. It is meant to thwart an adversary’s ability to conduct reconnaissance and
establish communications between devices on the network. MTD has been proven to be effective
at increasing the resilience of grid wide area networks against certain types of attacks.

Page 13 of 24
 PV Cyber Security Research
Sandia National Laboratories

Introducing artificial diversity into the Internet Protocol (IP) layer has been demonstrated to work
within a software-defined network (SDN) environment [32]. Information flows, based on
incoming port, outgoing port, incoming media access control (MAC) and outgoing MAC, are
introduced into software-defined switches from a controller system. The flows are specified within
the flow parameters and contain matching rules for each packet. If a match is made within a packet,
then the flow action is to rewrite source and destination IP addresses to random values. The packets
are rewritten dynamically while they are in flight traversing each of the software-defined switches.
Previously, performance metrics of prior proof-of-concept have been captured with latency
impacts of less than a millisecond, on average [33].

An example of this technology is shown in Figure 3. On the left is a utility subnet consisting of an
Advanced Distribution Management System (ADMS), Geographical Information System (GIS),
and DER management system (DERMS). On the right, is a collection of DER devices in a campus
or utility/commercial site on a single switch. There is an “IP Generator” computer in the bottom
that sends the new IP addresses to the switches in front of actual DER or computation devices.
The MTD changes the IP addresses of these switches but the utility-owned and DER nodes retain
static IP addresses. Actual implementation may require multiple MTD subsystems that
independently reconfigure the IP addresses of the utility subnet and DER devices. Since this
technology requires a separate network to be overlaid on the publicly-addressable one, it is likely
that DER will use cell modems or other out-of-band telecommunication services to update their
MTD/SDN settings. These additional communication channels and associated hardware are
already common in modern DER equipment.

Figure 3: Implementation of Moving Target Defense on a DER communication network.

To determine the security improvements of this technology compared to baseline experiments,

MTD was deployed on a virtualized environment using SCETPRE. SCEPTRE (capitalized, but
not an acronym) is a live, virtualized power system and control network co-simulation platform
Page 14 of 24
 PV Cyber Security Research
Sandia National Laboratories

developed at Sandia National Laboratories capable of investigating the tradeoffs between power
system performance and cyber resilience. SCEPTRE provides a comprehensive Industrial Control
System (ICS) and/or Supervisory Control and Data Acquisition (SCADA) modeling and
simulation Hardware-In-The-Loop (HIL) capability that captures the cyber-physical impacts of
controls system operations and targeted cyber events. Changes in the network are reflected in the
power simulation, and changes in the power simulation are reflected in the communication system,
thereby allowing researchers to analyze the complex interactions in a cyber-physical environment.
The virtual components in a SCEPTRE model are created and run as Virtual Machines (VM) using
Minimega, an Emulytics tool that was developed at Sandia for orchestrating distributed virtual
machines (VM) and producing host and network emulations. SCEPTRE leverages Minimega’s
hypervisor capabilities to deploy VMs on compute nodes. A virtual representation of PV inverters
was created with SunSpec Modbus Remote Terminal Units (RTUs). These virtual RTUs were
connected to a virtualized control network.

SCEPTRE also interfaced with and ran OpenDSS power simulations. These simulations couple to
the simulated control network demonstrate the performance of DER grid-support control functions
under different cybersecurity architectures, protocols, and additional security features. Sandia has
developed multiple OpenDSS models with DER device that can be leveraged for the power
simulations. When the DER settings were updated in the RTUs, an internal backend ZeroMQ
network transferred the new settings to the DERs in the OpenDSS distribution circuit simulation.
Similarly, the status of the power system at the location of the DER is transferred to the RTUs
using ZeroMQ when there is a load or solar generation solve.

To quantify the security improvements from MTD red team assessments of baseline networks and
MTD networks were performed.3 Red teaming is authorized, adversary-based assessments
conducted to measure defenses of a system. The assessment combined practices from multiple
sources: Sandia’s Information Design Assurance Red Team (IDART), NIST’s Guide to Industrial
Control Systems (ICS) Security Guidelines, best cyber security practices, and collective expertise
regarding the DER devices and network. The red team assessment focused on identifying and
compromising the PV inverters by turning them off, as well as disrupting network communications
and modifying grid-supported functions (e.g. Freq-Watt, Volt-VAR, Power-Factor). The results of
the assessment with the baseline case and MTD are shown in Table 4 and documented in [34-35].
The MTD implementation improved the security by reducing the risk of replay attacks.

The Moving Target Defense (MTD) environment is a difficult topology to conduct reconnaissance
because the networking stack implemented an IP-MAC-Port whitelisting that prevented network
visibility of the DERs and the IP addresses of the equipment regularly changed. However, a

3
I. Onunkwo, B. Wright, P. Cordeiro, N. Jacobs, C. Lai, J. Johnson, T. Hutchins, W. Stout, A. Chavez, B. T.
Richardson, K. Schwalm, “Cybersecurity Assessments on Emulated DER Communication Networks,” SAND2019-
2406, March 2019.
Page 15 of 24
 PV Cyber Security Research
Sandia National Laboratories

security weakness manifested itself through a vulnerable default switch proprietary

communication protocol. The Red Team was able to exploit the default configurations on the
switch connected to the ISP router to perform a VLAN hopping attack. This attack enabled the
Red Team to listen to all broadcasts on the VLANs to gain reconnaissance information (VLAN
information, IP addresses used by the SDN controller, and open ports). DoS attacks on the switch
were also successful in preventing traffic between the utility and the DER devices. MITM attack
was not successful because of the size of the IP address space that needed to be scanned for valid
addresses.

There were some challenges with the red team assessments. The MTD environment was built with
software defined networking (SDN) concepts inside of an Emulytics platform itself built on rapid
prototyping models of SDN, causing a fusion of certain network surfaces that would have been
separated in the real world. For instance, a real MTD system would protect the applications and
application plane communications with the interceding control plane, leaving the controller and
control plane communications as a new attack surface. Conflation of the Emulytics platform and
the MTD environment may have contributed to difficulties defining what elements were in scope
and what new attack surfaces were available. Without the identified security weakness (which can
exist in real networks), this virtual environment was far more challenging to craft MITM because
the target's IP address kept changing. Access control using network function virtualization in
software defined networking adds additional challenges to conducting reconnaissance on a
network. However, the MTD topology did not withstand many of the attempts at reconnaissance,
denial of service, packet replay, man-in-the-middle, or VLAN hopping. These attempts were prone
to causing system failure, which was attributed to the novelty of the integration of the complex co-
simulation sub-systems.

In theory, adding MTD should improve the security posture of the DER network. Adding MTD
and whitelisting prevents adversaries outside the subnet from accessing the devices and adversaries
with access to DER subnets from reaching into other enclaves. Encryption prevents replay and
MITM attacks because the adversary cannot authenticate the connection to the DERMS or DER.
Moving Target Defense further challenges the adversary because they cannot identify DER IP
address, ports, or protocols. Denial of Service attacks are very difficult to defend against but
whitelisting the DERMS and DER can help prevent these attacks. As shown in the Table 4,
theoretical risk scores were then calculated for Confidentiality based on the replay and MITM
attacks, Integrity based on the replay and MITM attacks, and Availability based on the DoS attack.

For the C-I-A triad columns, a scale of 1 to 5 was created to categorize the risk level on each
topology. A score of 1 indicated a low risk to all devices (green color code), whereas a score of 5
(red color code) indicates a high risk to a majority of the devices. Risk scores between 2 (light
green color code), 3 (yellow color code), and 4 (orange color code) indicates the varying levels
showing the progressive difficulty or scale of DER fleet compromise. Lower scores were issued if
the difficulty of the attack was substantial or the magnitude of compromise was not fleet-wide.

Page 16 of 24
 PV Cyber Security Research
Sandia National Laboratories

To determine the total score, the following vulnerability level metrics were loosely adapted from
the NIST CVSS v2.0 ratings:
• HIGH - means that means that an attack has fully succeeded. For this metric, a range of
values between 10-15 is assigned.
• MEDIUM - means that attacks have partly succeeded. For this metric, a range of values
between 5-9 is assigned.
• LOW - means that attacks have not succeeded. For this metric, a range of values between
0-4 is assigned.

The scores for the theoretical security were totaled for a security risk score between 3 and 15. In
this defined range, low risk scores between 3 and 4 have a green color code, medium risk scores
between 5 and 9 have an orange color code, and high risk scores between 10 and 15 have a red
color code. After the red team assessments, the actual scores for MTD defenses were much
different than anticipated. Moving Target Defense provided a couple of features that initially
inhibited red team traction. The use of SDN allowed on-switch access control. Packets not
matching the whitelist for the expected IP and MAC addresses on a particular switch port were not
transmitted by the switch. This gave the stance of the attacker no visibility to any devices or traffic
on the network besides the gateway router. This advantage was reduced when the Red Team
exploited a layer 2 (data layer) vulnerable default configurations which made the network
susceptible to some reconnaissance and DoS attacks used in disrupting communication paths. As
shown in Table 5, the Red Team was successful in conducting DoS attacks, but this topology
resisted replay and man-in-the-middle attacks. The DoS attack could be conducted in the emulated
system because the environment had an avoidable layer 2 unsecured default configuration that was
exploited.

Based on the red teaming experiments, the following are noted:

• Denial of service is difficult to prevent. Aggregators/utilities should implement firewall
whitelists to prevent these types of attacks.
• It is important that developers add layers of defense by reviewing and pushing secure code
to applications.
• MTD has the potential to drastically improve security for DER networks, but this is still an
area of research.

Page 17 of 24
 PV Cyber Security Research
Sandia National Laboratories

Table 4: Theoretical Security of Unsecured and MTD DER Networks

Attacks Results
Topology Encryption Total Score
Replay DoS MiTM C I A
Flat None ✔ ✔ ✔ 5 5 5 15
MTD with Whitelisting None 1 1 2 4

Table 5: Adversary-Based Assessment of SCETPRE DER Networks

Attacks Results
Topology Encryption Total Score
Replay DoS MiTM C I A
Flat None ✔ ✔ ✔ 5 5 5 15
MTD with Whitelisting None ✔ 1 1 5 7

Engineering Controls
One of the potential defense mechanisms for DER communications is engineering controls for
DER grid-support functions. If malicious or accidental undesired actions are taken by an adversary
or grid operator, the system will block this command and sent alerts to one or more monitoring
centers. This could be implemented as a real-time continuous cybersecurity situational awareness
monitoring system at the aggregator/utility, a check in the communication system like a bump-in-
the-wire solution, or a filter/check in the DER itself. For each of the advanced grid-support
functions (e.g., volt-var, freq-watt, specified power factor, etc.), the parameters that define these
functions should be required to fall within specific ranges that ensure the function has the desired
power system behavior. When parameters are set outside of these limits, the monitor would reject
the change if the parameter is outside the limits. For instance, the volt-var pointwise curves require
(V, Q) points; if points are assigned to be in Q1 and Q3 in the V-Q plane, they would be rejected,
as shown in Figure 4. These types of rules are currently implemented in some PV inverters, but
not standardized. Defining ranges of values for each of the parameters in the information models
(e.g., CSIP, DNP3 Application Note, SunSpec Modbus Models, IEC 61850-7-420) or in
interconnection standards would standardize the acceptable ranges for DER parameters and
vendors to write code that enforced these limits. The situational awareness monitor could exist as
a communication module inside the inverter, networking components, bump-in-the-wire
technology, gateway, or inverter microprocessor. Furthermore, once communications traffic
attempts to set the DER into an unsafe operating mode for the power system, the following actions
will be taken:
1. The traffic is blocked to the DERs.
2. Alarms will be sent to grid operators, aggregators, and public-private partnerships to alert
of a possible cyber attack, e.g., the Cybersecurity Risk Information Sharing Program
(CRISP).
3. The user who issued this command will be provide with an indication of the blocked traffic
and prevented from making additional changes for a period of time.

Page 18 of 24
 PV Cyber Security Research
Sandia National Laboratories

4. Any defense mechanisms that are included in the operational technology (OT) network
could be initiated, e.g., moving target defenses re-randomize IP or port numbers in the
network.
An example of one of these simple rules applied to the volt-var functions is shown in Figure 4.
The volt-var (VV) pointwise curves are defined by (V, Q) points. One cyber attack could invert
the VV curve such that at high voltages the DER injects reactive power and at low voltages the
DER absorbs reactive power. This would drive the voltage away from nominal and possibly cause
grid instabilities. The engineering control rules could be enforced at the DER or within the network
to ensure the points are assigned to be in Q2 and Q4 in the V-Q plane and rejected otherwise.

This concept was submitted as a provisional patent application in Dec 2018. It has also been
codified as an optional volt-var function check in the SunSpec SVP, when it sets the Modbus
registers using pysunspec.4 Additional work in this space is expected in the coming years.

4
SunSpec SVP pull request with the engineering controls applied to the VV curve. URL:
https://github.com/sunspec/svp_energy_lab/pull/12/commits/341e1b0952341fb2334f4d7798e40e
410f53fe3e
Page 19 of 24
 PV Cyber Security Research
Sandia National Laboratories

Figure 4: Example engineering control rules for VV curve parameters.

Significant Accomplishments and Conclusions

The work completed in this project represented the first steps in creating a robust DOE-funded
cybersecurity program for PV/DER equipment. The roadmap and primer were designed to act as
a starting place to guide national and local policy, standards, and public and private investment to
improve the resilience of the US power system. The roadmap specifically recommended hardening
PV control networks, developing and implementing detection technologies, and preparing to
rapidly respond to cyber attacks. Through collective implementation of these technologies the
security of photovoltaic control systems can be strengthened without compromising the
performance of the network.

This project also investigated and advanced multiple PV cybersecurity R&D topics. The team
researched and recommended new trust and encryption approaches for DER communications. The
team used a virtualized network and power system co-simulation to test the promising security
features of moving target defense applied to PV inverters networks. Lastly, the team developed,
refined, and began the patenting process for secure engineering controls for secure DER
information exchanges, which prevent grid misoperation and alert grid operators of attempts to
input incorrect settings.

Page 20 of 24
 PV Cyber Security Research
Sandia National Laboratories

Inventions, Patents, Publications, and Other Results

The following publications and patent application were created during this project:
1. J. Johnson, “Roadmap for Photovoltaic Cyber Security,” Sandia Technical Report,
SAND2017-13262, Dec 2017.
2. C. Lai, N. Jacobs, S. Hossain-McKenzie, C. Carter, P. Cordeiro, I. Onunkwo, J. Johnson,
"Cyber Security Primer for DER Vendors, Aggregators, and Grid Operators," Sandia
Technical Report, SAND2017-13113, Dec 2017.
3. I. Onunkwo, B. Wright, P. Cordeiro, N. Jacobs, C. Lai, J. Johnson, T. Hutchins, W. Stout,
A. Chavez, B. T. Richardson, K. Schwalm, “Cybersecurity Assessments on Emulated
DER Communication Networks,” SAND2019-2406, March 2019.
4. C. Carter, I. Onunkwo, P. Cordeiro, J. Johnson, “Cyber Security Assessments of
Distributed Energy Resources,” IEEE PVSC, Washington, DC, 25-30 Jun 2017.
5. J. Johnson, “Distributed Energy Resource Cyber Attack Detection and Mitigation Tool”
Provisional Patent Application 62/769,771, Filed 20 Nov 2018.

Page 21 of 24
 PV Cyber Security Research
Sandia National Laboratories

Path Forward
As described in the PV cybersecurity roadmap, sustained cyber security leadership and stakeholder
commitment are necessary to continuously improve PV equipment and networks, build effective
standards, maintain public-private information exchanges, and support government and
commercial R&D efforts. Maintaining positive momentum is the responsibility of all stakeholders.
Our desired end state is a world where grid operators, system owners, and aggregators
communicate with interoperable photovoltaic systems using safe, secure, resilient networks with
high availability, data integrity, and confidentiality.

This project investigated PV/DER security state-of-the-art practices and developed or investigated
multiple innovative technologies. It is important that the R&D developed in this project be
incorporated into field demonstrations and commercial products to harden DER networks. Future
work should include the following:
• Information in the PV Cybersecurity Roadmap and the Cybersecurity Primer should be
updated regularly to represented changes in the industry standards, regulations, and
approach to security. The roadmap should be revisited regularly to re-chart the planned
path forward for DOE and the solar industry.
• Additional assessments of DER equipment should be completed to understand security
practices of vendors and the weaknesses of equipment on the market. Findings should be
shared with vendors so they can better secure their equipment.
• New methods of authenticating and encrypting DER traffic should continue to be
investigated to keep DER current with state-of-the-art security practices.
• Moving Target Defense has shown promise in improving DER network security. This
technology should be assessed with a DER field demonstration soon.
• Sandia has begun the process of patenting the DER engineering controls intrusion
prevention and detection system with the hope that this can be commercialized by one or
more companies.

Page 22 of 24
 PV Cyber Security Research
Sandia National Laboratories

References
[1] SEIA, U.S. Solar Market Has Record-Breaking Year, Total Market Poised to Triple in Next
Five Years, Press Release, 8 Mar. 2017.
[2] IEEE Standard 1547-2003, Standard for Interconnecting Distributed Resources with
Electric Power Systems.
[3] A. Hoke, et al., Inverter Ground Fault Overvoltage Testing, 2015.
[4] R. Seguin, J. Woyak, D. Costyk, J. Hambrick, B. Mather, High-Penetration PV Integration
Handbook for Distribution Engineers, NREL Technical Report, NREL/TP-5D00-63114,
Jan 2016.
[5] A. Hoke, et al., The Frequency-Watt Function: Simulation and Testing for the Hawaiian
Electric Companies, July 2017.
[6] Pacific Gas and Electric Co., Electric Rule No. 21, Generating Facility Interconnections,
Filed with the CPUC, Jan. 20, 2015.
[7] Hawaiian Electric Company, “Inc. Rule No. 14, Service Connection and Facilities on
Customers Premises,” D&O No. 33258 filed Oct. 12, 2015, effective Oct 21, 2015.
[8] R. Bründlinger, “Advanced smart inverter and DER functions requirements in latest
European grid codes and future trends,” Solar Canada, 8 Dec. 2015.
[9] D. Rosewater, et al., “International development of energy storage interoperability test
protocols for renewable energy integration,” EU PVSEC, Hamburg, Germany, 14-18 Sept,
2015.
[10] J. Johnson, et al., “Collaborative Development of Automated Advanced Interoperability
Certification Test Protocols for PV Smart Grid Integration,” EU PVSEC, Amsterdam,
Netherlands, 22-26 Sept, 2014.
[11] CEC & CPUC, Recommendations for Utility Communications with Distributed Energy
Resources (DER) Systems with Smart Inverters, Phase 2 Recommendations, 28 Feb 2015.
[12] J.C. Boemer, et al., “Status of Revision of IEEE Std 1547 and 1547.1,” 6th Solar
Integration Workshop, Vienna, 14-17 Nov. 2016.
[13] B. Reaugh, R. Beckensten, D. Gross, D. Brearley, “SCADA Systems for Large-Scale PV
Plants” SolarPro, Issue 10.3, May/Jun 2017.
[14] M. Mills-Price, K. Hao, The Importance of Coordinated Control Systems in Solar
Generation Plants, PAC World Americas Conference, Raleigh, NC, 23-25 Sept, 2014.
[15] E. Syme, Power Industry Communication Protocol Features and Benefits, ProSoft
Technology, Inc. Accessed 10-19-2017, URL: https://scadahacker.com/
[16] S. Mohagheghi, J. Stoupis, Z. Wang, Communication Protocols and Networks for Power
Systems-Current Status and Future Trends, ABB US Corporate Research Center, Raleigh,
NC, 3 Mar 2011.
[17] SunSpec Alliance, SunSpec Specifications & Information Models, accessed 10-19-2017,
URL: https://sunspec.org/about-sunspec-specifications/
[18] IEEE Std 2030.5-2013, IEEE Adoption of Smart Energy Profile 2.0 Application Protocol
Standard, 11 Nov. 2013.
[19] IEEE Std 1815-2012, IEEE Standard for Electric Power Systems Communications-
Distributed Network Protocol (DNP3)," 10 Oct. 2012.

Page 23 of 24
 PV Cyber Security Research
Sandia National Laboratories

[20] OpenADR 2.0 Profile Specification B Profile, Revision 1.1. Document 20120912-1, 17
Nov 2015.
[21] P. Fairley, 800,000 MicroInverters Remotely Retrofitted on Oahu—in One Day, IEEE
Spectrum, 5 Feb 2015.
[22] A. Konkar, ‘Something Astounding Just Happened’: Enphase’s Grid- Stabilizing
Collaboration with Hawaiian Electric, Enphase Energy blog, 11 Mar 2015.
[23] GE Energy Consulting, Oahu Distributed PV Grid Stability Study, Part 1: System
Frequency Response to Generator Contingency Events, March 3, 2016.
[24] T. Fox-Brewster, “This Man Hacked His Own Solar Panels... And Claims 1,000 More
Homes Vulnerable,” Forbes, Aug. 1, 2016.
[25] F. Bret-Mounet, “All Your Solar Panels are Belong to Me,” DEF CON 24, Las Vegas, Aug
4-7, 2016.
[26] K. Leswing, A massive cyberattack knocked out major websites across the internet,
Business Insider, 21 Oct 2016.
[27] B. Seal, et al., “Final Report for CSI RD&D Solicitation #4 Standard Communication
Interface and Certification Test Program for Smart Inverters,” June 2016.
[28] J. Henry, et al., Cyber Security Requirements and Recommendations for CSI RD&D
Solicitation #4 Distributed Energy Resource Communications, Oct 2015.
[29] Sandia National Laboratories, The Information Design Assurance Red Team (IDART™),
2009. URL: http://www.idart.sandia.gov/
[30] K. Stouffer, V. Pillitteri, S. Lightman, M. Abrams, A. Hahn, “Guide to Industrial Control
Systems (ICS) Security,” NIST, May 2015.
[31] DHS, ICS-CERT, “Recommended Practices,” May 22, 2017. URL: https://ics-cert.us-
cert.gov/Recommended-Practices
[32] Al-Shaer, E., Duan, Q., Jafarian, J. "Random Host Mutation for Moving Target Defense," in
SecureComm, pp. 310-327, Springer, 2012.
[33] A. R. Chavez, J. R. Hamlet, W.M.S. Stout, “Artificial Diversity and Defense Security
(ADDSec) Final Report” SAND2018-4545, April 2018.
[34] I. Onunkwo, B. Wright, P. Cordeiro, N. Jacobs, C. Lai, J. Johnson, T. Hutchins, W. Stout,
A. Chavez, B. T. Richardson, K. Schwalm, “Cybersecurity Assessments on Emulated DER
Communication Networks,” SAND2019-2406, March 2019.
[35] J. Johnson, B. Richardson, K. Schwalm, I. Onunkwo, P. Cordeiro, B. Wright, N. Jacobs, C.
Lai, “Assessing DER Network Cybersecurity Defenses in a Power-Communication Co-
Simulation Environment,” IEEE Internet of Things Journal (in preparation).

Page 24 of 24

View publication stats