SCADA Cyber Security Testbed Development

C. M. Davis, J. E. Tate, H. Okhravi, C. Grier, T. J. Overbye, and D. Nicol

School of Electrical and Computer Engineering
University of Illinois Urbana-Champaign
Urbana, Illinois

Abstract- New technologies are increasing the vulnerability B. Future SCADA Architecture
of the power system to cyber security threats. Dealing with ...
these threats and determining vulnerabilities is an important task
for utilities. This paper presents the development of a testbed
Presen ilities are tructurediverydier
have been in the past. The transmission system allows open
an they
designed to assess the vulnerabilities introduced by using public access, meaning that anyone owning a generator is allowed
networks for communication. to supply power to the grid. Markets have been set up to
decide which generators are used instead of the utility owning
I. INTRODUCTION generation and determining dispatch. These changes mean
The proliferation of new computer technologies in the power the transmission system is used in a very different way. The
system has brought many advantages and risks. Increasingly, changes in the operation of the transmission system make it
powerful computers are becoming prevalent not just in control more important to securely share data among system operators,
centers in offices but also in the field in the form of IEDs while ensuring that only the appropriate market insensitive
(Intelligent Electronic Devices). These new devices allow for data can be accessed by marketeers. Thus, the restructuring of
efficient network based communications, the use of next gen- the utility industry has resulted in the need for varying levels
eration Supervisory Control and Data Acquisition (SCADA) of information access [1].
protocols, and more intelligent behavior. Unfortunately, using There is also a shift in the nature of SCADA systems.
these new devices also has a down side. Using standard The old-style vendor-specific SCADA protocols are being
networks and protocols opens the devices to possible cyber replaced by next generation standards based protocols like
attacks. IEC 61850. These next generation protocols are based on a
To address these new vulnerabilities, the TCIP (Trustworthy common information models (CIM) [2]. Common information
Cyber Infrastructure for the Power grid) project has been models are used to associate devices with services. This kind
started under the ITI (Information Trust Institute). TCIP is an of abstraction, made possible by the improved computational
NSF funded project consisting of researchers in various areas power of new SCADA hardware, allows for useful features like
of computer security and power systems. device discovery. Instead of micro-controller based hardware
Determining the vulnerabilities of systems using these programmed in assembly language, present day hardware
devices is a complicated process because of the complex runs more advanced real-time operating operating systems
hardware and software interactions that must be considered. (e.g. real-time Linux and vxWorks) [3]. Not only are the
One approach is to build a comparatively simple system that protocols and hardware changing, but the communications
captures the relevant complexity, i.e., a testbed. This work first links are evolving as well. Expensive dedicated phone lines
presents motivations for developing a testbed in the form of a and microwave links are being replaced by data networks [4].
brief review of cyber security basics. Next, several components C. Threats
developed for use in the testbed environment are discussed. In
the final section, the pieces are put together and a simulation There are many threats facing critical infrastructure today.
Of a cyber attack scenario is presented. Currently, the most famous threats are the threats posed by
terrorist groups and hostile nation states. These are organized
II. CYBER SECURITY BACKGROUND groups with a clear goal and some level of sophistication.
A. Traditional SCADA Architecture There is also a threat posed by a company's own employ-
ees. Company insiders have access to internal controls and
Historically electric utilities have been regulated, vertically data, and, either by accident or malicious intent, can cause
integrated monopolies. One company owned and controlled equipment outages. A third category of threat is the threat
everything from the generators to the distribution system. posed by casual hackers, known as "script kiddies". These are
Utilities knew their systems very well and data was shared people without great computer ability who download and use
only on a limited basis. SCADA systems often communicated prepackaged tools.
Over dedicated communication links like phone lines and mi-
crowave radio links. The SCADA system hardware possessed D. Vulnerabilities
limited processing power and often utilized vendor-specific The term vulnerabilities is used to refer to equipment that is
protocols. vulnerable to attack. This notion is distinct from threats. For

483

Authorized licensed use limited to: Intel Corporation via the Intel Library. Downloaded on September 26, 2009 at 23:20 from IEEE Xplore. Restrictions apply.
example, a vulnerability would be a hole in the fence, whereas 2) Capabilities: The client currently has the following key
a threat would be the person who wants to get through the capabilities:
fence. Power systems face a new array of cyber vulnerabilities . The client provides a graphical view of power system
as new equipment, running more standard, real-time operating states. The information used to drive the display is
systems, is phased in. These standard operating systems are obtained via TCP/IP from a server (see Section III-C
subject to a greater number of well known attacks. The below for a description of the protocol used). This mimics
Cyber Emergency Response Team (CERT) has been tracking a control room display that is obtaining SCADA data
computer vulnerabilities since the late 1980s. Their statistics from the power grid over a communications network.
show that the number of vulnerabilities has been increasing . The ability to control (rather than simply view) power
dramatically in recent years [5]. system elements is also a key component of real power
system operation. The client supports control actions,
E. Countermeasures such as opening and closing of lines, in addition to simple
To address vulnerabilities present in the power system, the isplay of data.
All data displayed on the client must first be communi-
North American Reliability Council (NERC), working with
the Department of Energy and the Department of Homeland cated over the network from the server to the client. This
Security and their Canadian counterparts, has developed a set decoupling of the display (the network client) from the
data source (the PowerWorld server) enables independent
ofcye seurt
standards....
[6]. These standards are a protocol
requiring companies to identify their vulnerabilities and risks
modification
m and testing of the display, communications
and take steps to mitigate them. This is done in several networks, and power system without affecting other com-
steps. First, the impact of the loss of assets is determined. ponents of the testing environment.
Next, the standard calls for the identification of vulnerabilities. An individual client can access any number of servers,
with a highly configurable scheduling mechanism for
Then, using this information,
. '. . ' companies
can preform risk
.................. . retrieving data. Data retrieval from the server can be
analysis to decide which vulnerabilities are most important aycrono ini.ta retr to oc t regular te
to protect against. Finally, companies decide which defenses
are most cost effective and begin to implement them. Common
intervals. By setting the intervals between retrievals to
defences against cyber attacks include application of firewalls a very small value, it is easy to stress the underlying
and authentication methods [7]. Peer-to-peer overlay routing is communications system to examine bandwidth effects.
another possible defence against Distributed Denial of Servce Suort for maor oeratin sstems Windows Mac OS
(DDoS) attacks [8]. X, Linux)
A sample screen shot of the network client displaying retrieved
III. SIMULATION ARCHITECTURE data for a 7 bus system is shown in Figure 1.
A. Network Client B. PowerWorld Server
1) Purpose: The PowerWorld server serves two purposes
1)ons Purpose.oThplemenetw
client provides several key func-

tions needed to implement an accurate testbed

testbed thatclosely
that closely
an accurate
which, when combined, allow it to serve as a surrogate for the
ra oe rdwe efrigeprmns
mimics real world operation of the power grid.
. The server simulates the power grid with a feature-rich
power flow solver. This allows us to simulate systems
01.81 mw F-Tfl with a high degree of modeling accuracy by taking

_6-09 d >0_99 deg

1.46 deg.
1.00 pu.

Q01 MW
i_ 4 Bus 5
-0c& deg. C-'.-

1i01 p_u.

Bu!3~/ Bus~
Bu5~~~~~~~~~~~~~ 4'~~~~~~~~~~~~..
E3u57
7 ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~.vq ~~~~~A..CBF' ;r

6 231 deg. RO JIdCn-eg.

2DO.)4 kW/ 200 64 hAN_A

Fig. 1. Network client screenshot - opening a line in a 7 bus case Fig. 2. PowerWorld server screenshot -serving data for a 7 bus case

484

Authorized licensed use limited to: Intel Corporation via the Intel Library. Downloaded on September 26, 2009 at 23:20 from IEEE Xplore. Restrictions apply.
 advantage of the advanced modeling facilities built into - Viewer iSSFNet Network Simulator
the PowerWorld Simulator [9] software. database Emam
The server provides the SCADA data that would typically S U ,,,
be fed into a control center display (represented by the Internet -=:
client). The server provides the simulated data to the DataSeret Simulator
client over a TCP/IP network using a custom networking Manager
protocol (see Section III-C below). 1RNSELbackup instance
2) Controls. The server also accepts control commands
sent by clients, e.g., the opening and closing of lines. The
server continuously solves the power flow, so network flow
impacts are instantly solved and propagated to all connected
clients. The ability to accept control commands from the client Fig. 3. RINSE Architecture
allows us to study the effects of various network attacks on
control actions.
Clients run Network Viewer which connects to the database
3) Data Provided: The server currently provides the fol- through the Data Server and gives each client a separate
lowing data to clients: view of the simulated network. This view the portion of
is
* Bus voltage magnitude and phase angle the simulated network the client is interested in and is able
* Line status to interact with. This feature can be used in exercises like
* Line flow the October 2003 Livewire cyber war exercise which was
* Generator status conducted by the Department of Homeland Security [12].
A sample screen shot of the PowerWorld server providing 2) Capabilities: In addition to parallel real-time large-
client data on the 7 bus case is shown in Figure 2. scale simulation, RINSE is capable of multi-resolution traffic
C. Client-Server Protocol simulation, meaning it can simulate traffic with varying levels
of detail. This makes it suitable for simulating high-volume
The protocol used for communicating between the client traffic and attacks. When traffic is presented and simulated
and server is a simple request/response protocol which uses in a multi-resolution fashion, traffic with important dynamic
the TCP/IP networking protocols. All network communication behavior (foreground traffic) is simulated with high-resolution
is initiated by the client, which can either send or receive an packet-level details whereas traffic showing other activities in
arbitrary amount of data in a single session. the network (background traffic) is simulated using a coarse-
grained fluid model [13] [14] . RINSE uses both resolutions
for different traffic at the same time [15], coupled with a
1) Purpose: The Real-time Immersive Network Simulation fixed point solution technique, resulting in several orders of
Environment for Network Security Exercises (RINSE) is a tool magnitude speed-up in simulation [16] [17]. For attack traffic
for realistic emulation of large networks as well as network (e.g., Denial of Service (DoS) attacks) the details of the traffic
transactions, attacks, and defenses [10]. are of little importance and we are only interested in the
RINSE has unique capabilities which make it suitable for coarse behavior (volume of the traffic). Therefore, a coarser,
cyber security and game-playing exercises including large- multi-resolution model is used to increase the efficiency of
scale real-time human/machine-in-the-loop network simulation simulation and to make real-time simulation possible [18].
support, multi-resolution network traffic models, and novel Another important feature of the RINSE architecture is
routing simulation techniques. the Network Viewer which gives clients the ability to have
RINSE consists of five components: different views of the simulated network and also to issue com-
* the iSSFNet network simulator, mands to the simulator (Figure 4). Five types of commands
* the Simulator Database Manager, are currently supported:
* a database, . Attacks: for initiating attacks (particularly DDoS attacks)
* the Data Server, in the network.
* and client-side Network Viewers . Defenses: for applying countermeasures against attacks.
The internal architecture of RINSE is shown in Figure 3. These commands include filtering packets at routers
The iSSFNet is the core network simulator which is built which can mitigate attack effects.
on top of the Scalable Simulation Framework (iSSF), an . Diagnostic Tools: which simulate common networking
Application Programming Interface (API) for parallel large- utilities such as ping.
scale simulation [11]. In this architecture, the Simulation . Device Controls: for controlling individual devices in the
Database Manager is responsible for collecting simulation data network.
from simulator nodes and putting it into the database. In . Simulator Data: for controlling the output of the simula-
the RINSE architecture, many simulator nodes can work in tor.
parallel to support large-scale real-time network simulation. RINSE is also capable of emulation, i.e., it can represent

485

Authorized licensed use limited to: Intel Corporation via the Intel Library. Downloaded on September 26, 2009 at 23:20 from IEEE Xplore. Restrictions apply.
 RINSE +VPN Server
Network Cliert

PowerWorld Server
Prox + VPN Ci ent

FigQ.~4. Ntwor viwe scrensho Fig85.ClentSeverRISE ntgraio Scem

40021:2~~~1
NOA M2111 4~~IN(~01 ICs !TABLE
relnoe wItviua nodesin, the siulte netor adCAAERICSTAPIN
generate real packets~~~~~ for,0,,8trnscton ithl the oTsFide world.
Emlaio is th key fetr of RINS tHaTTP has) benueHMduTeic oeWrlevc
inthe1 inegato of RINEitthteteEuainasBrnhRU oe/letasmsonie
needs som exr copoetswic redscsedi dtilGeeatrRT ea enrtontrato
beo .1 A 5;0- " 6

F,Protocol Converter~~~~~~~~~~~~~~~~~~~~~~~[C 23TC)

The protocol converter is a program designed to convert the in which there are virtual nodes representingPowerWorld~~~~~~~..........
the
cusomPoerord roocl7nt ralSCDAIrooclscTisseve ad henewok ciet().Upn rrva o teIacet
allows, the netor client2to, interfac with actua hadwre)t,te irua nderereeninte owrWrlNsrvr,th
Thepotoco conerteralso rovies a eansof teting imultor gnerats.rel.pacets.ith vrtualIP.adresss.an
protocols by grabbing~~~~~~~ th evrsotu adfradn tdeieste ote enl hs2akesaesn hog
acros a (pssbl sImuae)ntokFnte P unltotepoysre.Fnly h rx
On .ucto. ha.ut.eprfre.y.h.rooo.sre.tasatstevita.ades.n.enste.akt
converte is mapn bewe1h ipePwrol evrt h oerol evr h aepoeshpesih

SCADA protocol~~~~~~F_, [19. Aetfree opeen-souree Mdbssmlehadth networ5Client(S)epaserISE IthgroughnRIScemakngi

real nodes withvirtual nodes i thsetupulisethatwtheanetwork SERImulto (RPINSEG scmltl

FgInertegreatio fckesilaortasatrnsprenntothePowrWoldhervruadsteintwokwcien(s)
Emltheschem that isy faused for ineRaIoNS ofthathsbe sdMdueie PowerWorldSevc
servere ithegnetworkcletsf RINSE inoteestbd
n showntionFigur IVanc ATTACKoSen/coetaRsisionle
5nThis architectureasfumajorent whcomponenticus:d A.ScenarioeDesriptionT edgnraonomto

E. Neotworkl Clivent(serodpce obcm velae.A h aetm

Th prINSEoaondvterVNt erer issueormdeindtonettei whcommereands Undrta normal ciprcusetances thePoperatorl
proortls (hin nrceth trasmssonk
PusowerWorlwerverl viatocaprtox srverl onCADseife te systemntos reliev theroverloadIne thikes
snervaerwtheatranlhatesre theae
thioscase, petortk00) Thien prox t however, thde netwrkseattakianbine opeWrlevea,tors
the
destnaton fthse ackts t th vitualIP ddrss o th frmulknowlegeneaeofeth problems andh vremovedPthdeseirailt tod
proowerorldb serverbing the simulaed' network Theporackets are respond. them stuation detneriorThese pascprtsaetiont
equipmen
thens deliveredlthroughate)ntoknte RISpoer h velaeieoutyo servier. Aicascadinge poutag
VPN tunnel to fre
the
daemouncgrabs theacketstfro therINSEmend ofthe VPNoco hserbegu thasatewill vresualt adblackou for thendload pocket.

tunland injesCtsA
thminodtels sModulao usin eomunlato
th PoferWbudsesr1,e2
Forit this scenari, the loadfi pokeenthcnsst

capaIltgaity ofRISE.uatrIsE thnsiuatsalaguntor pn3swhihcanbhe seenoin Figultre 6.IThE)

inresin dopemand
486 prn otePweWrdsre n hentokcin()

Authorized licensed use limited to: Intel Corporation via the Intel Library. Downloaded on September 26, 2009 at 23:20 from IEEE Xplore. Restrictions apply.
 Packet Drop Percentage
100/
2 /0Atc
= 80 starts /
6M Th90 , 0/7MWc6 70 -
ZX \ Filtering
69 MW
15 Mvar 3
0 Mvar
S
200 MW4\ 5
\|fgXf O Mvar y a.> 60 )/ i
\starts
g~- 40-
r )30
20-
Fig. 6. Scenario one-line diagram 90 MW bus 1 load o0 - :

Time (s)
- - No Attack -Attack w/o Filter - -Attack with Filter

89MW
16 Mvar
1
3
175 Mvar
200 MW
Fig. 9. Packet drop percentage in simulated network for a lOOs run
141 Mvar

DDoS attack starts in the network by issuing a command to

Fig. 7. Scenario one-line diagram 110 MW bus 1 load RINSE.

Upon reception of the command, the attacker sends attack

is modeled by step changes of the load at bus 2. The load at signals to zombie hosts in the network and they start emitting
bus 2 starts at 90 MW. At this load level, the transmission packets to the victim server at the rate of 700 Kbits/s. The
line from bus 4 to bus 1 is loaded at 70.5% of capacity. The attack starts after 30 seconds of simulation and lasts for 100
next load level is 110 MW. The corresponding line loading seconds. With this scenario we study the effect of attack on the
is 89.4%. The final load level is 125 MW. At this load level power system that uses a public network as its communication
the transmission line is overloaded, operating at 103.5% of medium.
capacity. Images of the system under each loading level can The third scenario complements the second one by applying
be seen in figures 6, 7, and 8. countermeasures in the network to mitigate the negative effects
of the attack on the power grid. A defensive filter is enabled
B. Attack Description which drops all packets from the zombie hosts.
To study the effect of cyber attacks on the simulated power
system, we have used the architecture shown in Figure 5. A C Results
relatively large network with hundreds of hosts and routers
and many subnets is used for simulation with two of the The three scenarios described above were run and the effects
hosts representing the PowerWorld server and the network of a DDoS attack on the power system were studied. To
client. The network client represents the operator station in get an estimate of the responsiveness of the network under
the control room. normal condition, attack, and attack with filters, packet drop
Three different scenarios have been simulated to study percentage between the PowerWorld server and its client were
the effects of attack and defense. In the first scenario, the measured using "ping" and is shown in Figure 9.
network runs under normal conditions with some server/client When there is not an attack occurring, the operator at the
transactions and background traffic present. The goal of this network client sees data that is refreshed at the proper rate
scenario is to study the interaction of the PowerWorld server and has the ability to open and close lines. If an attack is
and client under normal operating conditions of the network. in progress the SCADA data and commands are prevented
In the second scenario, after a period of normal operation, a from getting to and from the network client. The DDoS attack
floods the network with packets, causing the real data to be
delayed or lost. This is evident in the divergence of the one-line
views between the network client and the PowerWorld server.
When an attack is under way, the network client continues to
-I
2 display old data showing that the system is operating safely
even though a transmission line is overloaded (i.e., the operator
105MW ~~ 1205 17°5MMWr 14Mvar
otne to see Figure 6instead of Figure 7or Figure
application of a filter iS one defense against a DDoS attack.
The 8),.
Applying a filter during an attack successfully mitigated the
attack and allowed SCADA data to transit the network as
Fig. 8. Scenario one-line diagram 125 MW bus 1 load illustrated in Figure 9.

487

Authorized licensed use limited to: Intel Corporation via the Intel Library. Downloaded on September 26, 2009 at 23:20 from IEEE Xplore. Restrictions apply.
 V. CONCLUSION [4] R. McClanahan, "IEEE industry applications magizine," Mar.-Apr. 2003.
[5] C. E. R. Team. (2006) Cert/cc statistics 1988-2006. [Online]. Available:
The experiment presented in this paper uses the network http://www.cert.org/stats
client to act as a control station, the PowerWorld server to act [6] N. A. E. R. Council. (2006) Critical infrastructure protection. [Online].
as the power system, and RINSE to act as the communication Available: http://www.nerc.com/cip.html
network. This experiment demonstrated the vulnerability of the [7] J. Pollet, "Developing a solid scada security strategy," in Proc. Sensors
for Industry Conference, Houston, TX, USA, 2002, pp. 148-156.
network client to a DDoS attack and the ability of filtering to [8] J. J. Farris and D. M. Nicol, "Evaluation of secure peer-to-peer overlay
mitigate an attack. The attack prevented data from being trans- routing for survivable scada systems," in Proceedings of the 2004 Winter
Simulation Conference, Washington D.C.,USA, Dec. 2004.
mitted across the network, causing the control display to dis- [9] PowerWorld Corporation. [Online]. Available: http://www.powerworld.
play incorrect data. So far this work presents a testbed created com
using only software. The next step is to more accurately model [10] M. Liljenstam, J. Liu, D. Nicol, Y Yuan,, G. Yan, and C. Grier, "Rinse:
the real-time immersive network simulation environment for network
the SCADA system by incorporating actual hardware (RTUs, security exercises," in In Workshop on Principles of Advanced and
relays, etc.) in the simulations. Using the protocol converter Distributed Simulation, 2005.
it is possible already to interface with devices communicating [11] J. Cowie, D. Nicol, and A. Ogielski, "Modeling the global internet,"
using the ModbusTCP protocol. Work is ongoing to implement Computing in Science and Engineering, vol. 1, pp. 42-50, Jan. 1999.
using the ModbusTCP Work Is ongoing to implement [12] A. Press. (2003, Nov.) T. bridis. gov't simulates terrorist cyberattack.
next generation protocols and to extend the functionality of the [Online]. Available: http://www.zone-h.org/en/news/-read/id=3728
network client and PowerWorld server (e.g., provide more data [13] B. Liu, D. R. Figueiredo, Y Guo, J. Kurose, and D. Towsley, "A
and more control commands). In its final form, the testbed will study of networks simulation efficiency: Fluid simulation vs. packet-
level simulation," in In Proceedings of IEEE Infocom, Apr. 2001.
consist of computer simulations, hardware, and people acting [14] G. Kesidis, A. Singh, D. Cheung, and W. Kwok, "Feasibility of
as controllers. fluid-driven simulation for atm network," in In Proceedings of IEEE
Globecom, Nov. 1996.
ACKNOWLEDGMENT [15] D. Nicol and G. Yan, "Discrete event fluid modeling of background
tcp traffic," ACM Transactions on Modeling and Computer Simulation,
The authors would like to thank the NSF for their support vol. 14, pp. 1-39, July 2004.
under award number CNS-0524695. [16] D. Nicol, J. Liu, M. Liljenstam, and G. Yan, "Simulation of large-scale
networks using ssf," in In Winter Simulation Conference (WSC), Dec.
REFERENCES 2003.
[17] D. Nicol and G. Yan, "Simulation of network traffic at coarse time-
[1] G. Zecevic and Z. Jovanovic, "Company intranet access to scada scales," in In Workshop on Principles of Advanced and Distributed
information," in Proc. Budapest International Conference on Electric Simulation, 2005.
Power Engineering, New York City, USA, Aug. 1999, p. 121. [18] , "Discrete event fluid modeling of background tcp traffic," in Proc.
[2] G.-S. Kim and H.-H. Lee, "A study on iec 61850 base communication ACM Workshop on Rapid Malcode, Oct. 2003.
for intellegent electronic devices," in Proc. IEEE 9th Russian-Korean [19] Modbus-IDA. (2005) Modbus-ida:the architecture for distributed
International Symposium on Science and Technology, vol. 1, Novosi- automation. [Online]. Available: http://www.modbus.org
birsk,Russia, Jun.-Jul. 2005, pp. 765-770. [20] D. Wimberger. (2004) jamod. [Online]. Available: http://jamod.
[3] C. Bowen, T. Buennemeyer, and R. Thomas, "Next generation scada sourceforge.net
security: Best practices and client puzzles," in Proc. IEEE Workshop
on Information Assurance and Security, vol. 1, United States Military
Academy, West Point, NY, 2005, pp. 426-427.

488

Authorized licensed use limited to: Intel Corporation via the Intel Library. Downloaded on September 26, 2009 at 23:20 from IEEE Xplore. Restrictions apply.