Kimura 1
Kimura 1
Mike MacLaren
Randy Kimura
AltaLink Management Ltd. AltaLink Management Ltd.
Calgary, Canada Calgary, Canada
Abstract— Time synchronization is a very interesting topic for the LAN Local Area Network
electric power industry. Our requirements and expectations of time MPLS Multiprotocol Label Switching
synchronization have evolved over time with new applications NTP Network Time Protocol
requiring improved accuracy and resiliency. New technologies and PTP Precision Time Protocol
techniques are required to achieve the additional accuracy and RTU Remote Terminal Unit
resiliency. SCADA Supervisory Control and Data Acquisition
SOE Sequence of Events
Index Terms—IRIG-B, Precision Time Protocol, Time
SNTP Simple Network Time Protocol
synchronization
III. ALTALINK
I. INTRODUCTION As Alberta’s largest regulated
In a traditional SCADA system time synchronization transmission company, AltaLink
accuracy is required to meet event and alarm reporting owns more than half the provincial
requirements, typically 1 millisecond. Newer applications, for transmission system with a
example synchrophasors and sampled values, may require a 212,000 square kilometre service
higher degree of accuracy and resiliency while other area. Our network of more than
applications, for example security event reporting, may 13,000 kilometres of transmission
tolerate a less accurate time synchronization methodology. lines and 300 substations delivers
This paper describes the approach used by AltaLink to align electricity safely, reliably and
time synchronization technologies with the accuracy efficiently to more than three
requirements and the capabilities of the devices. million Albertans. We transport
power from the generation
This paper discusses the evolution of time synchronization facilities where it is created to the
within AltaLink to meet past, present and future time communities, businesses and
synchronization requirements beginning with manufacturer industries that need it every day.
proprietary SCADA communication protocols. The pros and
cons of the more recent approaches, IRIG-B, NTP, and DNP3,
are discussed with a focus on the future feasibility of each
technology. AltaLink’s pilot project deploying Precision Time
Protocol (PTP) grandmaster and boundary clocks, using the To get a correct sequence of events across different places
TeleCom and Power Profiles, will be described. Future in the power system a time tagging with a precise global time
deployments which include transparent and slave clocks, has to be provided. Therefore, all related devices shall be
Parallel Redundancy Protocol (PRP) networks, and High- synchronized with the requested accuracy. Common for events
availability Seamless Redundancy (HSR) networks will be is 1 ms.[1]
discussed highlighting the concerns and triggers required for TABLE I summarizes the time synchronization classes and
future deployments. applications defined in IEC 61850 [1]. Time synchronization
class T1 must be achieved for the power system event time
II. NOMENCLATURE tagging.
CIP Critical Infrastructure Protection TABLE I. TIME SYNCHRONIZATION CLASSES
DNP Distributed Network Protocol
DST Daylight Saving Time Class Accuracy Application
(µsec)
GLONASS Global Navigation Satellite System
GNSS Global Navigation Satellite System TL >10000 Low time synchronization - miscellaneous
GPS Global Positioning System T0 10000 Time stamping of events with an accuracy of 10 ms
IEC International Electrotechnical Commission
IEEE Institute of Electrical and Electronics Engineers T1 1000 Time stamping of events with an accuracy of 1 ms
IRIG Inter-Range Instrumentation Group T2 100 Time tagging of zero crossings
Class Accuracy Application In order to time stamp power system events with one
(µsec) millisecond accuracy the equipment clock must be time
T3 25 Miscellaneous synchronized before it drifts a millisecond. For the RTU clock
with 100 ppm accuracy an update period less than 10 seconds
T4 4 Time tagging of synchronized samples is required. For the high accuracy oscillator an update period
T5 1 High precision time tagging of samples less than 500 seconds (8 minutes) is required.
TABLE III. UPDATE FREQUENCY
Accuracy One Millisecond Drift Update Frequency
IV. STANDALONE EQUIPMENT (ppm) (minutes) (seconds)
The early systems consisted of one or more pieces of 1 16.67 1000
substation equipment operating in isolation, relying on
2 8.33 500
someone manually setting the clocks. Setting the clock will
be very inconvenient as the equipment must be visited in order 5 3.33 200
to set the clock, therefore it should be assumed that the clock 10 1.67 100
will be infrequently set. It is likely that the clock will be
initially set during the commissioning/installation phase and 100 0.167 10
never updated. After an extended period of time, the
equipment's clock drift may affect the clock accuracy. A. Communication Path Delay
An RTU that previously had significant market share has a A communication protocol's use of millisecond resolution
free running clock accuracy of 100 ppm. TABLE II does not guarantee accurate equipment clocks. Whenever a
summarizes the potential clock drift accumulated over an Master Station issues a time synchronization command, it
hour, day and a year. After one hour the referenced RTU clock sends the current time. Upon receipt of the command the
has drifted 360 milliseconds making the 1 millisecond equipment will update its clock with the time and date
requirement unachievable. The accumulated error after a year contained in the command.
exceeds 52 minutes.
Figure 2 illustrates the communication protocol based time
The RTU manufacturer offered a high accuracy crystal synchronization sequence:
oscillator version with a clock accuracy of 2 ppm. Although
1. The Master Station sends a time synchronization
the high accuracy crystal oscillator version is a significant
command with the current time, t1.
improvement, the one millisecond time stamping requirement
remains unachievable after an hour. 2. Sometime later, t2, the equipment receives the last
TABLE II. CLOCK DRIFT byte of the time synchronization.
Accuracy Hourly Drift Daily Drift Yearly Drift 1 2
(ppm) (milliseconds) (milliseconds) (seconds)
1 3.6 86.4 31.536 t
t1 t2
2 7.2 172.8 63.072
Figure 2 Communication Protocol Time Synchronization
5 18 432 157.680
10 36 864 315.360
The time difference, t2 (the time the last byte of the time
synchronization command is received) – t1 (the time
100 360 8640 3153.600 synchronization message was built), is the communication
propagation time. If the clock is set using the time provided by
the Master Station, t1, than the error will be the
I. SCADA PROTOCOLS communication propagation time:
Communication protocols provided the ability to remotely
update equipment clocks, removing the inconvenience of
visiting the equipment. This system is automated, allowing for
terror t 2 t1
the clocks to be updated at a known frequency. Appropriately
selecting the frequency will eliminate the effects of clock drift Some communication protocols will attempt to measure
errors. the communication propagation time and adjust the clock to
account for this delay. TABLE IV summarizes some
communication protocol capabilities to adjust a clock for the
Master
Station communication path time.
TABLE IV COMMUNICATION PATH ADJUSTMENT
Protocol
RTU
Time Sync Protocol Clock Adjustment
Ack
Conitel 3000 Not supported.
Figure 1 Master Station Based Synchronization
Modbus Not supported.
Protocol Clock Adjustment Backup
Master
Master
DNP3 Calculates and adjusts for communication path Station
Station
delays.
Protocol 1 Protocol 2
RTU
A single master station may used to time synchronize Time Sync Time Sync
Ack Ack
devise in multiple substations (shown in Figure 3).
Figure 5 Backup Master Station
Master
Station
Figure 6 illustrates the difficulty accurately identifying the
order of events when the substation equipment is synchronized
Protocol Time Sync from two sources.
RTU
Ack
5
Time Sync
RTU
Ack 4 2
Time Sync
RTU 1 3
Ack
now set to a time 59,990 milliseconds before t1 (time Seconds Minutes Hours Day
= t1 + 10 – 60000). P0
1 2 4 8 10 20 40
P1
1 2 4 8 10 20 40
P2
1 2 4 8 10 20
P3
1 2 4 8 10 20 40 80
P4
100 200
P5
stamped with time t2. This time is 59,980 Control Functions Time of Day
P9
29 210 211 212 213 214 215 216
P10
a problem. Although the second event occurred 20 Figure 9 200-98 IRIG-B Format
milliseconds after the first event, the second event is time
stamped 59.980 milliseconds before the first event. The IEEE 1344 (IEEE Standard for Synchrophasors for Power
correct order, in which the events occurred or the time Systems) included year data in the control bits of the IRIG-B
between the two events, cannot be determined. The removal of format. This variation, shown in Figure 10, is commonly
the clock adjustment will eliminate the event reporting referred to as IEEE 1344 extensions.
problems. 50 60 70 80
1 2 4 8 10 20 40 80 LS DSP DST ± 1 2 4 8 P
Accurate system time is maintained using a satellite-based LSP ½H
produces an IRIG-B signal or SNTP message for time Figure 10 IEEE 1344 Extensions
synchronizing substation equipment as shown in Figure 8.
In 2004, IRIG Standard 200-04 was updated to include the
The United Stated Department of Defense satellite based year data.
Global Positioning System (GPS) time is received by a GPS
clock. The precise Positioning Service (PPS) level of accuracy It would impractical to use more than one external clock
is limited to authorized military users. The Standard source in a substation with multiple devices. The daisy chain
Positioning Service (SPS) level of accuracy is available to the approach, shown in Figure 11, may be used when there are a
general public and provides the high precision time. limited number of devices within close proximity to the clock
source. Clocks may support multiple IRIG-B outputs,
This satellite based system monitors 21 satellites in six permitting the implementation of a number of IRIG-B daisy
orbital planes 20000 km above earth. At least 4 satellites can chains.
be simultaneously monitored from every point on the globe.
The system computes the GPS time by accurately measuring
the propagation of signals between the satellites and GPS Clock
clock.
IED
Master
Station IED
Protocol IRIG-B
Figure 11 IRIG-B Daisy Chain
RTU Clock
Time Sync
Ack
Fiber-optic transceivers can be used to extend the IRIG-B
network over longer distances. The layout shown in Figure 12
Figure 8 Satellite Clock has been used to extend the IRIG-B network between control
Inter-range instrumentation group time codes, commonly buildings.
known as IRIG time codes, are standard formats for
transferring timing information. Atomic frequency standards R
T
Fiber
Transceiver
RS-232/485
Clock
and GPS receivers designed for precision timing are often
equipped with an IRIG output. The standards were created by R Fiber
Transceiver
RS-232/485
IED
T
IRIG-B
Uses a hard real time operating system.
RTU IED
The characteristics of RTU 2 are:
IRIG-B
IED
The hardware, application software, and operating
Figure 13 IRIG-B Sub-master software may be sourced from different suppliers
The application software and operating system may
A. Accuracy be independently patched.
The adoption of substation automation resulted in a DNP3
communication interface between the RTU and substation A soft real time operating system is used.
devices. Elimination of the IRIG-B network and time DNP3 time synchronization was rejected due to the
synchronization using the communication protocol was accuracy dependency on the RTU make, model, or firmware.
identified as a potential reduction in engineering complexity
and project delivery cost. This change required confirmation The addition of an IRIG-B time synchronization source is
that the communication protocol based time synchronization similar the addition or upgrade of a second Master Station.
could achieve accuracies comparable to the IRIG-B The traditional approach is disabling the time synchronization
implementation. service between the Master Station and RTU. When the IRIG-
B signal is lost, the RTU clock is free running and will drift
The introduction of a communication path error was over time. If the IRIG-B signal is lost for significant period of
described in Figure 2. Communication protocol capability to time the RTU clock will eventually become inaccurate.
measure and adjust for the communication path error is
summarized in TABLE IV. The setup, shown in Figure 14, The same technique can be implemented when the Master
was used to compare the two time synchronization methods. Station time synchronization functionality cannot be disabled.
The RTU returns a positive response to Master Station without
IRIG-B modifying the internal clock. The RTU is time synchronized
Clock I/O
using the IRIG-B signal.
IRIG-B Protocol
RTU I/O
Time Sync Master
Ack Station
-1 42
Protocol IRIG-B
RTU Clock
-1 42 Time Sync
Ack
-1 35
Figure 16 Master Station Time Failover
-1 44
The failover logic is straight forward. The RTU will accept
-1 44
and process the Master Station time synchronization messages
0 45 whenever:
The IRIG-B signal between the RTU and GPS clock
is lost.
IRIG-B processing is disabled in the RTU. these event logs the time synchronization accuracy
requirement was downgraded from one millisecond to several
seconds. Changing the accuracy permitted the use of Network
IRIG-B Signal
Time Protocol (NTP), a technology already supported by the
EMS Time Sync networking equipment.
IRIG-B Enabled
The Center for Internet Security Critical Security Control
Figure 17 IRIG-B Failover Logic
6.2 recommends:
GPS clocks may have the capability to detect the loss of Include at least two synchronized time sources from which
signal, for example a break in cabling between the antenna all servers and network equipment retrieve time
and the clock, destruction of the antenna or an obstacle information on a regular basis so that the timestamps in
blocking the signal to the antenna. The GPS clock can be logs are consistent [5].
configured to assert a relay contact when the loss of the GPS When possible, backup NTP servers are configured.
signal is detected. A generic I/O module is used to interface Improved redundancy is achieved by selecting NTP servers in
the relay contact, allowing the RTU to monitor the status of different geographical areas.
the antenna and GPS clock.
The time synchronization failover logic has been updated IV. PRECISION TIME PROTOCOL
to include the status between the antenna and GPS clock. The Precision Time Protocol (PTP) was developed to
RTU will accept and process the Master Station time simultaneously synchronize several devices on a local area
synchronization messages whenever: network (LAN) with sub-microsecond accuracy. The protocol
The IRIG-B signal between the RTU and GPS clock was developed to fill a void left by previous technologies.
is lost. Network Time Protocol (NTP) was not accurate enough to
meet a lot of industries technical requirements and multiple
RTU IRIG-B processing is disabled. GPS receivers in the system was not a cost effective solution.
The GPS signal between the antenna and GPS clock PTP was originally defined in IEEE 1588-2002 (1588v1)
is lost. in order to target LAN applications. In 2008, IEEE 1588-2008
(1588v2) was released to provide greater accuracy, precision
and robustness targeting telecommunication networks.
IRIG-B Signal
Unfortunately IEEE 1588v2 is not backwards compatible with
IRIG-B Enabled EMS Time Sync IEEE 1588v1.
Clock Relay Contact
PTP utilizes an algorithm to self-organize all the clocks in
a system into a master-slave hierarchy as shown in Figure 19.
Figure 18 IRIG-B and GPS Failover Logic
The free running accuracy of the GPS clock should be Grandmaster Clock
checked prior to adding the loss of GPS signal to the time
synchronization failover logic. The drift of the free running
GPS clock is often minimal and may fall within acceptable
limits.
Substation Clock
III. CIP EVENT LOGGING
CIP-007-5 security event monitoring requirement 4.1
requires the responsible entity to: Device Clock
Log events at the BES Cyber System level (per BES Cyber
System capability) or at the Cyber Asset level (per Cyber Figure 19 PTP Hierarchy
Asset capability) for identification of, and after-the-fact
Grandmaster Clock determines the time base for the system.
investigation of, Cyber Security incidents that includes, as
a minimum, each of the following types of events: Substation Clock is a slave to the Grandmaster Clock and
master to the Device clocks.
1. Detected successful login attempts;
Device Clock is a slave to the Substation Clock.
2. Detected failed access attempts and failed login
attempts; The grandmaster clock determines the time base for the
system and is typically synchronized using GPS/GLONASS
3. Detected malicious code.[4]
antennas. Each slave synchronizes to its master by
Some of the equipment does not support IRIG-B and an exchanging Sync, Delay_Req, Follow_Up, Delay_Resp
alternative technology is required to time stamp event logs. messages. The time of these message exchanges are recorded
The substation LAN equipment (Ethernet switches, serial port and the slave clock computes the time delay offset from its
servers, radios, etc.) support Syslog for event reporting. For master and adjusts its local clock. The synchronization and
time offset computation process can be seen in Figure 20 C. System Architecture
PTP Synchronization Offset. The telecom profile (G.8265.1) allows the user to specify
the IP addresses of multiple grandmaster clocks. The local
substation clock will actively evaluate each of these
grandmaster clocks to determine which the best available
master is at a given time. A “PTP Subnetwork” was
provisioned within the existing MPLS network where three (3)
grandmaster clocks and the local PTP substation clock are
assigned IP addresses within the subnet.
The local PTP substation clock has two independently
programmable Ethernet ports. The second port is assigned an
IP address on the local substation subnet to distribute the PTP
power profile (C.37.238-2011). All IED’s capable of
accepting the PTP power profile are synced over their existing
LAN connections. A second PTP clock is also synced using
the PTP power profile and it then distributes legacy IRIG-B to
devices not capable PTP time synchronization. The full
system architecture is shown in Figure 23.
Grandmaster Clock Grandmaster Clock Grandmaster Clock
Figure 20 PTP Synchronization Offset
PTP IRIG-B
Substation Clock
I/O I/O