Ethernet Communications

and Requirements for IEC

61850 Based Systems

Dr. Alexander Apostolov

Los Angeles, CA

© OMICRON
 Scope

• Discuss Ethernet architectures supporting

IEC 61850 projects
• Not defined in the standard but essential for a
real project !
• Background on Ethernet principles and
technologies
• Practical examples

© OMICRON Page: 2
 Transmission Definitions

SIMPLEX
Transmission in one direction only

HALF DUPLEX
Two way means of transmission but data
can only travel in one direction at a time

FULL DUPLEX
Transmission in both directions
simultaneously

© OMICRON Page: 3
 Transmission Definitions

PROTOCOL
Rules and procedures that
Communications networks use to
communicate on the communications
medium

CONNECTION
Communications are Connection
Oriented or Connectionless

© OMICRON Page: 4
Transmission Definitions
Point-to-point

Engineering Laptop
Station

Modem IED

Modem

IED

© OMICRON Page: 5
 Transmission Definitions
Shared Access
Substation
Ethernet HMI IED IED IED

Ethernet
Switch

Proxy Substation
Server HMI

RS485
Ethernet
Legacy IEC 61850 IEC 61850
IED IED IED

© OMICRON Page: 6
Transmission Definitions
NETWORK ACCESS

CARRIER SENSE METHODS

CSMA (Carrier Sense Multiple Access)/CD
(Collision Detection)
Medium Access Control (MAC)

TOKEN ACCESS METHOD

The device that has the Token has access to
transmit

RESERVATION METHOD
Each device has a predefined time slot to transmit

© OMICRON Page: 7
 Network Terminology

Network An interconnected group of nodes or

stations linked by communication channels

Node The interface point where one or more

functional units are connected

LAN Local area network (<5km)

WAN Wide area network

Network topology Pattern of nodes and their interconnection

© OMICRON Page: 8
 Transmission Definitions

Medium - Ethernet devices attach to a common medium that

provides a path along which the electronic signals will travel:
- historically, this medium has been coaxial copper cable
- more commonly a twisted pair
- fiber optic cabling.

Segment - a single shared medium as an Ethernet segment.

Nodes - devices that attach to that segment are stations or
nodes.

Frame - The nodes communicate in short messages called

frames, which are variably sized chunks of information.

© OMICRON Page: 9
Open Systems Interconnection
(OSI) Model
Application Selects appropriate service for application

Presentation Provides code conversion, data reformatting

Coordinates interaction between end

Session
application process
Provides for end to end data integrity and
Transport
quality of service

Network Switches and routes information

Transfers unit of information to other end

Data Link
of physical link

Physical Transmits bit stream to medium

© OMICRON Page: 10
 OSI Stack
PCI =Protocol Control Inform. Commands or data
PDU =Protocol Data Unit Application
H =Header Presentation
PCI PDU
T =Trailer
PCI PDU Session

PCI PDU Transport

PCI PDU Network

PCI PDU Data link

PDU PDU PDU Physical

H fragment
T H fragment
T H fragment
T

Frames (Ethernet, token ring, etc)

© OMICRON Page: 11
 Communications Process

Upper Layer Upper Layer

SAP SAP

Entity with Peer-to-peer dialog Entity with

services to offer services to offer

SAP SAP

Lower Layer Lower Layer

© OMICRON Page: 12
 What is Ethernet?
• Ethernet is the most widely used local area
network (LAN) technology.
• The original and most popular version of
Ethernet supports a data transmission rate
of 10 Mb/s.
• Newer versions of Ethernet called "Fast
Ethernet" and "Gigabit Ethernet" support
data rates of 100 Mb/s and 1 Gb/s (1000
Mb/s).
• An Ethernet LAN may use coaxial cable,
special grades of twisted pair wiring, or fiber
optic cable.

© OMICRON Page: 13
 What is Ethernet?
• "Bus" and "Star" wiring configurations are
supported.
• Ethernet devices compete for access to the
network using a protocol called Carrier
Sense Multiple Access with Collision
Detection (CSMA/CD).

© OMICRON Page: 14
Ethernet and the ISO/OSI Model
• The standard model for networking
protocols and distributed applications is the
International Standard Organization's Open
System Interconnect (ISO/OSI) model. It
defines seven network layers.
• Layer 1 - Physical
• Physical layer defines the cable or physical
medium itself, e.g., thinnet, thicknet,
unshielded twisted pairs (UTP).
• All media are functionally equivalent. The
main difference is in convenience and cost
of installation and maintenance.

© OMICRON Page: 15
Ethernet and the ISO/OSI Model
• Layer 2 - Data Link
• Data Link layer defines the format of data on
the network. A network data frame - packet,
includes checksum, source and destination
address, and data.
• The largest packet that can be sent through
a data link layer defines the Maximum
Transmission Unit (MTU).
• The data link layer handles the physical and
logical connections to the packet's
destination, using a network interface.

© OMICRON Page: 16
Ethernet and the ISO/OSI Model
• Layer 2 - Data Link
• A host connected to an Ethernet would have
an Ethernet interface to handle connections
to the outside world, and a loopback
interface to send packets to itself.
• Ethernet addresses a host using a unique,
48-bit address called its Ethernet address or
Media Access Control (MAC) address.

© OMICRON Page: 17
 Ethernet Frame
Pre SFD DA SA Lgt/Type MAC Data + Pad FCS
7 1 6 6 2 0 – n bytes 4

• Pre: The Preamble is an alternating pattern (7 bytes) of

1 and 0 that tells receiving stations that a frame is
coming
• SFD: Start-of-frame delimiter (1 byte: 10101011)
indicating that the next bit is the left-most bit in the left-
most byte of the destination address.
• DA: Destination address (6 bytes) identifies which
station(s) should receive the frame
• SA: Source addresses (6 bytes) identifies the sending
station

© OMICRON Page: 18
 Ethernet Frame
Pre SFD DA SA Lgt/Type MAC Data + Pad FCS
7 1 6 6 2 0 – n bytes 4

• Length/Type: Number of MAC-client data bytes that are

contained in the data field of the frame
• MAC Client Data: A sequence of n bytes (46=< n
=<1500) of any value. (The total frame minimum is 64
bytes). The Pad contains (if necessary) extra data bytes
in order to bring the frame length up to its minimum size.
A minimum Ethernet frame size is 64 bytes from the
Destination MAC Address field through the Frame Check
Sequence.
• FCS: The Frame Check Sequence is a 32-bit cyclic
redundancy check (CRC) value

© OMICRON Page: 19
 Extended Ethernet Frame
Pre SFD DA SA TagT TagC Lgt/Type MAC Data + Pad FCS
7 1 6 6 2 2 2 0 – n bytes 4

• Virtual LAN (VLAN) tags are used to identify the

VLAN to which the frame belongs. It allows
frames from stations to be assigned to logical
groups
• TagT: The first 2-bytes of the VLAN tag consist
of the "802.1Q Tag Type" - always set to a value
of 0x8100.
• TagC: VLAN Tag Control

© OMICRON Page: 20
 VLAN Tag Control
Pre SFD DA SA TagT TagC Lgt/Type MAC Data + Pad FCS
7 1 6 6 2 2 2 0 – n bytes 4

• The first 3-bits are a User Priority Field

that may be used to assign a priority
level to the Ethernet frame.
• The next 1-bit is a Canonical Format
Indicator (CFI) used in Ethernet frames
to indicate the presence of a Routing
Information Field (RIF).
• The last 12-bits are the VLAN Identifier
(VID) which uniquely identifies the
VLAN to which the Ethernet frame
belongs.
© OMICRON Page: 21
 Ethernet

• Media Access Control (MAC) Address -

This is the physical address of any device,
such as the NIC in a computer, on the
network.
• The MAC address has two parts, each 3
bytes long.
• The first 3 bytes identify the company that
made the NIC.
• The second 3 bytes are the serial number
of the NIC itself.

© OMICRON Page: 22
 Ethernet

• Unicast - A transmission from one node

addressed specifically to another node.
• Multicast - When a node sends a
packet addressed to a special group
address. Devices that are interested in
this group register to receive packets
addressed to the group.
• Broadcast - When a node sends out a
packet that is intended for transmission
to all other nodes on the network.

© OMICRON Page: 23
 Ethernet Basic Principle
1. Listen and broadcast if the wire is free
2. If there is a collision then re-transmit in an interval [0, 2N-1]

Ethernet

© OMICRON Page: 24
Switches principles

Internal Switches
Queue

© OMICRON Page: 25
Switches: Performances

Pure delay (store & forward): 5-10μs

Maximum frame size: 1536 bytes
1536 bytes @ 100 Mbps = 123 μs
Goose size << 1536 bytes

© OMICRON Page: 26
Switches: Unicast

© OMICRON Page: 27
Switches: Multicast

2 2

© OMICRON Page: 28
Switches: Broadcast

2 2 2 2

© OMICRON Page: 29
To Process
Binary Input
Wall-to-wall performances
Filtering [2-10]

Logical [1-50]
example (ms)
[5-75]

Communication [2-15]

Switching [0.1]

Communication [2-15]

Logical [1-50] [4-72]

Relaying [1-7]

Process To + 9/147
Binary Output

© OMICRON Page: 30
Switches: other features used in
Substation Automation

• Substation environment !
• Electro-magnetic compatibility
• DC power supply with 20ms voltage dips, possibly
redundant
• Priority management (802.1p):
• Capability to have priority queues in order to further
boost the communication performances
• Virtual LAN – VLAN (802.1q tag):
• Capability to create logical groups of devices in order
to filter the messages not belonging to one group

© OMICRON Page: 31
Switches: other features used in
Substation Automation

• Management
• Capability to detect a switch failure: SNMP
or watch-dog
• Redundancy management
• No standard today for hard real time
redundancy
• Redundancy between switches (not
applications)
• See further slides

© OMICRON Page: 32
Typical network architectures
STAR

ISD ISD ISD

Basic scheme since failure of the

central switch leads to the total
communication failure

© OMICRON Page: 33
Typical network architectures
RING

ISD ISD ISD

The most common scheme since good

performance/cost ratio (save 2 central switches
vs. start configuration)
Failure of a single fiber or device does affect
the system once reconfiguration is completed

© OMICRON Page: 34
Typical network architectures
DOUBLE STAR

ISD ISD ISD

The most secured scheme since multiple

failures are tolerated except the two central
switches simultaneously

© OMICRON Page: 35
 Redundancy management
• A way to improve system availability and reliability
• Behavior is not defined in IEC 61850
• Typical example where the architect needs to define what
shall be done… and first check on paper that it can be
supported by the IEDs

• May be applied to:

• Communication infrastructure: tolerate the loss of an optical
fiber or/and the loss of a central switch
• Clients: capability for a client to continue the tasks initiated
by a first client (example: Graphical user interface)
• Servers: capability for a client to switch to a redundant server
if the first one is not operational (example: CT/VT sensor)

• 99.99% availability requests the 3 types of redundancy

© OMICRON Page: 36
 Redundancy management at
communication level
• Defined between Ethernet switches
• If a device has two Ethernet port it must also have an integrated
switch
• Spanning tree mechanisms
• Recalculation of the route between switches using an internal
protocol between switches and avoiding loops
• Base: 802.3d. Typically 30 s reconfiguration time
• Fast: 803w. Typically 100 ms reconfiguration time, some
implementations claims 5 ms per switch (N x 5ms with a ring made of
N switches)
• Other mechanisms
• Detects the failure of the adjacent switch to reconfigure
• For ring topology, less than 1 ms reconfigurations
• Sends two signals at the same time and keep the first one arrived
• No switch-over time, but need to keep a correct buffer size (dual homing)

© OMICRON Page: 37
 Example (ring)
“Repeaters”
(IEE 802.3 Ethernet switch)
+ Self Healing Manager (SHM)
Ethernet Ports 10/100 Base TX

EEPROM

Port MII

SHM N°
Failsafe Self healing ring
output relays manager

100Mbs Full duplex

opto opto

Primary Fiber

Secondary Fiber
Optical Ring

© OMICRON Page: 38
 Example (ring)

X During nominal situation, Ethernet packet goes in

primary fibre always in the same direction, and
only a checking frame (4 bytes) is sent every 5ms
in secondary fibre in the opposite direction.
Primary fibre

1 2 3 4 5 6 7 8 9 10 11

switch switch switch

switch switch
RP EP
C D E
A B

ES RS

Secondary fibre

© OMICRON Page: 39
 Example (ring)
If the connection between 2 switches is broken
X The Ethernet network will continue to run correctly.
X Both SHM start immediately the network self-healing.
Š At one side, the messages received are no more emitted to
the primary fibre but to the secondary fibre.
Š On the other side of the cut off, the messages received to secondary
are emitted to primary and the new topological loop is closed.
Primary fibre

1 2 3 4 5 6 7 8 9 10 11

switch switch switch

switch switch
RP EP
C D E
A B

ES RS

Secondary fibre

© OMICRON Page: 40
Other communication elements
IEC Com. Com.
OSI Model Technology Protocol Services Element

Application MMS/SNTP Report, Control, Gateway/

File Proxy

Presentation

Session
Software

Transport TCP

Network IP Router

Bridge/Switch
Data Link 802.3 GOOSE
/Proxy
Hardware
Physical 802.3

Need to design the communication infrastructure

according to the system constraints and requirements

© OMICRON Page: 41
 IEC 62351
• “Data and Communication Security” series of
standard
• IEC 62351-6: Security for IEC 61850 profiles
• Relies on IEC 62351-4: Profiles including MMS
• Relies on IEC 62351-3: Profiles including TCP
IEC Com. Security
OSI Model Technology Protocol Services Services

Report, Control, Authentification,

Application MMS/SNTP
File Replay

Presentation

Session
Software
Authentification,
Transport TCP Encryption, Replay

Network IP

Authentification,
Data Link 802.3 GOOSE Replay

Hardware
Physical 802.3

© OMICRON Page: 42