1

Ultra-Low Latency (ULL) Networks: The IEEE

TSN and IETF DetNet Standards and Related 5G
ULL Research
Ahmed Nasrallah, Akhilesh S. Thyagaturu, Ziyad Alharbi, Cuixiang Wang, Xing Shao, Martin Reisslein, and
Hesham ElBakoury

Abstract—Many network applications, e.g., industrial control, tactile Internet [2], [3], and on the order of 100 microsec-
arXiv:1803.07673v3 [cs.NI] 24 Sep 2018

demand Ultra-Low Latency (ULL). However, traditional packet onds for the one-way fronthaul in wireless cellular networks.
networks can only reduce the end-to-end latencies to the order of For example, critical healthcare applications, e.g., for tele-
tens of milliseconds. The IEEE 802.1 Time Sensitive Networking
(TSN) standard and related research studies have sought to surgery, and transportation applications [4] require near real-
provide link layer support for ULL networking, while the time connectivity. Throughput requirements largely dependent
emerging IETF Deterministic Networking (DetNet) standards on the application needs, which may vary widely from small
seek to provide the complementary network layer ULL support. amounts of IoT data to large exchanges of media data transfers
This article provides an up-to-date comprehensive survey of the to and from the cloud (or the fog to reduce latency) [5].
IEEE TSN and IETF DetNet standards and the related research
studies. The survey of these standards and research studies is Additionally, autonomous automotive vehicles [6], augmented
organized according to the main categories of flow concept, and virtual reality (AR/VR), as well as robotic applications,
flow synchronization, flow management, flow control, and flow which are essential for Industrial IoT (IIoT), may require both
integrity. ULL networking mechanisms play a critical role in high data rates as well as ULL [7]–[10]. The high data rates
the emerging fifth generation (5G) network access chain from may be required for transporting video feeds from cameras
wireless devices via access, backhaul, and core networks. We
survey the studies that specifically target the support of ULL in that are used to control vehicles and robots [11]. Therefore, in
5G networks, with the main categories of fronthaul, backhaul, such heterogeneous environments and applications, a dedicated
and network management. Throughout, we identify the pitfalls mechanism to universally accommodate a diverse range of
and limitations of the existing standards and research studies. ULL requirements would be very helpful [12].
This survey can thus serve as a basis for the development of
standards enhancements and future ULL research studies that
address the identified pitfalls and limitations. B. Contributions and Organization of this Survey
Index Terms—Deterministic networking (DetNet), Preemption, This article provides a comprehensive up-to-date survey of
Time-sensitive networking (TSN), Time synchronization, Ultra-
low delay. standards and research studies addressing networking mecha-
nisms for ULL applications. Section III covers the IEEE TSN
standards that have grown out of the AVB standards and focus
I. I NTRODUCTION
primarily on the link layer, while Section IV covers the ULL
A. Motivation research studies related to TSN. Section V covers the Internet
Traditional networks which provide end-to-end connectivity Engineering Task Force (IETF) Deterministic Networking
to the users have only been successful in reducing the operat- (DetNet) standards developments, while Section VI covers
ing end-to-end latencies to the order of tens of milliseconds. the ULL research studies related to DetNet. This sequence
However, present and future applications demand Ultra-Low of the section on standards followed by the section on related
Latency (ULL). For instance, the end-to-end latencies should research studies is inspired by the temporal sequence of the
be on the order of a few microseconds to a few milliseconds development of the ULL field, where standard development
for industrial applications [1], around 1 millisecond for the has typically preceded research studies.
A large portion of the ULL applications will likely involve
Please direct correspondence to M. Reisslein.
A. Nasrallah, A.S. Thyagaturu, Z. Alharbi, and M. Reisslein are with the wireless communications, whereby the fifth generation (5G)
School of Electrical, Computer, and Energy Eng., Arizona State University, wireless systems will play a prominent role. In particular,
Tempe, AZ 85287-5706, USA, Phone: 480-965-8593, Fax: 480-965-8325, (e- the emerging tactile Internet paradigm with end-to-end target
mail: {ahnasral, athyagat, zalharbi, reisslein}@asu.edu).
C. Wang and X. Shao are with the Dept. of Internet of Things latencies below 1 ms is tightly coupled to the ongoing 5G
Eng., Yancheng Inst. of Techn., Yancheng, Jiangsu Province, P.R. China (e- developments [5], [13]–[16]. The support of 5G wireless ULL
mail: [email protected], [email protected]). C. Wang and X. Shao visited communications services will likely heavily rely on the TSN
Arizona State Univ., Tempe, while contributing to this study. C. Wang was
supported by the Overseas Training Program of the Yancheng Institute of and DetNet standards and research results. On the other hand,
Technology. X. Shao was supported by a Jiangsu Provincial Government due to prevalence and importance of wireless communications
Scholarship for Overseas Learning. in today’s society, the particular 5G wireless context and
H. ElBakoury is with Futurewei Technologies Inc., 2330
Central Expressway, Santa Clara, CA, 95050, USA (e-mail: requirements will likely influence the future development of
[email protected]). ULL standards development and research. We believe that for
 2

a thorough understanding of the complete ULL research area II. BACKGROUND

it is vital to comprehensively consider the ULL standards,
A. Latency Terminology
namely TSN and DetNet, as well as a main “application
domain” of ULL standards and research results. We anticipate Generally, latency refers to the total end-to-end packet delay
that 5G wireless communications will emerge as a highly from the instant of the beginning of transmission by the sender
important application domain of ULL standards and research (talker) to the complete reception by the receiver (listener).
results and we therefore survey ULL related standards and The term ultra-low latency (ULL) commonly refers to latencies
research studies for 5G wireless systems in Section VII. that are very short, e.g., on the order of a few milliseconds or
Section VIII identifies the main gaps and limitation of the less than one millisecond. ULL applications often require de-
existing TSN and DetNet standards as well as ULL related terministic latency, i.e., all frames of a given application traffic
5G standards and research studies and outlines future research flow (connection) must not exceed a prescribed bound [44],
directions to address these gaps and limitations. e.g., to ensure the proper functioning of industrial automation
systems. It is also possible that applications may require
probabilistic latency, i.e., a prescribed delay bound should
be met with high probability, e.g., for multimedia streaming
C. Related Literature systems [45], [46], where rare delay bound violations have
negligible impact of the perceived quality of the multimedia.
While to the best of our knowledge there is no prior survey Latency jitter, or jitter for short, refers to the packet latency
on time sensitive networking (TSN), there are prior surveys on variations. Often ULL systems require very low jitter. Latency
topics that relate to TSN. We proceed to review these related and jitter are the two main quality of service (QoS) metrics
surveys and differentiate them from our survey. for ULL networking. We note that there are a wide range of
A survey on general techniques for reducing latencies in ULL applications with vastly different QoS requirements, see
Internet Protocol (IP) packet networks has been presented Table I. For instance, some industrial control applications have
in [17]. The survey [17] gave a broad overview of the very tight delay bounds, e.g., only a few microseconds, while
sources of latencies in IP networks and techniques for latency other industrial control applications have more relaxed delay
reduction that have appeared in publications up to August bounds up to a millisecond.
2014. The range of considered latency sources included the
network structure, e.g., aspects of content placement and
service architectures, the end point interactions, e.g., aspects of B. IEEE 802.1 Overview
transport initialization and secure session initialization, as well Before we delve into the standardization efforts of the
as the delays inside end hosts, e.g., operating system delays. IEEE Time-Sensitive Network (TSN) Task Group (TG), we
In contrast, we provide an up-to-date survey of the IEEE Time briefly explain the organizational structure of the IEEE 802.1
Sensitive Networking (TSN) standard for the link layer and the Working Group (WG). The 802.1 WG is chartered to develop
Deterministic Networking (DetNet) standard for the network and maintain standards and recommended practices in the
layer, and related research studies. Thus, in brief, whereas the following areas: 1) 802 LAN/MAN architecture, 2) inter-
survey [17] broadly covered all latency sources up to 2014, networking among 802 LANs, MANs, and other wide area
we comprehensively cover the link and network layer latency networks, 3) security, 4) 802 overall network management, and
reduction standards and studies up to July 2018. 5) protocol layers above the MAC and LLC layers. Currently,
A few surveys have examined specific protocol aspects that there are four active task groups in this WG: 1) Time Sensitive
relate to latency, e.g., time synchronization protocols have Networking, 2) Security, 3) Data Center Bridging, and 4)
been surveyed in [18], [19], routing protocols have been OmniRAN.
surveyed in [20]–[22], while congestion control protocols have The main IEEE 802.1 standard that has been continuously
been covered in [23], [24]. Several surveys have covered revised and updated over the years is IEEE 802.1Q-2014 [58],
latency reduction through mobile edge and fog computing, formally known as the IEEE 802.1D standard. That is, IEEE
e.g., see [25]–[28]. Also, the impact of wireless protocols 802.1Q-2014, which we abbreviate to IEEE 802.1Q, is the
on latency has been covered in a few surveys [29]–[35], main Bridges and Bridged Networks standard that has incor-
while smart grid communication has been covered in [36]. porated all 802.1Qxx amendments, where “xx” indicates the
Low-latency packet processing has been surveyed in [37], amendment to the previous version of 802.1Q.
while coding schemes have been surveyed in [38], [39]. A 1) IEEE 802.1 Bridge: IEEE 802.1Q extensively utilizes
comprehensive guide to stochastic network calculus, which can the terminology “IEEE 802.1 bridge”, which we abbreviate to
be employed to analyze network delays has appeared in [40]. “bridge”. A bridge is defined as any network entity within
Several surveys have covered the Tactile Internet an 802.1 enabled network that conforms to the mandatory
paradigm [2], [3], [13], [41], which strives for latencies or optional/recommended specifications outlined in the stan-
on the order of one millisecond. The AVB standard, which is dard, i.e., any network node that supports the IEEE 802.1Q
a predecessor to the IEEE TSN standards development was functionalities. IEEE 802.1Q details specifications for VLAN-
surveyed in [42], [43]. In contrast to these existing surveys aware bridges and bridged LAN networks. More specifically,
we provide a comprehensive up-to-date survey of the IEEE IEEE 802.1Q specifies the architectures and protocols for the
TSN standards development and the related research studies. communications between interconnected bridges (L2 nodes),
 3

TABLE I
E ND - TO - END LATENCY AND JITTER REQUIREMENTS FOR TYPICAL ULL APPLICATIONS

QoS Requirements
Area Application
Latencies Jitter
Medical [47]–[49] Tele-Surgery, Haptic Feedback 3–10 ms < 2 ms
0.2 µs–0.5 ms for netw. with 1 Gbit/s link speeds meet lat. req.
Indust. Automation, Control Syst.
Industry [50] 25 µs–2 ms for netw. with 100 Mbit/s link speeds meet lat. req.
Power Grid Sys. approx. 8ms few µs
Banking [51] High-Freq. Trading < 1 ms few µs
Avionics [52] AFDX Variants 1–128ms few µs
Adv. Driver. Assist. Sys. (ADAS) 100–250 µs few µs
Automotive [53]–[56] Power Train, Chassis Control < 10µs few µs
Traffic Efficiency & Safety < 5 ms few µs
Augmented Reality 7–20 ms few µs
Infotainment [57]
Prof. Audio/Video 2–50 ms < 100 µs

TABLE II Technology (EtherCAT). In general, the Ethernet definitions

IEEE 802.1 T RAFFIC C LASSES lack the following aspects for supporting ULL applications:
Priority Traffic Class i) Lack of QoS mechanisms to deliver packets in real time
0 Background for demanding applications, such as real time audio and
1 Best effort
2 Excellent effort video delivery.
3 Critical application ii) Lack of global timing information and synchronization in
4 “Video” < 100 ms latency and jitter network elements.
5 “Voice” < 10 ms latency and jitter
6 Internetwork control, e.g., OSPF, RIP iii) Lack of network management mechanisms, such as band-
7 Control Data Traffic (CDT), e.g., from IACSs width reservation mechanisms.
iv) Lack of policy enforcement mechanisms, such as packet
filtering to ensure a guaranteed QoS level for an end-user.
and the inter-process communication between the layers and Motivated by these Ethernet shortcomings, the Institute of
sublayers adjacent to the main 802.1 layer (L2). Electronics and Electrical Engineers (IEEE) and the Internet
2) 802.1Q Traffic Classes: The IEEE 802.1Q standard Engineering Task Force (IETF) have proposed new definitions
specifies traffic classes with corresponding priority values that to introduce deterministic network packet flow concepts. The
characterize the traffic class-based forwarding behavior, i.e, IEEE has pursued the Time Sensitive Networking (TSN)
the Class of Service (CoS) [58, Annex I]. Eight traffic classes standardization [62] focusing mainly on physical layer (layer
are specified in the 802.1Q standard, whereby the priority one, L1) and link layer (layer two, L2) techniques within the
level ranges from lowest priority (0) to highest priority (7), TSN task group in the IEEE 802.1 working group (WG).
as summarized in Table II. The IETF has formed the DETerministic NETwork (DETNET)
working group focusing on the network layer (L3) and higher
layer techniques.
C. General Development Steps from Ethernet Towards TSN
Ethernet has been widely adopted as a common mode III. IEEE TSN S TANDARDIZATION
of networking connectivity due to very simple connection This section surveys the standardization efforts of the IEEE
mechanisms and protocol operations. Since its inception in the 802.1 TSN TG. IEEE 802.1 TSN TG standards and protocols
1970s [59], [60] and first standardization in the IEEE 802.3 extend the traditional Ethernet data-link layer standards to
standard in 1983 [61], Ethernet has kept up with the “speed guarantee data transmission with bounded ultra-low latency,
race” and today’s Ethernet definitions support connections up low delay variation (jitter), and extremely low loss, which is
to 400 Gbps. Due to the ever increasing demands, there is ideal for industrial control and automotive applications [63],
an ongoing effort to advance Ethernet connectivity technolo- [64]. TSN can be deployed over Ethernet to achieve the
gies to reach speeds up to 1 Tbps. The best-effort Ethernet infrastructure and protocol capabilities for supporting real-time
service reduces the network complexity and keeps protocol Industrial Automation and Control System (IACS) applica-
operations simple, while driving down the product costs of tions.
Ethernet units. Despite the enormous successes and wide- In order to give a comprehensive survey of the current state
spread adoption of Ethernet, the Ethernet definitions funda- of the art of TSN standardization, we categorize the stan-
mentally lack deterministic quality of service (QoS) properties dardization efforts for the network infrastructure supporting
of end-to-end flows. Prior to the development of the TSN ULL applications. We have adopted a classification centered
standards, ULL applications, e.g., industrial communications, around the notion of the TSN flow, which is defined as follows.
deployed point-to-point communication and circuit switching An end-to-end unicast or multicast network connection from
or specialized/semi-proprietary specifications, such as, fieldbus one end station (talker, sender) to another end station(s)
communication, e.g., IEEE 1394 (FireWire), Process Field (listener(s), receiver(s)) through the time-sensitive capable
Network (Profinet), or Ethernet for Controlled Automation network is defined as a TSN flow, which we often abbreviate
 4

to “flow” and some publications refer to as “stream”. We have In general, gPTP systems consist of distributed and inter-
organized our survey of the standardized TSN mechanisms and connected gPTP and non-gPTP devices. Time-aware bridges
principles in terms of the TSN flow properties, as illustrated and end points are gPTP devices, while non-gPTP devices
in Fig.1. Complementarily to the taxonomy in Fig.1, Fig. 2 include passive and active devices that do not contribute
provides a historical perspective of the TSN standards and the to time synchronization in the distributed network. gPTP is
ongoing derivatives and revisions. a distributed protocol that uses a master-slave architecture
to synchronize real-time clocks in all devices in the gPTP
domain with the root reference (GM) clock. Synchronization is
A. Flow Concept: PCP and VLAN ID Flow Identification accomplished through a two-phase process: The gPTP devices
A TSN flow (data link flow) is characterized by the QoS 1) establish a master-slave hierarchy, and 2) apply clock
properties (e.g., bandwidth and latency) defined for the traffic synchronization operations. In particular, gPTP establishes a
class to which the flow belongs. In particular, a TSN flow is master-slave hierarchy using the BMCA [80], which consists
defined by the priority code point (PCP) field and VLAN ID of two separate algorithms, namely data set comparison and
(VID) within the 802.1Q VLAN tag in the Ethernet header. state decision. Each gPTP device operates a gPTP engine,
The PCP field and VID are assigned based on the application i.e., a gPTP state machine, and employs several gPTP UDP
associated with the flow. Fig. 3 outlines the general QoS IPv4 or IPv6 multicast and unicast messages to establish the
characteristics of the traffic classes related to the Informational appropriate hierarchy and to correctly synchronize time [65].
Technology (IT) and Operational Technology (OT) domains. Any non-time aware bridge that cannot relay or synchronize
Furthermore, Fig. 3 provides the main features for each block, timing messages does not participate in the BMCA clock
including typical applications used. As IT and OT establish spanning tree protocol.
a converged interconnected heterogeneous network, the delay The time synchronization accuracy depends mainly on the
bottleneck must be diminished to tolerable levels for IACS accuracy of the residence time and link delay measurements. In
applications, i.e., the machine and control floor networks. order to achieve high accuracy, 802.1AS time-aware systems
correct the received upstream neighbor master clock’s timing
information through the GM’s frequency ratio, this process is
B. Flow Synchronization called logical syntonization in the standard. In the synchroniza-
1) IEEE 802.1AS Time Synchronization for Time-Sensitive tion context, frequency refers to the clock oscillator frequency.
Applications: Many TSN standards are based on a network- The frequency ratio is the ratio of the local clock frequency
wide precise time synchronization, i.e., an established common to the frequency of the time-aware system at the other end
time reference that is shared by all TSN network entities. of an attached link. 802.1AS achieves proper synchronization
The time synchronization is, for instance, employed to deter- between time-aware bridges and end systems using both the
mine opportune data and control signaling scheduling. Time frequency ratio of the GM relative to the local clock to
synchronization is accomplished through the IEEE 802.1AS compute the synchronized time, and the frequency ratio of
stand-alone standard [65], [79], which uses a specialized the neighbor CM relative to the local CS to correct any
profile (selection of features/configuration) of IEEE 1588-2008 propagation time measurements.
(1588v2) [80], the generic Precision Time Protocol (gPTP). IEEE802.1AS-REV introduces new features needed for
The gPTP synchronizes clocks between network devices by time-sensitive applications. These features include the ability
passing relevant time event messages [18]. The message to support multiple time domains to allow rapid switchover
passing between the Clock Master (CM) and the Clock Slaves should a GM clock fail, and improved time measurement
(CSs) forms a time-aware network, also referred to as gPTP accuracy.
domain, as illustrated in Fig. 4. The time-aware network 2) Summary and Lessons Learned: IEEE 802.1AS pro-
utilizes the peer-path delay mechanism to compute both the vides reliable accurate network wide time synchronization.
residence time, i.e., the ingress-to-egress processing, queuing, All gPTP systems compute both the residence time and
and transmission time within a bridge, and the link latency, the link latency (propagation delay) and exchange messages
i.e., the single hop propagation delay between adjacent bridges along a hierarchical structure centered around the selected
within the time-aware network hierarchy with reference to the GM clock to accurately synchronize time. Flow control and
GrandMaster (GM) clock at the root of the hierarchy [65, management components, e.g., IEEE 802.1Qbv and 802.1Qcc
Section 11]. The GM clock is defined as the bridge with the (see Sections III-D and III-C), can utilize the 802.1AS tim-
most accurate clock source selected by the Best Master Clock ing synchronization to provide accurate bounded latency and
Algorithm (BMCA) [80]. extremely low loss and delay variation for TSN applications.
For example, in Fig. 4, the bottom left-most 802.1AS An open aspect of time synchronization is that the frequent
end point receives time information from the upstream CM periodic exchange of timing information between the individ-
which includes the cumulative time from the GM to the ual network entities can stress and induce backpressure on the
upstream CM. For full-duplex Ethernet LANs, the path delay control plane. The control plane load due to the time synchro-
measurement between the local CS and the direct CM peer nization can ultimately impact ULL applications. A centralized
is calculated and used to correct the received time. Upon time synchronization system, e.g., based on a design similar to
adjusting (correcting) the received time, the local clock should software defined networking (SDN) [81], [82], with message
be synchronized to the gPTP domain’s GM clock. exchanges only between a central synchronization controller
 5

TSN Standardization, Sec. III

? ? ? ? ?
Flow Concept, Flow Synchronization, Flow Management, Flow Control, Flow Integrity,
Sec. III-A Sec. III-B Sec. III-C Sec. III-D Sec. III-E
TSN Data-Link Flow Network Timing YANG Models Credit Based Shaping Frame Replication and
Flow Characterization IEEE 802.1AS [65] IEEE 802.1Qcp [66] IEEE 802.1Qav [70] Elimination
Flow Identification Stream Reservation Frame Scheduler IEEE 802.1CB [76]
IEEE 802.1Qat [67] IEEE 802.1Qbv [71] Path Control
IEEE 802.1Qcc [68] Frame Preemption IEEE 802.1Qca [77]
Link-Local Reservation IEEE 802.3br [72] Per-Stream Filtering
IEEE 802.1CS [69] IEEE 802.1Qbu [73] IEEE 802.1Qci [78]
Cyclic Queuing
IEEE 802.1Qch [74]
Asynchronous Shaping
IEEE 802.1Qcr [75]
Fig. 1. Classification taxonomy of Time Sensitive Networking (TSN) standardization.

2010 2016 2016 2017

2012
IEEE 802.1Qat IEEE 802.1AC IEEE 802.1AB IEEE 802.1CB (FRER)
standards published AVB TG renamed
superseded superseded 802.1AB-2009 standard published
SRP for distributed to TSN TG
802.1AC-2012 Station and Media
resource reservation to broaden 802.1Q bridge
2014 Access Control
2011 specs. to IACS apps. 2017
IEEE 802.1Q-2014 Connectivity
IEEE 802.1AS IEEE 802.1Qch (CQF)
Most recent LAN/MAN Discovery (LLDP)
standard published IEEE 802.1Qci (PSFP)
standard for bridging LANs
2005 time synchronization standards published
AVB TG started

Many
In
Progress
2011 2012 2013 2014 2015 2016 2017
2010 2018

2015
2016
2012 IEEE 802.1Qbv (TAS)
IEEE 802.1Qca (PCA) IEEE 802.1Qbu (Frame preemption)
IEEE 802.1AC-2012 standard published 2015 IEEE 802.1Qbz (Bridging
MAC service definition that abstracts MAC Internetworking TG IEEE 802.1Qcd (App-
specific TLV) enhancements
data transfer services from upper layers merged into TSN TG to 802.11) standards published
standards published

Fig. 2. Timeline of IEEE TSN task group (TG), highlighting significant milestones and depicting the shift from Audio Video Bridging (AVB) to TSN.

and individual network entities could help mitigate the control 802.1 bridges and bridge components, including Media Access
plane overhead. However, such a centralized synchronization Control (MAC) Bridges, Two-Port MAC Relays (TPMRs),
approach may create a single-point of failure in the time Customer Virtual Local Area Network (VLAN) Bridges, and
synchronization process. The detailed quantitative study of Provider Bridges [66]. Additionally, IEEE 802.1Qcp is used to
these tradeoffs is an interesting direction for future research. support other TSN standard specifications, such as the Security
and Datacenter Bridging TG standards 802.1AX and 802.1X.
C. Flow Management YANG [83], [84] is a data modeling language for config-
Flow management enables users or operators to dynamically uration data, state data, remote procedure calls, and notifi-
discover, configure, monitor, and report bridge and end station cations for network management protocols, e.g., NETCONF
capabilities. and RESTCONF. NETCONF is the Network Configuration
1) IEEE 802.1Qcp YANG Data Model: The TSN TG Protocol [85] that provides mechanisms to install, manage,
has proposed the IEEE 802.1Qcp TSN Configuration YANG and delete the configurations of network devices. The industry
model standard to achieve a truly universal Plug-and-Play wide adoption of the YANG formalized data modeling lan-
(uPnP) model. The IEEE 802.1Qcp standard utilizes the Uni- guage, e.g., by the IETF and the Metro Ethernet Forum (MEF),
fied Modeling Language (UML), specifically the YANG [83], is an important motivation for integrating, automating, and
[84] data model. The YANG data model provides a frame- providing support for YANG data modeling in 802.1 bridges
work for periodic status reporting as well as for configuring and related services for upper layer components.
 6

Machine Floor Control Floor Production Floor Enterprise Floor

 High cost pressure
Features

 Dynamic allocation of network  High precision time synch  Universal Time

 Devices with integrated switches
resources  Strict separation between end  Strict separation between end
 Isochronous working clock
 Seamless redundancy stations and network stations and network
 Seamless redundancy

OT Domain Edge IT IT Domain

Typical Applications

 Manufacturing Execution
System (MES)  Enterprise Resource Planning
 PLC & Embedded Control  Cyber-physical Systems (CPS)
 Supervisory control and data (ERP)
Systems  Machine-to-Machine (M2M)
acquisition (SCADA)  Enterprise Content
 Individual real-time applications  Multiple real-time applications
 Condition monitoring Management (ECM)
 High diversity of L3 protocols

Bandwidth: 100 M – 1 G 1G 1 G – 10 G 10 G
Latency: 31.25 us – 16 ms 200 ms – 1ms 100 ms – 1 s 100 ms – sec

Fig. 3. Illustration of the broad range of QoS requirements according to the network setting (floor), whereby the machine floor requires the highest level of
determinism and the lowest latency. Traditional networking is deployed on the enterprise floor. The top row summarizes the features required at each floor,
while the bottom row illustrates typical example applications. TSN can, in principle, be deployed everywhere, but typically, TSN is most attractive for the
real-time systems in the OT Domain, i.e., the machine and control floors.

802.1 A Endpoint protocol that was introduced by the AVB TG to ensure that the
TIA Clock CM: Clock Master
(Current Grand Master) CS: Clock Slave AVB talker is guaranteed adequate network resources along its
Source
CM
EPON
transmission path to the listener(s). This is accomplished using
Backbone
Networks

Link the Multiple Registration Protocol (MRP) [58, Section 10],

CS CS where the traffic streams are identified and registered using
802.1 AS Bridge 802.1 AS Bridge a 48-bit Extended Unique Identifier (EUI-48). The EUI-48
CM CM
EPON
is usually the MAC source address concatenated with a 16-
Link bit handle to differentiate different streams from the same
CS 802.1 CS source and is also referred to as StreamID. The SRP reserves
802.1 AS Bridge AS resources for a stream based on the bandwidth requirement
CM Bridge CM
802.3
and the latency traffic class using three signaling protocols,
Wireless
Eth. Link Link namely 1) the Multiple MAC Registration Protocol (MMRP),
CS CS CS 2) the Multiple VLAN Registration Protocol (MVRP), and 3)
802.1 AS 802.1 AS 802.1 AS
Endpoint Endpoint
the Multiple Stream Registration Protocol (MSRP) [58], [67,
Endpoint
Section 35].
Fig. 4. Illustration of a typical gPTP domain operation and time sharing where MMRP and MVRP control the group registration prop-
the selected GM source distributes timing information to all downstream agation and the VLAN membership (MAC address infor-
802.1AS bridges. Each bridge corrects the delay and propagates the timing
information on all downstream ports, eventually reaching the 802.1AS end mation [58, Sections 10 and 11]), while MSRP conducts
points (end stations). The International Atomic Time (TIA) is the GM’s source the distributed network resource reservation across bridges
for timing information. and end stations. MSRP registers and advertises data stream
characteristics and reserves bridge resources to provide the
appropriate QoS guarantees according to the talker’s declared
2) IEEE 802.1Qat Stream Reservation Protocol (SRP) and propagation attributes, which include the SRP parameters that
IEEE 802.1Qcc Enhancements to SRP and Centralization are sent by the end station in MSRP PDUs (MSRPDUs).
Management: The IEEE 802.1Qat Stream Reservation Proto- A station (talker) sends a reservation request with the MRP,
col (SRP) [67], which has been merged into 802.1Q, provides i.e., the general MRP application which registers the stream
a fundamental part of TSN. In particular, IEEE 802.1Qat speci- resource reservation. The 802.1 TSN TG has developed the
fies the admission control framework for admitting or rejecting MRP Attribute Declaration (MAD) for describing the request
flows based on flow resource requirements and the available based on the stream characteristics. All participants in the
network resources. Moreover, IEEE 802.1Qat specifies the stream have an MSRP application and MAD specification and
framework for reserving network resources and advertising each bridge within the same SRP domain can map, allocate,
streams in packet switched networks over full-duplex Ethernet and forward the stream with the necessary resources using the
links. Most of the standards that use priorities, frame schedul- MRP attribute propagation (MAP) [67]. Fig. 5 illustrates the
ing, and traffic shaping protocols depend on SRP [67], since MRP architecture.
these protocols work correctly only if the network resources In essence, the SRP protocol ensures QoS constraints for
are available along the entire path from the sender (talker) each stream through the following steps:
to the receivers (listeners). IEEE 802.1Qat is a distributed 1) Advertise stream
 7

Centralized User
Configuration
(CUC)
Bridge End Station User/Network
MRP Participant MRP Participant MRP Participant Configuration Info
Centralized Network
MRP MRP MRP Configuration
Application Application Application
(CNC)
MAP
MAD MAD MAD
User/ Management
Network User/
LLC LLC LLC Configuration Network
MAC Relay Info Configuration
Entity Info
Frame Frame Frame Frame Frame Frame
Rx Tx Rx Tx Rx Tx
Talker
MRP: Multiple Registration Protocol Listener
(Transmitter)
MAD: MRP Attribute Declaration (Receiver)
LLC: Logical Link Control
Fig. 6. Illustration of Centralized Network Configuration (CNC): End stations
Fig. 5. Illustration of Multiple Registration Protocol (MRP) architecture: interact with the network entities via the User-Network Interface (UNI).
Each end station (illustrated on the right) declares the propagation attributes The CNC receives the requests, e.g., flow reservation requests, and provides
using the MRP Attribute Declaration (MAD) and the MRP Applications corresponding management functions. An optional CUC provides delay-
encapsulated as an MRP participant which gives end stations the ability to optimized configuration, e.g., for closed-loop IACS applications. The solid
register resources. The MRP participant entry is stored in bridges and mapped arrows represent the protocol, e.g., YANG or TLV, that is used as the UNI for
between all required ports using MRP Attribute Propagation (MAP). A bridge exchanging configuration information between Talkers/Listeners (users) and
mapping between two different interfaces in the LAN is illustrated on the left. Bridges (network). The dashed arrows represent the protocol, e.g., YANG or
TLV, that transfers configuration information between edge bridges and the
CNC.

2) Register paths of stream

3) Calculate worst-case latency
ration model of the original SRP protocol, i.e., allows for
4) Establish an AVB domain
centrally managed systems to coexist with decentralized ad-
5) Reserve the bandwidth for the stream.
hoc systems. In addition, 802.1Qcc supports a “hybrid” con-
Since the existing IEEE 802.1Qat (802.1Q Section 35) SRP figuration model, allowing a migration path for legacy AVB
features a decentralized registration and reservation proce- devices. This hybrid configuration management scheme when
dure, any changes or new requests for registrations or de- coupled with IEEE 802.1Qca Path Control and Reservation
registrations can overwhelm the network and result in intol- (PCR) (see Section III-E2) and the TSN shapers can provide
erable delays for critical traffic classes. Therefore, the TSN deterministic end-to-end delay and zero congestion loss.
TG has introduced the IEEE 802.1Qcc standard to improve
3) IEEE 802.1CS Link-Local Reservation Protocol (LRP):
the existing SRP by reducing the size and frequency of
To effectively achieve tight bounds on latency and zero con-
reservation messages, i.e., relaxing timers so that updates are
gestion loss, traffic streams need to utilize effective admission
only triggered by link state or reservation changes.
control policies and secure resource registration mechanisms,
Additionally, IEEE 802.1Qcc [68] provides a set of tools
such as the SRP [67] and the SRP enhancements and manage-
to manage and control the network globally. In particular,
ment standard [68]. While the MRP [58, Section 10] provides
IEEE 802.1Qcc enhances the existing SRP with a User
efficient methods for registering streams; the database holding
Network Interface (UNI) which is supplemented by a Cen-
the stream state information, is limited to about 1500 bytes. As
tralized Network Configuration (CNC) node, as shown in
more traffic streams coexist and the network scale increases,
Fig. 6. The UNI provides a common method of requesting
MRP slows significantly as the database proportionally in-
layer 2 services. Furthermore, the CNC interacts with the
creases which results in frequent cyclic exchanges through the
UNI to provide a centralized means for performing resource
MAD between all bridge neighbors.
reservation, scheduling, and other types of configuration via
a remote management protocol, such as NETCONF [85] or The Link-Local Reservation Protocol (LRP) [69] has been
RESTCONF [86]; hence, 802.1Qcc is compatible with the introduced by the 802.1 TSN TG to efficiently replicate an
IETF YANG/NETCONF data modeling language. MRP database between two ends of a point-to-point link and to
For a fully centralized network, an optional Centralized User incrementally replicate changes as bridges report new network
Configuration (CUC) node communicates with the CNC via developments or conditions. Additionally, the LRP provides
a standard Application Programming Interface (API), and can a purging process that deletes replicated databases when the
be used to discover end stations, retrieve end station capa- source of such databases remains unresponsive or the data gets
bilities and user requirements, and configure delay-optimized stale. Furthermore, the LRP is optimized to efficiently handle
TSN features in end stations (mainly for closed-loop IACS databases on the order of 1 Mbyte.
applications). The interactions with higher level reservation While MRP is considered application specific, i.e., the MRP
protocols, e.g., RSVP, are seamless, similar to how the AVB operations are defined by each registered application, LRP
Transport Protocol IEEE 1722.1 [87] leverages the existing is an application neutral transport protocol. Fig 7 illustrates
SRP. the LRP protocol architecture operating within bridges or end
802.1Qcc [68] still supports the fully distributed configu- points.
 8

Bridge End Station plane overheads. A centralized scheme can benefit from SDN
implementation and management but could result in new
LRP Application X LRP Application X
LRP Application Y
infrastructure cost for the operators. Nevertheless, the choice
LRP Application Y
LRP Application Z LRP Application Z of deployments can be based on the relative performance
LRP LRP LRP
levels among centralized and distributed nodes, as well as
the use of existing infrastructures and the deployment of new
LLC LLC LLC
infrastructures. Future research needs to thoroughly examine
these tradeoffs.
MAC Relay
Entity Another important future research direction is to examine
Frame Frame Frame Frame Frame Frame
Rx Tx Rx Tx Rx Tx
predictive models that estimate the resource reservation re-
quirements in bridges. Estimations may help in effectively
LRP: Link-Local Reservation Protocol managing queues and scheduling while efficiently utilizing the
LLC: Logical Link Control
network resources.
Fig. 7. Illustration of LRP Architecture: A Link-Local Reservation Protocol
(LRP) instance (illustrated by the blue LRP box) interacts with each appli-
cation and provides a generic transport service for multiple registered LRP D. Flow Control
applications, which are represented by yellow colored boxes near the top of
the illustration. Flow control specifies how frames belonging to a prescribed
traffic class are handled within TSN enabled bridges.
1) IEEE 802.1Qav Forwarding and Queuing of Time-
4) Resource Allocation Protocol (RAP)—Towards a Dis- Sensitive Streams: IEEE 802.1Qav specifies Forwarding and
tributed TSN Control Model: Although the SRP and the Queuing of Time Sensitive Streams (FQTSS), which has
related MSRP (MSRPv1 [68]) were designed for distributed been incorporated into 802.1Q. IEEE 802.1Qav serves as a
stream configuration (including registration, reservation, and major enhancement to the forwarding and queuing operation
provisioning), SRP is generally restricted to A/V applications in traditional Ethernet networks. IEEE 802.1Qav specifies
with a limited number of Stream Reservation (SR) classes, bridge operations that provide guarantees for time-sensitive
e.g., classes A and B for the Credit Based Shaper (CBS), (i.e., bounded latency and jitter), lossless real-time audio/video
see Section III-D1. SRP guarantees the QoS characterized (A/V) traffic [70]. The IEEE 802.1Qav standard [58], [70,
by each stream through the reservation in conjunction with Section 34], details flow control operations, such as per
shaper mechanisms, see Section III-D. IEEE 802.1Qcc pushed priority ingress metering and timing-aware queue draining
for more centralized configuration models, where all the algorithms.
newly established TSN features, e.g., shaping, preemption, IEEE 802.1Qav was developed to limit the amount of A/V
and redundancy, are supported through the CNC configuration traffic buffering at the downstream receiving bridges and/or
model. Any distributed model is currently restricted to CBS. end stations. Increasing proportions of bursty multimedia
The Resource Allocation Protocol (RAP) [88] leverages traffic can lead to extensive buffering of multimedia traffic, po-
the LRP to propagate TSN stream configuration frames that tentially resulting in buffer overflows and packet drops. Packet
include resource reservation and registration information in drops may trigger retransmissions, which increase delays,
a manner similar to MSRP. The MSRP (and MSRPv1) is rendering the re-transmitted packets obsolete and diminishing
geared towards AVB systems, while RAP is defined for TSN the Quality of Experience (QoE).
enabled systems for distributed stream configuration. The RAP IEEE 802.1Qav limits the amount of buffering required in
promises to improve scalability (through LRP), to support all the receiving station through the Stream Reservation Protocol
TSN features, to improve performance under high utilization, (SRP) [67] in conjunction with a credit-based shaper (CBS).
and to enhance diagnostic capabilities. The CBS spaces out the A/V frames to reduce bursting and
5) Summary and Lessons Learned: Flow management al- bunching. This spacing out of A/V frames protects best-effort
lows distributed (legacy SRP and RAP) as well as centralized traffic as the maximum AVB stream burst is limited. The
(802.1Qcc and 802.1CS) provisioning and management of spacing out of A/V frames also protects the AVB traffic
network resources, effectively creating protected channels over by limiting the back-to-back AVB stream bursts, which can
shared heterogeneous networks. Moreover, flow management interfere and cause congestion in the downstream bridge.
offers users and administrators Operations, Administration, The CBS shaper separates a queue into two traffic classes,
Maintenance (OAM) functions to monitor, report, and con- class A (tight delay bound) and class B (loose delay bound).
figure (802.1Qcp and 802.1Qcc) network conditions. This Each class queue operates according to the throttling mech-
allows for fine-grained support of network services while anism illustrated in Fig. 8. When no frame is available in
enforcing long term allocations of network resources with the queue, the credit for the queue is set to zero. A queue
flexible resource control through adaptive and automatic re- is eligible for transmission if the credit is non-negative. The
configurations. credit is increased by idleSlope when there is at least one frame
However, both centralized and distributed flow management in the queue, and decreased by sendSlope when a frame is
models have specific deployment advantages and disadvan- transmitted. The idleSlope is the actual bandwidth reserved (in
tages. For example, a centralized entity presents a single point bits per second) for the specific queue and traffic class within
of failure, whereas, distributed schemes incur extensive control a bridge [58, Section 34], while the sendSlope is the port
 9

CBS Queue Switching Fabric

Queue 0 Queue 1 Queue 2 Queue 7

Queued No Synchronized
Set Credit to 0 IPC or Audio/
Frames Best Effort Best Effort Clock
Control Video
Traffic Traffic
Traffic Traffic
Yes Gate Control List
T0: 00111111
T1:01000000
Credit ≥ 0 T2:10000000
No T3:00111111
Accumulate Yes T4: 01000000
Decrease
Credit Enhanced Transmission .
SendSlope Strict Priority Credit Based .
Increase Yes Transmission Selection
Channel Send Scheduling Shaper .
IdleSlope Selection Algorithm
Not Busy Frame T125:10000000
Time-Aware Time-Aware Time-Aware Time-Aware T126: REPEAT
Gate Gate Gate Gate
Fig. 8. Flow-chart illustration of the Credit-Based Shaper (CBS) operation Gate Control
for a given queue. A queue is permitted to transmit if both credits are greater Entity
than or equal to zero, and the channel is vacant. Transmission Selection

Transmissions are controlled Fig. 10. IEEE 802.1Qbv: Illustration of egress hardware queue with 8
by the Gate Control Lists (GCL) software queues, each with its unique transmission selection algorithm. The
transmissions are controlled by the Gate Controlled List (GCL) with multiple
Guard Guard
Gate Control Entries (GCEs) that determine which software queues are open.
band band
For instance, in time interval T0, the gates for queues 2 through 7 are open, and
Periodic Periodic the transmission selection at the bottom arbitrates access to the medium [58,
Best Effort (LP) Best Effort (LP)
Traffic (HP) Traffic (HP) Section 8.6.8]. In time interval T1, the gate opens for AV traffic from Queue 1,
and a credit based shaper (CBS) regulates the frame transmissions from
Queue 1. In time interval T2, the gate opens for Queue 0 and strict priority
Common Scheduled Traffic Common Scheduled Traffic scheduling selects the frames to transmit from Queue 0.
Slot Window Slot Window

Fig. 9. IEEE 802.1Qbv Time-Aware Shaper (TAS) [71]: Scheduled traffic Periodic Non-Preemptive
Guard band Best Effort (LP)
is sent over synchronized Time-Division Multiplexing “windows” within the Traffic (HP) Frame Transmission
Ethernet traffic. Yellow marked frames are time-sensitive high priority (HP) Preemption
traffic that have guaranteed reserved resources across the network, while the Overhead
blue frames correspond to best-effort low priority (LP) traffic.
Best Effort Periodic Best Effort Preemptive
Reduced (LP) Traffic (HP) (LP) Cont. Frame Transmission
Guard band
transmit rate (in bits per second) that the underlaying MAC
service supports. Furthermore, two key limiting parameters Scheduled
Traffic Window
are defined: i) hiCredit and ii) loCredit, which are functions
of the maximum frame size (in the case of loCredit) and Fig. 11. The IEEE 802.1Qbv transmission selection prevents low priority
(best effort) frames from starting transmission if the transmission cannot be
maximum interference size (in the case of hiCredit), the idleS- completed by the start of the scheduled traffic window. This transmission
lope/sendSlope (respectively), and the maximum port transmit selection essentially enforces a guard band (sized as a maximum size frame)
rate. Further details can be found in [70, Annex L]. The CBS to protect the scheduled traffic window. With preemption (IEEE 802.3br, IEEE
802.1Qbu) the guard band can be reduced to the smallest Ethernet frame
throttles each shaped traffic class to not exceed their precon- fragment.
figured bandwidth limits (75% of maximum bandwidth due
to bandwidth intensive applications, e.g., audio and video [70,
Section 34.3.1]). The CBS in combination with the SRP is 802.1Qav shortcomings, the TSN task group proposed a new
intended to bound delays to under 250 µs per bridge [67]. traffic shaper, namely the IEEE 802.1Qbv Time-Aware Shaper
Overall, the IEEE 802.1Qav Ethernet AVB standard guarantees (TAS) [71] along with the IEEE 802.1Qbu frame preemption
worst-case latencies under 2 ms for class A and under 50 ms technique [73] to provide fine-grained QoS [90]. The TAS
for class B up to seven network hops [70]. and frame preemption mechanisms are suitable for traffic with
However, some key CBS disadvantages are that the average deterministic end-to-end ULL requirements, e.g., for critical
delay is increased and that the delay can be up to 250 µs control or Interprocess Communication (IPC) traffic, with sub-
per hop, which may be too high for industrial control applica- microseconds latency requirements. In particular, the TAS
tions [89]. Also, CBS struggles to maintain delay guarantees schedules critical traffic streams in time-triggered windows,
at high link utilizations. which are also referred to as protected traffic windows or as
In order to address the CBS shortcomings, the TSN TG has time-aware traffic windows. Thus, TAS follows the TDMA
introduced other standards, e.g., IEEE 802.1Qbv, 802.1Qch, paradigm, similar to Flexible Time-Triggered Ethernet (FTT-
and 802.1Qcr, which are reviewed in the following subsec- E) [91], [92], whereby each window has an allotted trans-
tions. Also, addressing the CBS shortcomings is an active mission time as shown in Fig. 9. In order to prevent lower
research area, see Section IV-C. priority traffic, e.g., best effort traffic, from interfering with
2) IEEE 802.1Qbv Enhancements to Traffic Scheduling: the scheduled traffic transmissions, scheduled traffic windows
Time-Aware Shaper (TAS): As a response to the IEEE are preceded by a so-called guard band.
 10

TAS is applicable for ULL requirements but needs to have Queuing Frames
all time-triggered windows synchronized, i.e., all bridges from Transmission Transmission
sender to receiver must be synchronized in time. TAS utilizes a Selection Selection
gate driver mechanism that opens/closes according to a known MAC Control MAC Control
and agreed upon time schedule, as shown in Fig. 10, for each

802.3br
Express MAC Preemptable

(IET)
port in a bridge. In particular, the Gate Control List (GCL) in (eMAC) MAC (pMAC)
Fig. 10 represents Gate Control Entries (GCEs), i.e., a 1 or 0
MAC Merge Sublayer
for open or close for each queue, respectively. The frames of
a given egress queue are eligible for transmission according to PHY (unaware of preemption)
the GCL, which is synchronized in time through the 802.1AS Fig. 12. Illustration of the layering for the Ethernet MAC Merge Sublayer:
time reference. The GCL is executed in periodically repeating The MAC Merge Sublayer provides a Reconciliation Sublayer (RS) service
cycle times, whereby the each cycle time contains one GCL for pMAC and eMAC frames. The RS service supports two main ways to
hold the transmission of a pMAC frame in the presence of an eMAC frame:
execution. Within a cycle time, the time period during which By preempting (interrupting) the pMAC frame transmission, or by preventing
a gate is open is referred to as the time-aware traffic window. the start of the pMAC frame transmission.
Frames are transmitted according to the GCL and transmission
selection decisions, as illustrated in Fig. 10. Each individual
software queue has its own transmission selection algorithm, that are in transit, i.e., they are holding on to the resource
e.g., strict priority queuing (which is the default). Overall, the (transmission medium), can be preempted by express frames.
IEEE 802.1Qbv transmission selection at the bottom of Fig. 10 After the transmission of an express frame has completed, the
transmits a frame from a given queue with an open gate if: transmission of the preempted frame can resume.
(i) The queue contains a frame ready for transmission, (ii) With preemption, the guard band in Fig. 9 can be reduced
higher priority traffic class queues with an open gate do not to the transmission time of the shortest low priority frame
have a frame to transmit, and (iii) the frame transmission fragment. Thus, in the worst case, the transmission of the low
can be completed before the gate closes for the given queue. priority frame fragment can be completed before starting the
Note that these transmission selection conditions ensure that transmission of the next high priority frame. The transmission
low priority traffic is allowed to start transmission only if the of the leftover fragmented frame can then be resumed to
transmission will be completed by the start of the scheduled completion. Note that this preemption occurs only at the
traffic window for high priority traffic. Thus, this transmission link-level, and any fragmented frame is reassembled at the
selection effectively enforces a “guard band” to prevent low MAC interfaces. Hence the switches process internally only
priority traffic from interfering with high priority traffic, as complete frames. That is, any frame fragments transmitted
illustrated in Fig 11. over a physical link to the next bridge are re-assembled in
One critical TAS shortcoming is that some delay is incurred the link layer interface; specifically, the MAC merge sublayer
due to additional sampling delay, i.e., due the waiting time (see Fig. 12) in the link layer of the next bridge, and the next
until the next time-triggered window commences. This sam- bridge then only processes complete frames. Each preemption
pling delay arises when unsynchronized data is passed from an operation causes some computational overhead due to the
end-point to the network. Task and message scheduling in end- encapsulation processing by the bridge to suspend the current
nodes would need to be coupled with the TAS gate scheduling fragment and to transition the operational context to the
in the networks in order to achieve the lowest latencies. express traffic frame and vice versa, which is illustrated in
Moreover, synchronizing TSN bridges, frame selections, and Fig. 11. Note that this overhead occurs only in layer 2 in the
transmission times across the network is nontrivial in moder- link interface.
ately sized networks, and requires a fully managed network. 4) IEEE 802.1Qch Cyclic Queuing and Forwarding (CQF):
Also, the efficient use of bandwidth with TAS needs to be While the IEEE 802.1Qav FQTSS with CBS works well
thoroughly examined. Overall, TAS has high configuration for soft real-time constraints, e.g., A/V traffic, the existing
complexity. Future research needs to carefully examine the FQTSS has still several shortcomings, including, i) patholog-
scalability to large networks, runtime reconfiguration, and the ical topologies can result in increased delay, and ii) worst-
integration of independently developed sub-systems. case delays are topology dependent, and not only hop count
3) IEEE 802.3br and 802.1Qbu Interspersing Express Traf- dependent, thus buffer requirements in switches are topology
fic (IET) and Frame Preemption: To address the ULL la- dependent. The TSN TG introduced Cyclic Queuing and
tency requirements and the inverted priority problem, i.e., Forwarding (CQF) [74], also known as the Peristaltic Shaper
the problem that an ongoing transmission of a low priority (PS), as a method to synchronize enqueue and dequeue op-
frame prevents the transmission of high priority frames, the erations. The synchronized operations effectively allow LAN
802.1 TG along with the 802.3 TG introduced frame preemp- bridges to synchronize their frame transmissions in a cyclic
tion (802.1Qbu and 802.3br) [72], [73]. Frame preemption manner, achieving zero congestion loss and bounded latency,
separates a given bridge egress port into two MAC service independently of the network topology.
interfaces, namely preemptable MAC (pMAC) and express Suppose that all bridges have synchronized time, i.e., all
MAC (eMAC), as illustrated in Fig. 12. A frame preemption bridges are 802.1AS enabled bridges, and suppose for sim-
status table maps frames to either pMAC or eMAC; by plicity of the discussions that wire lengths and propagation
default all frames are mapped to eMAC. Preemptable frames times are negligible. Then, time sensitive streams are sched-
 11

Even Cycle Odd Cycle

which is based on the urgency-based scheduler (UBS) [89],
Interfering Traffic High Priority [93]. The ATS aims to smoothen traffic patterns by reshaping
Bridge 1 Interfering Traffic (LP)
(LP) Traffic
TSN streams per hop, implementing per-flow queues, and
prioritizing urgent traffic over relaxed traffic. The ATS operates
Bridge 2 Interfering Traffic (LP)
High Priority
Traffic
asynchronously, i.e., bridges and end points need not be
synchronized in time. Thus, ATS can utilize the bandwidth
efficiently even when operating under high link utilization with
Fig. 13. Illustration of Cyclic Queuing and Forwarding (CQF) without mixed traffic loads, i.e., both periodic and sporadic traffic.
preemption for a linear network: Each High Priority (HP) traffic frame The UBS is based on the Rate-Controlled Service Disci-
scheduled on a cycle (even or odd) is scheduled to be received at the next
bridge in the next cycle, whereby the worst-case HP frame delay can be two plines (RCSDs) [94]. RCSDs are a non-work conserving class
cycle times. In the illustrated example, the HP traffic is delayed due to low of packet service disciplines which includes Rate-Controlled
priority interfering traffic, but still meets the two cycle time delay bound. Static Priority [95] and Rate-Controlled Earliest Deadline
Preemption
First [96]. The RCSD packet scheduling consist of two compo-
Even Cycle Odd Cycle
Overhead nents: the rate controller implements the rate-control policies,
High Priority Best Effort (LP)
and the scheduler implements the packet scheduling according
Bridge 1
Traffic (preempted) to some scheduling policy, e.g., Static-Priority, First-Come-
First-Serve, or Earliest Due-Date First. By separating the rate
High Priority Best Effort (LP) controller and scheduler, the RCSD effectively decouples the
Bridge 2 Traffic (preempted)
Reduced
bandwidth for each stream from its delay bound, i.e., allocating
Guard Band a prescribed amount of bandwidth to an individual stream is
independent of the delay bound. Hence, RCSD can support
Fig. 14. Illustration of CQF with preemption for a linear network: A Guard
Band (GB) before the start of the cycle prevents any interfering (LP) traffic low delay and low bandwidth streams.
from affecting the High Priority (HP) traffic. The CQF without preemption in UBS adds a few improvements to RCSDs [94], namely:
Fig. 13 did not prevent the LP traffic from interfering with HP traffic, while 1) UBS provides low and predictable worst-case delays even
the CQF with preemption prevented the LP traffic from interfering with HP
traffic. Thus, preemption can improve the performance for HP traffic. at high link utilization, 2) low implementation complexity
due to the separation of per-flow queues from per-flow states
where flow state information, such as Head-of-Queue frame
uled (enqueued and dequeued) at each time interval or cycle and time stamp, is stored, and 3) independence from the global
time with a worst-case deterministic delay of two times the reference time synchronization; specifically, individual flow
cycle time between the sender (talker) and the downstream delays are analyzed at each hop, i.e., per-hop delay calculation,
intermediate receiver, as illustrated in Fig. 13. In essence, the and end-to-end delays are calculated based on the network
network transit latency of a frame is completely characterized topology and by the closed-form composition of the per-hop
by the cycle time and the number of hops. Therefore, the frame delays calculated initially.
latency is completely independent of the topology parameters The fundamental aim of the RCSD is to individually control
and other non-TSN traffic. frame selection and transmission at each hop between the
CQF can be combined with frame preemption specified in transmitter and receiver, i.e., per hop shaping. As pointed
IEEE 802.3Qbu, to reduce the cycle time from the transmission out by Specht et al. [89], the RCSD has multiple scalability
time of a full size frame to the transmission time of a minimum problems, including dynamic reordering of packets within
size frame fragment (plus all the TSN traffic), as illustrated separate queues according to the packets’ eligibility times, i.e.,
in Fig. 14. Note however that for CQF to work correctly, all priority queue implementation with non-constant complex data
frames must be kept to their allotted cycles, i.e., all transmitted structures, such as heaps. Specialized calender queues have
frames must be received during the expected cycle at the been proposed to achieve constant complexity [89]. However,
receiving downstream intermediate bridge [74]. Therefore, the calender queues require RCSD capable switches to have large
cycle times, the alignment of the cycle times among the memory pools, are difficult to control as the network size
bridges in the network, and the timing of the first and last scales up, and are ideal only for some specific applications
transmissions within a cycle need to be carefully considered with special properties. Therefore, Specht et al. [89] utilize
in order to ensure that the desired latency bounds are achieved. the RCSD concept with the outlined improvements and have
To this end, CQF in conjunction with IEEE 802.1Qci ingress proposed a novel UBS solution as the core of the ATS
policing and the IEEE 802.1Qbv TAS ensures that all frames standard.
are kept within a deterministic delay and guaranteed to be 6) Summary and Lessons Learned: Flow control mainly
transmitted within their allotted cycle time. enforces rules to efficiently forward and appropriately queue
5) IEEE 802.1Qcr Asynchronous Traffic Shaping (ATS): frames according to their associated traffic classes. All existing
While CQF and TAS provide ULL for critical traffic, they flow controls follow similar principles, namely, certain priv-
depend on network-wide coordinated time and, importantly, ileges are associated with TSN flows while non-TSN flows
due to the enforced packet transmission at forced periodic are delayed. Nearly all existing schedulers and shapers en-
cycles, they utilize network bandwidth inefficiently [89]. To force fair transmission opportunities according to each flow’s
overcome these shortcomings, the TSN TG has proposed traffic class. The transmission selection algorithm selects the
the IEEE 802.1Qcr Asynchronous Traffic Shaper (ATS) [75], appropriate stream within a given traffic class according to the
 12

network and traffic conditions. Flow control collaborates with

flow management, see Section III-C, and flow integrity, see
Disjoint Path A
Section III-E, to ensure adequate resources are available for
TSN streams. Packet 1 Packet 2 Packet 1 Packet 2 Packet 1 Packet 2
Overall, we can classify real-time TSN systems into event- Lost

triggered systems and time-triggered systems. For exam- Frame Flow Disjoint Path B Frame Flow
ple, IEEE 802.1Qbv is a time-triggered shaper, while IEEE Replication Packet 1 Packet 2 Elimination
802.1Qcr is an event-triggered shaper. An interesting future Lost
research direction is to explore whether both types of shapers
can be combined. That is, would it be efficient to dynamically
Fig. 15. Illustration of FRER operation: The first bridge replicates the frame
change a flow’s priority, individually or collectively, and to and transmits the duplicated frames on two disjoint paths. The FRER operation
reshape flows based on neighbor network conditions while can be started and ended at any bridge between the sender and receiver.
each flow is shaped by a centralized computed schedule incor-
porating time slots at each egress’s port? For example, a stream
initially sent with a certain high priority can be downgraded to and other loss-tolerant traffic is transmitted normally. FRER
low priority based on downstream network conditions while is compatible with industrial fault-tolerance architectures, e.g.,
adhering to each bridge’s time-aware scheduler and gating High Availability and Seamless Redundancy (HSR) [97] and
mechanism. the Parallel Redundancy Protocol (PRP) [98]. We note that
Also, it will be interesting to investigate whether frame duplication, routing, and elimination are non-trivial
IEEE 802.1Qbv can be replaced with an event-triggered shaper tasks that will likely require centralized management. Hence,
that guarantees an upper bound on latency, but not generally such protocols can be combined with other standards, e.g.,
a deterministic latency. Changing TAS into an event-triggered 802.1Qcc and 802.1Qca, to ensure seamless redundancy and
shaper can lead to more flexible and easily computed schedules fast recovery in time-sensitive networks.
since certain events, e.g., incoming frames or network changes, 2) IEEE 802.1Qca Path Control and Reservation (PCR):
can require schedule changes at runtime. IEEE 802.1Qca Path Control and Reservation (PCR) is based
on and specifies TLV extensions to the IETF Link State
Protocol (LSP), the Intermediate Station to Intermediate Sta-
E. Flow Integrity tion (IS-IS) protocol [99]. IEEE 802.1Qca allows the IS-IS
To accomplish the goals of deterministic ultra-low latency, protocol to control bridged networks beyond the capabilities
jitter, and packet loss, TSN streams need to deliver their frames of shortest path routing (ISIS-SPB) [58], [100, Section 28],
regardless of the dynamic network conditions, including phys- configuring multiple paths through the network [77], [101].
ical breakage and link failures. Several techniques have been IEEE 802.1Qca PCR aims to integrate control protocols
standardized to enable flow integrity. required to provide explicit forwarding path control, i.e.,
1) IEEE 802.1CB Frame Replication and Elimination for predefined protected path set-up in advance for each stream,
Reliability (FRER): IEEE 802.1CB Frame Replication and bandwidth reservation, data flow redundancy (both protection
Elimination for Reliability (FRER) [76], is a stand-alone and restoration), and distribution of control parameters for flow
standard that ensures robust and reliable communication using synchronization and flow control messages [77].
proactive measures for applications that are intolerant to packet In general, 802.1Qca specifies bridging on explicit paths
losses, such as control applications. 802.1CB FRER minimizes (EPs) for unicast and multicast frame transmission, and pro-
the impact of congestion and faults, such as cable breakages, tocols to determine multiple active topologies, e.g., Shortest
by sending duplicate copies of critical traffic across disjoint Path, Equal Cost Tree (ECT), Internal Spanning Tree (IST),
network paths, as shown in Fig. 15. If both frames reach their Multiple Spanning Tree Instance (MSTI), and Explicit Tree
destination, the duplicate copy is eliminated. If one copy fails (ET), in a bridged network. Explicit forwarding paths, as op-
to reach its destination, the duplicate message can still be posed to hop-by-hop forwarding, mitigate disruptions caused
received, effectively providing seamless proactive redundancy by the reconvergence of bridging protocols. PCR has similar
at the cost of additional network resources. goals and evolved from spanning tree protocols, e.g., the
In order to minimize network congestion, the packet repli- Rapid Spanning Tree Protocol (RSTP) [58, Section 13.4], the
cation can be selected based on traffic class and the path Multiple Spanning Tree Protocol (MSTP) [58, Section 13.5],
information acquired through the TSN stream identification and the Shortest Path Bridging (SPB) [58, Section 27].
(stream handle), plus a sequence generation function. The The IEEE 802.1Qca standard is based on Shortest Path
sequence generation function generates identification numbers Bridging (SPB) [58, Section 27] and incorporates a Soft-
for replicated frames to determine which frames to discard and ware Defined Networking (SDN) hybrid approach [77]. In
which frames to pass on so as to ensure correct frame recovery the hybrid approach, the IS-IS protocol in the data plane
and merging. The frame redundancy information is carried in handles basic functions, e.g., topology discovery and default
a Redundancy Tag [76]. Frame sequence numbers and timing path computation, while the SDN controller [102] in the
information are also needed to limit the memory needed for control plane manages the Explicit Paths (EPs), as shown in
duplicate frame detection and elimination. For example, FRER Fig. 16. In particular, the controller utilizes dedicated path
may only be employed for critical traffic, while best effort computation server nodes called Path Computation Elements
 13

CB – Core Bridge IS-IS process, see Section III-D4. The PSFP flow metering enforces
EB – Edge Bridge Shortest Path Tree predefined bandwidth profiles for streams. The metering may,
for instance, enforce prescribed maximum information rates
and burst sizes.
CB1 EB2 4) Summary and Lessons Learned: Flow integrity provides
PCE
Controller path redundancy, multi-path selection, as well as queue filter-
EB1 ing and policing. Flow integrity also prevents unauthorized or
EB3 mismanaged and rogue streams on bridged LAN networks.
CB2
In general, as network devices improve in terms of hard-
Fig. 16. Illustration of Explicit Paths (EPs): A control plane PCE SDN ware performance, they can be equipped with more state
controller installs computed Explicit Tree (ET) paths via the IS-IS data plane.
Two computed ET paths are shown represented by the green and blue lines. information within the core network. The increased state
information allows for fine granular QoS management at the
Incoming expense of control messages for efficient control dissemination
Frames in the network. Future research needs to carefully examine
the tradeoffs between disseminating more extensive control
Stream Filter messages and the resulting QoS management improvements.
Stream Gate
F. Discussion on TSN Standardization
Meter
The IEEE TSN TG has standardized deterministic net-
Queueing
working for Layer 2 Ethernet based bridging LANs. These
standards have been revised and continue to be updated to
reflect the convergence of the industrial and consumer mar-
Fig. 17. Illustration of PSFP flow: The flow is first filtered according kets. Overall, the TSN standards guarantee the required QoS
to per-flow policies. Then, a gating mechanism regulates the flow. Finally,
flow metering ensures bandwidth limitations before a frame is queued for requirements for data transmission and provide sufficient mea-
forwarding. sures to enable end-to-end functional communication safety in
the network. Essentially, the TSN standardization provides the
recommended practices for enabling low latency, jitter, and
(PCEs) [103], defined by the IETF PCE WG [103], to manage data loss, as well as redundancy and reservation. In addition,
the EPs. A PCE interacts with the IS-IS protocol to handle the TSN standardization provides mechanisms for bandwidth
and install requests for the network and can interact with the limitation, dynamic reconfiguration, centralized management,
SRP protocol, see Section III-C, to reserve resources along the and strict timing features.
EPs. Additionally, the PCEs can manage redundancy on the Timing measurement and sub-microsecond time synchro-
EPs, thus providing protection on top of the EPs by utilizing nization as basis for TSN standard mechanisms can be
alternate paths, e.g., Loop Free Alternates (LFAs) [77], that achieved with IEEE 802.1AS and the updated revised version
reroute in a few milliseconds. 802.1AS-REV. Essentially, all gPTP network entities con-
3) IEEE 802.1Qci Per-Stream Filtering and Policing tribute to distributing and correcting delay measurement timing
(PSFP): The IEEE 802.1Qci per-stream filtering and policing information based on the source GM. 802.1AS-REV provides,
(PSFP) standard [78], also known as ingress policing/gating among others, GM redundancy for fast convergence.
standard, filters and polices individual traffic streams based on Several flow management standards, including IEEE
rule matching. IEEE 802.1Qci prevents traffic overload condi- 802.1CB (FRER), 802.1Qca (PCR), 802.1Qci (PSFP),
tions, that are caused, for instance, by erroneous delivery due 802.1Qcc (Enhanced SRP and centralized Management),
to equipment malfunction and Denial of Service (DoS) attacks, 802.1CS (LRP) and RAP have been published or are in
from affecting intermediate bridge ports and the receiving end progress to enable redundancy, path reservation, bandwidth
station, i.e., improves network robustness. IEEE 802.1Qci may limitation, dynamic reconfiguration, as well as overall flow in-
be used to protect against software bugs on end points or tegrity and management. Although standard Ethernet provides
bridges, but also against hostile devices and attacks. IEEE redundancy features, e.g., through spanning tree protocols,
802.1Qci specifies filtering on a per flow (stream) basis by the convergence time in the event of a failure is too slow
identifying individual streams with a StreamID, which utilizes for real-time IACS applications. Therefore, FRER is used to
the 802.1CB stream handler method [76]. The identified indi- proactively enable seamless data redundancy at the cost of ad-
vidual streams can then be aggregated, processed, and finally ditional bandwidth consumption. Moreover, PCR in combina-
queued to an input gate. As illustrated in Fig. 17, each gate tion with FRER and 802.1Qcc enables fast recovery, efficient
performs three functions. path redundancy, and dynamic runtime flow management.
The PSFP stream filter performs per-flow filtering by match- Furthermore, PSFP manages, controls, and prevents rogue
ing frames with permitted stream IDs and priority levels, and flows from deteriorating the network performance. Since SRP
then applies policy actions. The PSFP stream gate coordinates and the related signaling protocols are fully distributed mech-
all streams such that all frames proceed in an orderly and anisms targeted towards AVB applications, the SRP and MRP
deterministic fashion, i.e., similar to the 802.1Qch signaling protocols are not scalable to large networks with real-time
 14

IACS applications due to a limited state information database Thus, 802.1 LAN devices can be authenticated and appropriate
for the registered flows, see Section III-C3. Therefore, LRP policies for transmission and reception of data and control
in conjunction with RAP as the signaling protocol features protocols to and from devices can be applied. The IEEE
a decentralized approach to support resource reservations for 802.1 Security TG is working on a couple of amendments
scalable TSN enabled networks. to address privacy concerns and to include a YANG model
To achieve low latency, several flow control standards have allowing configuration and status reporting for PNAC in 802.1
been released, including IEEE 802.1Qbv (TAS), 802.1Qch LANs. The integration of the security protocols and standards
(CQF), and IEEE 802.1Qcr (ATS). For TAS, IEEE 802.1Qbu with TSN enabled networks needs to be addressed in future
frame preemption can ensure that the transmission channel research and standardization. For instance, the impact of the
is free for the next express traffic transmission. CQF can security stack overhead on TSN flows and the impact of the
coordinate ingress and egress operations to reduce the TAS security overhead on OT related applications running over
configuration complexity, albeit at the expense of higher de- Ethernet LANs need to be investigated. Thus, there are ample
lays. Finally, ATS has been proposed to provide deterministic research opportunities for testing and benchmarking to ensure
operations independently of the reference time synchronization the efficient integration of legacy security protocols with TSN.
and low delays for high link utilization. The efficient dynamic The important area of networks for industrial applications
configuration of these flow control standard mechanisms, often employs cut-through switching techniques. An interest-
including IEEE 802.1Qbv, is an open challenge that requires ing future research direction is to investigate how networking
extensive future standardization and research efforts. with cut-through switching compares with networking based
The TSN mechanisms (and similarly the DetNet mecha- on the TSN standards (tool sets).
nisms) do not explicitly define mechanisms to specifically More broadly, even though many standards and recom-
reduce packet jitter. The various TSN mechanisms for ensuring mended practices addressing deterministic networking have
very short deterministic packet delays implicitly achieve very been published, significant testing and benchmarking is needed
low packet jitter. Moreover, resource reservation and admission to provide assurances to the industry and consumer markets.
control can further reduce end-to-end jitter by limiting interfer-
ing traffic, which is typically the main cause of jitter. Addition- IV. TSN R ESEARCH S TUDIES
ally, CQF can coordinate ingress and egress operations, which This section surveys the existing research studies towards
can cause jitter, to reduce delays to sub-microsecond levels achieving ULL in the context of the TSN standards. The
or to bound delays to within a few microseconds, effectively TSN standards provide tool sets to enable TSN characteristics,
eliminating jitter caused by the physical properties of links and such as flow synchronization and flow control (see Sec. III),
switching fabrics [104]. However, while it is very unlikely that in conventional networks. Based on the application require-
high jitters occur in a TSN network, in the event of high jitter, ments, various TSN standard tools can be independently
the TSN standards do not actively delay or throttle flows to and selectively adopted on network segments to enable TSN
compensate for the high jitter condition. Such specific jitter characteristics. Similar to the organization of the review of
control operations are an open issue for potential future TSN TSN standards in Fig. 1, we organize the survey of TSN related
standards development. research studies in Fig. 18 according to the same classification
The TSN standardization has so far excluded the spe- as the TSN standards in Fig. 1. To date there have been no
cific consideration of security and privacy. The IEEE 802.1 specific research studies on the TSN flow concept; therefore,
Security TG has addressed security and privacy in general we omit the flow concept category in Fig. 18.
IEEE 802.1 networks, i.e., functionalities to support secure
communication between network entities, i.e., end stations
and bridges. The TG has detailed a number of standards A. Flow Synchronization
and amendments, including 802.1X Port-based Network Ac- 1) Clock Precision: Most existing time synchronization
cess Control (PNAC) [105], [106] , 802.1AE MAC Security implementations are limited to clock precisions on the order
(MACsec) [107]–[110], and 802.1AR Security Device Iden- of sub-microseconds [139]. The global sharing of the timing
tity (DevID) [111], that focus on providing authentication, information across the network elements allows the clocks in
authorization, data integrity, and confidentiality. Specifically, the network elements to be precisely synchronized relative
PNAC utilizes industry standard authentication and authoriza- to each other (see Section III-B). The challenges associated
tion protocols enabling robust network access control and with network wide clock synchronization are not limited
the establishment of a secure infrastructure. Furthermore, to one particular network attribute. Rather, a wide set of
PNAC specifies the MACsec Key Agreement (MKA) [106] network attributes, including hardware capabilities, such as
protocol. MACsec specifies the use of cryptographic cipher clock stability, and isolation from environmental impacts e.g.,
suites, e.g., Galois/Counter Mode of Advanced Encryption temperature, and software implementations, e.g., for designing
Standard cipher with 128-bit key (GCM-AES-128), that al- an effective closed-loop system to track and correct the timing
low for connectionless user data confidentiality, frame data drifts, influence the synchronization quality in the network
integrity, and data origin authentication, essentially providing as a whole. As a result, most current deployments rely on
a set of protocols that ensures protection for data traversing sub-microseconds clock precision techniques. However, future
Ethernet LANs. For instance, DevID is a unique per-device trends in network applications require a tighter clock syn-
identifier that cryptographically binds a device to the DevID. chronization on to the order of sub-nanoseconds in Ethernet
 15

TSN Research Studies, Sec. IV

? ? ? ?
Flow Synchronization, Flow Management, Flow Control, Flow Integrity,
Sec. IV-A Sec. IV-B Sec. IV-C Sec. IV-D
Clock Precision [112] Resource Resrv. [116] Fault Tolerance. [137], [138]
Freq. Sync. [113] Reconfiguration [117]
Timing Accuracy [114], [115] Bandw. Alloc. [118] ? ? ?
Routing [119] Shaping Scheduling Preemption
SDN TSN [120], [121] Traf. Shap. Ana.. [122] TTEthern. vs. TSN [124] Preemption [134]
Shaping Overhead [123] Ctrl. Traf. Sch. [125] Pre. effect on non-CDT [135]
Opti. Sch. [126]–[128] Preempt. M/G/1 Analy. [136]
Urgency Sch. [89], [93]
Joint Rout. & Sch. [129]–[131]
Traf. Sch. Impact [132], [133]
Fig. 18. Classification of TSN research studies.

4-8 GB
ULL switching and routing based on synchronization. Their
2x USB 2.0 HS 16 buttons BNC
DDR3L 1x USB 3.0 SS 16 7-Seg. disp. Clock
design integrates a state-of-the-art FPGA with a standard x86-
ECC 1x SDHC/SDXC 1 console port In/Out 64 processor (which supports both 32 and 64 bit operation)
Memory to support TSN functions. The system provides frequency
Sys. Ctrl. & GPIO SyncE synchronization over standard Ethernet to the entire network.
JTAG
Intel Sys. Monit. LEDs Clock Frequency synchronization enables distribution of timing in-
Atom formation with low-jitter across the network. In the frequency
Processor Xilinx Artix-200T FPGA synchronization design illustrated in Fig. 19, datapaths are
RGMIIs enabled with one or more synchronous modules supported by
Watchdog 2 Gigabit
10 Standard Gigabit clock synchronization. These datapaths are allocated resources
Ethernet Ports
TPM Ethernet Ports with Clock
in terms of bit rate and packet rate based on the worst-case
Recovery traffic load. This design exploiting hardware synchronization
capabilities achieves cut-through latencies of 2 to 2.5 µs
Fig. 19. Illustration of frequency synchronization design supporting TSN with for twelve Gigabit Ethernet ports at full line rate packet
clock recovery and network wide synchronization [113]. processing [113]. The constituents of the observed latency
were identified as pipeline delay, arbitration delay, aggregation
delay, backpressure cycles, cross-clock domain synchroniza-
networks. For instance, the control system of the CERN Large tion cycles, datapath width adaptation cycles, and head-of-line
Hadron Collider (LHC) communication network has to operate blocking cycles. Emphasizing the importance of the hardware
with sub-nanosecond precision to share timing and perform implementation of the frequency synchronization process, Li et
time-trigger actions [140]. al. [113] suggest that their novel hardware implementation and
Gutierrez et al. [112] have analytically evaluated the syn- timing distribution process based on frequency synchroniza-
chronization process and the quality of the timing error estima- tion across networks can be easily extended to other custom
tion in large scale networks based on the IEEE 802.1AS TSN designs.
synchronization standard. In particular, Gutierrez et al. focused
on the clock synchronization quality with a small margin of 3) Timing Accuracy: Although TSN protocols offer very
error between each node for a large network consisting of accurate timing information for the inter clock alignment, the
a few thousand nodes with maximum distances between the validity and accuracy of the received timing information can
grandmaster clock and synchronized node clocks spanning up still be uncertain. That is, typically the timing information
to 100 hops. The study of the protocol behavior included vari- received from the grand master is blindly followed by the
ous network aspects, such as clock granularity, network topol- clock alignment process, which can potentially result in out-of-
ogy, PHY jitter, and clock drift. The results from probabilistic sync clocks if the received timing information is not accurate.
analytical modeling and simulation evaluations indicate that The detection of erroneous timing information by the receiving
implementation specific aspects, such as PHY jitter and clock node can potentially help time critical network applications to
granularity, have a significant impact on the clock precision re-trigger the verification, calibration, and re-synchronization
with deviations reaching 0.625 µs in the TSN synchronization process. Moreover, nodes can use this information to alert
process. Therefore, it is critical to ensure that the physical network applications to request a new path or to terminate
properties of the clock within each node are accurate so as critical operations that require timing precision. Therefore,
to ensure the overall quality of the synchronization process in timing accuracy is an essential aspect in TSN networks.
TSN networks that adopt IEEE 802.1AS. The time-error is the relative clock difference between the
2) Frequency Synchronization: Li et al. [113] have intro- slave and the grand master. The time-error can still exist
duced a novel networking device architecture that provides even if the slave node applies the timing corrections based
 16

on timing error estimation. The timing accuracy represents Reservation message

the overall quality of the timing distribution throughout the transmission
network. The timing accuracy at a node can be estimated in
two ways: i) by receiving the timing information from another
Reservation completion and timer
source and periodically comparing to check the accuracy, and initialization (timestamp = 0)
ii) keeping track of the node’s self error and (ingress and
egress) port latencies to predict the inaccuracy in the received
timing. Noseworthy et al. [114] have specifically addressed the Utilizing Yes
timing inaccuracy of a Precision-Time Protocol (PTP) node reservation
with the help of an auxiliary node. The proposed network-
based system monitors and measures the timing errors and n=n+timer No
port latencies to track the self errors independently of the PTP
protocol and network application. Such a system can share No
Timestamp > N
the information with other nodes so that the other nodes can
estimate the timing errors. In addition to the timing error of Yes

a PTP node, the ingress and egress delays in the PTP nodes Release bandwidth reserved for
for a specific TSN flow have been estimated and used in the timed out stream
process of clock reference maintenance. A PTP extension to
wireless networks has been investigated in [115] while related De-registration
measurement techniques have been examined in [141], [142]. completion
4) Summary and Lessons Learnt: An important aspect of Fig. 20. The automatic flow de-registration process monitors the network for
timing and synchronization in TSN networks is to estimate transmission activity and removes the resource reservations when a flow is
the relative timing difference between two nodes. Timing idle for more than a threshold duration [116].
differences may arise because of clock errors, synchroniza-
tion errors, as well as tracking and estimation errors [18].
Clock errors are caused by the timing drifts resulting from 1) Resource Reservation: A resource reservation process is
hardware imperfections. Synchronization errors are caused by typically applied across the network elements so as to ensure
false timing information and wrong interpretation of timing that there are sufficient resources for processing TSN flow
information. Tracking and estimation errors can, for instance, frames with priority. The TSN IEEE 802.1 Qat protocol de-
arise due to sleep states for power savings. In deep-sleep states, fines the resource reservation mechanism in TSN networks, see
only a minimal set of sub-systems is kept alive. Moreover, the Section III-C2. Park et al. [116] revealed that the TSN IEEE
clock system is typically switched from high resolution and 802.1 Qat standard lacks effective procedures for terminating
high precision to low resolution and low precision, which may reserved resources. The existing standardized resource release
incur large clock drifts. The repeated switching of the clocking mechanism involves signaling among TSN nodes to establish
system may accumulate significant synchronizing errors that a distributed management process, such that the connection
need to be corrected by external sources. In order to achieve reservations are torn-down and the resources released when
high-order precision in the clock implementation for TSN the TSN flow is no longer needed. Similarly, when there is
applications, all aspects of the clock errors must be considered a renewed need for the TSN flow, the connection with its
to mitigate the effects arising from incorrect local timing. resource reservation is re-initiated based on the flow’s traffic
The clock synchronization in the network requires signif- requirements. For networks with a few nodes and short end-
icant bandwidth, i.e., imposes a significant overhead in the to-end delays, the management process has relatively low
network. The synchronization data needs to be propagated signaling complexity and does not significantly impact the
throughout the network in a deterministic fashion. Hence, TSN flows. However, Park et al. [116] found that the numbers
the synchronization traffic interferes with the scheduled and of nodes that are typical for in-vehicle networks result in a
regular traffic. Therefore, the design of TSN networks requires pronounced increase of the overall control message exchanges
careful consideration of the overhead resulting from the syn- for the tear-down and re-initiation of connections.
chronization process and efforts to reduce the overhead. On Therefore, Park et al. [116] have proposed an automatic
the other hand, the effectiveness of the protocol that facilitates de-registration to tear down reservations. All participating
the synchronization process is limited by the node capability nodes run the algorithm to de-register the reserved resources
to preserve a synchronized local clock. If the local clock in a synchronized manner across the entire network based
skew is high compared to the frequency of the synchronization on the network wide synchronization capability in TSN net-
process, then the local clock will often have the wrong timing. works. Figure 20 presents the flow chart of the automatic
Therefore, the future design of synchronization protocols and de-registration process: A timer is initialized to track the
the frequency of synchronization should be based on the node idle times for a specific TSN flow. Once the timer meets a
characteristics. predefined threshold, the resource reservations of the flow are
automatically torn-down by all the participating nodes. The de-
registration process is simultaneously performed throughout
B. Flow Management
the network based on the synchronized timers. The downside
 17

of such an automatic de-registration process is the overhead Time series

analysis, machine Routing
for the re-activation process of the resource reservation for learning etc.
CONTROL algorithms, load
TSN flows which were deactivated due to a short period of balancing etc.
PLANE
inactivity. Thus, for highly bursty traffic, the automatic de-
Computation load, Analysis and
registration process may negatively impact the overall network Planning
network topology,
performance since the idle times between traffic bursts may communication, Monitoring Execute Switch forwarding
trigger the automatic de-registration. latency etc. tables, virtual
machine to host
Raagaard et al. [117] have examined GCL reconfiguration in DATA
mapping etc.
PLANE
the context of CNC and CUC (see Section III-C2). The actual
underlying scheduling mechanism is an elementary greedy Work-stations on
earliest deadline first heuristic. That is, flows with earlier the shop floor
equipped with
deadlines are scheduled first. A weakness of the approach sensors and
appears to be the long reconfiguration time. Despite the actuators
algorithmic simplicity, reconfigurations take between several
seconds to up to a minute. Dynamic runtime management and Research &
Development Manufacturing
reconfiguration of the IEEE 802.1Qbv GCL schedules thus Personnel
continue to pose a significant challenge and are an important Fig. 21. Software Defined Networking (SDN) based Time Sensitive Net-
topic for future research [143]–[150]. working (TSN) in industrial network setting: Monitoring sensors from var-
ious factory locations deliver information to the centralized controller. The
2) Bandwidth Allocation: Bandwidth allocation reserves centralized controller applies the time sensitive networking rules across the
the physical transmission resources required to meet the delay industrial networks to support critical connectivity paths [120].
requirements of an end-to-end flow. A specific bandwidth
allocation challenge in TSN arises from the multiple traffic
classes, such as the different priority levels for scheduled
4) Software Defined Networking for TSN: The centralized
traffic and best-effort non-scheduled traffic.
computation and management of routing of an end-to-end
Ko et al. [118] have developed a simulation model to study TSN flow follows similar principles as the central control in
the impact of the Maximum Transmit Unit (MTU) size of TSN the SDN paradigm. A formal adoption of the SDN paradigm
traffic packets on the performance for scheduled traffic within in TSN networks has been presented by Nayak et al. [120].
a specific bandwidth allocation framework. Specifically, Ko Nayak et al. employed SDN principles to evaluate the routing
et al. have examined bandwidth allocations for the scheduled of TSN flows and to apply the evaluated routes to the network
traffic based on TSN definitions. Ko et al. assume that 75% of nodes. As shown in Fig. 21, the proposed SDN controller
the bandwidth is allocated to the different QoS traffic classes, implements four main management functions, namely monitor,
while the remaining 25% of the bandwidth are allocated to analyze, plan, and execute to establish and control the TSN
best-effort traffic. In particular, two classes of QoS traffic were flows. Nayak et al. have conducted delay and flexibility
considered, namely scheduled traffic and audio/video traffic. simulation evaluations of several routing mechanisms with the
Bandwidth is allocated such that the total bandwidth allocated SDN approach and without the SDN approach to quantify the
to scheduled and audio/video is always 75%, i.e., the allocation benefits offered by SDN. Based on simulations, Nayak et al.
ratio between QoS traffic and best-effort traffic is maintained have proposed the adoption of SDN to existing processes for
constant (75% to 25%). The study varies the bandwidth ratio the network management of time-sensitive applications.
between the scheduled traffic and the audio/video traffic. The
While SDN inherently provides management flexibil-
bandwidth allocation for the scheduled traffic was varied by
ity [151]–[153], the actual deployment characteristics of SDN
varying its MTU size. The simulations for a specific in-
for TSN still need to be carefully characterized. Towards
vehicle network scenario found that an MTU size of 109 bytes
this goal, Thiele et al. [121] have presented the challenges
(corresponding to a bandwidth allocation of 7% to scheduled
in adapting SDN for TSN networks. Specifically, Thiele et
traffic), optimally allocated bandwidth to the scheduled traffic,
al. have performed a timing analysis of an end-to-end TSN
which achieved an average end-to-end latency of 97.6 µs.
flow in the SDN framework to verify the limitations of SDN,
3) Routing: In contrast to routing mechanisms in conven- such as overhead, scalability, and control plane delay in
tional networks, Arif et al. [119] have proposed a computa- meeting the TSN requirements for in-vehicle networks. Thiele
tionally efficient optimization method to evaluate the routing et al. used a compositional performance analysis framework
paths for a TSN end-to-end connection. The proposed solution to model the SDN network performance for TSN. The SDN
considers an optimality criterion that minimizes the routing deployment requires a centralized controller for the global
path delays which effectively reduces the end-the-end latency management of the TSN network from flow establishment
of the TSN flows across the network. The proposed approach to tear-down. The placement of the controller among the
also considers multipath jitter, as well as the probability of TSN nodes can be challenging since the control signalling
loop occurrence while evaluating the end-to-end routing path communication between nodes and controller can span across
of the TSN flow. The main purpose of the routing is to load the entire network. Each TSN flow establishment process re-
balance the TSN flows in the network nodes and thus to reduce quires the exchange of control messages between a TSN node
the routing path delays. and the controller. As the numbers of TSN nodes and flows
 18

increase, the overhead due to control message exchanges could is buffered and scheduled for transmissions at an available
increase, affecting the overall TSN performance. Moreover, opportunity. (In contrast, traffic policing simply drops the
the flow setup process requires the TSN node to request the exceeding traffic.) The downside of traffic shaping is queuing
flow rules from the SDN controller which can increase the delay, while the downside of policing is that excess frame
flow setup time as compared to a static non-SDN scenario. dropping can affect the TCP transmission windows at the
Therefore, to determine the feasibility of SDN for in-vehicle sender, reducing the overall network throughput.
TSN networks an analytical formulation was verified through 1) Traffic Shaping: Control-Data Traffic (CDT) is the TSN
simulations. The simulation results demonstrate that the worst- traffic class for transmissions of control traffic with the shortest
case SDN network configuration delay is 50 ms, which is possible delay. In addition to the CDT class, TSN distinguishes
typically tolerable for admission control and fault recovery in traffic class A and class B. Collectively, these traffic classes
conventional Ethernet networks. A related SDN based control are shaped by the traffic shapers in the TSN nodes to meet
plane architecture has recently been proposed in [154]. the delay requirements. The traffic shapers ensure that i) the
5) Summary and Lessons Learnt: In addition to dynamic CDT is allocated resources with strict priority, ii) the TSN
flow establishment based on current network characteristics, traffic is isolated from the regular traffic, and iii) the wait
flow management ensures that TSN networks preserve the times for enqueued frames are bounded. Towards these goals,
time-sensitive characteristics, such as low end-to-end delay, various traffic shaping methods have been standardized, see
when the network characteristics, such as topology and number Section III-D, in order to satisfy the requirements of the flows
of nodes, change. The adaptability of the network to changes in based on their traffic classes.
network characteristics is an important network design aspect a) Shaping Analysis: Thangamuthu et al. [122] have con-
that needs to be examined in detail in future research. This ducted a detailed comparison of the standard TSN traffic shap-
future research needs to address the control plane as well as ing methods. In particular, Thangamuthu et al. have compared
the data plane. the burst limiting shaper (BLS, a variation of CBS, which
Currently, IEEE 802.1Qcc has centralized management, but was considered in research but not incorporated into the TSN
does not preclude distributed management. The TSN TG has standard), the time aware shaper (TAS), and the peristaltic
started the process of chartering a project to standardize RAP, shaper (PS), see Section III-D. The simulations show that for
see Section III-C4, which uses distributed management. Gener- typical 100 Mbps Ethernet network deployments the in-vehicle
ally, centralized management can reduce the traffic overhead delay requirements are met for most applications, except
and reduce the management complexity. The detailed inves- for applications with strict delay requirements. Therefore,
tigation of the tradeoffs between centralized and distributed additional ULL mechanisms are recommended, in addition to
management is an important direction for future research. the traffic shaping, to satisfy strict application requirements.
The static allocation of link resources to a TSN flow can Complementarily, Thiele et al. [155]–[157] have conducted a
result in low network efficiency. Dynamic link resource allo- formal timing analysis and worst-case latency analysis of the
cation provides more efficiency and flexibility. More specif- different shapers for an automotive Ethernet topology, while
ically, a flow management technique can be implemented an avionics context has been considered in [158]. Moreover,
to statistically multiplex several flows sharing common net- general latency and backlog bounds have recently been derived
work resources, while the worst-case flow performance is in [159]–[165]. As alternative to CBS and TAS shaping, a pre-
still bounded by a maximum prescribed value. A pitfall that shaping approach at the senders has been explored in [166]. A
needs to be carefully addressed is the network complexity complementary analysis of the ATS shaper has bee conducted
in developing and deploying flow management techniques in in [167]. Pre-shaping has been found to be effective for a
actual networks. SDN may be a promising technology for the low number of hops. However, the pre-shaping effectiveness
management of dynamic resource allocation in TSN networks. decreases with increasing hop count. Also, pre-shaping does
SDN also provides an inherent platform to design advanced not protect the shaped traffic flows from other unshaped or
TSN flows management mechanisms, such as admission con- misbehaving flows in the network. The wireless fronthaul
trol and security mechanisms. context, see Section VII-A1, has been considered in [168].
b) Traffic Shaping Overhead: Traffic shaping, in particu-
C. Flow Control lar the TAS can significantly impact the configuration overhead
The overall temporal characteristics of a TSN flow are throughout the network, especially for temporary (short lived)
dictated by the flow control mechanisms that are applied in TSN flows. Typically, the TSN flows that originate from plug-
the intermediate nodes. The flow control mechanisms imple- and-play devices attached to the TSN network are temporary
mented at each TSN node directly impact the process of frame in nature. The transmission schedule for TAS gate control
traversal through each node that a particular flow is defined must be evaluated and maintained at each traversed TSN
to pass through. A variety of flow control mechanisms are node corresponding to each temporary flow. The schedule
employed in the intermediate nodes before an enqueued frame information at each node is generated and managed as a
is scheduled for transmission over the physical link. The most network configuration. These network configurations must be
critical flow control mechanisms in TSN nodes are traffic applied across the network to establish an end-to-end TSN
shaping as well as scheduling and preemption. flow. The temporary TSN flows resulting from plug-and-
Traffic shaping limits the traffic rate to a maximum allowed play connections can create a deluge of management traffic
rate, whereby all traffic exceeding the maximum allowed rate overhead.
 19

To address this overhead issue, Farzaneh et al. [123] have modules. The simulation evaluations indicated significant la-
presented an ontology based automatic configuration mecha- tency reductions by up to 50% for the control traffic flows,
nism. Application management service and TSN management i.e., the scheduled traffic flows, compared to non-scheduled
service entities coordinate the connection establishment and traffic. A limitation of the Bello et al. [125] study is that it
tear-down procedures, managing the control plane actions for considered only the automotive network domain and did not
the TSN network. A TSN knowledge database is implemented consider the wider applicability and potential of TSN.
to track and manage new, existing, and previous connections. c) Optimization Based Scheduling: An important short-
For each connection, QoS requirements, assignments, and coming of the IEEE 802.1Qbv standard, which defines the
source details, such as port, related topics and devices are transmission of scheduled traffic in TSN, is that there are no
identified and analyzed to build an ontology of TSN flows specific definitions of algorithms to determine the transmission
corresponding to an application and device. Thus, whenever schedule of frames on a link. In addition, the IEEE 802.1Qbv
the plug-and-play event for a specific device occurs in the net- standard enforces a time spacing, i.e., guard bands, between
work, the TSN configurations are automatically retrieved and the scheduled traffic types. The guard bands isolate scheduled
applied, lowering the overhead compared to the conventional traffic belonging to a specific class from other traffic classes,
connection management scheme. Although the automatic con- including the best-effort traffic class. A critical pitfall in the
figuration mechanism is similar to the principles of SDN, IEEE 802.1Qbv standard is that as the number traffic classes
Farzaneh et al. have discussed the process based automatic increases, there can potentially be a large number of guard
configuration mechanism independently of SDN. Nevertheless, band occurrences during the traffic transmissions over the link.
the ontology based automatic configuration mechanism can be Traffic schedules with frequent guard bands waste bandwidth
easily adapted to SDN by implementing the proposed applica- and can contribute to latency increases. Hence, an important
tion management service and TSN management functions as future work direction is to develop traffic transmission sched-
an SDN application. ules with reduced numbers of guard band occurrences in order
2) Scheduling: to prevent wasted bandwidth and to keep latencies low.
a) TTEthernet vs. TSN: Craciunas et al. [124] have Dürr et al. [126] have modeled TSN scheduling as a no-
presented an overview of scheduling mechanisms for Time- wait job-shop scheduling problem [172]. Dürr et al. then have
Triggered Ethernet (TTEthernet) [169]–[171] and TSN. In adapted the Tabu search algorithm [173]–[175] to efficiently
the TTEthernet switch, the incoming frames for an outgoing compute optimal TSN transmission schedules while reducing
egress port are temporarily stored in a buffer, and wait for the the occurrences of guard bands. The simulations evaluations
scheduler to assign a transmission-slot based on the precom- indicate that the proposed algorithm can compute the near-
puted schedule. In contrast, the incoming frames in TSN are optimal schedules for more than 1500 flows on contemporary
directly inserted into priority queues, and these priority queues computing systems while reducing the guard band occurrences
are served based on prescribed schedules. The fundamental by 24% and reducing the overall end-to-end latency for
difference between TTEthernet and TSN is the scheduling TSN flows. With the minimal duration of guard bands, see
procedure, whereby the TTEthernet buffer is served based Section IV-C3, the receivers have to be actively synchronized
on global static scheduling information, i.e., a tt-network- for the correct reception of TSN frames. The existence of
schedule assigned to meet the end-to-end delay requirements. guard bands in the traffic flows provides an inherent secondary
In contrast, TSN employs a dynamic schedule local to each synchronization for the receivers. However, it should be noted
node for control frame transmissions from priority queues. that the implementation of such optimization algorithms can
TSN switches may be synchronized to network timing and increase the network node complexity as well as protocol
can preempt an ongoing lower priority transmission, which is operations, increasing the overall operational cost of the de-
not possible in a TTEthernet switch. Thus, the deployment vice. These scheduling principles have been further developed
of TSN switches as opposed to TTEthernet switches can in [104] towards the incremental addition of new flows.
improve support for delay critical applications. However, the Craciunas et al. [127] have examined the scheduling of real-
implementation cost and complexity (due to synchronization) time traffic, whereby the transmission schedules are computed
of TSN is typically higher than for TTEthernet. through optimization methods. The constraints for the opti-
b) Control Traffic Scheduling: Bello et al. [125] have mization problem formulation are based on the generalized
presented an overview of TSN standards and examined the TSN network configuration in terms of the characteristics of
scheduling of control traffic flows in intra-vehicular Ethernet the Ethernet frames, physical links, frame transmissions, end-
networks. More specifically, Bello et al. focused on the IEEE to-end requirements, and flow isolation. While considering a
802.1Qbv standard for scheduled traffic. Bello et al. have comprehensive set of parameters, the optimization problem is
implemented the scheduled traffic mechanism for automotive modeled to compute transmission schedules in online fashion
connectivity applications by utilizing the time-sensitive prop- (i.e., is frame arrival event driven) to achieve low latency and
erties of TSN. In particular, flow prioritization has been used to bounded jitter. While a complex optimization problem can
prioritize the control traffic flows over regular data flows. The provide a near optimal solution, it is also important to consider
traffic flows are separated into multiple priority queues and the required computation times Addressing the complexity
scheduling procedures are applied across the queues. Bello aspect, Craciunas et al. have proposed several extensions to
et al. [125] developed a simulation model for an automotive the optimization process and outlined the implications for
network to study the behaviors of TSN supported network the computation time. Craciunas et al. [127] have conducted
 20

simulation evaluations for various network loads and config- interference with traditional end-to-end timing analysis ap-
urations. The simulation results indicate that an optimization proaches to reduce the computation times. Such an integrated
process can be scalable while achieving the desired level of approach can estimate an upper bound on the scheduled
scheduling benefits, i.e., bounded latency and jitter for an interference for various scheduling types, and the evaluations
end-to-end connection carrying real-time traffic. Craciunas et show significant computation time reductions.
al. have further developed this optimal scheduling problem A complementary study by Park et al. [133] has investigated
in [176], [177] A related scheduling approach based on a the performance of scheduled traffic as opposed to the non-
graphical model has recently been examined by Farzaneh et scheduled traffic. Park et al. preformed extensive simulations
al. [128], while a recent study by Kentis et al. [178] has focusing on TSN to verify whether the end-to-end flow
examined the impact of port congestion on the scheduling. requirements are impacted by increasing numbers of TSN
d) Joint Routing and Scheduling: TSN frame transmis- nodes in the presence of non-scheduled traffic. The simula-
sions out of the queues can be controlled through gating (see tions employed the general network wide synchronous event-
Section III-D2), whereby a predefined event triggers the gate triggered method for frame transmissions in TSN networks.
to transmit a frame from a queue according to a prescribed The simulations for an in-vehicle network based on the event
scheduling policy. With event triggering, the frame transmis- triggered scheduling for various traffic types show that the
sions follow the predefined time triggered pattern, resulting in delay requirements of control traffic can be successfully met
so-called time triggered traffic [92], [170], [179], [180]. Pop for up to three hops. However, the scheduled traffic needs
et al. [129] have designed a joint routing and scheduling opti- to be transferred within at most five hops to meet the typical
mization that evaluates the time trigger events to minimize the 100 µs delay requirement for critical control data in in-vehicle
worst-case end-to-end frame delay. The time trigger schedule networks.
is based on an optimization problem formulated with integer At a given TSN node, the events to trigger an action
linear programming. The proposed optimization problem com- that is then utilized for traffic scheduling can either be
prehensively considers the network topology as well as time generated by a processing unit within the TSN node or by
trigger flows and AVB flows. The time trigger flows follow the an external control entity. With the development and prolif-
shortest route, while AVB flows follow a greedy randomized eration of SDN, future research can develop various event
adaptive search approach. Simulation evaluations indicate that generation techniques based on the centralized SDN control
the compute time to evaluate the time triggered scheduling and management. The generated events can trigger various
and AVB routing optimization is acceptable as compared to TSN specified actions, such as frame transmissions, frame
the timing of the frame flows. A limitation of the approach dropping, or frame preemption, enabling new applications for
by Pop et al. is that the optimizations are not scalable and SDN control and management. To the best of our knowledge,
flexible when there are changes in the properties of network event triggering methods based on SDN have not yet been
infrastructures, e.g., topology changes. When there are such investigated in detail, presenting an interesting direction for
network infrastructure changes, then the entire optimization future research. However, SDN based management of TSN
process must be reconfigured. The recent related study by has already proposed and we discuss the applicability of SDN
Smirnov et al. [130] has focused on mixed criticality levels for managing TSN flows in Sec. IV-B4.
while the study by Mahfouzi et al. [131] has focused on the While scheduled TSN transmissions provide low latency
stability aspects of joint routing and scheduling. for prioritized traffic, lower-priority traffic which is also TSN
e) Impact of Traffic Scheduling: Although TSN networks scheduled can be significantly affected by higher priority
provide a pathway to achieve ULL through enhancements to traffic. In order to advance the understanding of the behaviors
the existing Ethernet standards, the benefits are limited to of traffic shapers on low-priority TSN traffic, Maixum et
TSN flows as opposed to best-effort traffic. That is, in case al. [181] have analyzed the delay of Ethernet frames that
of mixed transmissions, where the TSN defined transmissions are scheduled according to a hierarchical CBS or TAS in
are multiplexed with non-scheduled best effort traffic trans- TSN switches. The evaluations by Maixum et al. indicate
missions, there are no guarantees for the effective behavior of that the traffic scheduling for higher priority TSN flows can
the non-scheduled best-effort traffic. If there are requirements potentially result in traffic burstiness for lower priority TSN
for the non-scheduled traffic, such as a hard deadline for flows, increasing the overall delay for the lower priority traffic.
frame delivery in an end-to-end connection, the application This is because, long bursts of higher priority traffic starve
can be severely affected due to the interference from the the scheduling opportunities for lower priority frames, leading
scheduled TSN traffic. The behavior characterization of non- to the accumulation of low priority traffic. In addition to the
scheduled traffic can be challenging and unpredictable due static scheduling order, Maixum et al. have also studied the
to the interference from scheduled TSN traffic. Therefore, effects of changing the scheduling orders in terms of end-to-
Smirnov et al. [132] have provided a timing analysis to study end delay for both higher and lower priority levels. The formal
the uncertainty of critical non-scheduled traffic in presence worst-case delay analysis and simulation results indicate that
of scheduled TSN traffic interference. The challenge in the low priority traffic is severely affected by the scheduled higher
characterization of scheduled interference is to consider all priority traffic. Simulations of an automotive use-case indicate
possible traffic scenarios, such as all possible scheduling types, a worst-case delay for the prioritized traffic of 261 µs, while
resulting in long computation times. Smirnov et al. propose the worst-case delay for low priority traffic is 358 µs.
an approach to integrate the analysis of worst-case scheduled 3) Preemption:
 21

a) Preemption Mechanism: Lee et al. [134] have ex- reducing the frame delays for express non-preemptable traffic
amined the preemption mechanism (see Section III-D3) in relative to preemptable traffic; the average frame delays of
conjunction with the TSN timing and synchronization char- the express traffic are one to over three orders of magnitude
acteristics to estimate the transmission properties of CDT shorter than for preemptable traffic. Zhou et al. have also
and non-CDT frames. In particular, Lee et al. have proposed provided the VHDL design layout of the transmit unit and
to insert a special preemption buffer into the transmission receive unit for frame preemption for an FPGA based hardware
selection module that operates across all the different queues implementation.
at the bottom in Fig. 10 to aid with the preemption mechanism. 4) Summary and Lessons Learnt: Flow control mechanisms
Lee et al. have then analyzed the timing dynamics of the ensure that intermediate nodes support the end-to-end behavior
preemption. Lee et al. note that in actual deployments there of a TSN flow. Traffic shaping controls the frame transmission
are likely timing synchronization errors which impact the over the egress port in a TSN switch. Each traffic shaper
frame boundary calculations. Therefore, a minimum safety strives to transmit a frame from a priority queue within
margin that avoids collisions should be maintained while the shortest possible deadline while minimizing the impact
implementing the preemption mechanism. Lee et al. [134] on the transmissions from other queues. A finer resolution
advocate for a safety margin size of 20 bytes, accounting of priority levels, i.e., a higher number of priority levels
5 bytes for an error margin and 15 bytes for synchronization provides increasingly fine control over frame transmissions
errors. The simulation evaluations justify the impact of the from multiple queues. As a limiting scenario, an independent
synchronization errors on the safety margin duration and end- queue can be implemented for each individual flow in a TSN
to-end delay. Related preliminary preemption analyses have node. However, such fine-grained prioritization would require
been conducted in [182]. extensive computation and memory resources in each TSN
b) Preemption Effect on Non-CDT: Preemption priori- node. To overcome this, virtual queues can be implemented by
tizes CDT frame transmissions over the transmission of regular marking the frames in a single queue, eliminating the need for
Ethernet frames. Thus, preemption of non-CDT traffic can neg- a number of queues equal to the number of TSN flows. Each
atively impact the end-to-end characteristics of non-CDT traf- marked frame can be scheduled based on the marking value.
fic. In addition, low priority CDT frames can be preempted by As low priority flows can potentially face long delays due to
high priority CDT frames. Hence, the preemption process can resource starvation from the scheduling of high priority flows,
impact the end-to-end delay differently for the different prior- dynamic (i.e., changeable) priority values can be assigned
ity levels even within the CDT traffic. Thiele et al. [135] have to virtual queues. Dynamic priorities can prevent prolonged
formulated an analytical model to investigate the implications delays for flows that were initially assigned low priority. The
of preemption on the end-to-end delay characteristics of CDT priority levels can be dynamically changed based on the wait
and non-CDT traffic. Thiele et al. have compared standard time or the total transit delay of a frame compared to a
Ethernet with preemption (IEEE 802.1Q + IEEE 802.3br) and predefined threshold. Advanced dynamic priority techniques,
TSN Ethernet with time triggered scheduling and preemption such as priority inversion, could be implemented such that
(IEEE 802.1Qbv + IEEE 802.3br) with the baseline of standard the worst-case delay of low priority traffic is kept within
Ethernet (IEEE 802.1Q) without preemption. The worst-case prescribed limits.
end-to-end latency of CDT with preemption was on average
60% lower than for 802.1Q without preemption. Due to the
CDT prioritization, the worst-case latency of non-CDT traffic D. Flow Integrity
increased up to 6% as compared to the baseline (802.1Q) due 1) Fault Tolerance: The AVB task group was mainly intro-
to the overhead resulting from the preemption process. Hence, duced to add real-time capabilities to the best effort Ethernet
the impact of preemption of non-CDT traffic is relatively service. Industrial control networks expect more reliable and
minor as compared to the performance improvements for stricter QoS services as compared to best effort Ethernet
CDT traffic. Additionally, the latency performance of standard network service. Fault tolerance is a critical part of industrial
Ethernet with preemption is comparable to that of Ethernet networks. The general principle for enabling fault tolerance in
TSN with preemption. Therefore, Thiele et al. [135] suggest a network is to introduce redundancy.
that standard Ethernet with preemption could be an alternative Following this general principle, TSN provides fault toler-
to TSN for CDT traffic. Standard Ethernet would be much ance through redundancy mechanisms, such as frame replica-
easier to deploy and manage than TSN, as TSN requires the tion and elimination as well as path control and reservations,
design and maintenance of the IEEE 802.Qbv gate scheduling see Section III-E. Kehrer et al. [137] have conducted research
processes along with time synchronization across the network. on possible fault-tolerance techniques for TSN networks. The
c) Preemption Analysis and Hardware Implementation: main challenges associated with fault tolerance mechanisms
Zhou et al. [136] conducted a performance analysis of frame in TSN networks are the restoration processes for the end-
transmission preemption. In particular, Zhou et al. adapted to-end link failures while preserving the network topology,
a standard M/G/1 queueing model to estimate the long run i.e., without causing any significant break in continuous
average delay of preemptable and non-preemptable frame network connectivity. To address this, Kehrer et al. have
traffic and evaluated the frame traffic through simulations. The compared two approaches: i) decoupled stream reservation and
numerical results from the adapted M/G/1 queueing model and redundancy [183], and ii) harmonized stream reservation and
the simulations indicate that preemption is very effective in redundancy (which corresponds to IEEE 802.1CB).
 22

In the decoupling approach, the stream reservation proto- Therefore, smooth interoperability between TSN and different
col registers and reserves the streams independently of the external networks is essential for TSN operation in hetero-
redundancy requirements. This decoupled approach allows for geneous network scenarios. Ideally, the connectivity between
arbitrary redundancy protocols to be utilized. In contrast, the TSN and non-TSN networks should be able to accommodate
harmonized approach integrates establishment of the reserva- similar characteristics as TSN to ensure the overall end-to-end
tion and the redundancy requirements. More specifically, the connection requirements in heterogeneous deployments.
IEEE 802.1Qca stream reservation protocol is coupled with 1) V2X Communication: Juho et al. [185] have proposed
the IEEE 802.1CB frame duplication. iTSN, a new methodology for interconnecting multiple TSN
The main pitfall to avoid is to understand the application networks for large-scale applications. The iTSN methodology
requirements in terms of flexibility before choosing the re- utilizes wireless protocols, such as IEEE 802.11p, for the inter-
dundancy approach. Specific industrial automation networks networking between different TSN networks. In particular,
may have peculiar reliability requirements that may be more the sharing of global timing and synchronization information
flexibly met with the decoupled approach. On the other hand, across the interconnected network is important for establishing
the decoupled approach has a higher protocol overhead and a common timing platform to support TSN characteristics
requires more network bandwidth due to the distributed and in the external networks. The iTSN network uses the IEEE
independent mechanisms along with the lack of coordination 802.11p WAVE short message protocol to share the timing
between stream reservation and redundancy, as opposed to the information between different TSN networks. Critical rapid
integrated approach. A related fault tolerance approach based alert messages can be prioritized not only within a given TSN
on redundant packet transmissions has been examined in [138] network, but also across multiple interconnecting networks.
while a mixing of temporal and spatial redundancy has been Thus, the iTSN methodology enables, for instance, vehicular
proposed in [184]. networks to transmit safety critical messages to control nodes,
2) Summary and Lessons Learnt: Failure recovery and fault e.g., Road Side Units (RSUs) [186], with delays on the order
tolerance are key aspects of reliable network design. However, of microseconds in a heterogeneous deployment. Through
to date there has been only very scant research to address the the adoption of such reliable inter-connectivity techniques,
critical challenges of resource reservation for fault tolerance the vehicle braking safety distance can be achieved in much
while considering ULL requirements. Future research has to shorter (microseconds) time spans than the currently feasible
investigate the wide range of tradeoffs and optimizations that range of milliseconds. Overall, TSN and an interconnecting
arise with reliability through frame replication. For instance, technique, such as iTSN, can create a communication platform
high priority flows could have reservations of dedicated re- for safe autonomous driving systems.
sources, while low priority flows could share a common 2) Network Modeling: Although TSN standards have re-
reserved resource. The dedicated resources would enable the ceived significant attention in networks for automotive driving,
instantaneous recovery of the high priority TSN flows; albeit, a major challenge in network deployment is managing the
at the expense of a slight reduction of the overall network complexity. As automotive driving technology progresses,
efficiency due to the redundancy. In the event of failure for a more requirements are imposed on the existing in-vehicle
low priority traffic flow, the connection could be reestablished network infrastructure. As the number of sensors increase in
with a new flow path considering that the flows can tolerate an in-vehicle network, the increasing connectivities and band-
delays on the order of the connection reestablishment time. width requirements of the sensors should be correspondingly
Centralized SDN management can also provide the flexibility accommodated in the network planning. However, the dynamic
of dynamic path computation and resource reallocation in changes in the network requirements for an in-vehicle control
the event of failures. Therefore, the area of flow integrity system could require a more extensive network infrastructure,
requires immediate research attention to design and evaluate resulting in higher expenditures. Considering the complexities
the performance of efficient recovery processes based on of automotive networks, Farzaneh et al. [187] have proposed
priority levels. a framework to analyze the impact of adding new sensors
to an existing infrastructure that supports critical applica-
tions. In particular, the network configuration that fulfills all
E. General TSN Research Studies the requirements, including newly added sensors, must be
TSN is being widely adopted in critical small-scale closed dynamically evaluated and implemented. Towards this end,
automotive and industrial networks to establish reliable ULL the Farzaneh et al. [187] framework involves a design and
end-to-end connections. However, a key TSN limitations is verification tool based on a Logic Programming (LP) method
exactly this focus on closed networks, e.g., in-vehicle networks to support the reconfiguration and design verification processes
and small-scale robotic networks. The network applications for an in-vehicle TSN network. The proposed framework
running in robots and in in-vehicle networks often involve sig- consists of comprehensive logical facts and rules from which
nificant interactions with external non-TSN networks. Robotic a user can query the database with the requirements to obtain
and vehicular network applications require a tight integration configurations that satisfy the requirements. A key character-
with mobility handling procedures by the external network. If istic of the proposed approach is that the network modeling
advanced network features, such as mobility, are not properly process considers the most accurate logical facts and rules of
supported in the external network, then the TSN benefits the TSN applications and requirements to obtain an efficient
are fundamentally limited to small-scale closed networks. configuration and verification process.
 23

3) TSN Simulation Framework: Heise et al. [188] have the overall TSN node performance compared to a software-
presented the TSimNet simulation framework to facilitate only implementation. The performance evaluations from a
the development and verification of TSN networks. TSimNet prototype implementation based on a Virtex-6 FPGA showed a
was primarily implemented to verify industrial use-cases in significant reduction in the CPU load compared to a software-
TSN networks. The simulation framework is based on OM- only implementation. Additionally, the precision of the time-
NeT++, whereby the non time-based features, such as policy triggered event generation in the hardware implementation was
enforcement and preemption are implemented in a modular improved by a factor of ten compared to software triggered
fashion to increase the flexibility of designing new network events.
mechanism suitable for industrial networks. For instance, the 5) Summary and Lessons Learnt: The general aspects of
initial evaluation of the simulation framework for frame pre- TSN that determine the overall success of TSN designs and
emption mechanisms indicates that the end-to-end latency can implementations are the inter-interoperability with heteroge-
be increased if the network is not configured in an optimized neous network architectures, such as LANs, WANs, and core
way for critical functions, such scheduling and traffic shaping. networks. Most of the research on TSN to date has focused on
Heise et al. have evaluated the computational cost of the in-vehicle networks which are independent and isolated from
TSimNet framework for various network function simulations, external networks. Another limitation of the TSN research field
such as policing, recovery, and preemption in terms of CPU is the lack of a simulation framework that encompasses large-
and memory requirements. The simulation framework also scale heterogeneous network architectures. Valid use cases
features Application Programming Interfaces (APIs) for TSN that include both localized and external network interactions,
mechanisms that do not require time synchronization, such such as automotive driving, should be created and considered
as stream forwarding, per-stream filtering, as well as frame in benchmark evaluations. Currently, the general use-case in
replication and recovery. APIs can be invoked by the sim- most TSN research studies is an in-vehicle network supporting
ulation framework through a profile notification. The basic on-board sensor connectivity and audio/video transmission
framework modules also include the TSimNet Switch Model, for infotainment. Future custom TSN simulation frameworks
which can identify streams based on MAC, VLAN, and/or should be based on networks that support next-generation
IP addresses, while the TSimNet Host Model implements applications with localized and external network interactions,
complex functions, such as ingress and egress policy, as well as such as automotive driving. Similarly, the SDN based TSN
traffic shaping. Related simulation evaluations with OMNeT++ management could exploit hierarchical controller designs to
have been reported in [189], while a TSN simulation model extend the management from localized networks, such as in-
based the OPNET simulation framework has been presented vehicle networks, to external networks, such as vehicle-to-any
in [190]. (V2X) networks.
4) Hardware and Software Design: Hardware and soft-
ware component designs to support TSN functions, such as
scheduling, preemption, and time-triggered event generation F. Discussion on TSN Research Studies
in TSN nodes require significant engineering and develop- The TSN network infrastructure and protocols have to
ment efforts. Hardware implementations are highly efficient support bounded end-to-end delay and reliability, to support
in terms of computational resource utilization and execution basic features related to critical applications of IoT, medical,
latency but result in rigid architectures that are difficult to automotive driving, and smart homes. TSN based solutions
adapt to new application requirements. On the other hand, for addressing the requirements of these applications result in
software implementations can flexibly adapt to new application complex network infrastructures supporting various protocols.
requirements, but can overload CPUs due to the softwarization Hence, simplified TSN network management mechanisms are
of network functions, such as time-triggered scheduling and essential to reduce the complexity while achieving the critical
hardware virtualization. needs of the ULL applications.
Gross et al. [191] have presented a TSN node architecture The deterministic TSN network behavior has so far been
design where the time-sensitive and computationally inten- generally applied to a closed network, i.e., a network spanning
sive network functions are implemented in dedicated hard- only the scope of a particular application, for instance, in-
ware modules to reduce the CPU load. The proposed hard- vehicle networks. However, the connectivity to external net-
ware/software co-design approach flexibly allocates network works, such as cellular and WLAN networks, enhances the
function to be executed completely in hardware, completely capabilities of TSN networks. For instance, in automotive driv-
in software, or in both hardware and software based on the ing, the application requirements can be controlled by weather
dynamic load. The flexible allocation is limited to network data from the cloud or by sharing information with a neighbor-
functions that independently scale with the timing require- ing TSN in-vehicle network. Therefore, reliable, secure, and
ments, such as the synchronization protocol. More specifically, low-latency communication between multiple TSN networks
Gross et al. have considered time-triggered transmissions, is essential to support a wide range of future applications. The
frame reception and timestamping, and clock synchroniza- lack of TSN standards for connecting and communicating with
tion. The hardware modules can produce the time-triggered external TSN and non-TSN networks is impeding the research
events nearly jitter free, implement frame reception and time- activities in inter-operating networks and needs to be urgently
stamping in real-time, and synchronize clocks with a high addressed. In summary, we identify the following main future
degree of precision. Thus, the hardware modules improve design requirements for TSN research:
 24

i) Support for a wide range of applications spanning from A. Flow Concept

time-sensitive to delay tolerant applications with flow Similar to the TSN TG, DetNet flows are specified by
level scheduling capabilities. their QoS classes. DetNet defines each flow’s QoS by 1)
ii) Connectivity between multiple closed TSN architectures. the maximum and minimum end-to-end latency, and 2) the
iii) Flexible and dynamic priority allocations to ensure packet loss probability requirements [198]. DetNet strives
bounded end-to-end latency for lower priority traffic. to transport unicast and multicast ULL data flows for real-
iv) Adoption of SDN for the centralized management of TSN time applications with extremely low packet loss. In essence,
functions with a global network perspective. DetNet emulates point-to-point links over a packet switched
v) Efficient timing information sharing and accurate clock network, where each link can be shared between multiple
design through self-estimation and correction of local DetNet and non-DetNet flows, each with varying flow require-
clock skewness. ments and properties. A key aspect of DetNet flow control
vi) Computationally efficient hardware and software designs. and management is ensuring that non-DetNet flows have no
Generally, TDM can enforce a deterministic (100%) latency influence on DetNet flows. Maintaining each DetNet flow’s
bound, but the TDM average delay is typically somewhat QoS is achieved through the mechanisms surveyed in this
higher than the statistical multiplexing average delay (and section.
TDM has low utilization for bursty traffic). With proper ad- 1) DetNet Flow Types: Before introducing the DetNet flow
mission control, statistical multiplexing can provide statistical types, we first give a brief overview of two main layers of
guarantees for latency bounds [192], e.g., the probability for the DetNet architecture stack model. The DetNet Transport
exceeding the delay bound can be very low, e.g., less than Layer has an option to provide congestion protection (see
10−4 probability that the delay bound is violated. These rare Section V-D). The DetNet Service Layer provides service
occurrences of violating the delay bound “buy” usually much protection, e.g., through flow duplication (see Section V-E).
higher utilization (throughput) than TDM and lower average Four main DetNet flow types have been identified [198]:
delay (for bursty data traffic) [193]–[197]. An interesting
1) App-flow: The native data flow between the source
future research direction is to examine the tradeoffs between
and destination end systems within a DetNet enabled
deterministic and probabilistic delay bound assurances in
network.
detail for ULL traffic served with TSN mechanisms.
2) DetNet-t-flow: The specific data flow format bound to
the transport layer within a DetNet enabled network.
V. D ET N ET S TANDARDIZATION The DetNet-t-flow contains the specific data attributes
In this section, we present a detailed overview of the that provide features for congestion protection.
current standardization of the IETF Deterministic Networking 3) DetNet-s-flow: The specific data flow format bound to
(DetNet) WG. The IETF DetNet WG collaborates with the the service layer within a DetNet enabled network. The
IEEE 802.1 TSN TG to define a common architecture for DetNet-s-flow contains the specific data attributes that
layers 2 and 3, whereby the TSN TG focuses on layer 2 provide features for replication and elimination functions
bridged networks and the DetNet WG focuses on layer 3 supporting service protection.
routed segments. Similar to the TSN goals, DetNet aims to 4) DetNet-st-flow: The specific data flow format that is
support deterministic worst-case bounds on latency, packet bound to both the transport and service layers within
delay variation (jitter), and extremely low/zero packet loss. a DetNet enabled network. The DetNet-st-flow signals
Moreover, both TSN and DetNet strive for high reliability and the appropriate forwarding function utilizing both the
redundancy over disjoint paths targeted towards IACS real- service and transport layer attributes.
time applications. 2) DetNet Flow Identification: In contrast to a conventional
The charter of the DetNet WG is to specify an overall strictly layered network architecture, DetNet nodes inten-
architecture that standardizes the data plane and the Opera- tionally violate “layering norms” so that lower layers can
tions, Administration, and Maintenance (OAM) for layer 3 detect and become aware of higher layer flow types. This
ULL support. This charter includes the time synchronization, awareness of the higher layer flow types helps to provide
management, control, and security operations that enable specific queuing, shaping, and forwarding services as flows
multi-hop routing. Moreover, the DetNet charter includes the are transported across multiple technology domains. However,
various forms of dynamic network configuration (automated violating the layering norms creates new layering and re-
and distributed as well as centralized and distributed) and layering complexities. Therefore, DetNet must provide a way
the multi-path forwarding. In general, DetNet focuses on to easily and correctly identify flows and their associated
extending the TSN data and control plane into the layer 3 types. DetNet is architected to allow nodes within the network
domain, thus expanding the scope of TSN beyond LANs. data plane to distinguish DetNet flows based on the flow ID
Since the DetNet WG has only been established recently and DetNet Control Word (CW), i.e., sequencing information,
(started in October 2014, and became a WG in October 2015), appended in the packet header, whereby the CW is used for
no IETF RFCs exist yet. However, at the time of writing this replication and elimination purposes.
article, several IETF drafts have become available and will be To achieve accurate flow detection and identification, the
covered in the following subsections to provide a comprehen- flow attribute mapping between layers and across technology
sive overview of the ongoing IETF DetNet standardizations. domains has to be standardized. For each forwarding of a Det-
 25

DetNet Standardizations, Sec. V

? ? ? ? ?
Flow Concept [198], Flow Synchronization, Flow Management, Flow Control, Flow Integrity,
Sec. V-A Sec. V-B Sec. V-C Sec. V-D Sec. V-E
DetNet Flow Types Jitter Reduction [198] DetNet Configuration and DetNet Data Plane [200] Packet Replication and
DetNet Flow Identi. YANG Model [199] DetNet Traffic Elimination [198], [201]
Network Resource Engineering [198] Enforced Heterogeneity [198]
Advertisement and Queuing, Shaping, Fault Mitigation [198]
Distribution [198] Scheduling, IGP-TE Extensions [202]
Centralized Path Setup [198] Preemption [198] Flow Information Model [203]
Distributed Path Setup [198] QoS Performance Security and Privacy [204], [205]
Guarantees [198]
Fig. 22. Classification of DetNet Standardization.

Net flow between different technology domains, the relay node 1) DetNet Configuration and YANG Model: In order for
(i.e., router) needs to acquire upper layer information related to DetNet to enable seamless configuration and reconfiguration
the flow type and corresponding attributes. For example, when across various DetNet enabled network entities, a uniform
a DetNet flow is forwarded between two Label Switching and scalable configuration model needs to be defined. The
Routers (LSRs) that interconnect different Layer 2 bridged Internet draft [199] defines distributed, centralized, and hybrid
domains, then at each domain boundary, the higher layer flow configuration models, related attributes, and the YANG model
information is passed down to the node for correct forwarding. for DetNet.
Three main forwarding methods are considered in DetNet: 1) a) DetNet Configuration Model: Three configuration
IP routing, 2) MPLS label switching, and 3) Ethernet bridging. models have been introduced [199]: fully distributed, fully
For forwarding across technology domains, each DetNet App- centralized, and hybrid. For a fully distributed configuration
flow packet is appended or encapsulated with multiple flow- model, UNI information is sent over a DetNet UNI protocol,
IDs (IP, MPLS, or Ethernet). This enables DetNet routing i.e., sent using the flow information model discussed in Sec-
and forwarding between different and disparate IP and non-IP tion V-E. A distributed DetNet control plane propagates the
networks, essentially providing network interoperability. UNI and configuration information to each data plane entity.
In the centralized configuration model, the CUC sends the
B. Flow Synchronization UNI information to the CNC, similar to the IEEE 802.1Qcc
centralized configuration model, see Section III-C. For the
The main objective of DetNet is to expand the TSN ca- hybrid configuration approach, a combination of distributed
pabilities to layer 3 routing segments. DetNet relies heavily and centralized protocols within the control planes are used
on the services of the IEEE TSN TG mechanisms. Flow to coordinate configuration information. The fully distributed
synchronization with respect to the DetNet flow architectural and hybrid configuration models are not covered in [199] and
model has not been specifically addressed in [198]. Therefore, are left for future work.
it is likely that DetNet will ensure timing synchronization b) DetNet Configuration Attributes: Depending on the
between DetNet capable network entities (bridges, routers, configuration model and control plane associated protocols
and end systems) through various existing synchronization (i.e., IGP and RSVP-TE, or CNC and CUC), different config-
techniques and profiles, e.g., IEEE 802.1AS and IEEE 1588v2. uration parameters or attributes are used. The following main
Applications in the mission critical latency traffic class attributes have been defined for the centralized configuration
require extremely low delay variations (jitter). High jitter model [199]:
can lead to packet loss downstream and in the worst-case,
1) DetNet Topology Attributes specify topology related
loss of human life in factory networks. DetNet strives to
attributes, such as the node type, whether it is Packet
support minimal jitter by bounding the minimum and maxi-
Replication and Elimination Function (PREF) capable
mum latency [198], which is challenging in large scale packet
or not, and the queueing management algorithm.
switched networks. DetNet specifies jitter reduction through
2) DetNet Path Configuration Attributes specify the
two main principles: 1) sub-microsecond time synchroniza-
networked path related attributes, such as the constraints
tion between network entities, and 2) time-of-execution fields
(required min/max latency), and explicit routes using a
embedded within the application packets [198]. While no spe-
PCE (with PREF).
cific specifications regarding time synchronization for DetNet
3) DetNet Flow Configuration Attributes specify the
network devices exist, the DetNet WG have overall hinted
DetNet flow attributes, such as the flow ID, priority,
at using other Standardization Development Organization’s
traffic specification, and encapsulation method.
(SDO), e.g., IEEE TSN’s 802.1AS methods, see Section III-B.
4) DetNet Status Attributes specify the flow status feed-
back attributes, such as the flow performance (delay,
C. Flow Management loss, policing/filtering), and the PREF status.
Flow management describes and specifies the mechanisms DetNet YANG Model: Similar to IEEE 802.1Qcp (see
for discovering and configuring node capabilities. Section III-C1), a DetNet YANG model has been defined [199]
 26

for the centralized configuration model to convey network to the admission control mechanisms researched within the
configuration parameters. IETF IntServ framework [193]–[197], must be researched to
2) Network Resource Advertisement and Distribution: To operate within the DetNet framework. Based on the admission
supplement the DetNet Congestion Protection mechanisms control, network resources must be managed such that ULL
(which are defined for DetNet as flow control mechanisms, applications/traffic that is marked with higher priorities than
including shaping, scheduling, and preemption), and to ac- other traffic can be allocated the appropriate resources.
curately provision network resources for DetNet flows, i.e.,
admission control, each node (or central controller in a cen- D. Flow Control
tralized setup) needs to share and alert nearby networks of its While most control functions for DetNet flows follow the
(end system and/or transit node) capabilities [198] including: same principles used for IEEE TSN TG deterministic flows,
1) System capabilities, e.g., shaping and queuing algorithm key integration mechanisms and several differences are out-
used, buffer information, and worst-case forwarding lined as follows.
delay 1) DetNet Data Plane: To better understand how DetNet
2) Dynamic state of the node’s DetNet resources services operate, we first provide a brief overview of the Det-
3) Neighbor nodes and the properties of their relationships, Net data plane. A DetNet capable network is composed of in-
i.e., the properties of the links connecting them, e.g., terconnected end systems, edge nodes, and relay nodes [198].
length and bandwidth. Transit nodes (e.g., routers or bridges) are used to interconnect
How this information is carried over the control plane and DetNet-aware nodes, but are not DetNet-aware themselves.
the implementation specification is not available nor standard- Transit nodes view linked DetNet nodes as end points. DetNet
ized yet. However, with this information, PCE’s automatic path is divided into two main layers: 1) the DetNet service layer,
installation (distributed or centralized) can handle each DetNet and 2) the DetNet transport layer. The DetNet service layer
flow’s QoS requirement assuming that enough resources are is the layer responsible for specific DetNet services, such as
available, which is enforced by admission control mechanisms congestion and service protection, while the DetNet transport
similar to the TSN SRP (MRP) protocols (see Section III-C2). layer is responsible for optionally providing congestion pro-
3) Centralized Path Setup: Similar to IEEE TSN’s cen- tection for DetNet flows over paths provided by the underlying
tralized management model (802.1Qcc, see Section III-C2), network [198]. More specifically, the service layer can apply
DetNet’s centralized path setup leverages PCEs and packet specific services, such as packet sequencing, flow replica-
based IP or non-IP network information dissemination to tion/duplicate elimination, and packet encoding, while the
enable global and per-flow optimization across the DetNet transport layer can apply congestion protection mechanisms
enabled network. The DetNet WG [198] has addressed several (through the underlaying subnetworks, e.g., MPLS TE, IEEE
related key issues, such as the installation of the paths cor- 802.1 TSN, and OTN) and explicit routes. DetNets can have
responding to the received path computation (whether by the several hierarchical DetNet topologies where each lower layer
Network Management Entity (NME) or end systems), and how services the higher layers. Furthermore, DetNet nodes (end
a path is set up, i.e., through direct interactions between the systems and intermediary nodes) are inter-connected to form
forwarding devices and the PCEs, or by installing the path on sub-networks. These sub-networks, e.g., Layer 2 networks, can
one end of the path through source-routing or explicit-routing support DetNet traffic through compatible services, e.g., IEEE
information [198]. 802.1 TSN or point-to-point Optical Transport Network (OTN)
4) Distributed Path Setup: The DetNet WG has developed service in 5G systems [198].
initial design specifications for a distributed path setup (similar There are currently various protocol and technology options
to the 802.1Qat, 802.1Qca, and MRP signaling protocols) under consideration for DetNet service and transport layer pro-
utilizing Interior-Gateway Protocol Traffic Engineering (IGP- tocols. Table III provides an overview of these protocol can-
TE) signaling protocols, defined in Section V-D, e.g., MPLS- didates for the DetNet service and transport layers, including
TE, RSVP-TE, OSPF-TE, and ISIS-TE [198]. A key issue a brief description of each protocol and the latency impact on
is how the interactions and integration between layer 2 sub- a DetNet flow. Although no official solution has emerged yet
network peer protocols for TE and path installation will be for the DetNet data plane encapsulation at the network layer, a
defined, since significant work has been accomplished by the couple of proposals exist to tackle this problem. According to
IEEE 802.1 TSN TG regarding distributed and centralized Korhonen et al. [200], two of the most prominent deployment
protocols on path and multi-path setup and signaling protocols. candidates for the data plane protocols are either a UDP/TCP
5) Summary and Lessons Learned: Before controlling a service layer over a native-IP (IPv6) transport layer or a
DetNet flow, the node’s capabilities need to be distributed to PseudoWire-based (PW) [206] service layer over an MPLS
the PCE in the control plane. To efficiently disseminate the Packet Switched Network (PSN) transport layer. While many
node capability information, a configuration and YANG model options exist for DetNet data encapsulation, it is imperative to
need to be standardized to allow for dynamic reconfiguration, test and discern the corresponding performance overhead for
management, and status collection in large scale IP/non-IP each proposed DetNet node’s packet manipulation technique.
based networks. 2) DetNet Traffic Engineering: The IETF Traffic Engineer-
Additionally, as networks under the control of DetNet ing Architecture and Signaling (TEAS) WG considers Traffic
related services and mechanisms may become saturated with Engineering (IE) architectures for packet and non-packet net-
flows, effective admission control mechanism, e.g., similar works [208], essentially allowing network operators to control
 27

TABLE III
C ANDIDATE P ROTOCOLS FOR D ET N ET S ERVICE AND T RANSPORT L AYERS . A PROMINENT DEPLOYMENT CANDIDATE IS A UDP SERVICE LAYER OVER
AN IP TRANSPORT LAYER .

Layer Candidate Protocol Description Latency Imp.

PseudoWire (PW) Emulates networking services across packet switched networks (PSNs), delivers bare Moderate
minimum network service functionality on physical infrastructure with some degree
of fidelity.
Service User Datagram Prot. (UDP) Connection-less transmission of packets with low overhead, though no feedback Low
services provided.
Generic Rout. Encap. (GRE) Tunneling protocol that encapsulates arbitrary network layer protocol over another Moderate
network layer protocol, e.g., IPv6 over IPv4.
Internet Prot. Ver. 4 (IPv4) Connection-less protocol for use in PSNs supporting best-effort services. Moderate
Internet Prot. Ver. 6 (IPv6) Similar to IPv4 but with a larger address space, includes a few improvements and Moderate
simplifications.
Multi-Prot. Label Swit. Label Routing prot. that forwards labeled packets that define the source-destination paths Moderate
Swit. Path (MPLS LSP) without routing table look-ups. Instead, at each hop, the label is used as an index
and a new label is generated and sent along the packet to the next hop.
Transport
Bit Ind. Explicit Rep. (BIER) An alternative multicast forwarding technique that does not use per-flow forwarding High
entries. Instead, a BIER header is used to identify the packet’s egress nodes in the
BIER domain. A bit string that is set at each ingress node is used, and the flow is
replicated at each egress node represented by the bit string.
BIER-Traffic Engin. (BIER-TE) Operates similarly to BIER but does not require an Interior Gateway Protocol. TE High
by explicit hop-by-hop forwarding and loose hop forwarding [207] of packets is
supported.

traffic traversing their networks. Since DetNet operates with mon Control and Measurement Plane (CCAMP) standardized
explicit paths, the DetNet WG has drafted a TE architec- by the IETF CCAMP WG, where the aggregate control plane,
tural design for DetNet utilizing similar methodology as the i.e., the control and management planes, is distinctly split
Software Defined Networking (SDN) paradigm. The DetNet between management and measurement entities within the
WG defines three main planes [198]: 1) the (user) application control plane. Additionally, the control plane leverages PCEs
plane, 2) the control plane, and 3) the network plane. The and NMEs. PCEs are considered the core of the control plane.
network plane conforms with the specification of the Internet Given the relevant information through the network interface,
Research Task Force (IRTF) RFC 7426 [209] that details the the PCEs compute the appropriate deterministic path that is
structure and architecture of the SDN networking paradigm. installed in the network plane devices.
This DetNet SDN approach shares similarities with the IEEE c) Network Plane: The aggregate network plane consti-
TSN’s 802.1Qcc management scheme (see Section III-C2) and tutes the operational (control), forwarding (data), and parts of
centralized SDN approach. the applications plane aspects under the RFC 7426 standard.
The network plane interconnects all the Network Interface
a) Application Plane: The collection of applications and
Cards (NICs) in the end systems and intermediate nodes
services that define the network behavior constitute the appli-
(i.e., IP hosts and routers/switches). Additionally, UNIs and
cation plane. For example, network services, such as network
Network-to-Network (NNI) interfaces are used for TE path
topology discovery, network provisioning, and path reserva-
reservation purposes. A network interface is used to enable
tion, are all part of network applications that can be utilized
communication between the network plane and the control
through the application plane and can be accessed by a user-
plane, whereby the control plane can describe and install the
application interface or by other services through the service
physical topology and resources in the network plane.
interface [198]. Moreover, the DetNet WG has defined a user
In general, this DetNet TE architecture envisions a highly
agent application for passing DetNet service requests from
scalable, programmable, and uPnP scheme, where network
the application plane via an abstraction Flow Management
functionality and configurations are easily implemented and
Entity (FME) to the network plane. The management interface
extended.
handles the negotiation of flows between end systems, where
3) Queuing, Shaping, Scheduling, and Preemption: While
requested flows are represented by their corresponding traffic
identifying the appropriate data and control plane solutions
specification (Tspec), i.e., the flow characteristics. The appli-
is imperative for correct operations in DetNet environments,
cations in the application plane communicate via the service
flow control principles (e.g., queuing, shaping, scheduling,
interface with the entities in the control plane
and preemption) must be defined to enable DetNet flows to
b) Control Plane: The collection of functions respon- achieve deterministic bounded latency and packet loss [198].
sible for controlling (e.g., flow installation and processing Flow control usually involves admission control and network
in the forwarding plane) and managing (e.g., monitoring, resource reservation, i.e., bandwidth and buffer space alloca-
configuring, and maintaining) network devices constitute the tion. However, a key aspect of reservation is to standardize
control plane. The DetNet TE architecture utilizes the Com- reservations across multi-vendor networks, such that any la-
 28

tency in one system that differs in another system is accounted planes are defined and specified. This allows users and opera-
for and handled appropriately. tors to easily control, measure, and manage flows dynamically
DetNet flow control will accordingly leverage the IEEE while introducing fast recovery and deterministic bounds on
802.1 TSN queuing and enhanced transmission and traffic QoS parameters.
shaping techniques surveyed in Section III-D. These TSN In contrast to the TSN flow control operations and services
mechanisms include the credit-based shaper (802.1Q, Sec- which are contained within a given L2 network segment, we
tion 34), the time-gated or time-aware transmission selection anticipate that the DetNet flow control operations will have
(802.1Qbv), the cyclic queuing and forwarding or peristaltic significantly larger scale and higher complexity. DetNet flow
shaper (802.1Qch), the asynchronous traffic shaper (802.1Qcr), control will pose several challenges in areas of interoperability,
and the preemption within bridges (802.1Qbu and 802.3br). control data overhead, and, importantly, in guaranteeing QoS
These techniques (except for packet preemption) can relatively metrics across a wide range of L2 network segments. In
easily be implemented in DetNet networks and are a focus of addition, there may arise complex contractual aspects of QoS
collaboration between the DetNet WG and the TSN TG. Service-Level Agreements (SLAs) among owners of different
4) QoS Performance Guarantees between Synchronous and network segments.
Asynchronous DetNet Flows: DetNet flows, similar to TSN
flows, can be transmitted synchronously or asynchronously. E. Flow Integrity
Each method has advantages and disadvantages with respect to
congestion protection. Synchronous DetNet flows traverse Det- DetNet flow integrity follows similar principles and methods
Net nodes that are closely time synchronized (e.g., better than used in IEEE TSN standards and recommended practices.
one microsecond accuracy). The time synchronized DetNet However, some key differences include terminology, L2/L3
nodes can transmit DetNet flows belonging to different traffic integration, and security/privacy considerations.
classes in a coordinated timely fashion, i.e., based on repeated 1) Packet Replication and Elimination Function: The
periodic schedules that are synchronized between the DetNet Packet Replication and Elimination Function (PREF) shares
nodes. This synchronized transmission follows the same prin- several similarities with the TSN TG 802.1CB standard and
ciples as the TSN time-aware gated mechanism (802.1Qbv) is derived from the IETF HSR and PRP mechanisms. PREF
where buffers are shared based on the coordinated time among operates in the DetNet service layer with three main func-
the nodes. A main disadvantage of synchronous transmission is tions [198]:
that there is a tradeoff between fine-grained time synchronized a) Packet Sequencing Information: Packet sequencing
schedules and the required network resource allocation [198]. adds sequence numbers or time-stamps to each packet belong-
In contrast, asynchronous DetNet flows are relayed based on ing to a DetNet flow once. The sequence numbers are used
the judgment of a given individual node. More specifically, the to identify the duplicates if two or more flows converge at a
node assumes the worst-case latency interference among the transit or relay node. Moreover, these sequence numbers can
queued DetNet flows and characterizes flows based on three be used to detect packet loss and/or reordering.
properties: b) Replication Function: Flows are replicated at the
source, i.e., with explicit source routes, whereby a DetNet
1) The maximum packet size of each DetNet flow stream is forwarded on two disjoint paths directed to the same
2) The observational interval, i.e., the time a DetNet flow destination.
is occupying the resource c) Elimination Function: Flow elimination is performed
3) The maximum number of transmissions during the ob- at any node in the path with the intent of saving network
servational interval. resources for other flows further downstream. However, most
Based on the DetNet packet properties and the various commonly, the elimination point is at the edge of the DetNet
header fields resulting from the employed protocol stack, the network, near or on the receiver end system. The receiving
transmission control limits the DetNet flow’s transmission port selectively combines the replicated flows and performs
opportunities to a prescribed number of bit times per observa- packet-by-packet selection of which to discard based on the
tional interval. DetNet’s design goal of deterministic operation packet sequence number.
with extremely low packet loss dictates that each flow must PREF is a proactive measure to reduce or even nullify
be regulated in terms of consumed bandwidth. Furthermore, packet loss. However, the PREF replication mechanism needs
any unused bandwidth can be allocated to non-DetNet flows, at least two disjoint paths to ensure reliability. Therefore, in
and not to any other DetNet flow since each DetNet flow has an effort to enable PREF over networks lacking disjoint paths,
its own resource reservation allowance. Huang et al. [201] defined a single-path PREF function. The
5) Summary and Lessons Learned: DetNet specifies the single-path PREF function does not replicate the DetNet flow
control parameters and properties that can integrate with lower over multiple paths; instead, it uses the same path as the
layer L2 network transport functionalities. These specifications original flow. Therefore, only the terminating or edge node
enable deterministic bounds on QoS flow requirements across has to apply PREF on the flow. The main rationale behind
L3 networks that consist of multiple L2 network segments. using such a technique is that if parts of a flow on the same
DetNet defines a high-level TE architecture that follows an path is corrupted or lost, then the replicated flow can cross-
SDN approach, where key concepts and functions that control check and rebuild the original flow’s corrupted or lost packets,
and manage DetNet flows and the relationships between the essentially performing error correction and remediation. Since
 29

more packets are sent on the same link for a single flow than 6) Security and Privacy Considerations: While ensuring
usual, more bandwidth is needed. Therefore, the technique is bounded worst-case latency and zero packet loss are the
mainly used for applications that require low-rate bursty or main goals of DetNet, security and privacy concerns are
constant traffic services, e.g., blockchain and IoT constrained also important [198]. DetNet is envisaged as a converged
protocols. network that integrates the IT and OT domains. Technologies
2) Enforced Heterogeneity: Similar to its TSN counterpart, that once operated in isolation or with very limited Internet
DetNet enforces bandwidth discrimination between DetNet connectivity, e.g., cyber-physical systems (CPSs), such as the
and non-DetNet flows. The DetNet network dedicates 75% of power grid as well industrial and building control, are now
the available bandwidth to DetNet flows [198, Section 3.3.1]. interconnected [210]. The interconnection makes these CPS
However, to keep bandwidth utilization high, any bandwidth applications susceptible to external attacks and threats that are
that has been reserved for DetNet flows, but is not utilized widespread on consumer IT-based networks [204]. Since any
can be allocated to non-DetNet flows (though not to other potential attack can be fatal and cause considerable damage,
DetNet flows). Thus, DetNet’s architectural model ensures CPS applications present attractive targets for cyber-attackers.
proper coexistence between differentiated services and appli- Mizrahi et al. [204] have defined a threat model and
cations [198]. Additionally, DetNet flows are transmitted in analyzed the threat impact and mitigation for the DetNet
a way that prevents non-DetNet flows from being starved. architecture and DetNet enabled network. The attacks that are
Moreover, some flow control properties from Section V-D are associated with several use cases have been detailed in [205].
employed so as to guarantee the highest priority non-DetNet Since security models and threat analysis are outside the scope
flows a bounded worst-case latency at any given hop. of this paper, we only briefly note that the three main DetNet
3) Fault Mitigation: In addition to the flow replication and security aspects are (i) protection of the signaling or control
bandwidth discrimination, DetNet networks are designed with protocol, (ii) authentication and authorization of the physical
robustness that reduces the chances of a variety of possible controlling systems, and (iii) identification and shaping of
failures. One of the key mechanisms for reducing any disrup- DetNet flows and protection from spoofing and Man-in-the-
tion of DetNet flows is applying filters and policies, similar Middle (MITM) attacks and refer to [204] for further details.
to IEEE 802.1Qci (PSFP), that detect misbehaving flows and 7) Summary and Lessons Learned: The integrity and pro-
can flag flows that exceed a prescribed traffic volume [198]. tection of DetNet flows against possible failures, including
Furthermore, DetNet fault mitigation mechanisms can take intentional and non-intentional failures, is imperative for the
actions according to predefined rules, such as discarding envisaged convergence of the IT and OT domains, i.e., the
packets, shutting down interfaces, or entirely dropping the linking of CPSs with the consumer/enterprise systems. Fur-
DetNet flow. The filters and policers prevent rogue flows from thermore, the secure information dissemination across DetNet
degrading the performance of conformant DetNet flows. enabled networks, including access control and authentication,
must be addressed.
4) IGP-TE Extensions for DetNet Networks: To effectively
Future work should examine whether it would be feasi-
utilize DetNet techniques, i.e., explicit routes as well as con-
ble to ensure reliability without explicit packet replication.
gestion and resource protection, important network informa-
The underlying idea of replication is to proactively replicate
tion, such as node capabilities, available resources, and device
packets for mission-critical applications, since ULL packets
performance, needs to be communicated to and processed at
become stale if retransmissions are used. Therefore, replication
the control entities [202]. The DetNet WG utilizes a PCE
is the easiest way to achieve reliability, albeit at the added
where the necessary network information is fed as input,
cost of bandwidth. State-of-the-art Ethernet technology has
and the PCE can effectively compute a path that satisfies
now been standardized to allow up to 400 Gbps bandwidth.
the QoS requirements of the DetNet flow. Additionally, some
Hence, there should be enough bandwidth for replication for
information can be distributed and collected using already
low to moderate proportions of mission-critical applications. If
defined TE metric extensions for OSPF and ISIS.
mission-critical applications account for large portions of the
Key parameters, including the employed congestion con- applications, then alternative reliability mechanisms based on
trol method, the available DetNet bandwidth, as well as the low-latency coding, e.g., low-latency network coding [211]–
minimum and maximum queuing delay are embedded in sub- [219], may be required.
TLVs [202]. Based on these parameters, OSPF and ISIS
can accurately compute the path according to the perceived
network topology and status. F. Discussion on DetNet Standardization
5) Flow Information Model: In order to simplify imple- DetNet strives to extend and integrate L2 techniques and
mentations and to enable DetNet services to operate on Layers mechanisms with the aim of enabling end-to-end determin-
2 and 3, a DetNet flow information model must be defined istic flows over bridges and routers, i.e., DetNet L3 nodes
to describe the flow characteristics such that nodes within beyond the LAN boundaries. DetNet is envisioned to run
L2 or L3 provide support flows properly between the sender over converged packet switched networks, in particular IP-
and receiver end systems [203]. Farkas et al. [203] have based networks. Essentially, the DetNet architecture provides
specified a DetNet flow and service information model based deterministic properties, e.g. bounded worst-case latency, jitter,
on the data model described in the IEEE 802.1Qcc centralized and packet loss, with the goal of IT and OT convergence
management and reservation standard (see Section III-C2). requiring L2 and L3 capabilities.
 30

The DetNet WG has so far mainly focused on flow man- External IoT
agement, Sec. V-C, and flow integrity, Sec. V-E. The DetNet Network G/W
NME
specifications to date provide correct end-to-end navigation PCEPCE
and encapsulation, including the DetNet data plane and overall LLN: Low-power and
DetNet architecture utilizing stable well-known standards, i.e., Lossy wireless Network Subnet
NME: Network Backbone
IETF RFCs and IEEE standards. For instance, DetNet employs
Management Entity
PCE for path computation, HSR and PRP for path redundancy, PCE: Path Computing BBR_2
as well as SDN and centralized approach to the overall DetNet BBR_1
Element
network. BBR: Backbone Track
As DetNet integrates IT and OT, security is an important Routers
aspect of the DetNet architecture and protocols. While pre-
vious OT network topologies and designs have “air gapped” LLN
security, i.e., completely isolated OT networks from the out-
side world, the convergence of IT and OT will place emphasis Replication
on legacy security protocols and consequently require exten-
sible, flexible, and power efficient security stacks that can be
ported onto OT network components. Furthermore, with the Fig. 23. IPv6 Time Slotted Channel Hopping (6TiSCH) Architecture [221]:
Software Defined Networking (SDN) based applications for Deterministic
emerging “fog” computing platforms, i.e., essentially moving Networking (DetNet) include the Path Computing Element (PCE) for cen-
IT (physical datacenters) close to the OT (physical operation tralized computing of paths supporting frame replication for reliability in
points), it becomes imperative to closely inspect traffic and low-power and lossy networks.
monitor conditions since any intrusion can potentially lead to
catastrophic situations.
IoT access methods [223] because of its simplicity and re-
silience to interference [224]. Moreover, IoT wireless devices
VI. D ET N ET R ESEARCH S TUDIES have widely adopted IPv6 as their default IP layer. 6TiSCH
Only very few research studies have examined DetNet is a scheduling mechanism [225] based on TSCH supporting
aspects. In particular, the flow control aspect of scheduling, IPv6 to achieve DetNet characteristics. Thumbert et al. [221]
and flow integrity through replication have been studied, as have identified the challenges associated with centralized
surveyed in this section. scheduling in 6TiSCH based on SDN to design end-to-end low
latency connectivity. The Path Computing Element (PCE) in
A. Flow Control: Scheduling the 6TiSCH architecture conducts the centralized monitoring
and scheduling management of a TSCH network. The PCE
An important aspect of the deterministic characteristics of
also interacts with the Network Management Entity (NME) to
the packet flow is the centralized network wide scheduling.
compute the optimal allocations and to assign the transmission
The centralized network wide scheduling has already been
resources to the devices. The challenges in applying DetNet for
adopted by many low-latency end-to-end connectivity tech-
6TiSCH include dynamic network topology changes and the
nologies, such as MultiProtocol Label Switching (MPLS).
corresponding runtime modifications of the network resources
In case of MPLS, the Path Computing Element (PCE) is a
in response to network topology changes. Additionally, the
centralized network entity that computes the optimal end-to-
traffic classification should be uniformly supported between
end path based on global topology information. The PCE also
low power wireless links and wired networks.
agrees with the principles of SDN, where all the network
control decisions are centralized. Thus, the PCE can achieve
the characteristics of DetNet. Alternatively, advanced wireless B. Flow Integrity
protocols, especially for industrial applications that require Industrial applications require determinism, i.e., a bounded
deterministic characteristics, such as ISA100.11a and wireless and deterministic delay value, along with reductions in the
HART, already use centralized routing mechanisms [220]. end-to-end packet latency. Towards this end, Armas et al. [226]
Adopting wired technologies, such as DetNet, to wireless have examined a path diversity mechanism with packet repli-
networks poses challenges due to the possibility of hidden cation. Armas et al. have conducted a comprehensive perfor-
and exposed nodes. Additionally, the wireless node mobility mance evaluation to understand the influence of the number of
makes it more complicated to track the delay characteristics. nodes and the number of replications on the energy consump-
As a result, for wireless technologies supporting DetNet, a tion and the end-to-end packet delay. Armas et al. implemented
promising method for enabling determinism is by scheduling a centralized scheduler based on SDN principles in the DetNet
all transmissions through a centralized decision entity. Time architecture framework to compute the disjoint paths and
Slotted Channel Hopping (TSCH) is a physical layer access to apply the flow rules on networks with up to 80 nodes.
technique where multiple devices access the physical resources The packet loss over the network was evaluated through
in terms of time and frequency slots [222]. However, every simulations. The results indicated that with a packet replication
subsequent physical layer access over the same channel hops factor of one, where each packet is duplicated once, the packet
to a different frequency slot to achieve independence from loss was reduced by 90% on average, showing the potential
interference and jamming. TSCH has been widely adopted for of packet replication. As the packet replication factor was
 31

further increased, the packet loss was completely eliminated. Wireless Fronthaul Backhaul
For a given network deployment, the complete packet loss Segment Remote Segment Segment
elimination can be achieved with some combination of a Radio Cloud Optical Core
Optical
degree of disjoint paths and a packet replication factor; any Head -RAN Netw.
additional replication would then waste resources. The energy Eth.
consumption almost doubles (is ∼1.863 times higher) for a
packet replication factor of 1, while the packet replication
5G Ecosystem
factor 4 increases the energy consumption by almost 3 times
(∼2.914), showing significant energy consumption increases Fig. 24. The main network segments that constitute the 5G ecosystem are the
due to packet replication. In addition to the reliability, the wireless segment, the fronthaul segment, as well as backhaul segment with
simulation evaluations have found end-to-end packet latency corresponding and core network. In addition to various research efforts on
the wireless segment, a variety of research efforts have been conducted on
reductions of up to 40% with a packet replication factor of one, the fronthaul as well as the backhaul and corresponding core network. In this
demonstrating the latency reduction potential of path diversity. article we focus mainly on the ULL techniques in the fronthaul and backhaul
Pitfalls of packet replication include bandwidth shortages network segments.
that arise from the competition between replicated packet
traffic and non-replicated traffic, potentially increasing con-
gestion and delays. Also, as the number of flows with packet As illustrated in Fig. 24, the overall 5G ecosystem can be
replication increases in the network, the flow management classified in terms of wireless access, fronthaul, as well as
process becomes extremely difficult in the event of failures that backhaul segment with corresponding and core network. The
require the reallocation of resources. Therefore, addressing the wireless access is responsible for the wireless connectivity
packet replication challenges is a critical aspect of designing between the devices and the radio nodes. The fronthaul con-
reliability mechanisms. SDN based flow management mech- nects the radio nodes to the radio baseband processing units,
anism can potentially optimize the replication factor while while the backhaul connects the radio baseband processing
minimizing the bandwidth utilization, consumed energy, and units to core networks. The core network interconnects with
end-to-end packet latency. the Internet at large, including data centers, to provide end-
to-end services to devices. A large number of 5G research
efforts have been conducted in the wireless access domain;
C. Discussion on DetNet Research Studies
additionally, many articles have presented overviews of the
Overall, there has been relatively little DetNet research to 5G advancements [227]–[239].
date, leaving a wide scope for future research on architectural The recent survey on low latency characteristics in 5G by
and protocol improvements. Key future research challenges Parvez et al. [34] focuses on waveform designs, wireless pro-
include the control plane management, virtualization, and the tocol optimizations, microwave backhaul architectures, SDN
inter-operation with external networks. DetNet depends on architectures for backhaul and core networks, and content
TSN to support deterministic L2 layer support, and hence caching mechanism for 5G. To the best of our knowledge,
requires strict scheduling techniques for resource sharing over there is no prior survey that comprehensively covers the ULL
L2 layers. Moreover, flow synchronization and flow control aspects across the 5G network segments from the fronthaul
(e.g., for traffic shaping) are generally L2 features and hence to the core networks focusing on the transport mechanisms
DetNet does not address these aspects. On the other hand, flow of the user data and the control plane signalling. We fill this
management is a fundamental aspect of DetNet to oversee the gap by providing a comprehensive survey of ULL techniques
management of end-to-end flow connections. SDN inherently across the 5G wireless access, fronthaul, as well as backhaul
provides a centralized management platform to manage the and core networks in this section.
end-to-end connections through continuous monitoring and 5G ULL mechanisms are motivated by applications that
network reconfigurations to preserve the deterministic network require ultra low end-to-end latency. As discussed by Lema
service characteristics. SDN can also play an important role et al. [240], the business use cases for low latency 5G
in integrating DetNet with external networks, as well as networks include health-care and medical applications, driving
in operating in both small scale and large scale wide area and transport, entertainment, and industry automation. Remote
networks. There has also been a lack of use case definitions health-care and medical interventions, including robotic tele-
in emerging markets, such as automatic driving and industrial surgery, require reliable communication with ultra-low latency.
control networks. Assisted and automatic driving require high data rates for
sensor data processing as well as low latency to ensure
VII. 5G U LTRA -L OW L ATENCY (ULL) quick responses to changing road conditions. Immersive and
5th Generation (5G) cellular technology is a paradigm shift integrated media applications, such as Augmented Reality
in the network connectivity as 5G is expected to comprehen- (AR) and Virtual Reality (VR) for gaming and entertainment
sively overhaul the network infrastructure by establishing an require high data rates for video transmissions and extremely
end-to-end ultra-reliable and ultra-low latency connection [3], low latency to avoid jitter in the video and audio. With
[41]. 5G is also expected to improve the network efficiency these demanding business needs and application requirements,
in terms of network utilization, control plane overhead, and 5G is expected to continuously evolve to support ultra and
energy savings. extremely-low latency end-to-end connectivity.
 32

TABLE IV MAC MAC

L ATENCY COMPARISON AT MULTIPLE COMPONENTS OF NETWORK
CONNECTIVITY OVER 3G (H IGH S PEED PACKET ACCESS (HSPA)), 4G Coding DeCoding
(LTE), 4.9G ( PRE 5G), AND 5G [241].
Rate Match. Rate Demat.
Delay Comp. (ms) 3G 4G 4.9G 5G
Scrambling Split ID (eCPRI) De-Scram.
DL Trans. 2 1 0.14 0.125
UL Trans. 2 1 0.14 0.125 Modulation De-mod.
Frame alig. 2 1 0.14 0.125
Scheduling 1.3 0–18 Pre-sch. Pre-sch. Lay. Map. Ch. Est. IDFT
UE proc. 8 4 0.5 0.250 Split IID/ IU
eNB proc. 3 2 0.5 0.250 Pre-Coding (eCPRI)

PHY
PHY
Trans.+Core 2 1 0.1 0.1
Total Delay (ms) 20 10–28 1.5 1 RE Map. RE Demap.

Digital BF Digital BF
Antenna Fronthaul Backhaul
Radio Equipment (RE) Radio Equipment Control (REC) IFFT/CP Add Split E (CPRI) FFT/CP Rmv.
C&M Sync. User C&M Sync. User
Plane Plane Plane Plane Plane Plane D/A A/D

RF
RF
Layer 2 Layer 2
Layer 1 Layer 1 Analog BF Analog BF

Fig. 25. Common Public Radio Interface (CPRI) system overview [242]: The Fig. 26. Split options defined by eCPRI the steps above the horizontal
Radio Equipment Control (REC) connects to the Radio Equipment (RE) via dashed line are processed at the BBU and the steps below the dashed
the CPRI interface. The REC is part of the Base Band Unit (BBU) and the line are processed at the RRH: Split E corresponds to the CPRI data,
RE is part of the Remote Radio Head (RRH) in the Cloud-RAN architecture. split ID corresponds to the eCPRI downlink data after scrambling, split
IID corresponds to the eCPRI downlink data after pre-coding, and split IU
corresponds to the eCPRI uplink data after RE-demap [243].
A. 5G ULL Standardization
In this section, we identify the key components in 5G
standards for supporting ULL mechanisms. Various standard- rates. For instance, a massive MIMO connectivity with 64 an-
ization organizations contribute to the development of 5G tennas for both transmission and reception would require more
standards, including the IEEE and IETF, as well as the than 100 Gbps [245]. Additionally, the CPRI Service Level
Third Generation Partnership Project (3GPP), and the Euro- Agreements (SLAs) require delays below 75 µs. Therefore,
pean Telecommunications Standards Institute (ETSI). We first CPRI poses severe scalability issues as the required data rate
discuss the standards related to the 5G fronthaul interface, increases drastically with the number of antennas for massive
and subsequently we present the 5G architecture components MIMO which are widely considered for 5G networks [245].
which include the backhaul. The fundamental latency limits Dense Wavelength Division Multiplexing (DWDM) and Op-
of 5G standards are summarized in Table IV. The 4.9G tical Transport Networks (OTNs) can support the stringent
corresponds to the optimization efforts for LTE towards 5G, CPRI SLA requirements. However, dense deployments of
where a drastic more than 10 fold reduction in the latency is 5G radio nodes due to the short mmWave range require
achieved. The current standardization efforts have targeted the fiber connectivity to large numbers of radio nodes. Therefore,
total delay for 5G to be 1 ms or lower. eCPRI, an enhanced version of CPRI, has been proposed to
1) Common Public Radio Interface (CPRI and eCPRI): address the scalability issues of CPRI [246]. The 5G fronthaul
a) CPRI: The Common Public Radio Interface enabled by eCPRI will not only reduce the required fronthaul
(CPRI) [243] is a digital interface for transporting information bandwidths, but also relax latency requirements compared to
between Radio Equipment (RE) and Radio Equipment Control CPRI.
(REC). The RE resides at the Remote Radio Head (RRH) b) eCPRI: eCPRI reduces the effective data rate carried
and is responsible for the transmission of radio signals while over the L1 connection between RE and REC. eCPRI also
the baseband signal processing is conducted at the BaseBand removes the mandatory L1 requirements, thus allowing oper-
Unit (BBU) which implements the REC. In particular, CPRI ators to implement low-cost Ethernet links. More specifically,
provides the specifications for packing and transporting the data rate reduction is achieved by various functional
baseband time domain In-phase/Quadrature (I/Q) samples. split options as shown in Fig. 26. The split option defines
Figure 25 illustrates the connectivity of BBU and REC the allocation of the RF and PHY processing steps to the
with the RRH and RE using the CPRI. CPRI mandates RRH and BBU. The steps above the split indicated by a
the physical layer (L1) to be optical Ethernet transmissions horizontal dashed line in Fig. 26 are conducted at the BBU,
over fiber, while the MAC layer can include control and while the steps below the split are conducted at the RRH.
management, synchronization, and user data. CPRI has been Accordingly the split option governs the type of signal (and
widely adapted for LTE and 4G deployments due to the its corresponding QoS requirements) that has to be transmitted
protocol simplicity and readily available dark fiber owned by over the fronthaul network. eCPRI primarily defines two split
cellular operators [244]. options in the downlink. The ID split performs PHY layer bit
5G is expected to support high bandwidth connections up scrambling at the BBU, while RF transmissions are modulated
to several Gbps, resulting in very high effective I/Q CPRI data at the RRH. Similarly, the IID split conducts pre-coding,
 33

TABLE V stack description of eCPRI services over IP and Ethernet

CPRI S PLIT E AS WELL AS E CPRI SPLITS ID , IID ( DOWNLINK ), AND IU is illustrated in Fig. 27. The eCPRI specific protocol layer
( UPLINK ) ONE - WAY PACKET DELAY AND PACKET LOSS
REQUIREMENTS [247]. transports the time domain I/Q data for split E, or frequency
domain I/Q data for splits ID and IU . eCPRI messages are
One-way max. One-way transmitted as UDP/IP packets whereby the eCPRI header
CoS name Example use
packet delay pkt. loss ratio
High User Plane 100 µs 10−7
and data constitute the UDP packet payload. The UDP packet
User Plane (slow), headers contain both the source and destination IP addresses of
Medium 1 ms 10−7
C&M Plane (fast) the eCPRI nodes. Various message types control the overall
Low C&M Plane 100 ms 10−6
operation of eCPRI over Ethernet links, including one-way
delay measurement, remote reset, and event indication.
eCPRI Services Unlike CPRI, which requires point-to-point and point-to-
User Real-Time Other Conn.
C&M Sync. multipoint operation in a master-to-slave configuration, eCPRI
Data Ctrl. eCPRI Serv. OAM
is agnostic to the network topology which may encompass
E.g. local area networks, as well as public routers and switches.
eCPRI Protocol Layer PTP SyncE
SNMP
The logical topologies that are possible with eCPRI include:
• Point-to-point, i.e., one BBU to one RRH which is similar
UDP, TCP,
UDP UDP
SCTP etc. to CPRI.
• Point-to-multi-point, i.e., one BBU to multiple RRHs

IP (IPSec) ICMP
(supported in CPRI as well).
• Multi-point-to-multi-point, i.e., multiple BBUs to multi-
ple RRHs (mesh configuration), unique to eCPRI.
Ethernet MAC, VLAN (priority tag), and MACSec In a generalized Ethernet network carrying multiple traffic
Eth.
CPRI

OAM types (including best effort traffic), the user plane I/Q data and
Ethernet PHY
the real time O&M data require high priority transmissions.
TSN mechanisms, see Sec. III, can enable Ethernet networks
Fig. 27. The eCPRI protocol stack consists of the eCPRI protocol layer, which to meet the eCPRI delay requirements. eCPRI management
transports the data from various split options over generic UDP and IP protocol messages and user plane data can be regarded as Control Data
layers. The lower layers, PHY and MAC, are equivalent to the CPRI protocol.
The eCPRI services as well as the eCPRI control and management data along Traffic (CDT) that is transmitted with high priority scheduling
with synchronization are supported by the eCPRI protocol stack [247]. over the TSN network. Traffic requirements for user plane data
vary for the different split options, which can be assigned
different TSN priority levels. For instance, the C&M data
Resource Element (RE) mapping, digital Bandpass Filter (BF), is typically not as delay sensitive as user plane data; hence,
and IFFT/FFT and Cyclic Prefix (CP) at the BBU. In contrast a lower priority can assigned to C&M traffic. However for
to the downlink, eCPRI defines only one split option in the critical C&M data, such as remote reset while troubleshooting
uplink IU , whereby the PHY layer functions, from the channel a Remote Equipment (RE) problem may require higher priority
estimation to the decoding, are conducted at the BBU, while levels than user data. Therefore, two priority levels can be
RE demapping to RF transmissions are processed at the RRH. assigned to C&M traffic, i.e., a priority level higher than user
In contrast to eCPRI, CPRI only carries the output from the data and another priority level lower than user data. These
IFFT/FFT and Cyclic Prefix (CP) at the BBU to the RF priority levels can be readily supported by TSN networks,
Digital to Analog (D/A) converter at the RRH. The delay which accommodate eight independent priority queues.
requirements for the various Classes of Service (CoS) for c) Summary and Lessons Learned: 5G technology sup-
the ID and IID splits (eCPRI) and the E split (CPRI) are ports diverse applications with a wide range of data rates and
summarized in Table V. The high CoS corresponding to split latency requirements, which directly translate to requirement
E (CPRI) requires the one way maximum packet delay to be for a flexible and scalable fronthaul. CPRI and eCPRI provide
on the order of 100 µs. The split E transports the I/Q data and standardized protocols for inter-operating with existing cellular
in-band Control and Management (C&M) information. The infrastructures. CPRI may not be suitable for supporting
medium CoS, which supports both the user and C&M plane massive broadband services due to the very high required I/Q
data, requires 1 ms delay. The low CoS for the uplink eCPRI data rates. Also, the CPRI latency requirements need to be
IU split requires 100 ms delay. carefully considered and may require the judicious use of the
The eCPRI services include: scheduled traffic concept [168]. eCPRI overcomes the data rate
i) User plane I/Q data transport between BBU and RRH, issue through functional splits but increases the complexity of
user plane control and management (C&M), and support remote radio nodes. Another shortcoming of eCPRI is that
services, such as remote reset. the system considers asymmetrical OFDM in the downlink
ii) Time synchronization between BBU and RRH. and uplink, i.e., single-carrier OFDM (SC-OFDM) in the
iii) Operation and management (OAM), including eCPRI uplink. Symmetrical OFDM systems are being investigated
connection setup, maintenance, and tear-down. for increased spectral uplink efficiency [245]. However, there
eCPRI supports various message formats to transport I/Q is no specific split defined for symmetrical OFDM systems
data according to the adopted split option. The protocol in eCPRI. Remote spectrum analysis for troubleshooting RF
 34

Separated fronthaul flow responsible for the internal frame forwarding from ingress
E.g., user (I/Q), C&M to egress port.
Packet ii) Queueing (interference) delay tQueuing due to ongoing
domain via
Radio transmissions of higher priority frames.
Eth. Bridges
Equip. iii) Self queuing delay tSelf Queuing due to frames of the
Radio Equip. same class that arrive across multiple ports and need to
Controller be sequentially queued.
IEEE 802.1CM IEEE 802.1/802.3 iv) Periodic Constant Bit Rate (CBR) high priority data flow
Fronthaul Profiles Eth. Encapsulation
delay tMaxGoldFrameSize+Pre+SFD+IPG . IQ data flows are
Fig. 28. IEEE 802.1 CM defines the support for Ethernet-based fronthaul in referred to as gold flows in IEEE 802.1 CM. The CBR
a bridged network. Flows are separated into different classes and a specific
fronthaul profile is applied to each class to transport the flows over the Ethernet data delay tMaxGoldFrameSize+Pre+SFD+IPG of a gold
bridges based on the flow requirements [248]. frame corresponds to an IQ data frame with maximum
frame size with Preamble (Pre), Start Frame Delimiter
(SFD), and Inter Packet Gap (IPG).
issues is possible in CPRI; whereas, eCPRI does not provide
such remote RF evaluation capabilities, although splits IU and The total worst-case self-queuing delay in a bridge can be
ID allow for remote RF management. Hence, mechanisms for evaluated based on the number Np of ingress ports that can
the transmission of sampled time domain I/Q samples from receive interfering gold frames which need to be transmitted
the RRH back to the BBU must be developed for advanced over egress port p, and the total number of flows Fi,p supported
troubleshooting. between ingress port i and egress p. Let Gi,pk denote the max-
2) IEEE 802.1CM: Time-Sensitive Networking for Fron- imum number of frames belonging to a gold flow k traversing
thaul: The IEEE 802.1CM standard [248] is a CPRI-IEEE from ingress port i to egress port p that can be grouped into
802.1 collaboration to provide bridged Ethernet connectivity a single time window before the reception of frames at the
for fronthaul networks, as illustrated in Fig. 28. An 802.1CM ingress edge port of the bridge network. The resulting worst-
bridge must support a data rate of 1 Gbps or higher on each case self-queuing delay at port j can be evaluated as
port. The IEEE 802.1CM requirements are derived from CPRI tj,p
Self Queueing = tMaxGoldFrameSize+Pre+SFD+IPG
and eCPRI so as to support various splits, such as splits at NP Fi,p
the FFT, demapping, and scrambling radio functions. IEEE X X
× Gi,p
k . (1)
802.1CM defines mechanisms for end stations, bridges, and
i=1,i6=j k=1
LANs to establish Ethernet networks that can support the time
sensitive transmissions of fronthaul streams. In current cellu- Without preemption, the maximum queuing delay tQueuing
lar network deployments, the separation between RRH and incurred by gold flows depends on the maximum size of the
BBU requires connectivity with stringent latency and capacity low priority frame along with preamble (Pre), Start Frame
requirements. These fronthaul connection requirements could Delimiter (SFD), and the Inter Packet Gap (IPG), which results
not be readily provided by today’s bridged Ethernet networks. in tQueuing = tMaxLowFrameSize+Pre+SFD+IPG . However with
IEEE 802.1CM provides specific mechanisms, such as preemption, a high priority frame is transmitted right after the
scheduling, preemption and synchronization mechanisms, to transmission of the fragment of the preemptable frame, which
satisfy the fronthaul requirements. With IEEE 802.1CM, mo- includes the Cyclic Redundancy Check (CRC) and Inter Frame
bile operators can utilize large segments of existing bridged Gap (IFG). Therefore, the total worst-case delay tMaxBridge for
networks to support 5G fronthaul networks, reducing capital gold flows in a bridge can be evaluated as
expenditures. Moreover, centralized management mechanisms tMaxBridge = tMaxGoldFrameSize+Pre+SFD+IPG
can be employed for automatic network reconfigurations,
+tSF + tQueuing + tSelfQueuing . (2)
reducing the operational expenditures compared to manual
network configuration. IEEE 802.1CM distinguishes Class 1 b) Fronthaul Profiles: In general, the fronthaul flows in
traffic for CPRI and Class 2 traffic for eCPRI. In terms a bridged network are classified into High Priority Fronthaul
of network synchronization, the IEEE 802.1 CM standard (HPF), Medium Priority Fronthaul (MPF), and Low Priority
specifies two mechanisms: i) packet timing using protocols, Fronthaul (LPF) flows. The HPF corresponds to class 1 I/Q
such as the Precision Time Protocol (PTP) for point-to-point data and class 2 user plane data with the requirement of 100 µs
synchronization distribution from a remote common master, end-to-end one-way latency. Similarly, the MPF corresponds
and ii) co-located common master for both BBU and RRH. to the class 2 user plane (slow) data and class 2 C&M (fast)
a) Latency Components of a Bridge: A bridge supporting data. The LPF could include the C&M data of class 1 and 2
fronthaul network functionalities needs to tightly control the traffic. IEEE 802.1 CM defines two profiles, namely profiles A
latency and synchronize its functions. The latency for a single and B, to service different fronthaul technologies supporting
hop in a bridge network is the time duration from the arrival both class 1 and 2. The MPF data is typically assigned a
of the last bit of a given frame at a given bridge port to the priority level immediately below the HPF; whereas, the LPF
arrival of the last bit of the same frame at a particular port at data is assigned a priority immediately below the MPF data.
the next hop bridge. The main delay component are: In contrast to the traffic classes which are designed based on
i) Store and forward delay tSF due to all the elements the relative priorities, the profiles are designed based on the
 35

worst-case end-to-end delay within a given traffic class.

Profile A: The goal of profile A is to simplify the deploy- NGFI-I NGFI-II Backhaul
ments and support only strict priority, focusing on the transport
of I/Q user data as high priority traffic and C&M data with Remote Radio Central EPC, e.g.,
New: Digital
lower priority. The maximum fame size for all traffic is 2000 Head (RRH) Unit (DU) Unit (CU) CRAN BBU
octets. (a)
Profile B: Profile B adopts advanced TSN features, in-
cluding frame preemption, as defined in IEEE 802.3br and NGFI-I CU Backhaul
RRH BBU
+DU
802.1Qbu, as well as strict priorities to carry I/Q user data as
high priority traffic and C&M data as low priority preemptable (b)
traffic. The maximum frame size for user data is 2000 octets, RRH + NGFI-II Backhaul
CU BBU
while all other traffic can have variable maximum frame sizes. DU
c) Summary and Lessons Learned: IEEE 802.1CM pri- (c)
marily supports CPRI and eCPRI connectivity over bridged
networks. IEEE 802.1CM enables cellular operators to use
Fig. 29. Illustration of two-level architecture options for next-generation
the existing Ethernet infrastructure reducing the capital and fronthaul transport network: (a) RRH is connected via NGFI-I fronthaul
operational expenditures. However, the lack of support for interface to Digital Unit (DU) and DU is connected via NGFI-II interface
generalized fronthaul networks limits the applicability of the to Central Unit, (b) RRH is connected via NGFI-I interface to integrated CU
and DU, and (c) DU is integrated with RRH and connected via NGFI-II to
IEEE 802.1CM standard to a wider set of 5G applications, CU [250].
such as crosshaul [249]. The relative performance of the low
priority C&M traffic as compared to the high priority I/Q user
data traffic (i.e., the ULL traffic) still needs to be thoroughly a simplified fronthaul architecture that can utilize the existing
investigated to understand the behavior of traffic classes when standard Ethernet deployments of cellular operators. However,
operating at high load levels that approach the link capacities. IEEE 1914.1 does not define the functional split aspects of
Although the delay and synchronization aspects have been the fronthaul, while aligning with 3GPP to support functional
specified in the standards, the security and reliability issues splits suitable for 5G.
have not yet been considered in detail. Hence, security and b) Two-Level Fronthaul Architecture: IEEE P1914.1 de-
reliability present a wide scope for future research and stan- fines a two-level fronthaul architecture that separates the tradi-
dards development. These security and reliability issues should tional RRU to BBU connectivity in the CRAN architecture into
be investigated by the fronthaul task force which is responsible two levels, namely levels I and II. Level I connects the RRH
for the IEEE 802.1 CM standards development. via a Next Generation Fronthaul Interface-I (NGFI-I) to a new
We note that a cellular operator may choose to change network element, the so-called Digital Unit (DU). Level II con-
priority levels as desired. A potential pitfall is that regular nects the DU via an NGFI-II interface to the newly introduced
(non-fronthaul) traffic could be assigned higher priority than Central Unit (CU), as shown in Fig. 29(a). Figs. 29(b) and
fronthaul user data or C&M traffic. Such a priority assignment (c) show different deployment options with integrated RRH
would increase the self-queuing and queuing delays for the and DU, and with integrated CU and BBU, respectively. The
fronthaul traffic. Thus, the relative priority levels of the dif- purpose of the two-level architecture is to distribute (split)
ferent traffic priority classes need to be carefully considered the radio node (i.e., eNB/base station) protocol functions
in the network resource allocation. between CU and DU such that latencies are relaxed, giving
3) Next Generation Fronthaul Interface (NGFI): more deployment flexibilities. In general, NGFI-I is targeted
a) Overview: Although the IEEE 802.1 CM, CPRI, and to interface with the lower layers of the function split which
eCPRI fronthaul protocols provide implementation directions have stringent delay and data rate requirements. In contrast,
for fronthaul networks, the lack of fronthaul architectural stan- NFGI-II is targeted to interface with the higher layers of the
dardizations has prompted the IEEE standards group to com- function split relative to NGFI-I, relaxing the requirements for
mission the IEEE 1914 Working Group (WG) [250] to define the fronthaul link.
the standards for packet-based Fronthaul Transport Networks The NGFI is designed to mainly address:
(FTN). In particular, the IEEE 1914 WG has defined two i) Scalability: To enable C-RANs and Virtual-RANs that are
standards: i) IEEE P1914.1 focusing on architectural concepts functional split and traffic independent.
related to both data and management fronthaul traffic in an ii) Resource Utilization: To achieve statistical multiplexing
Ethernet based mobile FTN networks, and ii) IEEE P1914.3 by supporting variable MIMO and Coordinated Multi-
focusing on the encapsulation of I/Q data for Radio Over Eth- point (CoMP) for 5G.
ernet (RoE). In comparison to IEEE 1914.3, the latency impact iii) Flexibility: To operate in a radio technology agnostic
on the fronthaul deployment is mainly influenced by IEEE manner while supporting SDN controlled dynamic recon-
P1914.1. Hence, we primarily focus on architectural concepts, figurations.
protocol operations, traffic management, and requirements as iv) Cost Effective: To utilize existing cellular network infras-
well as the definitions for fronthaul links as defined by IEEE tructure.
P1914.1. The goals of IEEE P1914.1 are to support 5G critical Additionally, NGFI supports connectivity to Heterogeneous
use cases, such as massive broadband services and to design Networks (HetNets) by decoupling the transport requirements
 36

TABLE VI

S11

S6b
NGFI T RANSPORT C LASSES OF S ERVICE ; L OW SPLIT, M ED . SPLIT, AND
H IGH SPLIT ARE RELATIVE TO THE POSITIONING OF THE SPLIT IN F IG . 26, Serving PDN
WHEREBY THE LOW SPLIT IS CLOSER TO THE BOTTOM OF F IG . 26. Gateway-C Gateway-C

Class Sub Class Max. Lat. Pri. App.

Sxb
Sync. TBD TBD
C&M
Low Lat. RAN Radio Serving PDN Op. Net.
100 µs 2 S1-U
ctrl. plane Acc. Net Gateway-U Gateway-U (e.g., IMS)
Subclass 0 50 µs 0 ULL data
Subclass 1 100 µs 1 Low split.
Data Plane Subclass 2 1 ms 2 Med. split
Subclass 3 3 ms 3 High split Fig. 30. Illustration of Control and User Plane Separation (CUPS) for the
Legacy EPC as proposed by the 3GPP. The Serving-GW (S-GW) functions and
Subclass 4 10 ms 4 the PDN-Gateway (P-GW) functions in the EPC are split between S-GW-
Backhaul
Trans. Net. Trans. Net. C (i.e., control), S-GW-U, and P-GW-U (i.e., user) to increase the flexibility
1 ms 2 of existing EPC networks [254].
C&M ctrl. plane

from the radio technologies. Thus, multiple traffic classes, as change the radio control state to idle. Thus, within a single
summarized in Table VI, can be transported by the NGFI ULL flow session, there can be multiple user device state
network, mainly to support latencies according to the applica- transitions between idle and active. The core network manages
tion demands. The C&M class supports low-latency control the control plane signalling of the radio technology whereby
plane data for radio node signalling. Data plane latencies advanced methods can be implemented to reduce the state
vary according to the different subclasses 0–4 to support transition overhead during flow setup, thereby reducing the
multiple technologies and deployment versions with multiple latency. For ULL flows, irrespective of whether the traffic is
split options. Subclass 0 requires the highest priority with intermittent or has a constant bit rate, the end-to-end latency
50 µs of maximum allowed latency, while subclass 4 has the should be minimized for both flow setup and steady state traffic
lowest priority and a 10 ms maximum delay bound. Subclass flow.
4 can, for instance, be used for the legacy backhaul over the An efficient backhaul network design can reduce control
NGFI interface. The traffic of each subclass is independently plane signalling for both initial ULL flow setup and steady
transported between the end points without any mutual inter- state traffic. We give brief overviews of the two standardiza-
ference while achieving statical multiplexing gain among the tion efforts that efficiently implement the 5G core network
subclasses. functions for setting up and supporting ULL flows, namely
c) Summary and Lessons Learned: The NGFI primarily Control and User Plane Separation (CUPS) of EPC and Next
addresses the scalability and cost issues with the current Generation (NG) Core.
fronthaul solutions, such as CPRI. With NGFI, connections b) Control and User Plane Separation (CUPS) of EPC:
between DU and CU can be directly connected by an Ethernet The SDN paradigm of separating the control and data plane
link supporting IEEE P1914.1 specifications. The NFGI L2 functions while centralizing the overall control plane has
subclass 0 transport service can readily accommodate the provided substantial advantages in traditional networks. The
requirements of the existing CPRI deployments without any 3GPP has proposed Control and User Plane Separation
changes to the infrastructure deployments. Thus, NGFI is (CUPS) [254] for the Evolved Packet Core (EPC) backhaul
expected to play a significant role in the unification of hetero- of the LTE radio technology, see Fig. 30, to adapt SDN
geneous radio technologies at the transport level and support principles in the cellular backhaul core networks to achieve
converged fronthaul and backhaul networks for converged and similar benefits. Current network deployments are facing
coexisting 4G and 5G technologies. An important aspect to increased capital and operational expenditures when scaling
investigate in future research is the tradeoff between link the infrastructures to meet the capacity demands from the
utilization and multiplexing gain for the standard Ethernet users. This infrastructure scaling problem is exacerbated by
networks while adopting these new fronthaul support archi- the integrated control and user plane functions in the existing
tectures and protocols. backhaul networks. CUPS targets i) flexible deployments in
4) Backhaul Networks: both distributed and centralized control plane, and ii) inde-
a) Overview: The backhaul networks consisting of core pendent scaling of control and user plane functions.
network elements play a critical role in setting up the end- CUPS plays an important role in reducing the overall
to-end flows. Core networks control the user data scheduling end-to-end latency through the cellular operator networks by
in both uplink and downlink. The control signalling of the selecting the user plane nodes that are close to the RAN
radio technology, e.g., LTE, can contribute to flow latency node. In particular, the data is transported without having to
when user devices transition among various states, e.g., idle to interact with the control plane nodes for the path setup, which
active (connected) and vice versa [251]–[253]. For scenarios is especially beneficial for user mobility scenarios. That is,
with intermittent data activity, devices typically implement a the flow paths of user plane nodes are dynamically adapted
state transitioning mechanism from active to idle to conserve according to the requirements and mobility, without having
computing and wireless resources. For instance, if the inter to negotiate with control plane entities, such as SGW-C and
packet delay is more than 40 ms, the device can pro-actively PGW-C. This capability will greatly increase the backhaul
 37

redesigned to separate and isolate the network nodes based on

AUSF N13 UDM
service functions, i.e., functions related to the radio service,
such as user authentication and session management. While
the EPC core provides the network functionality for the LTE
AMF N11 SMF N7 PCF AF backhaul, the NG core specifically provides the backhaul for
the standalone 5G New Radio (NR) technology [256]. A non-
N4 standalone 5G would operate in coexistence with EPC and
LTE support.
UE RAN UPF DN
The existing EPC core collectively implements the LTE
PCF: Policy Control Func. radio service functions in a combined fashion within the
UE : User Equip. AUSF: Ath. Server Func. backhaul network gateways, such as S-GW and P-GW. In
UPF: User Plane Func. AF: Application Func. contrast, the NG core separates the service functions at the
SMF: Sess. Mgt. Func DN : Data Net.
AMF: Acc. and Mob. Mgt. Func.
network nodes level. The service function concept is akin
RAN: Radio Acc. Net.
to Network Function Virtualization (NFV) in that multiple
Fig. 31. Illustration of 3GPP Next Generation (NG) Core: Point-to-point refer- virtualized network functions are needed to implement a single
ence architecture based on service functions to support 5G radio nodes [255].
service function.
b) NG Core Elements: The point-to-point NG core archi-
flexibility of the existing LTE radio technology deployments. tecture is based on service functions supporting the 5G radio
New interfaces, namely Sxa, Sxb, and Sxc, see Fig. 30, have nodes, as show in Fig. 31. The fundamental motivation of the
been introduced to communicate between the control and user NG core is to support advanced network implementations and
planes of the Serving-GW (S-GW). The main advantages of network management schemes, such as network slicing, NFV,
CPUS in comparison to the existing EPC are: network service function chaining, and SDN to address the
scalability and flexibility of the core network. Each NG core
i) Removal of GPRS Tunneling Protocol (GTP) and session element is connected to other elements through Nx interfaces.
management between control plane entities. Critical NG core elements include:
ii) A cross connection interface between control and user i) The Access and Mobility Function (AMF) implements
plane, such that any control function can interact with the access control and mobility aspects of the user con-
any user function. text.
iii) A UE is served by a single control plane, but the data ii) The Session Management Function (SMF) is responsible
flow path may traverse multiple data plane functions. for the data path setup and tracking and terminating based
iv) A control plane function is responsible for creating, on the policy function.
managing, and terminating a flow over the user plane iii) The User Plane Function (UPF) defines the data path
functions. All 3GPP control functions, such as PCC, characteristics based on the users requirements and pol-
charging, and admission control are supported within icy.
control plane function, while the user plane is completely iv) The Policy Control Function (PCF) controls the user
agnostic to the 3GPP control functions. policy, such as roaming and network resource allocations,
v A legacy EPC consisting of S-GW and PDN-GW can for network management, including network slicing.
be replaced with new user plane and control plane split v) The Unified Data Management (UDM) manages the sub-
nodes without any impact on existing implementations. scriber information which is used for admission control
c) Summary and Lessons Learned: CUPS provides a and for defining the data path policies.
mechanism to adapt advanced resource management func- vi) The Network Repository Function (NRF) maintains the
tions, such as SDN, to existing networks while improving registry of service functions distributed throughout the
the flexibility. The reduction of data plane and control plane network.
overhead, particularly the removal of GTP tunneling, allows c) Summary and Lessons Learnt: The NG core decou-
user data to be transported without encapsulation and without ples the network service functions from the gateway nodes,
GTP sessions. Moreover, the user device state transitions allowing the core network to implement the network nodes
trigger control plane activities in the core networks. Therefore, based on service functions, which enhances the deployment
the separation of control and data plane not only increases flexibility. As a result, operators have more freedom in tran-
the flexibility, but also reduces the radio control signalling sitioning from an existing core network to the NG core by
to support ULL flows. Thus, cellular operators can incre- separating the core network elements based on the service
mentally upgrade towards 5G deployments. For distributed functions. However, future research needs to thoroughly exam-
deployments, future research needs to thoroughly examine ine the overhead of the control plane management, e.g., virtu-
the placement and implementation of control and user plane alization [257]. For instance, the overhead directly influences
entities without impacting the overall EPC system behavior. power consumption, and network efficiency for the ULL flow
5) Next Generation (NG) Core: setup in the core network data path, which must be carefully
a) Overview of NG Core Architecture: The 3GPP Next evaluated. Therefore, performance, resource utilization, and
Generation (NG) core [255] is equivalent to the LTE Evolved overhead must be considered while designing the optimal
Packet Core (EPC). However, the NG core network has been infrastructure deployment.
 38

6) Discussion on 5G ULL Standardization: In this section addressing ULL in the fronthaul, backhaul, and network
we have provided a brief overview of key components in the management of fronthaul and backhaul.
5G standardization efforts that contribute to ULL connectivity. 1) 5G Wireless Access ULL Research Studies: In this sec-
Several wireless connectivity and signalling optimizations tion we give a brief overview of the main research directions
have reduced the latency overhead in the data and control on ULL techniques in the 5G wireless access segment. Efforts
planes of the wireless air interface. Also, the new Radio to reduce the latency in the wireless access segment have
Resource Control (RRC) inactive state reduces the signalling been mainly focused on two aspects: i) shortening of the
for the RRC inactive to active state transition (compared to Transmission Time Interval (TTI), and ii) reduced processing
the conventional LTE RRC idle to RRC active transition). A time for each TTI [272]. The TTI is the fundamental time
wide variety of options, e.g., functional splits of CPRI and unit for the protocol operations, e.g., transmissions, in a given
NGFI for the fronthaul, exist for meeting the requirements of wireless technology, e.g., LTE. A shorter TII contributes to
5G components. Therefore, the design of an end-to-end 5G an overall reduced Round-Trip Time (RTT) due to shorter
supported system requires a comprehensive latency analysis cycles. For example, in LTE, the number of OFDM symbols
across all segments to select the right candidate set of transport in one TTI can be reduced from 7 to 2 or 3 OFDM symbol to
mechanisms, protocols, and architectural solutions. reduce the latency [273]. In contrast to LTE which uses only
Broadly speaking, the improvements that the TSN standards Orthogonal Frequency Division Multiplexing (OFDM) based
bring to bridged networks can feed into novel standard de- waveforms, the New Radio (NR) access technology [274]
velopments for Ultra-Reliable and Low Latency Communi- for 5G provides a platform to design and implement more
cations (URLLC) in cellular networks in two main areas: i) flexible waveforms based on both OFDM and non-OFDM over
backhaul network, and ii) fronthaul network. In traditional a wide range of spectrum resources, including microwave and
cellular networks, the various backhaul network nodes, such mmWave [275].
as the Home Subscriber Service (HSS) and the Radio Network In terms of reducing the TTI processing time, if a given
Controller (RNC), are typically interconnected by bridged TTI is divided into multiple sub-blocks, and each block is
networks. The adoption of TSN improves the capabilities independently processed in a pipelined fashion, the overall
and enhances the performance of the bridged networks that processing time can be reduced [272], as illustrated in Fig. 33.
interconnect the backhaul nodes. In contrast, fronthaul nodes, The independent processing of sub-blocks incurs an overhead
such as the Remote Radio Head (RRH) and the Cloud- in terms of both the physical wireless resources (i.e., Resource
RAN (C-RAN), were typically interconnected by point-to- Element (RE)) mapping and the processing overhead for
point optical links (as opposed to the bridged networks) as demapping. The mapping and demapping operations mainly
the fronthaul interconnections have very strict latency and involve table lookups and minimal arithmetic computations.
throughput requirements. The introduction of TSN enables Thus, current hardware implementation can readily accom-
bridged networks to provide the strict latency and throughput modate this mapping and demapping processing overhead.
requirements needed for the fronthaul. Thus, TSN can enable Without pipelined processing, the radio node has to wait for
the end-to-end URLLC support across both the fronthaul and the entire TTI frame to arrive before starting to process the
the backhaul for cellular networks. symbol, incurring the delay.
Overall, the adaptability of each solution for 5G deployment Alternatively, the OFDM sub carrier spacing in the fre-
could impact the end-to-end ULL flow latency. Flexibility quency domain can be increased, thus inherently reducing the
could improve the scalability and network utilization, but TTI duration in the time domain, i.e., reducing the OFDM
the control plane separation requires careful consideration of symbol duration. However, such techniques require increased
control plane overhead and latency. Similarly, deployments of guard bands in both the frequency and time domains to protect
new architectures, such as NG core, could result in efficient from inter-carrier and inter-symbol interferences as well as in-
backhaul management to support ULL mechanism with min- creased hardware complexity in terms of tight synchronization
imal overhead, but may require high expenditures for cellular and sensitive receiver designs.
operators. Nevertheless, as deployment options vary widely The next generation Node B radio node in the context of
based on the implementation, relative performance evalua- 5G is often referred to as gNB; this gNB is equivalent to
tion based on distances between different nodes, interfaces, the eNB in 4G LTE. For simplicity, we follow the common
protocol overhead, transport mechanisms, and architectural eNB terminology to refer to the radio node in both legacy
consideration need to be conducted in future research as and 5G technology. The wireless link latency in 5G networks
ground work towards optimal 5G system design. can typically be attributed to two sources: i) user plane latency
when the User Equipment (UE) is in CONNECTED state (i.e.,
active radio link is established between UE and radio node
B. 5G ULL Research Studies (eNB/gNB)), and ii) control plane latency when device is in
This section surveys the research studies on 5G ULL mech- idle state (i.e., no active radio link connectivity exists). The
anisms following the classification in Fig. 32. In particular, we user plane latency in the uplink consists of the delays for the
first give a brief overview of the main ULL research directions scheduling, and the UE to eNB transport, including the packet
in the 5G wireless access segment and refer to the extensive processing. The wireless control plane latency consists mainly
5G wireless access literature for more details [34], [35], [229], of the delays for the state change from IDLE to CONNECTED
[258]–[260]. Then we survey in detail the research studies through a signaling process, such as PAGING and Random
 39

5G ULL Research Studies, Sec. VII-B

? ? ?
Fronthaul Sec. VII-B2 Backhaul Sec. VII-B3 Network Management Sec. VII-B4
Optical Trans. Tech. [244] Int. Fronthaul and Backhaul [249] Int. Fronthaul and Backhaul Arch. [269]
Freq. Domain Wind. [261] mmWave Backhaul [266] Optical-Wireless Net. [245]
Pack. and Sch. over Eth. [262], [263] In-band mmWave Backhaul [267] SDN based EPC core [270]
TDM-PON DBA [264] TCP over mmWave [268] Dynamic Gateway Placement [271]
Traffic Statistics DBA [265]

Fig. 32. Classification of 5G Research Studies.

Transport block of a given TTI

the Round Trip Time (RTT) for a Scheduling Request (SR)
and GRANT is at least 4 ms, resulting in data transmission
Actual proc.
Sub-block1 Sub-block2 Sub-block3 Sub-block4 delays of 8 ms or more. Proactive granting can reduce the
time
packet delay to less than 4 ms by eliminating the SR and
GRANT procedures.
Proc. of
Sub-block1
sub-block1
2) Fronthaul: The fronthaul segment connects the radio
nodes, i.e., radio transmission nodes, to the radio processing
Sub-block2
Proc. of Proc. of nodes, i.e., radio signal processing [279]. Typically, radio
sub-block2 entire block nodes are referred to as Remote Radio Units (RRUs) and
Proc. of radio processing nodes are referred to as Base Band Units
Sub-block3
Pipeline processing sub-block3
of sub-blocks
(BBUs). Cloud-RAN (CRAN) technology [280] centralizes
Proc. of and virtualizes the BBU functions such that a given BBU
Sub-block4
sub-block4 can connect to and serve several RRHs. Initial CRAN designs
entirely virtualized the BBU functions and transported only
time domain In-Phase/Quadrature (I/Q) samples to RRHs.
Fig. 33. A given frame can be divided into multiple sub-blocks. Each sub- However, the time domain I/Q transport technology was lim-
block is independently processed without having to wait for the entire frame
to arrive to start the processing, reducing the overall latency [272]. ited by strict delay and bandwidth requirements that hampered
the scalability of deployments. Recent CRAN designs feature
flexible BBU function separation between CRAN and RRU
Access CHannel (RACH). With increasing numbers of devices to meet scalability and latency demands [281], [282]. While
connecting to 5G networks, robust scheduling mechanisms are there exist extensive discussions on fronthaul challenges and
essential to preserve the fairness among all the devices in terms future designs [283]–[286], we focus on the key aspects of
of latency and data rate. Intermittent data generation, e.g., in fronthaul techniques supporting ULL connectivity.
IoT, increases the control plane signaling due to the IDLE a) Optical Transport Techniques: The Common Public
to CONNECTED transitions [276]. Furthermore, in small cell Radio Interface (CPRI) [243], see Section VII-A1, imposes
environments, the device mobility, e.g., for automotive and an overall fronthaul link delay limit of 5 ms, excluding the
industrial robot applications, can result in additional data and propagation delay [243]. Typically, the distance between BBU
control plane delays. The additional data plane delay in mobil- and RRU is 20 km with a delay tolerance of 100 µs and
ity scenarios is associated with the wireless link discontinuity a frequency accuracy within 2 ppm (parts per billion). In
during the handover process. Whereas, the control plane delay addition to the CPRI requirements, the deployment consid-
in the mobility scenarios is associated with the signaling over eration should also consider the availability of fiber, cost effi-
the core network due to device transitions between eNBs. ciency, CPRI propagation delay, as well as administration and
Robotic systems in industrial networks require ULL for management, since fiber providers are typically different from
control system loops. As compared to unlicensed wireless mobile network providers. The main topology consideration
access (e.g., WiFi), the licensed LTE and 5G technologies not for deployments are the point-to-point, daisy chain, multi-
only provide ultra reliable and low latency connectivity for path ring, and mesh topologies. Point-to-point links provide
a closed ecosystem of industrial networks, but also support dedicated fiber resources for the fronthaul connectivity, but
seamless mobility for robotic systems [277]. The scheduling can be expensive. The daisy chain topology allows the fiber
of data from the devices is a MAC layer procedure which resources to be shared among multiple RRUs; however, a
incurs significant delays in 5G wireless networks. To address link failure can impact all the connected RRUs. Multipath
the scheduling delay, pro-active granting, similar to Semi- ring and mesh topologies provide generally a better balance
Persistent Scheduling (SPS) [278], i.e., periodic grants to between fiber availability, cost, and resilience to link failures.
device for transmission, can be employed. However, pro-active Fronthaul data can be transported through several optical
granting could reduce the overall link utilization due to over- transport techniques [244]:
provisioning of scheduling resources. In LTE with 1 ms TTI, Optical Transport Network (OTN): The OTN uses a TDM
 40

approach over a single wavelength which can be extended adds wait time for the data filling up the maximum frame
to multiple wavelengths through dense wavelength division size. Hence, reducing the latency requires the transmission of
multiplexing (DWDM). OTN has relatively high power con- short frames.
sumption, as OTN equipment requires power for the optical The scheduling of Ethernet frames can provide multiplexing
transmissions at both receiver and transmitter. gain through resource sharing, however, the scheduling can
Passive Optical Network (PON): PONs may provide incur queuing delays. Therefore, to achieve low latency the
a cost-effective option for fiber deployments, if PONs are overhead, packetization latency, and scheduling delay must be
already deployed for fiber to the home connections. Recent carefully considered. Chang et al. [262] have evaluated the
PON developments [287]–[292] support both high bit rates CRAN performance in terms of packetization and scheduling
and low latencies to meet the fronthaul requirements. PON on the Ethernet fronthaul. For functional splits along layer
technology is also power efficient as compared to the OTN. boundaries, for instance when the complete PHY layer is
Point-to-Point with CWDM: Point-to-point links with implemented in the RRH, or the complete MAC and PHY
a wavelength multiplexer for Coarse Wavelength Division layers are implemented in the RRH, an RRH Ethernet gateway
Multiplexing (CWDM) are generally cheaper than an OTN has been introduced to perform the scheduling, aggregate the
with DWDM. Motivated by diverse optical transport options, traffic from RRH nodes, and discard the packets which are past
Chanclou et al. [244] have proposed a WDM optical network their deadlines. For instance, look-ahead depth packetization
solution to meet the data rates and latency requirements of packs channel estimation I/Q data such that the channel esti-
the CRAN fronthaul. Automatic wavelength assignment is mation data precedes regular payload data in the demodulation.
enabled by passively monitoring the RRUs through a self- That is, demodulation does not wait for all the frame I/Q data
seeded approach [293] that considers the bit rates, latencies, to process the I/Q data related to channel estimation.
jitter and synchronization, as well as fiber availability of the In contrast, the prefetch method [262] waits uniformly
CPRI links. over all the I/Q data for the packetization to receive the
b) Frequency Domain Windowing: The general 5G end- Reference Signal (RS) symbols consisting of I/Q for channel
to-end latency guideline is 1 ms, while the total fronthaul estimation. More specifically, the packetization process is
link (propagation) delay budget is 200 µs [294]. Consider a performed for transporting the I/Q data to the base band
20 km fronthaul link, then the processing delay (for CPRI processing module only when all the required I/Q symbols
signal and protocol processing) would need to be significantly corresponding to the RS within the look-ahead depth buffer
lower than the link (propagation) delay, i.e., on the order of have been received. Thus, transporting the I/Q data needed
a few µs. The general consideration for the processing delay for the channel estimation has priority as compared to regular
in the fronthaul is 5 µs. In an effort to further reduce the I/Q data. Various scheduling policies were applied to study the
processing delay of 5 µs, Liu et al. [261] have designed impact of the packetization process based on first-come, first-
an optical transport system supporting the CPRI-equivalent served (FCFS), shortest processing time (SPT), least remaining
rate of 59 Gbps. 48 LTE RF signals of 20 MHz each were bit (LRB), earliest due date (EDD), and least slack time (LST).
transmitted through a single WDM channel with an effective The performance analysis evaluated the maximum number of
RF bandwidth of 1.5 GHz. The processing delay was reduced RRHs supported over the RRH link for a given Ethernet link
through a Frequency Domain Windowing (FDW) technique capacity, packet size, scheduling policy, and functional split.
that reduces the overall FFT/IFFT size in the process of chan- The simulation results showed that packetization techniques
nel aggregation and de-aggregation. FDW is applied to each (e.g., look-ahead depth and prefetch) while employing the
N -point IFFT corresponding to every aggregated channel. LRB scheduling policy with packet discarding provided a sig-
The FDW technique attenuates the high-frequency components nificant multiplexing gain and supported the maximum number
such that the inter-channel crosstalk is reduced. As a result, the of RRHs. In a related research effort, Hisano et al. [263] have
effective FFT/IFFT size can be reduced, thereby reducing the adapted the gating mechanism (see Section III-D2) to support
overall processing latency. The experimental results for the low-latency 5G fronthaul.
fronthaul distance of 5 km have shown an overall fronthaul d) TDM-PON Dynamic Bandwidth Allocation: In a PON
delay reduction from 5 µs to 2 µs. system, distributed Optical Network Units (ONUs) connects to
c) Packetization and Scheduling over Ethernet: Similar a central Optical Line Terminal (OLT) via a shared optical
to optical transport of I/Q data from BBU to RRH, I/Q data fiber. The transmissions from the ONUs to the OLT are
can be digitized and packetized for the transmissions over controlled by a scheduler implemented at the OLT. In a
Ethernet. Radio over Ethernet (RoE) [295]–[298] defines the TDM-PON system, the OLT coordinates the transmissions
process of converting radio signal I/Q data to packets which from multiple ONUs such that there are no collisions on
can be transported over Ethernet. The main issues associated the shared fiber. The Dynamic Bandwidth Allocation (DBA)
with the packetization process while encapsulating the I/Q mechanism assigns the transmission resources to ONUs based
data over the fronthaul link are: i) overhead, ii) packetization on the QoS deadlines. For each DBA polling cycle, each
latency, and iii) scheduling delay. The packetization overhead ONUs transmits a REPORT message indicating the queue
results from the frame and packet headers. Therefore, to reduce size to the OLT. The OLT processes the REPORT messages
the overhead, each frame must be created with the maximum from all ONUs to determine the transmission schedule. The
I/Q data possible such that the overall number of packets and transmission schedule is then sent to all the ONUs with
Ethernet frames is minimized. However, a large frame size GRANT messages indicating the exact transmission details
 41

Optical band. eval. several hours. The allocated bandwidth is then adapted based
BBU Uplink on the estimated long term mean and variance, which can,
optical for instance, be obtained through monitoring the packet traffic
grant with software defined networking based techniques [301]–
OLT DL Data
[303], the bandwidth allocation requests [304], [305], or
+Uplink
wireless Uplink monitoring the optical signal levels [306]. The estimated
grant optical bandwidth allocation is applied over the subsequent time
Tx period, and a new bandwidth allocation is estimated for each
ONU ONU ONU
time period. The experimental results demonstrated end-to-end
fronthaul latencies of 35 µs, while the effective link bandwidth
RRH RRH RRH utilization was increased by 58% compared to FBA.
However, one of the shortcomings of the proposed band-
Devices width allocation based on traffic statistics is that it does not
consider the specific fronthaul split option. For a traditional
Fig. 34. DBA scheme optimizing latency: Grants for the optical transmissions CRAN, where the RF I/Q samples are transported from RRH
are evaluated in advance and sent to ONUs based on the wireless uplink to BBU, a constant bit rate is required at all times; thus the
information which is known to the BBU [264].
FBA can efficiently meet the fronthaul requirements. Traffic
variations according to varying user activity occur only for
higher order functional CRAN splits. Therefore, traffic statis-
for each specific ONU. This polling DBA mechanism consists tics based bandwidth allocation is limited to higher functional
of reporting the demands and waiting for the grants from split fronthauls with a split position towards the upper end of
centralized scheduler; therefore, typically, the total end-to-end Fig. 26.
PON delay is on the order of milliseconds [299], [300], i.e., f) Summary and Lessons Learnt: In a typical CRAN
much higher than the fronthaul requirements of a few micro deployment where the RF I/Q is transported from RRH to
seconds. A PON system in the CRAN framework connects BBU, the fronthaul traffic is independent of the user data
the RRHs to ONUs, and the BBU to the OLT. Thus, the BBU which results in a constant bit rate over fronthaul links at
can schedule transmissions from the RRHs. Due to the PON all times to support the normal operations of BBU and RRH.
delay characteristics, the PON system is not readily suitable Hence, there can be significant power consumption overhead
for fronthaul application. for the CRAN deployment [307], [308]. Therefore, the new
To address the PON delay, Tashiro et al. [264] have pre- designs of fronthaul solutions should consider the overall en-
sented a novel DBA mechanism specifically for fronthaul ergy consumption in addition to the end-to-end latency [309].
applications. As the BBU assigns the grants for wireless Several advanced physical layer techniques, such as, modu-
upstream transmissions of the devices attached to an RRH lation, detection, and DSP (e.g., I/Q compression) for fiber
(i.e., ONU), the RRH upstream bandwidth requirements are transmissions have been proposed as part of energy efficient
known at the BBU (i.e., OLT) ahead in time. In wireless designs [310]–[312]. While the higher order functional splits
LTE systems, the request reporting to grant reception (related provide statistical multiplexing gains, the worst-case delay
to wireless scheduling) is separated by 4 ms in the protocol must be analyzed to ensure that latency is within the delay
operations, similarly the grant reception to RF transmissions budget of the fronthaul link. The fronthaul infrastructure is
is separated by 4 ms. Hence, the total protocol delay from typically non-flexible and must support the deployments of
request to transmission is 8 ms. As illustrated in Fig. 34, future 5G networks [313]. Therefore, the fronthaul designs,
concurrent to the grant evaluation for wireless transmissions, such as bandwidth allocation and resource sharing mechanism
grants for the optical transmissions of the RRHs (i.e., ONUs) designs, should be able to readily accommodate new devel-
can also be evaluated and transmitted to the RRHs ahead of opments in the 5G technology. Although several techniques
time, eliminating the report and grant cycle between ONUs exists to mitigate the delay in fronthaul networks, there has
and OLT. The experimental evaluation of a TDM PON system been no research yet to address the synchronization of RRH
with advance scheduling has demonstrated average end-to-end and BBU to a universal timing. Flexible fronthaul techniques
latencies of less than 40 µs, and packet jitters of less than 25 µs can be developed based on reconfigurable network functions
for fronthaul distances up to 20 km. and physical layer entities, such as modulators and transparent
e) Traffic Statistics Based Bandwidth Allocation: Fixed spectral converters, in the framework of Software Defined
Bandwidth Allocation (FBA) can address the overhead and Optical Networks (SDON) [314]. For instance, Cvijetic et
scheduling delay incurred by the DBA mechanism, but fixed al. [315] have proposed an SDN based topology-reconfigurable
bandwidth allocations may waste resources due to over pro- optical fronthaul architecture. The dynamic reconfiguration of
visioning. For variable traffic, statistical multiplexing can be fronthaul can support low latency inter BS communications
employed to increasing the bandwidth and resource utilization. necessary for bidirectional Coordinated MultiPoint (CoMP)
Based on this principle, Kobayashi et al. [265] have proposed for inter-cell interference cancellation and inter-cell D2D.
a TDM-PON bandwidth allocation scheme based on the traffic 3) Backhaul:
statistics of the variable fronthaul traffic. The proposed scheme a) Integrated Fronthaul and Backhaul: The backhaul
considers the long term traffic characteristics on the order of connects Radio Access Networks (RANs) to core networks,
 42

e.g., the LTE backhaul connects the RAN eNB node (base may examine whether it is possible to exploit the high capacity
station) to the Evolved Packet Core (EPC) core network. mmWave transport for fronthaul. Also, mmWave transport
Typically, in CRAN technology, the RRH only implements may be suitable for particular 5G connectivity scenarios, e.g.,
a split part of the eNB functions, for instance, the eNB PHY for connecting a Customer Premises Equipment (CPE) home
layer is implemented at the RRH, while the MAC and higher gateway to an external serving gateway, e.g., a 5G base station
layers are implemented at the BBU. Thus, the RRH, the BBU, (gNB).
and the fronthaul connecting them, jointly constitute an eNB. Gao et al. [266] have presented a mmWave based backhaul
Thus, if the endpoints of a link in a 5G network are the for 5G using massive-MIMO to support a high number of
RRH and BBU, then the link operates as a fronthaul. On the radio nodes, i.e., Base Stations (BSs). The proposed approach
other hand, if the endpoints are the eNB and EPC, then the exploits Beam Division Multiplexing (BDM) whereby an
link operates as a backhaul. With the centralization of the independent beam is dedicated to a BS, thus creating a
computing in the core network, such as in a CRAN, the BBU backhaul link through spatial multiplexing. Each mmWave
and EPC can be implemented at a single physical location beam supports a high capacity link, hence, a Time Division
which enables the deployment of a common infrastructure Multiplexing (TDM) scheduling can be employed to share the
in an architecture to support both eNBs and RRHs over a resources within a single beam, supporting multiple BSs over
common platform. a single link. However, the scheduling of BDM resources with
The crosshaul (Xhaul) architecture [249] provides a com- TDM can incur significant end-to-end latency as compared to
mon platform to support both fronthaul and backhaul using an BDM without TDM, and therefore must be carefully evaluated
Xhaul transport network. In the SDN framework, the Xhaul specific to the backhaul latency requirements.
transport network provides reconfigurability while operating c) In-band mmWave Backhaul: The in-band mmWave
over heterogeneous switches and links, such as microwave, technique shares the spectrum resources with the wireless
mmWave, optical, and high speed Ethernet. In an effort to access (i.e., BS to device), and backhaul (i.e., BS to BS and
ensure the ULL capability of configurable integrated fronthaul BS to core network). Since the wireless access and backhaul
and backhaul networks, Li et al. [316] have proposed an resources compete for the same spectrum resources in the
X-Ethernet based on Flexible Ethernet [317] technology for in-band communication, there can be significant overhead in
the Xhaul architecture. The experimental demonstration of X- terms of capacity and latency. To analyze the in-band mmWave
Ethernet has demonstrated an average latency of 640 ns as communications in terms of capacity, Taori et al. [267] have
compared to 30–50 µs in a traditional Ethernet switch, indi- conducted a feasibility study and showed that 25% of the
cating that X-Ethernet can be deployed as a part of the Xhaul resources of the mmWave link is sufficient to support the user
data plane. As the control plane latency of X-Ethernet for data rates over the wireless link up to 0.8 Gbps. Typically,
reconfigurations has not been identified, the overall suitability in the in-band backhaul connectivity, the resources are shared
of X-Ethernet for Xhaul needs further investigations. in TDM fashion between wireless and backhaul applications
b) MillimeterWave (mmWave) Backhaul: Millimeter- impacting both wireless and backhaul end-to-end connectivity
Wave (mmWave) radio technology for wireless communica- during congestion. Although the suitability of in-band commu-
tions operates in the spectrum between 30 and 300 GHz [275], nication is justified in terms of capacity, the implications of in-
[318], [319]. mmWaves have relatively short wavelengths and band communication on the latency has not been characterized,
thus suffer pronounced signal attenuation with propagation and hence can compromise the performance of the entire
distance and due to obstacles. Also, mmWaves exhibit high end-to-end connectivity if not carefully considered. Taori et
directionality. Therefore, mmWave technology exploits beam- al. [267] have also proposed a point-to-multipoint transmission
forming by focusing the signal energy in a narrow spatial for BS to BS (inter-BS) communication based on in-band
beam to support longer propagation distances. Nevertheless, mmWave backhaul connectivity. Inter-BS communication is
the typical operational range of mmWave links is in the necessary to support mobility features, such as handover and
range of several hundred feet. Longer distances require several redirection, as well as advanced radio features, such as inter
intermediate repeaters which increase the latency. On the cell interference cancellation using Coordinated MultiPoint
positive side, the high attenuation property of mmWave signals (CoMP) and self organizing networks. As the deployments
facilitates geographical frequency reuse; thus saving the oper- of BS increase to meet the capacity demands through small
ators spectrum resources by avoiding co-channel interference. cells, the demand for coordination among neighboring BSs
The availability of high bandwidths in the mmWave spec- will increase. Hence, inter-BS communication is an important
trum can provide high capacity links which are potentially aspect of 5G that needs be addressed in a flexible, simple and
suitable for both fronthaul and backhaul. To date, mmWave cost effective manner. In-band mmWave connectivity provides
research in the context of 5G networks has mainly focused on a cost effective solution for inter-BS connectivity along with
the backhaul [267], [320] and we survey the mmWave based flexibility due to a wireless connection, as compared to the
techniques that specifically target ULL transport. Generally, physical deployment of optical fiber infrastructure. Point-to-
the latency requirements in the backhaul are relaxed compared multipoint mmWave connectivity results in a simpler and
to the very strict latency requirements for the I/Q user data cost effective solution through a dynamic reconfiguration of
transport in the fronthaul. Thus, mmWave transport with its re- mmWave links based on the requirements.
quired repeaters for covering distances beyond a few hundred d) TCP over 5G mmWave Links: mmWave links have
feet is generally better suited for backhaul. Future research typically high bandwidths, but are prone to outages as they
 43

require Line-of-Sight (LoS). Thus, there are high chances for fronthaul for 5G applications.
temporary link disruptions, which can result in temporary In contrast, optical wireless communication [325] utilizes
congestion. TCP congestion control could negatively impact the visible light with similar characteristics as mmWave. In
the overall capacity and the latency when a link is temporarily addition to the directionality (LoS) and spatial multiplexing
interrupted as a result of buffer bloating. Active Queue Man- properties, optical wireless communication suffers from inter-
agement (AQM) can be applied to adaptively drop packets ferences due to ambient light sources. Similar to mmWave
from the queue such that the queue size is contained for a designs, the system design should be robust to accommodate
particular flow to keep the end-to-end delay on average below disruptions due to temporary link obstructions. Future designs
a threshold. Control Delay (CoDel) [321] is an AQM technique should also ensure synchronization on the order of 65 ns [168],
which ensures short packet sojourn delays, i.e., short packet [242], [326] while supporting the shared fronthaul and back-
delays from ingress to egress. Each packet is time-stamped at haul.
the ingress and elapsed time is evaluated for the packet drop 4) Network Management: ULL mechanisms are closely
decision. Building on the well-known non-linear relationship related to network management for meeting the flow demands
between drop rate and throughput in TCP [322], the time in terms of resource allocation, reliability, congestion control,
interval between packet drops is reduced inversely proportional and end-to-end QoS. The increasing number of protocols
to the square root of the number of drops so as to linearly vary that support the fronthaul and backhaul connectivity in a
the throughput in relation to the drop count [321]. single end-to-end path creates a heterogeneous environment.
To investigate the impact of temporary 5G mmWave link The comprehensive (end-to-end) management of this het-
disruptions on end-to-end network connections, Pieska et erogeneous network environment can be complex without
al. [268] have evaluated the TCP performance tradeoff be- the support of an inter-operative mechanism. Management
tween capacity and latency. The evaluation indicated that the mechanisms based on Software Defined Network (SDN) could
disruption duration and frequency directly impact the TCP provide a single platform for the coordination of a multitude
performance in addition to the aggressiveness of the TCP of protocols [327]–[330].
variant, such as TCP Reno, TCP Illinois, TCP cubic, and a) Integrated Fronthaul and Backhaul Architecture: Both
TCP Scalable. Although CoDel is a promising technique in Distributed-RAN (DRAN) and CRAN offer unique deploy-
curtailing the buffer bloat in regular TCP networks, Non- ment options for cellular operators to enable cellular con-
LoS (NLOS) occurrences of a mmWave link can result in nectivity to the users. DRAN conducts the baseband signal
significant throughput loss of TCP over mmWave links due to processing at the remote Base Station (BS). As a result, the
extensive CoDel packet dropping, especially for a single flow BS to core network (backhaul) connectivity has relaxed QoS
of the TCP Reno variant. However, the evaluations indicated requirements and thus can be leased in the access network do-
that CoDel can achieve low latency and fast recovery for flows main. On the other hand, CRANs require dedicated fiber links
with short RTTs and disruption durations. Nevertheless, to (typically owned by the cellular operator) for connecting the
avoid the implications of buffer bloat, new TCP designs should radio nodes to the core networks. Therefore, 5G networks are
be able to accommodate short link disruptions, specifically for expected to uniformly support DRAN and CRAN architectures
5G mmWave connectivity for access, fronthaul, and backhaul. for enabling cellular connectivity to the users.
e) Summary and Lessons Learnt: Small cells where the Jungnickel et al. [269] have proposed an integrated fronthaul
devices are close to the radio nodes are widely adopted to and backhaul based on SDN to commonly support DRAN and
save power and to offload the burden on the macro wire- CRAN deployments for cellular operators. Traditional Ethernet
less cells [323]. However, the small cell traffic needs to be deployment strategies [331], such as the E-tree, can be adapted
eventually aggregated at the backhaul, resulting in demanding for the CRAN, and the E-LAN for D-RAN based on their
requirements for the small cell connectivity with the core topology support. To utilize the existing fiber, independent
networks. The connectivity can be provided through fiber wavelengths can be used to meet the latency and capacity
backhaul links that can be shared through FiWi techniques requirements of the fronthaul and backhaul. For example, the
among multiple wireless nodes [239]. mmWave technology is backhaul can use TDM within a single wavelength that is
another promising technology for meeting the high bandwidth shared among multiple radio nodes, and the fronthaul requires
and ULL requirements for next generation connectivity, such a dedicated wavelength between radio node and CRAN. How-
as, small cell backhaul supporting 5G, and fronthaul and ever, the sharing of traditional access networks in E-Tree and
backhaul sharing [324]. mmWave wireless links support i) E-LAN mode can cause security issues. Nevertheless, SDN
high throughputs with short symbol and frame durations, provides both flexibility of statistical multiplexing in both
and ii) high user numbers at a given radio node. Thus, the optical and electrical domains, and security through the
mmWave backhaul can increase the overall capacity of cel- virtualization of the network infrastructure. In a similar study,
lular networks in terms of supported flows with low-latency Ameigeiras et al. [332] have proposed a hierarchical SDN
QoS. As compared to the power consumption of optical architecture based on virtualization, as well as Ethernet and
communications, the power consumption of mmWave links is IPv6 technologies focusing on low latency.
typically significantly higher due to the scattering of wireless b) Optical Wireless Networking: The inter-working of
transmissions as compared to the guided optical waves in optical and wireless technologies has been explored in FiWi
a fiber. Therefore, mmWave requires new energy efficient networks [333]–[335] and in the general context of optical-
methods in resource management and shared backhaul and wireless integration in access and metro networks [336]–
 44

ROADM the S-GW control plane functions. The S-GW data plane
ROADM functions are replaced by the SDN supported switching nodes.
UDWDM Thus, the SDN architecture eliminates the data and control
PON UDWDM bearer based connectivity [341] by replacing the large GTP
PON messages with small SDN control messages. Additionally, the
ROADM SDN based switching nodes can assist in attach and mobility
UDWDM SDN
PON SDN Contoller procedures to reduce the overall load on the EPC core. As
Contoller a result, the overall end-to-end latency can be reduced by
ROADM
UDWDM reducing the data plane and control plane latency introduced
PON by the intermediate nodes in the EPC core.
UDWDM
PON
d) Dynamic Gateway Placement: Lakkakorpi et al. [271]
ROADM have proposed a low latency technique in an SDN based
backhaul network architecture that is fully reconfigurable. The
ROADM: Reconfigurable Optical Add-Drop Multiplexer
UDWDM: Ultra-Dense Wavelength Multiplexing gateway functions and queue management are configured to
achieve low latency by minimizing the flow reestablishment
Fig. 35. Simplified version of ULL optical wireless architecture where
procedures. The SDN controller dynamically programs the
WDM ring connects to wireless nodes and SDN controller through PON switching nodes to implement the network functions based on
framework [245]. the flow characteristics. More specifically, an anchor switching
node is dynamically selected to implement the gateway func-
tions and AQM based on the flow mobility characteristics.
[340]. As next-generation applications demand ULL and high For instance, in case of frequent handovers, the flow path
reliability, there is a great need to integrate optical and must often be reconfigured to pass from one gateway function
wireless technologies with minimal impact on the traditional node to another. Therefore, the gateway functions can be
cellular infrastructures, such as 4G LTE. Towards this end, implemented deeper in the core networks for the specific flows
the 5G STEP-FWD project [245] has been funded by the with frequent handovers, such that only the path routing is
European Commission to develop novel networking solutions updated during handovers. This implementation of the gateway
that closely integrate the optical and wireless technologies functions in the core networks also distributes the gateway
within the 5G framework. functions across the switching nodes, reducing the overall
Vardakas et al. [245] have proposed a high capacity and burden on the core network.
low latency 5G backhaul architecture as illustrated in Fig. 35. e) Summary and Lessons Learnt: In addition to the
Network densification is supported by small cells which are optimization of handover latencies in the wireless access, the
connected to macro BSs through PONs, mainly: i) Optical backhaul architecture should support lower handover laten-
Line Terminals (OLTs) connected through fiber links, ii) cies. Chen et al. [342] have discussed the need for efficient
point-to-point dedicated links, and iii) local Optical Network backhaul architecture to support ultra-short handover latencies.
Unit (ONU) connections through a fiber protection ring offered However, the discussions are limited to DBA mechanisms in
by dark fiber. The dark fiber utilization provides a cost effec- PONs for optimizing the LTE X2 and S1 interfaces.
tive solution as the infrastructure already exists. The wireless In 5G technology, handovers can cause temporary disrup-
access by the small cells and backhaul connectivity supported tions to large data flows which can result in buffer-bloat
by PONs are controlled by a unified SDN management frame- problems across the network. New congestion control mech-
work. mmWave-UDWDM technology effectively utilizes the anisms must be adapted to address the short and temporary
wavelength and space division multiplexing, while PONs pro- disruptions due to handovers during large data transfers. SDN
vide effective backhaul connectivity. The SDN management based strategies can help to address these challenging handover
can support dynamic reconfigurability to support advanced problems [343]. However, existing studies have not considered
network features, such as self-organization and self-healing the control plane latency and complexity, which may signifi-
for ultra-reliable infrastructure networks. cantly impact the overall end-to-end latency. Therefore while
c) SDN Based Evolved Packet Core (EPC) Networks: ensuring the flexibility and reliability in 5G networks, it is also
Pagé et al. [270] have presented an SDN architecture for the important to consider the end-to-end latency, through infras-
LTE Evolved Packet Core (EPC) to support low-latency to- tructure based solutions, such as, dense wavelength-division
wards an evolutionary 5G core network. OpenFlow technology multiplexed (DWDM) optical ring transport networks [344]
has been integrated into the switching nodes that connect the using dark fiber, which is both energy and cost efficient.
BSs (i.e., eNBs) to the EPC. The advantages of SDN based 5) Discussion on ULL 5G Research Studies: There have
switching include reduced need for protocol based transport been numerous research efforts in the wireless access segment
services, such as GTP, elimination of the Serving-Gateway of 5G networks. However, there is still a need for research
(S-GW) which conventionally provides flow based services, to solve compelling technical challenges [345] in enabling
such as buffering and connection management. In contrast to ultra-reliable ULL communication. These research challenges
the conventional LTE backhaul connectivity, where the S-GW include infrastructure reuse, as well as cost and power effi-
anchors the connections of the eNBs to the P-GW, the SDN ciency. Throughout, the implications of wireless access tech-
based EPC is managed by an SDN controller, which replaces niques on ULL services should be carefully considered. For
 45

instance, the emerging 5G New Radio (NR) platform proposes A. Time Sensitive Networks (TSN)
new waveform designs. The symbol and frame durations as
well as the guard band durations (e.g., cyclic prefixes in 1) Inter-Scheduler Coordination: Time aware sharpers im-
the OFDM symbol) in these new waveforms would directly plement local scheduling principles specific to each TSN node.
impact ULL services. Increasingly complex waveforms would The end-to-end time sensitive characteristics of a flow are
require longer symbols and longer frames, not only because of established under the assumption that each TSN node in
limited receiver processing capabilities, but also to maintain the flow path guarantees the time sensitive characteristics.
the synchronous delay between uplink and downlink messages. However, if an intermediate TSN node fails to enforce the TSN
Thus, increasing the waveform complexity would tend to characteristics due to overload, or due to scheduler or timing
increase the wireless round trip delay. Moreover, the channel inaccuracies, the overall end-to-end flow characteristics can
characteristics, such as the maximum (mobility) speed of be compromised. This situation may be more likely for TSN
5G user devices and the cell size influence the guard band nodes that are positioned where multiple flows can aggregate
duration. For example, a high speed train scenario requires a as opposed to the edge nodes (that are traversed by only few
relatively long doppler correction. Similarly, rural deployments flows).
require large cells. In both situations, a long guard band (cyclic To address this shortcoming, future research should develop
prefix) is preferred such that the inter symbol interference can a robust inter-scheduler coordination mechanism. The coordi-
be minimized. A long guard band (cyclic prefix) would imply nation mechanism should facilitate interactions between the
relatively long symbols and frames which could negatively time aware shapers in the TSN nodes in a flow path to ensure
affect ULL services. Thus, the new waveform designs in the the overall end-to-end time sensitive characteristics of the
5G platform should carefully consider the impact on ULL flow. For instance, upon frame reception at the destination, the
services throughout the development process. overall end-to-end latency can be estimated and the informa-
tion can be fed back to the nodes. The TSN nodes can then
With the radio node densification, user mobility between
establish a self performance profile. The interactions of the
radio nodes is expected to increase dramatically, which can
time aware shapers would enable inter-scheduler coordination
significantly increase the control plane complexity in terms
such that each TSN node can guarantee the time sensitive
of user context updates in the core networks. Therefore, a
scheduling relative to the end-to-end behavior of the flow path
light weight (i.e., reduced user context) user information set
similar to time-triggered scheduling [92].
must be managed by the core networks, as opposed to intense
policy and security mechanisms that contribute to control However, time-triggered scheduling depends on time syn-
plane complexity. End-to-end security can reduce the burden chronization to synchronously trigger the scheduling over the
of security measures by the core network. Similarly, user entire flow path. In contrast, the inter-scheduler coordination
activities can be tracked by the radio node to enforce the policy enables dynamic changes of the scheduler policies, such as
and QoS measures across the network. timing adjustments of frame transmissions (i.e., to delay or
advance the transmissions in the scheduled time slots) cor-
SDN plays an important role not only for managing fron-
recting the synchronization inaccuracy. Thus, the time aware
thaul, backhaul, and core networks, but also for reducing
scheduler depends not only on the time synchronization, but
the network complexity by reducing the network function
also on the end-to-end flow characteristics. The inter-scheduler
implementation in dedicated entities, such as policy enforce-
coordination can be enabled through a centralized mechanism.
ment and user authentication. SDN can also integrate the
For instance, an SDN based control can monitor the end-to-end
heterogeneous protocol operations through dynamic packet
characteristics of the flows, and configure the timing advances
header manipulation such that the protocol overheads are
and corrections of the time aware schedulers at specific TSN
minimized.
nodes as required.
Content caching in edge nodes has been widely discussed
2) In-band Control Plane Overhead: Control plane data in
for reducing the delivery latency in fog-RAN and edge com-
TSN network corresponds to the data generated from the con-
puting domains [346]–[349]. SDN provides a platform for
trol functions, e.g., for setting up connections, synchronizing
caching content across the entire network as well as based on
nodes, managing flows, and tearing down connections. The
user demands, optimizing both content caching and latency.
impact of control plane data in TSN networks has been largely
Although 5G technology is primarily focused on power opti-
ignored to date in research and standardization. Control plane
mization of user devices and wireless radio nodes [350]–[352],
traffic could be transported with the in-band connectivity of the
the overall energy consumption of the network responsible for
high priority Control Data Traffic (CDT) class, which carries
the end-to-end packet delivery should also be considered in
time critical information from data sources, such as sensors.
future designs.
However, the control plane traffic would then compete with
the CDT traffic.
Resource reservations in TSN networks to enable the de-
VIII. F UTURE W ORK D IRECTIONS terministic time-sensitive properties are typically estimated
based on CDT traffic requirements. Since the control plane
In this section we discuss the main open TSN and DetNet traffic rates are generally significantly lower than the CDT
research problems and outline directions for future research traffic, the in-band control plane traffic is generally ignored
efforts in TSN and DetNet networks. in the system design and resource reservations. However,
 46

new use TSN cases, such as robotics and automated drones, automotive and industrial environments. However, most use
may require the establishment of short lived TSN flows with cases require external connectivity to inter-operate with other
commensurate frequent triggering of control plane activities. networks. So far, no mechanism exists for establishing a
Thus, new use cases may significantly increase control plane common platform for the inter-operation of TSN networks
data traffic. Therefore, new resource reservations designs, with external non-TSN networks. We envision the inter-
especially for the in-band control plane data transport should operation of TSN networks with non-TSN networks in two
consider both the control plane data traffic as well as the CDT ways: i) centralized SDN management, and ii) ingress and
traffic in evaluating the resource reservation requirements. We egress based management for the TSN network. In case of
anticipate that it will be particularly challenging to ensure the the centralized SDN, a TSN flow outside the TSN network
requirements of the varying and dynamic control plane data can be distinguished and apply for resource reservations to
as compared to the steady CDT traffic. ensure the delay sensitive characteristics. In case of ingress
3) Low Priority Deadline Traffic: TSN nodes preempt and egress based management, an outside flow that requires
an ongoing low priority frame transmission for transmitting TSN properties while traversing through a TSN network can
an incoming high priority frame to guarantee the absolute be identified and configured over the entire flow path such that
minimum TSN node transit delay of the high priority frame. the end-to-end flow integrity is preserved.
Depending on the intensity of the high priority traffic, a low 6) TSN Performance for 5G Fronthaul Applications:
priority frame can be preempted several times. As a result, Fronthaul networks transport the highly delay sensitive In-
the end-to-end delay characteristics of the low priority traffic phase/Quadrature (I/Q) symbol information between the cen-
cannot be guaranteed as the preemption occurrences depend tral base band processing units and the remote radio heads.
directly on the high priority traffic intensity. If the high priority Therefore, typical deployments prefer optical fiber to establish
traffic intensity is significantly higher than the low priority high capacity and low latency links. Although traditional Eth-
traffic intensity, then the end-to-end delay of the low priority ernet can meet the capacity requirements, delay requirements
traffic can be greatly increased. Generally, low priority traffic are challenging to achieve with Ethernet networks. However,
carries delay sensitive data, that is less critical than high due to time sensitive properties, TSN Ethernet is being con-
priority traffic data, but still should be delivered within a worst- sidered as a potential candidate L2 protocol for 5G fronthaul
case deadline. In the current state of the art, there exists no applications as an alternative to the Common Public Radio
mechanism in research nor standards to ensure the worst-case Interface (CPRI) and eCPRI [243] protocols. The adoption
end-to-end delay of low priority traffic under preemption. of TSN for existing Ethernet infrastructures could result in
Therefore, future research needs to develop new mecha- significant capital and operating expenditures for new fiber
nisms to ensure a bounded worst-case delay for low priority deployments. But, the actual performance of TSN networks
traffic in TSN networks. A key challenge in designing a for fronthaul applications has not yet been investigated for the
bounded worst-case delay for low priority traffic is to not various fronthaul splits [285]. The PHY and sub-PHY splits
degrade the performance of high priority traffic. Rather, the require strict deadlines on the order of sub-microseconds. On
new mechanisms should opportunistically accommodate low the other hand, function splits in the MAC, Radio Resource
priority traffic transmissions to meet a worst-case deadline. Control (RRC), and higher cellular protocol layers relax the
4) Impact of Synchronization Inaccuracy: Several tech- delay requirements to the order of milliseconds. A compre-
niques for improving the synchronization accuracy while hensive performance evaluation considering the full range of
minimizing the synchronization errors have been developed aspects of fronthaul applications, such as relative performance
for TSN networks. However, there is a lack of studies that between Ethernet Passive Optical Networks (EPONs) and
quantify the implications of synchronization inaccuracies on TSN Ethernet, packetization, functional split, and fronthaul
the TSN network performance in terms of end-to-end delay distances for a Cloud Radio Access Network (CRAN) system
and throughout. For low cost devices which are typically could provide deep insight towards deployment considerations
employed in large scale networks and for remote applications for mobile operator networks. The ULL requirements of a wide
in IoT scenarios, the synchronization may not be as accurate as range of 5G wireless network applications and services have
for industrial and robotic applications. Due to synchronization been extensively documented, see e.g., [3], [41], [231], [233]–
errors in TSN nodes, the transmissions scheduled by the time- [240], [272], [276], [277], [345], [346], [350], [353]–[370].
aware shaper over a particular time slot, can extend or advance Thus, there is an extensive need to research latency reductions
to adjacent time slots, which can impact the overall scheduling for 5G wireless networks. Investigating the combined impacts
mechanism in a TSN node. For instance, in a time-triggered of the various latency reduction techniques developed in future
network, where all the TSN nodes schedule a flow based on 5G wireless network studies in conjunction with TSN based
synchronized timing information, synchronization errors can fronthaul is an important direction for future research.
offset the time-triggers which can miss the schedule of a very 7) TSN Applied to Wide Area Networks: The time-sensitive
short frame depending on the timing offset duration. Therefore, protocol mechanisms that are applied to micro-environments,
the performance impact due to synchronization errors for such as automotive networks, can also be applied to macro-
multiple priority traffic classes, frame sizes, and timing offset environments, such as Wide Area Networks (WANs). In most
durations requires a close investigation. situations, the end-to-end network delay is dominated by the
5) Ingress and Egress Nodes for TSN: TSN networks wait time in the queues (buffers) of intermediate forwarding
are typically implemented in closed environments, such as nodes. With the TSN rules applied to nodes, the overall end-to-
 47

end delay of a flow over a WAN network can be significantly typically readily accommodate the reverse back propagation
reduced. However, WAN networks typically handle large num- of the small drop-packet notifications. Future research would
bers of flows and operate at very large capacities, making need to conduct a rigorous performance study of the proposed
the TSN flow management very challenging. Despite these drop-packet approach for a wide range of network conditions,
challenges, WAN networks should, in principle, be capable of such as number of flows, relative delay in diversity paths, and
supporting TSN characteristics for specific flows that require numbers of intermediate paths.
strict end-to-end latency bounds, such as remote surgery in 2) Virtualization: L2 Independent Mechanisms: Although
health-care applications, where a doctor could operate on a DetNet focuses on the network layer (L3) and higher layers,
patient across a WAN network. One possible approach to DetNet relies on the time sensitive link layer (L2) to establish
handle the challenging flow management could be through the deterministic L3 packet flow properties. Therefore, pro-
SDN based control. The large geographical WAN area would moting DetNet mechanisms which are independent of the time
likely require an SDN control hierarchy consisting of multiple sensitive link layer could result in the wide adoption of DetNet
control plane entities, such as, local and root controllers, as due to the simple and cost-effective infrastructure support. For
well an orchestrator. instance, packet replication and fragmentation do not require
timing information and can be implemented independently of
the link layer. One way to achieve independence from the
B. Deterministic Networking (DetNet)
link layer is through Network Function Virtualization (NFV),
1) Packet Replication and Elimination: Packet replication which can dynamically scale the resource reservations based
inherently increases the flow reliability by increasing the prob- on the flow demands. However, such NFV mechanisms would
ability of packet delivery to the end destination. Additionally, require hypervisor and control plane management [371]–[375].
packet replication can reduce the overall end-to-end latency NFV also provides a platform for centralized control plane
due to disjoint paths [226]. However, a major disadvantage of management through the SDN framework. Thus, through a
packet replication is the increase in the effective bandwidth unique combination of SDN and NFV, DetNet can be inde-
required for a flow. The required bandwidth can be decreased pendently adapted to networks without time sensitive link layer
by reducing the degree of replication, which can effectively properties.
reduce the reliability. Thus, a balance between bandwidth and 3) Inter-networking: The DetNet inter-networking with an
degree of packet replication must be ensured to operate the external network (i.e., a non-DetNet network) is still an open
network within the required bandwidth (capacity) and latency issue. Generally, a DetNet requires an centralized in-domain
limitations. controller to establish an end-to-end placket flow. Therefore, if
Towards this end, we propose a reverse packet elimination an external flow needs to traverse a DetNet network, the flow
mechanism in which the destination node triggers an instruc- requirements must be configured within the DetNet network.
tion to the nodes in the reverse path to apply a packet drop Ingress and egress nodes could be introduced to manage the
action. For instance, consider the forward direction of a flow configurations for the incoming and outgoing external flows.
with four disjoint paths, i.e., each packet is replicated four In particular, an ingress node could perform admission control
times. These replicated packets traverse independently across to make flow accept/reject decisions. The ingress node would
the disjoint paths through the network to reach the common then also track and manage the packet flows. During this
end destination. We can assume that one packet will arrive process, the ingress node could cooperate with the DetNet cen-
earlier than the others, considering that multiple packets will tralized control entity, e.g., a Path Computing Element (PCE),
likely arrive at the destination. In the current implementation, to accomplish the flow setup over the DetNet network. Thus,
the other packets are discarded when they eventually arrive the cooperation between ingress node and PCE would enable
at the destination. Thus, the effective bandwidth is four fold the inter-networking of DetNet and non-DetNet networks. Of
increased in the forward direction. particular importance will be the study of interactions with
In a DetNet/SDN framework, the destination node can be data center networks. Latency reduction techniques for data
made aware of the exact nodes traversed by the different paths. center networks have received increasing attention in recent
That is, for a given path with node 0 denoting the source years, see for instance [376]–[406]. Thus, it will be important
node and node n denoting the destination node, the destination the extend DetNet into the data center networking domain.
node knows the n − 1 intermediate nodes. If there is sufficient 4) Application-adaptive Resource Reservations: With the
bandwidth in reverse direction, the destination node can send increasing number of applications on end user devices that
a short drop-packet in the reverse direction on paths through require network connectivity, the diversity of the traffic types
which the destination nodes has not yet received the packets; has been increasing. Traditional data included voice and user-
upon reception drop-packet, the intermediate nodes drop the data, such as files and media, while the present data sources
forward packet. This drop-packet would traverse backwards include sensor data as well as tracking and analytics informa-
through the nodes n − 1, n − 2, . . . towards the source node tion. Time sensitive advanced applications in the automotive
while applying the rule to drop until the drop-packet meets the and industrial sectors require special transmission resource
forward packet. Thus, because of the reverse back propagation reservations to meet their ULL requirements. Therefore, we
of the drop-packet, some of the forward direction bandwidth believe application-based resource reservations in L2 and
is freed up. In many networking scenarios, the ratio of uplink PHY (i.e., proactive grants, periodic grants and semi-persistent
traffic to downlink traffic is low, and therefore the uplink can scheduling) across the entire network are a promising tech-
 48

nique to achieve the fundamental limits of ULL end-to-end begun to investigate in detail the performance characteristics
latency for the users. and limitations of these link and network layer ULL mech-
5) Integration and Support for 5G Backhaul Networks: To anisms. Aside from this link and network layer perspective,
meet the growing data demands of ubiquitous mobile devices, extensive standardization and research efforts have approached
5G networks are expected to increase the infrastructure de- ULL support from the perspective of the common wireless
ployments through small cells. The small cells are deployed device-to-core network communication chain. In particular,
close to the users/devices, such as in shopping malls, stadiums, the emerging fifth generation (5G) wireless systems provide
and on university campuses. However, the deployment of extensive support mechanisms for ULL applications.
large numbers of small cells increases the backhaul network The survey has revealed numerous gaps and limitations of
complexity. Backhaul for small cells requires deterministic the existing ULL networking mechanisms that present a wide
latency for establishing secure IP layer connectivity with the range of avenues for future research. Aside from addressing
core networks. DetNet can provide backhaul connectivity for the limitations of the individual ULL support mechanisms,
the small cells in 5G networks. However, the integration of there is an urgent need to comprehensively evaluate the coop-
DetNet at the protocol level (e.g., GPRS Tunneling Protocol eration of the various developed ULL mechanisms. Judicious
(GTP) and IPSec) with the existing cellular networks is yet configuration and cooperation of the various ULL mechanisms
to be thoroughly investigated. Key challenges are to achieve a will likely be critical for providing effective ULL services to
low complexity overall control plane management as well as the end users.
to keep the impact on the existing 5G control plane minimal.
R EFERENCES
[1] M. Wollschlaeger, T. Sauter, and J. Jasperneite, “The future of indus-
C. 5G Networks trial communication: Automation networks in the era of the Internet
of Things and Industry 4.0,” IEEE Industrial Electronics Magazine,
1) Seamless Networks Access: Although 5G is envisioned vol. 11, no. 1, pp. 17–27, Mar. 2017.
to support ULL and high data rates in both the wireless air [2] G. P. Fettweis, “The tactile internet: Applications and challenges,”
interface and the core networks, the seamless network access IEEE Vehicular Technology Magazine, vol. 9, no. 1, pp. 64–70, Mar.
2014.
across multiple operators and connectivity technologies, such [3] M. Maier, M. Chowdhury, B. P. Rimal, and D. P. Van, “The tactile
as cable and DSL networks, is still an open issue in terms Internet: Vision, recent progress, and open challenges,” IEEE Commu-
of inter-networking functions. The inter-networking functions nications Magazine, vol. 54, no. 5, pp. 138–145, May 2016.
[4] A. Finzi, A. Mifdaoui, F. Frances, and E. Lochin, “Incorporating
across multiple networks and technology domains must be able TSN/BLS in AFDX for mixed-criticality avionics applications: Speci-
to negotiate the same set of services while the devices are fication and analysis,” arXiv preprint arXiv:1707.05538, 2017.
operating in the 5G domain. [5] P. Schulz, M. Matthe, H. Klessig, M. Simsek, G. Fettweis, J. Ansari,
S. A. Ashraf, B. Almeroth, J. Voigt, I. Riedel, A. Puschmann,
2) Network Session Migration: The current network con- A. Mitschele-Thiel, M. Muller, T. Elste, and M. Windisch, “Latency
nectivity technology trends, including the 5G technology critical IoT applications in 5G: Perspective on the design of radio
trends, enumerate several network interfaces that concurrently interface and network architecture,” IEEE Communications Magazine,
vol. 55, no. 2, pp. 70–78, Feb. 2017.
connect a user device to different networks, such as WiFi, LTE, [6] S. Samii and H. Zinner, “Level 5 by Layer 2: Time-sensitive networking
3G, and Ethernet. However, the actual network characteristics for autonomous vehicles,” IEEE Communications Standards Magazine,
of each interface change over time. For instance, in cellular vol. 2, no. 2, pp. 62–68, Jun. 2018.
[7] D. Delaney, T. Ward, and S. McLoone, “On consistency and network
communications the transmit power is proportional to the dis- latency in distributed interactive applications: A survey-Part I,” Pres-
tance from the base stations. Hence due to device mobility, the ence: Teleoperators & Virtual Environments, vol. 15, no. 2, pp. 218–
transmit power varies based on the relative distance between 234, Apr. 2006.
[8] ——, “On consistency and network latency in distributed interactive
base station and device. While there exists a static way of applications: A survey-Part II,” Presence: Teleoperators and Virtual
choosing the network interface based on application require- Environments, vol. 15, no. 4, pp. 465–482, Aug. 2006.
ment [407], a dynamic selection based on the network interface [9] C. S. V. Gutiérrez, L. U. S. Juan, I. Z. Ugarte, and V. M. Vilches, “Time-
sensitive networking for robotics,” arXiv preprint arXiv:1804.07643,
characteristics in real time remains an open research challenge. 2018.
Additionally, once a session is established over an interface, [10] F. Prinz, M. Schoeffler, A. Lechler, and A. Verl, “Dynamic real-time
any changes in the network characteristics that impede the orchestration of I4.0 components based on time-sensitive networking,”
Procedia CIRP, vol. 72, pp. 910–915, 2018.
connection quality would negatively impact the end-to-end [11] C. Bachhuber, E. Steinbach, M. Freundl, and M. Reisslein, “On the
latency. To maintain low latencies, an active session should minimization of glass-to-glass and glass-to-algorithm delay in video
be handed over to a different interface without interrupting communication,” IEEE Transactions on Multimedia, vol. 20, no. 1, pp.
238–252, Jan. 2018.
the session. [12] E. Gardiner, “The Avnu alliance theory of operation for TSN-enabled
industrial systems,” IEEE Commun. Standards Mag., vol. 2, no. 1, pp.
IX. C ONCLUSION 5–5, Mar. 2018.
[13] A. Aijaz, M. Simsek, M. Dohler, and G. Fettweis, “Shaping 5G for
This survey has comprehensively covered networks sup- the tactile internet,” in 5G Mobile Communications. Springer, Cham,
porting ultra-low latency (ULL) applications. Providing ULL Switzerland, 2017, pp. 677–691.
[14] A. Aijaz, M. Dohler, A. H. Aghvami, V. Friderikos, and M. Frodigh,
support requires specialized network protocol mechanisms that “Realizing the tactile internet: Haptic communications over next gener-
have been standardized for the link layer in the IEEE Time ation 5G cellular networks,” IEEE Wireless Communications, vol. 24,
Sensitive Networking (TSN) set of standards and for the no. 2, pp. 82–89, Apr. 2017.
[15] K. Antonakoglou, X. Xu, E. Steinbach, T. Mahmoodi, and M. Dohler,
network layer in the IETF Deterministic Networking (DetNet) “Towards haptic communications over the 5G tactile internet,” IEEE
specifications. In addition, extensive research studies have Communications Surveys & Tutorials, in print, 2018.
 49

[16] R. A. Delgado, K. Lau, R. H. Middleton, and T. Wigren, “Networked tions and open issues,” IEEE Communications Surveys & Tutorials,
delay control for 5G wireless machine-type communications using vol. 17, no. 1, pp. 179–197, First Qu. 2015.
multiconnectivity,” IEEE Transactions on Control Systems Technology, [37] D. Cerovic, V. D. Piccolo, A. Amamou, K. Haddadou, and G. Pujolle,
in print, 2018. “Fast packet processing: A survey,” IEEE Communications Surveys &
[17] B. Briscoe, A. Brunstrom, A. Petlund, D. Hayes, D. Ros, J. Tsang, Tutorials, in print, 2018.
S. Gjessing, G. Fairhurst, C. Griwodz, and M. Welzl, “Reducing [38] A. Badr, A. Khisti, W. T. Tan, and J. Apostolopoulos, “Perfecting
internet latency: A survey of techniques and their merits,” IEEE protection for interactive multimedia: A survey of forward error cor-
Communications Surveys & Tutorials, vol. 18, no. 3, pp. 2149–2196, rection for low-delay interactive applications,” IEEE Signal Processing
Third Qu. 2016. Magazine, vol. 34, no. 2, pp. 95–113, Mar. 2017.
[18] M. Lévesque and D. Tipper, “A survey of clock synchronization [39] A. Douik, S. Sorour, T. Y. Al-Naffouri, and M. S. Alouini, “Instantly
over packet-switched networks,” IEEE Communications Surveys & decodable network coding: From centralized to device-to-device com-
Tutorials, vol. 18, no. 4, pp. 2926–2947, Fourth Qu. 2016. munications,” IEEE Communications Surveys & Tutorials, vol. 19,
[19] Y. Luo, F. J. Effenberger, and N. Ansari, “Time synchronization over no. 2, pp. 1201–1224, Second Qu. 2017.
Ethernet Passive Optical Networks,” IEEE Communications Magazine, [40] M. Fidler and A. Rizk, “A guide to the stochastic network calculus,”
vol. 50, no. 10, pp. 136–142, Oct. 2012. IEEE Communications Surveys & Tutorials, vol. 17, no. 1, pp. 92–105,
[20] J. W. Guck, A. Van Bemten, M. Reisslein, and W. Kellerer, “Unicast First Qu. 2015.
QoS routing algorithms for SDN: A comprehensive survey and perfor- [41] M. Simsek, A. Aijaz, M. Dohler, J. Sachs, and G. Fettweis, “5G-
mance evaluation,” IEEE Communications Surveys & Tutorials, vol. 20, enabled tactile internet,” IEEE Journal on Selected Areas in Commu-
no. 1, pp. 388–415, First Qu. 2018. nications, vol. 34, no. 3, pp. 460–473, Mar. 2016.
[21] S. M. Laursen, P. Pop, and W. Steiner, “Routing optimization of AVB [42] J. Eveleens, “Ethernet AVB overview and status,” in Proc. IEEE
streams in TSN networks,” ACM SIGBED Review, vol. 13, no. 4, pp. SMPTE Technical Conf. Exhibition, Oct. 2014, pp. 1–11.
43–48, Sep. 2016. [43] M. D. J. Teener, A. N. Fredette, C. Boiger, P. Klein, C. Gunther,
[22] R. B. da Silva and E. S. Mota, “A survey on approaches to reduce D. Olsen, and K. Stanton, “Heterogeneous networks for audio and
BGP interdomain routing convergence delay on the internet,” IEEE video: Using IEEE 802.1 audio video bridging,” Proceedings of the
Communications Surveys & Tutorials, vol. 19, no. 4, pp. 2949–2984, IEEE, vol. 101, no. 11, pp. 2339–2354, Nov. 2013.
Fourth Qu. 2017. [44] K. Qian, F. Ren, D. Shan, W. Cheng, and B. Wang, “Xpresseth:
[23] J. Luo, J. Jin, and F. Shan, “Standardization of low-latency TCP with Concise and efficient converged real-time ethernet,” in Quality of
explicit congestion notification: A survey,” IEEE Internet Computing, Service (IWQoS), 2017 IEEE/ACM 25th International Symposium on.
vol. 21, no. 1, pp. 48–55, Jan.-Feb. 2017. IEEE, 2017, pp. 1–6.
[24] C. Xu, J. Zhao, and G. M. Muntean, “Congestion control design [45] M. Amjad, M. H. Rehmani, and S. Mao, “Wireless multimedia cogni-
for multipath transport protocols: A survey,” IEEE Communications tive radio networks: A comprehensive survey,” IEEE Communications
Surveys & Tutorials, vol. 18, no. 4, pp. 2948–2969, Fourth Qu. 2016. Surveys & Tutorials, vol. 20, no. 2, pp. 1056–1103, Second Qu. 2018.
[25] P. Mach and Z. Becvar, “Mobile edge computing: A survey on archi- [46] R. Trestian, I. S. Comsa, and M. F. Tuysuz, “Seamless multimedia
tecture and computation offloading,” IEEE Communications Surveys delivery within a heterogeneous wireless networks environment: Are
&Tutorials, vol. 19, no. 3, pp. 1628–1656, Third Qu. 2017. we there yet?” IEEE Communications Surveys & Tutorials, vol. 20,
[26] C. Mouradian, D. Naboulsi, S. Yangui, R. H. Glitho, M. J. Morrow, no. 2, pp. 945–977, Second Qu. 2018.
and P. A. Polakos, “A comprehensive survey on fog computing: State- [47] B. Cizmeci, X. Xu, R. Chaudhari, C. Bachhuber, N. Alt, and E. Stein-
of-the-art and research challenges,” IEEE Communications Surveys & bach, “A multiplexing scheme for multimodal teleoperation,” ACM
Tutorials, vol. 20, no. 1, pp. 416–464, First Qu. 2018. Trans. on Multimedia Computing, Communications, and Applications
[27] J. Ni, K. Zhang, X. Lin, and X. S. Shen, “Securing fog computing (TOMM), vol. 13, no. 2, pp. 21:1–21:28, May 2017.
for Internet of Things applications: Challenges and solutions,” IEEE [48] S. Xu, M. Perez, K. Yang, C. Perrenot, J. Felblinger, and J. Hubert,
Communications Surveys & Tutorials, vol. 20, no. 1, pp. 601–628, “Determination of the latency effects on surgical performance and the
First Qu 2018. acceptable latency levels in telesurgery using the dV-Trainer simulator,”
[28] T. Taleb, K. Samdanis, B. Mada, H. Flinck, S. Dutta, and D. Sabella, Surgical Endoscopy, vol. 28, no. 9, pp. 2569–2576, Sep 2014.
“On multi-access edge computing: A survey of the emerging 5G [49] Q. Zhang, J. Liu, and G. Zhao, “Towards 5G enabled tactile robotic
network edge cloud architecture and orchestration,” IEEE Communi- telesurgery,” arXiv preprint arXiv:1803.03586, 2018.
cations Surveys & Tutorials, vol. 19, no. 3, pp. 1657–1681, Third Qu. [50] “IEC/IEEE TSN Profile for Industrial Automation, Use Cases
2017. IEC/IEEE 60802 V0.61,” Apr. 2018.
[29] I. Al-Anbagi, M. Erol-Kantarci, and H. T. Mouftah, “A survey on [51] C. C. Moallemi and M. Sağlam, “The cost of latency in high-frequency
cross-layer quality-of-service approaches in WSNs for delay and trading,” Operations Research, vol. 61, no. 5, pp. 1070–1086, Oct.
reliability-aware applications,” IEEE Communications Surveys & Tuto- 2013.
rials, vol. 18, no. 1, pp. 525–552, First Qu. 2016. [52] S. Schneele and F. Geyer, “Comparison of ieee avb and afdx,” in Proc.
[30] M. Bennis, M. Debbah, and H. V. Poor, “Ultra-reliable and low- IEEE/AIAA Digital Avionics Systems Conf. (DASC), 2012, pp. 7A1–1–
latency wireless communication: Tail, risk and scale,” arXiv preprint 7A1–9.
arXiv:1801.01270, 2018. [53] Y. Kim and M. Nakamura, “Automotive Ethernet Net-
[31] M. Doudou, D. Djenouri, and N. Badache, “Survey on latency issues work Requirements,” Mar. 2011, last accessed 07/21/2018.
of asynchronous MAC protocols in delay-sensitive wireless sensor [Online]. Available: http://www.ieee802.org/1/files/public/docs2011/
networks,” IEEE Communications Surveys & Tutorials, vol. 15, no. 2, new-avb-KimNakamura-automotive-network-requirements-0311.pdf
pp. 528–550, Second Qu. 2013. [54] J. Takeuchi, “Requirements for Automotive AVB System Profiles,
[32] R. Ford, M. Zhang, M. Mezzavilla, S. Dutta, S. Rangan, and M. Zorzi, White Paper Contributed to AVnu Alliance,” Mar. 2011, last accessed
“Achieving ultra-low latency in 5G millimeter wave cellular networks,” 07/21/2018. [Online]. Available: http://avnu.org/wp-content/uploads/
IEEE Communications Magazine, vol. 55, no. 3, pp. 196–203, Mar. 2014/05/Contributed-Automotive-Whitepaper April-2011.pdf
2017. [55] A. Osseiran, F. Boccardi, V. Braun, K. Kusume, P. Marsch, M. Mater-
[33] P. Park, S. C. Ergen, C. Fischione, C. Lu, and K. H. Johansson, nia, O. Queseth, M. Schellmann, H. Schotten, H. Taoka, H. Tullberg,
“Wireless network design for control systems: A survey,” IEEE Com- M. A. Uusitalo, B. Timus, and M. Fallgren, “Scenarios for 5G mobile
munications Surveys & Tutorials, vol. 20, no. 2, pp. 978–1013, Second and wireless communications: The vision of the METIS project,” IEEE
Qu. 2018. Communications Magazine, vol. 52, no. 5, pp. 26–35, May 2014.
[34] I. Parvez, A. Rahmati, I. Guvenc, A. I. Sarwat, and H. Dai, “A survey [56] J. Huang, M. Zhao, Y. Zhou, and C. C. Xing, “In-vehicle networking:
on low latency towards 5G: RAN, core network and caching solutions,” Protocols, challenges, and solutions,” IEEE Network, in print, pp. 1–7,
IEEE Communications Surveys & Tutorials, in print, 2018. 2018.
[35] G. Sutton, J. Zeng, R. Liu, W. Ni, D. Nguyen, B. Jayawickrama, [57] D. Chatzopoulos, C. Bermejo, Z. Huang, and P. Hui, “Mobile aug-
X. Huang, M. Abolhasan, and Z. Zhang, “Enabling ultra-reliable mented reality survey: From where we are to where we go,” IEEE
and low latency communications through unlicensed spectrum,” IEEE Access, vol. 5, pp. 6917–6950, 2017.
Network, vol. 32, no. 2, pp. 70–77, March-April 2018. [58] “IEEE Standard for Local and metropolitan area networks–Bridges
[36] M. Erol-Kantarci and H. T. Mouftah, “Energy-efficient information and and Bridged Networks,” IEEE Std 802.1Q-2014 (Revision of IEEE Std
communication infrastructures in the smart grid: A survey on interac- 802.1Q-2011), pp. 1–1832, Dec. 2014.
 50

[59] R. M. Metcalfe and D. R. Boggs, “Ethernet: Distributed packet Std 802.1Q-2014/Cor 1-2015, IEEE Std 802.1Qbv-2015, IEEE Std
switching for local computer networks,” Communications of the ACM, 802.1Qbu-2016, and IEEE Std 802.1Qbz-2016), pp. 1–65, Sep. 2017.
vol. 19, no. 7, pp. 395–404, Jul. 1976. [79] K. B. Stanton, “Distributing deterministic, accurate time for tightly
[60] R. M. Metcalfe, “Computer/network interface design: Lessons from coordinated network and software applications: IEEE 802.1AS, the
Arpanet and Ethernet,” IEEE Journal on Selected Areas in Communi- TSN profile of PTP,” IEEE Communications Standards Magazine,
cations, vol. 11, no. 2, pp. 173–180, Feb. 1993. vol. 2, no. 2, pp. 34–40, Jun. 2018.
[61] D. Wright, “The history of the IEEE 802 standard,” IEEE Communi- [80] “IEEE Standard for a Precision Clock Synchronization Protocol for
cations Standards Magazine, vol. 2, no. 2, pp. 4–4, Jun. 2018. Networked Measurement and Control Systems,” IEEE Std 1588-2008
[62] J. Farkas, L. L. Bello, and C. Gunther, “Time-sensitive networking (Revision of IEEE Std 1588-2002), pp. 1–300, Jul. 2008.
standards,” IEEE Communications Standards Magazine, vol. 2, no. 2, [81] R. Alvizu, G. Maier, N. Kukreja, A. Pattavina, R. Morro, A. Capello,
pp. 20–21, Jun. 2018. and C. Cavazzoni, “Comprehensive survey on T-SDN: Software-
[63] N. Finn, “Introduction to Time-Sensitive Networking,” IEEE Commu- defined networking for transport networks,” IEEE Communications
nications Standards Magazine, vol. 2, no. 2, pp. 22–28, Jun. 2018. Surveys & Tutorials, vol. 19, no. 4, pp. 2232–2283, Fourth Qu. 2017.
[64] J. L. Messenger, “Time-sensitive networking: An introduction,” IEEE [82] B. A. A. Nunes, M. Mendonca, X.-N. Nguyen, K. Obraczka, and
Communications Standards Magazine, vol. 2, no. 2, pp. 29–33, JUNE T. Turletti, “A survey of software-defined networking: Past, present,
2018. and future of programmable networks,” IEEE Communications Surveys
[65] “IEEE Standard for Local and Metropolitan Area Networks - Timing & Tutorials, vol. 16, no. 3, pp. 1617–1634, Third Qu. 2014.
and Synchronization for Time-Sensitive Applications in Bridged Local [83] M. Bjorklund, “YANG - A Data Modeling Language for the Network
Area Networks,” IEEE Std 802.1AS-2011, pp. 1–292, Mar. 2011. Configuration Protocol (NETCONF),” RFC 6020, Oct. 2010. [Online].
[66] M. Holness, “IEEE Draft Standard for Local and metropolitan Available: https://rfc-editor.org/rfc/rfc6020.txt
area networks–Media Access Control (MAC) Bridges and Virtual [84] A. Bierman, “Guidelines for Authors and Reviewers of YANG
Bridged Local Area Networks Amendment: YANG Data Model,” IEEE Data Model Documents,” RFC 6087, Jan. 2011. [Online]. Available:
P802.1Qcp/D0.7 December 2016, Dec. 2016. https://rfc-editor.org/rfc/rfc6087.txt
[67] “IEEE Standard for Local and Metropolitan Area Networks—Virtual [85] R. Enns, M. Bjorklund, A. Bierman, and J. Schnwlder, “Network
Bridged Local Area Networks Amendment 14: Stream Reservation Configuration Protocol (NETCONF),” RFC 6241, Jun. 2011. [Online].
Protocol (SRP),” IEEE Std 802.1Qat-2010 (Revision of IEEE Std Available: https://rfc-editor.org/rfc/rfc6241.txt
802.1Q-2005), pp. 1–119, Sep. 2010. [86] A. Bierman, M. Bjorklund, and K. Watsen, “RESTCONF Protocol,”
[68] “IEEE Draft Standard for Local and metropolitan area networks– RFC 8040, Jan. 2017. [Online]. Available: https://rfc-editor.org/rfc/
Media Access Control (MAC) Bridges and Virtual Bridged Local rfc8040.txt
Area Networks Amendment: Stream Reservation Protocol (SRP) En- [87] “IEEE Standard for Device Discovery, Connection Management, and
hancements and Performance Improvements,” IEEE P802.1Qcc/D2.0, Control Protocol for IEEE 1722(TM) Based Devices,” IEEE Std
October 2017, pp. 1–207, Jan. 2017. 1722.1-2013, pp. 1–366, Oct. 2013.
[69] N. Finn, “IEEE Draft Standard for Local and metropolitan area [88] F. Chen, “Resource Allocation Protocol (RAP) based on LRP
networks–Media Access Control (MAC) Bridges and Virtual Bridged for Distributed Configuration of Time-Sensitive Streams,” 2017,
Local Area Networks Amendment: Link-local Registration Protocol,” http://ieee802.org/1/files/public/docs2017/tsn-chen-RAP-whitepaper-
IEEE P802.1CS/D1.2 December 2017, Dec. 2017. 0917-v01.pdf.
[70] “IEEE Standard for Local and Metropolitan Area Networks - Virtual [89] J. Specht and S. Samii, “Urgency-based scheduler for time-sensitive
Bridged Local Area Networks Amendment 12 Forwarding and Queuing switched ethernet networks,” in Proc. IEEE Euromicro Conf. on Real-
Enhancements for Time-Sensitive Streams,” IEEE Std 802.1Qav-2009 Time Systems, Jul. 2016, pp. 75–85.
(Amendment to IEEE Std 802.1Q-2005), pp. C1–72, Jan. 2009. [90] C. Simon, M. Maliosz, and M. Mate, “Design aspects of low-latency
[71] “IEEE Standard for Local and metropolitan area networks – Bridges services with time-sensitive networking,” IEEE Communications Stan-
and Bridged Networks - Amendment 25: Enhancements for Scheduled dards Magazine, vol. 2, no. 2, pp. 48–54, Jun. 2018.
Traffic,” IEEE Std 802.1Qbv-2015 (Amendment to IEEE Std 802.1Q— [91] P. Pedreiras, P. Gai, L. Almeida, and G. C. Buttazzo, “FTT-Ethernet:
as amended by IEEE Std 802.1Qca-2015, IEEE Std 802.1Qcd-2015, A flexible real-time communication protocol that supports dynamic
and IEEE Std 802.1Q—/Cor 1-2015), pp. 1–57, Mar. 2016. QoS management on Ethernet-based systems,” IEEE Transactions on
[72] “IEEE Standard for Ethernet Amendment 5: Specification and Manage- Industrial Informatics, vol. 1, no. 3, pp. 162–172, Aug. 2005.
ment Parameters for Interspersing Express Traffic,” IEEE Std 802.3br- [92] P. Meyer, T. Steinbach, F. Korf, and T. C. Schmidt, “Extending IEEE
2016 (Amendment to IEEE Std 802.3-2015 as amended by IEEE 802.1 AVB with time-triggered scheduling: A simulation study of the
St802.3bw-2015, IEEE Std 802.3by-2016, IEEE Std 802.3bq-2016, and coexistence of synchronous and asynchronous traffic,” in Proc. IEEE
IEEE Std 802.3bp-2016), pp. 1–58, Oct. 2016. Vehicular Networking Conf. (VNC), 2013, pp. 47–54.
[73] “IEEE Standard for Local and metropolitan area networks – Bridges [93] J. Specht and S. Samii, “Synthesis of queue and priority assignment
and Bridged Networks – Amendment 26: Frame Preemption,” IEEE for asynchronous traffic shaping in switched ethernet,” in Proc. IEEE
Std 802.1Qbu-2016 (Amendment to IEEE Std 802.1Q-2014), pp. 1–52, Real-Time Systems Symp. (RTSS), Dec 2017, pp. 178–187.
Aug. 2016. [94] H. Zhang and D. Ferrari, “Rate-controlled service disciplines,” Journal
[74] “IEEE Standard for Local and metropolitan area networks–Bridges of High Speed Networks, vol. 3, no. 4, pp. 389–412, 1994.
and Bridged Networks–Amendment 29: Cyclic Queuing and Forward- [95] ——, “Rate-controlled static-priority queueing,” in Proc. IEEE Info-
ing,” IEEE 802.1Qch-2017 (Amendment to IEEE Std 802.1Q-2014 as com, 1993, pp. 227–236.
amended by IEEE Std 802.1Qca-2015, IEEE Std 802.1Qcd(TM)-2015, [96] L. Georgiadis, R. Guérin, V. Peris, and K. N. Sivarajan, “Efficient net-
IEEE Std 802.1Q-2014/Cor 1-2015, IEEE Std 802.1Qbv-2015, IEEE work QoS provisioning based on per node traffic shaping,” IEEE/ACM
Std 802.1Qbu-2016, IEEE Std 802.1Qbz-2016, and IEEE Std 802.1Qci- Transactions on Networking, vol. 4, no. 4, pp. 482–501, Aug. 1996.
2017), pp. 1–30, Jun. 2017. [97] H. Kirrmann, K. Weber, O. Kleineberg, and H. Weibel, “HSR: Zero
[75] J. Specht, “IEEE Draft Standard for Local and metropolitan area recovery time and low-cost redundancy for industrial ethernet (high
networks–Media Access Control (MAC) Bridges and Virtual Bridged availability seamless redundancy, IEC 62439-3),” in Proc. IEEE Conf.
Local Area Networks Amendment: Asynchronous Traffic Shaping,” on Emerging Techn. & Factory Automation (ETFA), 2009, pp. 1–4.
IEEE P802.1Qcr/D0.2, November 2017, Nov. 2017. [98] H. Kirrmann, M. Hansson, and P. Muri, “IEC 62439 PRP: Bumpless
[76] “IEEE Standard for Local and metropolitan area networks–Frame recovery for highly available, hard real-time industrial networks,” in
Replication and Elimination for Reliability,” IEEE Std 802.1CB-2017, Proc. IEEE Conf. on Emerging Technologies and Factory Automation
pp. 1–102, Oct. 2017. (ETFA), 2007, pp. 1396–1399.
[77] “IEEE Standard for Local and metropolitan area networks– Bridges and [99] D. Oran, “OSI IS-IS Intra-domain Routing Protocol,” RFC 1142, Feb.
Bridged Networks - Amendment 24: Path Control and Reservation,” 1990. [Online]. Available: https://rfc-editor.org/rfc/rfc1142.txt
IEEE Std 802.1Qca-2015 (Amendment to IEEE Std 802.1Q— as [100] P. Unbehagen, N. Bragg, D. Allan, D. Fedyk, and P. J. Ashwood-
amended by IEEE Std 802.1Qcd-2015 and IEEE Std 802.1Q—/Cor Smith, “IS-IS Extensions Supporting IEEE 802.1aq Shortest Path
1-2015), pp. 1–120, Mar. 2016. Bridging,” RFC 6329, Apr. 2012. [Online]. Available: https:
[78] “IEEE Standard for Local and metropolitan area networks–Bridges and //rfc-editor.org/rfc/rfc6329.txt
Bridged Networks–Amendment 28: Per-Stream Filtering and Policing,” [101] J. Farkas, N. Bragg, P. Unbehagen, G. Parsons, P. J. Ashwood-Smith,
IEEE Std 802.1Qci-2017 (Amendment to IEEE Std 802.1Q-2014 as and C. Bowers, “IS-IS Path Control and Reservation,” RFC 7813,
amended by IEEE Std 802.1Qca-2015, IEEE Std 802.1Qcd-2015, IEEE Jun. 2016. [Online]. Available: https://rfc-editor.org/rfc/rfc7813.txt
 51

[102] F. Bannour, S. Souihi, and A. Mellouk, “Distributed SDN control: IEEE Information Technology, Electronics and Mobile Communication
Survey, taxonomy, and challenges,” IEEE Communications Surveys & Conf., Oct. 2016, pp. 1–8.
Tutorials, vol. 20, no. 1, pp. 333–354, First Qu. 2018. [124] S. S. Craciunas and R. S. Oliver, “An overview of scheduling mecha-
[103] J. Vasseur, A. Farrel, and G. Ash, “A Path Computation Element nisms for time-sensitive networks,” Real-time Summer School, LEcole
(PCE)-Based Architecture,” RFC 4655, Aug. 2006. [Online]. Available: dEte Temps Reel (ETR), Tech. Rep., 2017.
https://rfc-editor.org/rfc/rfc4655.txt [125] L. L. Bello, “Novel trends in automotive networks: A perspective on
[104] N. G. Nayak, F. Dürr, and K. Rothermel, “Incremental flow scheduling ethernet and the IEEE audio video bridging,” in Proc. IEEE Emerging
and routing in time-sensitive software-defined networks,” IEEE Trans. Technology and Factory Automation, Sep. 2014, pp. 1–8.
on Industrial Informatics, vol. 14, no. 5, pp. 2066–2075, May 2018. [126] F. Dürr and N. G. Nayak, “No-wait packet scheduling for IEEE time-
[105] “IEEE Standard for Local and metropolitan area networks–Port-Based sensitive networks (TSN),” in Proc. ACM Int. Conf. on Real-Time
Network Access Control,” IEEE Std 802.1X-2010 (Revision of IEEE Networks and Systems, 2016, pp. 203–212.
Std 802.1X-2004), pp. 1–205, Feb. 2010. [127] S. S. Craciunas, R. S. Oliver, M. Chmelı́k, and W. Steiner, “Scheduling
[106] “IEEE Standard for Local and metropolitan area networks – Port-Based real-time communication in IEEE 802.1 Qbv time sensitive networks,”
Network Access Control Amendment 1: MAC Security Key Agreement in Proc. ACM Int. Conf. on Real-Time Networks and Systems, 2016,
Protocol (MKA) Extensions,” IEEE Std 802.1Xbx-2014 (Amendment to pp. 183–192.
IEEE Std 802.1X-2010), pp. 1–107, Dec 2014. [128] M. H. Farzaneh, S. Kugele, and A. Knoll, “A graphical modeling tool
[107] “IEEE Standard for Local and Metropolitan Area Networks: Media supporting automated schedule synthesis for time-sensitive network-
Access Control (MAC) Security,” IEEE Std 802.1AE-2006, pp. 1–150, ing,” in Proc. IEEE Int. Conf. on Emerging Techn. and Factory Autom.
Aug 2006. (ETFA), Sept 2017, pp. 1–8.
[108] “IEEE Standard for Local and metropolitan area networks–Media [129] P. Pop, M. L. Raagaard, S. S. Craciunas, and W. Steiner, “Design
Access Control (MAC) Security Amendment 1: Galois Counter Mode– optimisation of cyber-physical distributed systems using IEEE time-
Advanced Encryption Standard– 256 (GCM-AES-256) Cipher Suite,” sensitive networks,” IET Cyber-Physical Systems: Theory Applications,
IEEE Std 802.1AEbn-2011 (Amendment to IEEE Std 802.1AE-2006), vol. 1, no. 1, pp. 86–94, 2016.
pp. 1–52, Oct 2011. [130] F. Smirnov, M. Glaß, F. Reimann, and J. Teich, “Optimizing message
[109] “IEEE Standard for Local and metropolitan area networksMedia Ac- routing and scheduling in automotive mixed-criticality time-triggered
cess Control (MAC) Security Amendment 2: Extended Packet Num- networks,” in Proc. ACM/EDAC/IEEE Design Autom. Conf. (DAC),
bering,” IEEE Std 802.1AEbw-2013 (Amendment to IEEE Std 802.1AE- June 2017, pp. 1–6.
2006), pp. 1–67, Feb. 2013. [131] R. Mahfouzi, A. Aminifar, S. Samii, A. Rezine, P. Eles, and Z. Peng,
[110] “IEEE Standard for Local and metropolitan area networks–Media “Stability-aware integrated routing and scheduling for control applica-
Access Control (MAC) Security - Amendment 3:Ethernet Data En- tions in ethernet networks,” in 2018 Design, Automation Test in Europe
cryption devices,” IEEE Std 802.1AEcg-2017 (Amendment to IEEE Std Conference Exhibition (DATE), March 2018, pp. 682–687.
802.1AE-2006 as amended by IEEE Std 802.1AEbn-2011 and IEEE Std [132] F. Smirnov, M. Glaß, F. Reimann, and J. Teich, “Formal timing analysis
802.1AEbw-2013), pp. 1–143, May 2017. of non-scheduled traffic in automotive scheduled TSN networks,” in
[111] “IEEE Standard for Local and metropolitan area networks - Secure Proc. IEEE Design, Automation Test in Europe Conf. Exhibition, Mar.
Device Identity,” IEEE Std 802.1AR-2009, pp. 1–77, Dec 2009. 2017, pp. 1643–1646.
[112] M. Gutiérrez, W. Steiner, R. Dobrin, and S. Punnekkat, “Synchroniza- [133] C. Park, J. Lee, T. Tan, and S. Park, “Simulation of scheduled traffic for
tion quality of IEEE 802.1 AS in large-scale industrial automation the IEEE 802.1 time sensitive networking,” in Information Science and
networks,” in Proc. IEEE Real-Time and Embedded Technology and Applications, LNEE Vol. 376. Springer, Singapore, 2016, pp. 75–83.
Applications Symposium (RTAS), 2017, pp. 273–282. [134] H. Lee, J. Lee, C. Park, and S. Park, “Time-aware preemption to
[113] C. Liß, M. Ulbricht, U. F. Zia, and H. Mller, “Architecture of a synchro- enhance the performance of audio/video bridging (AVB) in IEEE 802.1
nized low-latency network node targeted to research and education,” TSN,” in Proc. IEEE Int. Conf. on Computer Commun. and the Internet,
in Proc. IEEE Int. Conf. on High Performance Switching and Routing, Oct. 2016, pp. 80–84.
Jun. 2017, pp. 1–7. [135] D. Thiele and R. Ernst, “Formal worst-case performance analysis of
[114] B. Noseworthy, “Network-based application-independent time-error time-sensitive ethernet with frame preemption,” in Proc. IEEE 21st Int.
and direct port latency measurement,” in Proc. IEEE Int. Symp. Conf. on Emerging Techn. and Factory Automation (ETFA), 2016, pp.
on Precision Clock Synchronization for Measurement, Control, and 1–9.
Communication, Sep. 2016, pp. 1–6. [136] Z. Zhou, Y. Yan, S. Ruepp, and M. Berger, “Analysis and implementa-
[115] D. Shrestha, Z. Pang, and D. Dzung, “Precise clock synchronization in tion of packet preemption for time sensitive networks,” in Proc. IEEE
high performance wireless communication for time sensitive network- Int. Conf. on High Performance Switching and Routing (HPSR), Jun.
ing,” IEEE Access, vol. 6, pp. 8944–8953, 2018. 2017, pp. 1–6.
[116] D. Park, J. Lee, C. Park, and S. Park, “New automatic de-registration [137] S. Kehrer, O. Kleineberg, and D. Heffernan, “A comparison of fault-
method utilizing a timer in the IEEE 802.1 TSN,” in Proc. IEEE Int. tolerance concepts for IEEE 802.1 time sensitive networks (TSN),” in
Conf. on Computer Commun. and the Internet, Oct. 2016, pp. 47–51. Proc. IEEE Emerging Techn. and Factory Autom., Sep. 2014, pp. 1–8.
[117] M. L. Raagaard, P. Pop, M. Gutirrez, and W. Steiner, “Runtime [138] I. Alvarez, J. Proenza, M. Barranco, and M. Knezic, “Towards a time
reconfiguration of time-sensitive networking (TSN) schedules for fog redundancy mechanism for critical frames in time-sensitive network-
computing,” in Proc. IEEE Fog World Congress (FWC), Oct 2017, pp. ing,” in Proc. IEEE Int. Conf. on Emerging Techn. and Factory Autom.
1–6. (ETFA), Sept 2017, pp. 1–4.
[118] J. Ko, J.-h. Lee, C. Park, and S.-k. Park, “Research on optimal [139] S. Sommer, A. Camek, K. Becker, C. Buckl, A. Zirkler, L. Fiege,
bandwidth allocation for the scheduled traffic in IEEE 802.1 AVB,” M. Armbruster, G. Spiegelberg, and A. Knoll, “RACE: A centralized
in Proc. IEEE Int. Conf. on Vehicular Electronics and Safety (ICVES), platform computer based architecture for automotive applications,” in
2015, pp. 31–35. Proc. IEEE Int. Elec. Vehi. Conf., 2013, pp. 1–6.
[119] F. A. R. Arif and T. S. Atia, “Load balancing routing in time-sensitive [140] D. Cussans, “Trigger Logic Unit (TLU) design ready,” Advanced
networks,” in Proc. IEEE Int. Scientific-Practical Conf. Problems of European Infrastructures for Detectors at Accelerators (AIDA-2020),
Infocommunications Science and Technology, Oct. 2016, pp. 207–208. AIDA-2020-MS43, Tech. Rep., Feb. 2017.
[120] N. G. Nayak, F. Dürr, and K. Rothermel, “Software-defined environ- [141] T. Kovácsházy, “Towards a quantization based accuracy and precision
ment for reconfigurable manufacturing systems,” in Proc. IEEE Int. characterization of packet-based time synchronization,” in Proc. IEEE
Conf. on the Internet of Things, Oct. 2015, pp. 122–129. Int. Conf. Precision Clock Sync. for Meas., Ctrl., and Commun., 2016,
[121] D. Thiele and R. Ernst, “Formal analysis based evaluation of software pp. 1–6.
defined networking for time-sensitive ethernet,” in Proc. Design, Au- [142] T. Kovacshazy and A. E. Hollos, “Low cost field test measurement
tomation Test in Europe Conf. Exhibition, Mar. 2016, pp. 31–36. method and prototype measurement device implementation for timing
[122] S. Thangamuthu, N. Concer, P. J. L. Cuijpers, and J. J. Lukkien, accuracy evaluation of IEEE 1588 solutions,” in Proc. IEEE Workshop
“Analysis of Ethernet-switch traffic shapers for in-vehicle networking on Metrology for Industry 4.0 and IoT, Apr. 2018, pp. 72–77.
applications,” in Proc. IEEE Design, Automation Test in Europe Conf. [143] R. Bhagavatula and P. Bhagra, “Centrally managed time sensitive fog
Exhibition, Mar. 2015, pp. 55–60. networks,” Mar. 1 2018, US Patent App. 15/687,396.
[123] M. H. Farzaneh and A. Knoll, “An ontology-based plug-and-play [144] S. S. Craciunas, R. S. Oliver, and W. Steiner, “Demo abstract: Slate
approach for in-vehicle time-sensitive networking (TSN),” in Proc. XNS–an online management tool for deterministic TSN networks,” in
 52

Proc. IEEE Real-Time and Embedded Techn. and Applications Symp. [166] N. Navet, J. Migge, J. Villanueva, and M. Boyer, “Pre-shaping bursty
(RTAS), Apr. 2018, pp. 103–104. transmissions under IEEE802.1Q as a simple and efficient QoS mech-
[145] V. Gavrilut and P. Pop, “Scheduling in time sensitive networks (TSN) anism,” SAE Technical Paper 2018-01-0756, Tech. Rep., 2018.
for mixed-criticality industrial applications,” in Proc. IEEE Int. Work- [167] Z. Zhou, Y. Yan, M. Berger, and S. Ruepp, “Analysis and modeling
shop on Factory Commun. Systems (WFCS), Jun. 2018, pp. 1–4. of asynchronous traffic shaping in time sensitive networks,” in Proc.
[146] M. Gutierrez, A. Ademaj, W. Steiner, R. Dobrin, and S. Punnekkat, IEEE Int. Workshop on Factory Commun. Systems (WFCS), Jun. 2018,
“Self-configuration of IEEE 802.1 TSN networks,” in Proc. IEEE Int. pp. 1–4.
Conf. on Emerging Techn. and Factory Autom. (ETFA), Sept 2017, pp. [168] T. Wan and P. Ashwood-Smith, “A performance study of CPRI over
1–8. Ethernet with IEEE 802.1Qbu and 802.1Qbv enhancements,” in Proc.
[147] P. Heise, F. Geyer, and R. Obermaisser, “Self-configuring deterministic IEEE Globecom, 2015, pp. 1–6.
network with in-band configuration channel,” in Proc. IEEE Int. Conf. [169] S. S. Craciunas and R. S. Oliver, “Combined task-and network-level
on Software Defined Systems (SDS), 2017, pp. 162–167. scheduling for distributed time-triggered systems,” Real-Time Systems,
[148] R. S. Oliver, S. S. Craciunas, and W. Steiner, “IEEE 802.1Qbv gate vol. 52, no. 2, pp. 161–200, Mar. 2016.
control list synthesis using array theory encoding,” in Proc. IEEE Real- [170] H. Kopetz, A. Ademaj, P. Grillinger, and K. Steinhammer, “The time-
Time and Embedded Techn. and Applications Symp. (RTAS), Apr. 2018, triggered ethernet (TTE) design,” in Proc. IEEE Int. Symp. on Object-
pp. 13–24. Oriented Real-Time Distr. Computing (ISORC), 2005, pp. 22–33.
[149] P. Pop, M. L. Raagaard, M. Gutierrez, and W. Steiner, “Enabling fog [171] W. Steiner, G. Bauer, B. Hall, M. Paulitsch, and S. Varadarajan,
computing for industrial automation through time-sensitive networking “TTEthernet dataflow concept,” in Proc. IEEE Int. Symp. on Network
(TSN),” IEEE Communications Standards Magazine, vol. 2, no. 2, pp. Computing and Applications (NCA), 2009, pp. 319–322.
55–61, Jun. 2018. [172] T. Yamada and R. Nakano, “Job shop scheduling,” in Genetic Algo-
[150] M. Schmidt, R. Obermaisser, and C. Wurmbach, “Dynamic resource rithms in Engineering Systems, IEE Control Engineering Series 55,
allocation of switched ethernet networks in embedded real-time sys- A. Zalzala and P. Fleming, Eds. The Institution of Electrical Engineers,
tems,” in Proc. Int. Conf. on Inform. Techn. in Biomedicine. Springer, 1997, pp. 134–160.
Cham, Switzerland, 2018, pp. 353–364. [173] R. Battiti and G. Tecchiolli, “The reactive tabu search,” ORSA Journal
[151] R. Amin, M. Reisslein, and N. Shah, “Hybrid SDN networks: A survey on Computing, vol. 6, no. 2, pp. 126–140, Spring 1994.
of existing approaches,” IEEE Communications Surveys & Tutorials, [174] F. Glover, “Tabu search–Part I,” ORSA Journal on Computing, vol. 1,
in print, 2018. no. 3, pp. 190–206, Summer 1989.
[152] T. Huang, F. R. Yu, C. Zhang, J. Liu, J. Zhang, and Y. Liu, “A survey [175] R. Macchiaroli, S. Mole, and S. Riemma, “Modelling and optimization
on large-scale software defined networking (sdn) testbeds: Approaches of industrial manufacturing processes subject to no-wait constraints,”
and challenges,” IEEE Communications Surveys & Tutorials, vol. 19, International Journal of Production Research, vol. 37, no. 11, pp.
no. 2, pp. 891–917, Second Qu. 2017. 2585–2607, 1999.
[153] C. Trois, M. D. Del Fabro, L. C. de Bona, and M. Martinello, “A [176] S. S. Craciunas, R. S. Oliver, and W. Steiner, “Formal scheduling con-
survey on SDN programming languages: Toward a taxonomy,” IEEE straints for time-sensitive networks,” arXiv preprint arXiv:1712.02246,
Communications Surveys & Tutorials, vol. 18, no. 4, pp. 2687–2712, 2017.
Fourth Qu. 2016. [177] W. Steiner, S. S. Craciunas, and R. S. Oliver, “Traffic planning for time-
[154] S. Schriegel, T. Kobzan, and J. Jasperneite, “Investigation on a sensitive communication,” IEEE Communications Standards Magazine,
distributed SDN control plane architecture for heterogeneous time vol. 2, no. 2, pp. 42–47, Jun. 2018.
sensitive networks,” in Proc. IEEE Int. Workshop on Factory Commun. [178] A. M. Kentis, M. S. Berger, and J. Soler, “Effects of port congestion
Systems (WFCS), Jun. 2018, pp. 1–10. in the gate control list scheduling of time sensitive networks,” in Proc.
[155] D. Thiele, R. Ernst, and J. Diemer, “Formal worst-case timing analysis Int. Conf. on the Netw. of the Future (NOF), Nov 2017, pp. 138–140.
of ethernet TSN’s time-aware and peristaltic shapers,” in Proc. IEEE [179] S. Einspieler, B. Steinwender, and W. Elmenreich, “Integrating time-
Vehicular Networking Conference (VNC), 2015, pp. 251–258. triggered and event-triggered traffic in a hard real-time system,” in
[156] D. Thiele and R. Ernst, “Formal worst-case timing analysis of ethernet Proc. IEEE Industrial Cyber-Physical Systems (ICPS), 2018, pp. 122–
TSN’s burst-limiting shaper,” in Proc. IEEE Design, Automation Test 128.
in Europe Conf. Exhibition, Mar. 2016, pp. 187–192. [180] S. Rumpf, T. Steinbach, F. Korf, and T. C. Schmidt, “Software stacks
[157] J. Migge, J. Villanueva, N. Navet, and M. Boyer, “Insights on the for mixed-critical applications: Consolidating IEEE 802.1 AVB and
performance and configuration of AVB and TSN in automotive Ethernet time-triggered ethernet in next-generation automotive electronics,” in
networks,” in Proc. Embedded Real-Time Software and Systems, 2018, IEEE Int. Conf. on Consumer Electronics–Berlin (ICCE-Berlin), 2014,
pp. 1–10. pp. 14–18.
[158] F. He, L. Zhao, and E. Li, “Impact analysis of flow shaping in [181] D. Maxim and Y.-Q. Song, “Delay analysis of AVB traffic in time-
Ethernet-AVB/TSN and AFDX from network calculus and simulation sensitive networks (TSN),” in Proc. ACM Int. Conf. on Real-Time
perspective,” Sensors, vol. 17, no. 5, pp. 1–33, 2017. Networks and Systems (RTNS), Oct. 2017, pp. 18–27.
[159] J. Cao, M. Ashjaei, P. J. L. Cuijpers, R. J. Bril, and J. J. Lukkien, “An [182] W.-K. Jia, G.-H. Liu, and Y.-C. Chen, “Performance evaluation of IEEE
independent yet efficient analysis of bandwidth reservation for credit- 802.1 Qbu: Experimental and simulation results,” in Proc. IEEE Conf.
based shaping,” in Proc. IEEE Int. Workshop on Factory Commun. on Local Computer Networks (LCN), 2013, pp. 659–662.
Systems (WFCS), Jun. 2018, pp. 1–10. [183] O. Kleineberg, P. Fröhlich, and D. Heffernan, “Fault-tolerant audio
[160] A. Finzi, A. Mifdaoui, F. Frances, and E. Lochin, “Incorporating and video bridging (AVB) Ethernet: A novel method for redundant
TSN/BLS in AFDX for mixed-criticality applications: Model and stream registration configuration,” in Proc. IEEE Conf. on Emerging
timing analysis,” in Proc. IEEE Int. Workshop on Factory Commun. Technologies & Factory Automation (ETFA), 2012, pp. 1–8.
Systems, 2018, pp. 1–10. [184] I. Alvarez, J. Proenza, and M. Barranco, “Mixing time and spatial
[161] ——, “Network calculus-based timing analysis of AFDX networks redundancy over time sensitive networking,” in Proc. IEEE/IFIP Int.
with strict priority and TSN/BLS shapers,” in Proc. Int. Symp. on Conf. on Dependable Systems and Networks Workshops (DSN-W), Jun.
Industrial Embedded Sys., 2018, pp. 1–10. 2018, pp. 63–64.
[162] Y. Jiang, “A basic result on the superposition of arrival processes in [185] J. Lee and S. Park, “New interconnection methodology of TSNs
deterministic networks,” arXiv preprint arXiv:1804.10973, 2018. using V2X communication,” in Proc. IEEE Computing and Commun.
[163] E. Mohammadpour, E. Stai, M. Mohiuddin, and J.-Y. L. Boudec, “End- Workshop and Conf., Jan. 2017, pp. 1–6.
to-end latency and backlog bounds in time-sensitive networking with [186] Z. MacHardy, A. Khan, K. Obana, and S. Iwashina, “V2X access
credit based shapers and asynchronous traffic shaping,” arXiv preprint technologies: Regulation, research, and remaining challenges,” IEEE
arXiv:1804.10608, 2018. Communications Surveys & Tutorials, vol. 20, no. 3, pp. 1858–1877,
[164] L. Zhao, P. Pop, Z. Zheng, and Q. Li, “Timing analysis of AVB traffic Third Qu. 2018.
in TSN networks using network calculus,” in Proc. IEEE Real-Time [187] M. H. Farzaneh, S. Shafaei, and A. Knoll, “Formally verifiable
and Embedded Techn. and Applications Symp. (RTAS), Apr. 2018, pp. modeling of in-vehicle time-sensitive networks (TSN) based on logic
25–36. programming,” in Proc. IEEE Vehicular Netw. Conf., Dec. 2016, pp.
[165] L. Zhao, P. Pop, and S. S. Craciunas, “Worst-case latency analysis for 1–4.
IEEE 802.1Qbv time sensitive networks using network calculus,” IEEE [188] P. Heise, F. Geyer, and R. Obermaisser, “TSimNet: An industrial time
Access, vol. 6, pp. 41 803–41 815, 2018. sensitive networking simulation framework based on OMNeT++,” in
 53

Proc. IEEE Int. Conf. on New Technologies, Mobility and Security, [207] T. Eckert, G. Cauchie, W. Braun, and M. Menth, “Traffic Engineering
Nov. 2016, pp. 1–5. for Bit Index Explicit Replication (BIER-TE),” Internet Engineering
[189] S. Nsaibi, L. Leurs, and H. D. Schotten, “Formal and simulation-based Task Force, Internet-Draft draft-ietf-bier-te-arch-00, Jan. 2018, work
timing analysis of industrial-ethernet Sercos III over TSN,” in Proc. in Progress. [Online]. Available: https://datatracker.ietf.org/doc/html/
IEEE/ACM Int. Symp. on Distr. Simulation and Real Time Appl. (DS- draft-ietf-bier-te-arch-00
RT), 2017, pp. 1–8. [208] Y. Lee, D. Ceccarelli, T. Miyasaka, J. Shin, and
[190] M. Pahlevan and R. Obermaisser, “Evaluation of time-triggered traffic K. koog Lee, “Requirements for Abstraction and Control
in time-sensitive networks using the OPNET simulation framework,” of TE Networks,” Internet Engineering Task Force, Internet-
in Proc. Euromicro Int. Conf. on Parallel, Distr. and Network-based Draft draft-ietf-teas-actn-requirements-08, Jan. 2018, work in
Processing (PDP), March 2018, pp. 283–287. Progress. [Online]. Available: https://datatracker.ietf.org/doc/html/
[191] F. Groß, T. Steinbach, F. Korf, T. C. Schmidt, and B. Schwarz, draft-ietf-teas-actn-requirements-08
“A hardware/software co-design approach for ethernet controllers to [209] E. Haleplidis, K. Pentikousis, S. Denazis, J. H. Salim, D. Meyer, and
support time-triggered traffic in the upcoming IEEE TSN standards,” O. Koufopavlou, “Software-Defined Networking (SDN): Layers and
in Proc. IEEE Int. Conf. on Consumer Electr.–Berlin, 2014, pp. 9–13. Architecture Terminology,” RFC 7426, Jan. 2015. [Online]. Available:
[192] Y. Chen, H. Zhang, N. Fisher, L. Y. Wang, and G. Yin, “Probabilistic https://rfc-editor.org/rfc/rfc7426.txt
per-packet real-time guarantees for wireless networked sensing and [210] V. Watson, A. Tellabi, J. Sassmannahausen, and X. Lou, “Interoper-
control,” IEEE Trans. on Industrial Informatics, vol. 14, no. 5, pp. ability and security challenges of Industry 4.0,” in INFORMATIK 2017.
2133–2145, May 2018. Gesellschaft für Informatik, Bonn, Germany, 2017, pp. 973–985.
[193] E. W. Knightly and N. B. Shroff, “Admission control for statistical [211] J. Acevedo, R. Scheffel, S. Wunderlich, M. Hasler, S. Pandi, J. Cabrera,
QoS: Theory and practice,” IEEE Network, vol. 13, no. 2, pp. 20–29, F. H. P. Fitzek, G. Fettweis, and M. Reisslein, “Hardware acceleration
Mar/Apr 1999. for RLNC: A case study based on the Xtensa processor with Tensilica
[194] J. Liebeherr, D. E. Wrege, and D. Ferrari, “Exact admission control instruction-set extension,” Electronics, in print, 2018.
for networks with a bounded delay service,” IEEE/ACM Transactions [212] P. Garrido, D. Leith, and R. Aguero, “Joint scheduling and coding over
on Networking, vol. 4, no. 6, pp. 885–901, Dec. 1996. lossy paths with delayed feedback,” Preprint arXiv:1804.04921, 2018.
[195] M. Reisslein, K. W. Ross, and S. Rajagopal, “Guaranteeing statistical [213] F. Gabriel, S. Wunderlich, S. Pandi, F. H. Fitzek, and M. Reisslein,
QoS to regulated traffic: The single node case,” in Proc. INFOCOM, “Caterpillar RLNC with feedback (CRLNC-FB): Reducing delay in
vol. 3, 1999, pp. 1061–1072. selective repeat ARQ through coding,” IEEE Access, in print, 2018.
[196] ——, “A framework for guaranteeing statistical QoS,” IEEE/ACM [214] S. Pandi, F. Gabriel, J. A. Cabrera, S. Wunderlich, M. Reisslein, and
Transactions on Networking, vol. 10, no. 1, pp. 27–42, Feb. 2002. F. H. Fitzek, “PACE: Redundancy engineering in RLNC for low-latency
[197] Z.-L. Zhang, Z. Liu, J. Kurose, and D. Towsley, “Call admission control communication,” IEEE Access, vol. 5, pp. 20 477–20 493, 2017.
schemes under generalized processor sharing scheduling,” Telecommu- [215] V. Roca, B. Teibi, C. Burdinat, T. Tran-Thai, and C. Thienot, “Block
nication Systems, vol. 7, no. 1-3, pp. 125–152, Jun. 1997. or convolutional AL-FEC codes? a performance comparison for robust
[198] N. Finn, P. Thubert, B. Varga, and J. Farkas, “Deterministic low-latency communications,” HAL Inria, Id: hal-01395937v2, Tech.
Networking Architecture,” Internet Engineering Task Force, Rep., 2017.
Internet-Draft draft-ietf-detnet-architecture-04, Oct. 2017, work in [216] H. Shin and J.-S. Park, “Optimizing random network coding for
Progress. [Online]. Available: https://datatracker.ietf.org/doc/html/ multimedia content distribution over smartphones,” Multimedia Tools
draft-ietf-detnet-architecture-04 and Applications, vol. 76, no. 19, pp. 19 379–19 395, Oct. 2017.
[199] X. Geng and M. Chen, “DetNet Configuration YANG Model,” [217] J. K. Sundararajan, D. Shah, M. Medard, and P. Sadeghi, “Feedback-
Internet Engineering Task Force, Internet-Draft draft-geng-detnet- based online network coding,” IEEE Trans. on Information Theory,
conf-yang-00, Oct. 2017, work in Progress. [Online]. Available: vol. 63, no. 10, pp. 6628–6649, Oct. 2017.
https://datatracker.ietf.org/doc/html/draft-geng-detnet-conf-yang-00 [218] S. Wunderlich, J. A. Cabrera, F. H. Fitzek, and M. Reisslein, “Network
[200] J. Korhonen, L. Andersson, Y. Jiang, N. Finn, B. Varga, J. Farkas, coding in heterogeneous multicore IoT nodes with DAG scheduling of
C. J. Bernardos, T. Mizrahi, and L. Berger, “DetNet Data Plane parallel matrix block operations,” IEEE Internet of Things Journal,
Encapsulation,” Internet Engineering Task Force, Internet-Draft draft- vol. 4, no. 4, pp. 917–933, 2017.
ietf-detnet-dp-sol-00, Oct. 2017, work in Progress. [Online]. Available: [219] S. Wunderlich, F. Gabriel, S. Pandi, F. H. Fitzek, and M. Reisslein,
https://datatracker.ietf.org/doc/html/draft-ietf-detnet-dp-sol-00 “Caterpillar RLNC (CRLNC): A practical finite sliding window RLNC
[201] D. Huang, “Single-path PREF,” Internet Engineering Task Force, approach,” IEEE Access, vol. 5, pp. 20 183–20 197, 2017.
Internet-Draft draft-huang-detnet-single-path-pref-00, Dec. 2017, work [220] M. Nixon and T. Round Rock, “A comparison of WirelessHART and
in Progress. [Online]. Available: https://datatracker.ietf.org/doc/html/ ISA100. 11a,” Whitepaper, Emerson Process Management, pp. 1–36,
draft-huang-detnet-single-path-pref-00 2012.
[202] X. Geng and M. Chen, “IGP-TE Extensions for DetNet [221] P. Thubert, M. R. Palattella, and T. Engel, “6TiSCH centralized
Information Distribution,” Internet Engineering Task Force, Internet- scheduling: When SDN meet IoT,” in Proc. IEEE Conf. on Standards
Draft draft-geng-detnet-info-distribution-01, Sep. 2017, work in for Communications and Networking (CSCN), Oct. 2015, pp. 42–47.
Progress. [Online]. Available: https://datatracker.ietf.org/doc/html/ [222] R. T. Hermeto, A. Gallais, and F. Theoleyre, “Scheduling for
draft-geng-detnet-info-distribution-01 IEEE802.15.4-TSCH and slow channel hopping MAC in low power
[203] J. Farkas, B. Varga, R. Cummings, Y. Jiang, and Y. Zha, “DetNet industrial wireless networks: A survey,” Computer Communications,
Flow Information Model,” Internet Engineering Task Force, Internet- vol. 114, pp. 84–105, Dec. 2017.
Draft draft-ietf-detnet-flow-information-model-00, Jan. 2018, work [223] T. Qiu, N. Chen, K. Li, M. Atiquzzaman, and W. Zhao, “How can
in Progress. [Online]. Available: https://datatracker.ietf.org/doc/html/ heterogeneous Internet of Things build our future: A survey,” IEEE
draft-ietf-detnet-flow-information-model-00 Communications Surveys & Tutorials, vol. 20, no. 3, pp. 2011–2027,
[204] T. Mizrahi, E. Grossman, A. J. Hacker, S. Das, J. Dowdell, H. Austad, Third Qu. 2018.
K. Stanton, and N. Finn, “Deterministic Networking (DetNet) Security [224] T. Watteyne, M. Palattella, and L. Grieco, “Using IEEE 802.15.
Considerations,” Internet Engineering Task Force, Internet-Draft draft- 4e time-slotted channel hopping (TSCH) in the Internet of Things
ietf-detnet-security-01, Oct. 2017, work in Progress. [Online]. Avail- (IoT): Problem Statement,” RFC 7554, May 2015. [Online]. Available:
able: https://datatracker.ietf.org/doc/html/draft-ietf-detnet-security-01 https://rfc-editor.org/rfc/rfc7554.txt
[205] E. Grossman, C. Gunther, P. Thubert, P. Wetterwald, J. Raymond, [225] P. Thubert, T. Watteyne, R. Struik, and M. Richardson, “An architecture
J. Korhonen, Y. Kaneko, S. Das, Y. Zha, B. Varga, J. Farkas, for IPv6 over the TSCH mode of IEEE 802.15. 4. draft-ietf-6tisch-
F.-J. Goetz, J. Schmitt, X. Vilajosana, T. Mahmoodi, S. Spirou, architecture-10,” IETF Draft, March, 2015.
P. Vizarreta, D. Huang, X. Geng, D. Dujovne, and M. Seewald, [226] J. d. Armas, P. Tuset, T. Chang, F. Adelantado, T. Watteyne, and X. Vi-
“Deterministic Networking Use Cases,” Internet Engineering Task lajosana, “Determinism through path diversity: Why packet replication
Force, Internet-Draft draft-ietf-detnet-use-cases-13, Sep. 2017, work makes sense,” in Proc. IEEE Int. Conf. on Intelligent Networking and
in Progress. [Online]. Available: https://datatracker.ietf.org/doc/html/ Collaborative Systems, Sep. 2016, pp. 150–154.
draft-ietf-detnet-use-cases-13 [227] M. Amjad, F. Akhtar, M. H. Rehmani, M. Reisslein, and T. Umer,
[206] P. Pate and S. Bryant, “Pseudo Wire Emulation Edge-to-Edge “Full-duplex communication in cognitive radio networks: A survey,”
(PWE3) Architecture,” RFC 3985, Mar. 2005. [Online]. Available: IEEE Communications Surveys & Tutorials, vol. 19, no. 4, pp. 2158–
https://rfc-editor.org/rfc/rfc3985.txt 2191, Fourth Qu. 2017.
 54

[228] N. Bhushan, J. Li, D. Malladi, R. Gilmore, D. Brenner, A. Damn- [249] A. De La Oliva, X. C. Pérez, A. Azcorra, A. Di Giglio, F. Cavaliere,
janovic, R. Sukhavasi, C. Patel, and S. Geirhofer, “Network densi- D. Tiegelbekkers, J. Lessmann, T. Haustein, A. Mourad, and P. Iovanna,
fication: the dominant theme for wireless evolution into 5G,” IEEE “Xhaul: toward an integrated fronthaul/backhaul architecture in 5G
Communications Magazine, vol. 52, no. 2, pp. 82–89, 2014. networks,” IEEE Wireless Communications, vol. 22, no. 5, pp. 32–40,
[229] M. Agiwal, A. Roy, and N. Saxena, “Next generation 5G wireless Oct. 2015.
networks: A comprehensive survey,” IEEE Communications Surveys & [250] “Next generation fronthaul interface (1914) working group,” IEEE.
Tutorials, vol. 18, no. 3, pp. 1617–1655, Third Qu. 2016. [Online]. Available: http://sites.ieee.org/sagroups-1914
[230] E. Hossain and M. Hasan, “5G cellular: key enabling technologies and [251] S. Chen, R. Ma, H.-H. Chen, H. Zhang, W. Meng, and J. Liu,
research challenges,” IEEE Instrumentation & Measurement Magazine, “Machine-to-machine communications in ultra-dense networksa sur-
vol. 18, no. 3, pp. 11–21, Jun. 2015. vey,” IEEE Communications Surveys & Tutorials, vol. 19, no. 3, pp.
[231] S. Dutta, M. Mezzavilla, R. Ford, M. Zhang, S. Rangan, and M. Zorzi, 1478–1503, 2017.
“Frame structure design and analysis for millimeter wave cellular [252] M. A. Mehaseb, Y. Gadallah, A. Elhamy, and H. Elhennawy, “Classi-
systems,” IEEE Transactions on Wireless Communications, vol. 16, fication of LTE uplink scheduling techniques: An M2M perspective,”
no. 3, pp. 1508–1522, Mar. 2017. IEEE Communications Surveys & Tutorials, vol. 18, no. 2, pp. 1310–
[232] I. Kakalou, K. E. Psannis, P. Krawiec, and R. Badea, “Cognitive radio 1335, Second Qu. 2016.
network and network service chaining toward 5G: Challenges and [253] R. R. Tyagi, F. Aurzada, K. Lee, and M. Reisslein, “Connection
requirements,” IEEE Communications Magazine, vol. 55, no. 11, pp. establishment in LTE-A networks: Justification of Poisson process
145–151, Nov. 2017. modeling,” IEEE Systems Journal, vol. 11, no. 4, pp. 2383–2394, Dec.
[233] M. Luvisotto, Z. Pang, and D. Dzung, “Ultra high performance wireless 2017.
control for critical applications: Challenges and directions,” IEEE
[254] “Control and user plane separation of EPC nodes (CUPS),” 3GPP.
Transactions on Industrial Informatics, vol. 13, no. 3, pp. 1448–1459,
[Online]. Available: http://www.3gpp.org/news-events/3gpp-news/
Jun. 2017.
1882-cups
[234] C. She, C. Yang, and T. Q. S. Quek, “Radio resource management for
ultra-reliable and low-latency communications,” IEEE Communications [255] J. Kim, D. Kim, and S. Choi, “3GPP SA2 architecture and functions
Magazine, vol. 55, no. 6, pp. 72–78, Jun. 2017. for 5G mobile communication system,” ICT Express, vol. 3, no. 1, pp.
[235] G. Durisi, T. Koch, J. Östman, Y. Polyanskiy, and W. Yang, “Short- 1–8, 2017.
packet communications over multiple-antenna rayleigh-fading chan- [256] “3GPP Specification series for 5G NR,” 3GPP. [Online]. Available:
nels,” IEEE Transactions on Communications, vol. 64, no. 2, pp. 618– http://www.3gpp.org/DynaReport/38-series.htm
629, Feb. 2016. [257] U. Habiba and E. Hossain, “Auction mechanisms for virtualization
[236] O. Ploder, N. Palaoro, B. Etzlinger, and A. Springer, “A cross-layer in 5G cellular networks: Basics, trends, and open challenges,” IEEE
approach for ultra-low-latency machine type communication,” in Proc. Communications Surveys & Tutorials, vol. PP, no. 99, pp. 1–1, 2018.
IEEE ICC, 2017, pp. 1–6. [258] Y. Cai, Z. Qin, F. Cui, G. Y. Li, and J. A. McCann, “Modulation and
[237] P. Mogensen, K. Pajukoski, E. Tiirola, J. Vihriala, E. Lahetkangas, multiple access for 5G networks,” IEEE Communications Surveys &
G. Berardinelli, F. M. Tavares, N. H. Mahmood, M. Lauridsen, D. Cata- Tutorials, vol. 20, no. 1, pp. 629–646, First Qu. 2018.
nia, and A. Cattoni, “Centimeter-wave concept for 5G ultra-dense small [259] L. Dai, B. Wang, Z. Ding, Z. Wang, S. Chen, and L. Hanzo, “A survey
cells,” in Proc. IEEE VTC, 2014, pp. 1–6. of non-orthogonal multiple access for 5G,” IEEE Communications
[238] F. Pflug and T. Fingscheidt, “Robust ultra-low latency soft-decision Surveys & Tutorials, vol. 20, no. 3, pp. 2294–2323, Third Qu. 2018.
decoding of linear PCM audio,” IEEE Transactions on Audio, Speech, [260] Z. Ding, X. Lei, G. K. Karagiannidis, R. Schober, J. Yuan, and
and Language Processing, vol. 21, no. 11, pp. 2324–2336, Nov. 2013. V. K. Bhargava, “A survey on non-orthogonal multiple access for 5G
[239] H. Beyranvand, M. Lvesque, M. Maier, J. A. Salehi, C. Verikoukis, and networks: Research challenges and future trends,” IEEE Journal on
D. Tipper, “Toward 5G: FiWi enhanced LTE-A HetNets with reliable Selected Areas in Communications, vol. 35, no. 10, pp. 2181–2195,
low-latency fiber backhaul sharing and WiFi offloading,” IEEE/ACM Oct. 2017.
Transactions on Networking, vol. 25, no. 2, pp. 690–707, Apr. 2017. [261] X. Liu, H. Zeng, N. Chand, and F. Effenberger, “Experimental demon-
[240] M. A. Lema, A. Laya, T. Mahmoodi, M. Cuevas, J. Sachs, J. Mark- stration of high-throughput low-latency mobile fronthaul supporting
endahl, and M. Dohler, “Business case and technology analysis for 5G 48 20-MHz LTE signals with 59-Gb/s CPRI-equivalent rate and 2-
low latency applications,” IEEE Access, vol. 5, pp. 5917–5935, 2017. µs processing latency,” in Proc. IEEE European Conf. on Optical
[241] “Low latency in 4.9G/5G: Solutions for millisec- Communication, 2015, pp. 1–3.
ond latency,” Nokia Corporation. [Online]. Avail- [262] C.-Y. Chang, N. Nikaein, and T. Spyropoulos, “Impact of packetization
able: https://onestore.nokia.com/asset/201407/Nokia Low-latency in and scheduling on C-RAN fronthaul performance,” in Proc. IEEE
4dot9 and 5G Networks White Paper EN.pdf GLOBECOM, 2016, pp. 1–7.
[242] “CPRI specification,” Common Public Radio Interface. [Online]. [263] D. Hisano, Y. Nakayama, T. Kubo, T. Shimizu, H. Nakamura, J. Terada,
Available: http://www.cpri.info/ and A. Otaka, “Gate-shrunk time aware shaper: Low-latency converged
[243] A. De la Oliva, J. A. Hernández, D. Larrabeiti, and A. Azcorra, “An network for 5G fronthaul and M2M services,” in Proc. IEEE Global
overview of the CPRI specification and its application to C-RAN-based Commun. Conf. (GLOBECOM), 2017, pp. 1–6.
LTE scenarios,” IEEE Communications Magazine, vol. 54, no. 2, pp.
[264] T. Tashiro, S. Kuwano, J. Terada, T. Kawamura, N. Tanaka, S. Shige-
152–159, Feb. 2016.
matsu, and N. Yoshimoto, “A novel DBA scheme for TDM-PON based
[244] P. Chanclou, A. Pizzinat, F. Le Clech, T.-L. Reedeker, Y. Lagadec,
mobile fronthaul,” in Proc. OSA/IEEE Optical Fiber Commun. Conf.
F. Saliou, B. Le Guyader, L. Guillo, Q. Deniel, S. Gosselin et al.,
and Exhibition (OFC), 2014, pp. 1–3.
“Optical fiber solution for mobile fronthaul to achieve cloud radio
access network,” in Proc. IEEE Future Network and Mobile Summit, [265] T. Kobayashi, D. Hisano, T. Shimada, J. Terada, and A. Otaka, “Band-
2013, pp. 1–11. width allocation scheme based on simple statistical traffic analysis
[245] J. S. Vardakas, I. T. Monroy, L. Wosinska, G. Agapiou, R. Brenot, for TDM-PON based mobile fronthaul,” in Proc. OSA Optical Fiber
N. Pleros, and C. Verikoukis, “Towards high capacity and low latency Commun. Conf., 2016, pp. W3C–7–1–W3C–7–3.
backhauling in 5G: The 5G STEP-FWD vision,” in Proc. IEEE Int. [266] Z. Gao, L. Dai, D. Mi, Z. Wang, M. A. Imran, and M. Z. Shakir,
Conf. on Transparent Optical Networks. IEEE, 2017, pp. 1–4. “Mmwave massive-MIMO-based wireless backhaul for the 5G ultra-
[246] P. Monti, Y. Li, J. Mårtensson, M. Fiorani, B. Skubic, Z. Ghebretensaé, dense network,” IEEE Wireless Communications, vol. 22, no. 5, pp.
and L. Wosinska, “A flexible 5G RAN architecture with dynamic 13–21, Oct. 2015.
baseband split distribution and configurable optical transport,” in Proc. [267] R. Taori and A. Sridharan, “Point-to-multipoint in-band mmwave
IEEE Int. Conf. Transparent Optical Netw., 2017, pp. 1–1. backhaul for 5G networks,” IEEE Communications Magazine, vol. 53,
[247] “Requirements for the eCPRI transport network v1.0,” no. 1, pp. 195–201, Jan. 2015.
Common Public Radio Interface. [Online]. Avail- [268] M. Pieska and A. Kassler, “TCP performance over 5G mmWave links-
able: http://www.cpri.info/downloads/Requirements for the eCPRI tradeoff between capacity and latency,” in Proc. IEEE Int. Conf. on
Transport Network V1 0 2017 10 24.pdf Wireless and Mob. Comp., Netw. and Commun., 2017, pp. 385–394.
[248] “IEEE P802.1CM/D2.0 time-sensitive networking for fronthaul,” [269] V. Jungnickel, K. Habel, M. Parker, S. Walker, C. Bock, J. F. Riera,
LAN/MAN Standards Committee of the IEEE Computer Society. V. Marques, and D. Levi, “Software-defined open architecture for
[Online]. Available: http://www.ieee802.org/1/files/private/cm-drafts/ front-and backhaul in 5G mobile networks,” in Proc. IEEE Int. Conf.
d2/802-1CM-d2-0.pdf Transparent Optical Networks. IEEE, 2014, pp. 1–4.
 55

[270] J. Pagé and J.-M. Dricot, “Software-defined networking for low-latency [291] M. P. McGarry, M. Reisslein, F. Aurzada, and M. Scheutzow, “Shortest
5G core network,” in Proc. IEEE Int. Conf. Military Commun. and propagation delay (SPD) first scheduling for EPONs with hetero-
Inform. Sys., 2016, pp. 1–7. geneous propagation delays,” IEEE Journal on Selected Areas in
[271] J. Lakkakorpi, H. Flinck, J. Heinonen, P. Korja, T. Partti, and K. So- Communications, vol. 28, no. 6, pp. 849–862, Aug. 2010.
ranko, “Minimizing delays in mobile networks: With dynamic gateway [292] M. Xu, X. Liu, N. Chand, F. Effenberger, and G.-K. Chang, “Flex-frame
placement and active queue management,” in Proc. IEEE Wireless timing-critical passive optical networks for delay sensitive mobile and
Days, 2016, pp. 1–3. fixed access services,” in Proc. OSA Optical Fiber Commun. Conf.,
[272] S. Nagata, L. H. Wang, and K. Takeda, “Industry perspectives,” IEEE 2017, pp. Th4B–6–1–Th4B–6–3.
Wireless Communications, vol. 24, no. 3, pp. 2–4, Jun. 2017. [293] F. Saliou, G. Simon, P. Chanclou, M. Brunero, L. Marazzi, P. Parolari,
[273] P. K. Agyapong, M. Iwamura, D. Staehle, W. Kiess, and A. Benjebbour, M. Martinelli, R. Brenot, A. Maho, S. Barbet, G. Gavioli, G. Parladori,
“Design considerations for a 5G network architecture,” IEEE Commu- S. Gebrewold, and J. Leuthold, “Self-seeded RSOAs WDM PON field
nications Magazine, vol. 52, no. 11, pp. 65–75, 2014. trial for business and mobile fronthaul applications,” in Proc. OSA
[274] S. Parkvall, E. Dahlman, A. Furuskar, and M. Frenne, “NR: The new Optical Fiber Commun. Conf., 2015, pp. M2A–2–1–M2A–2–3.
5G radio access technology,” IEEE Commun. Standards Mag., vol. 1, [294] X. Liu and F. Effenberger, “Emerging optical access network technolo-
no. 4, pp. 24–30, Dec. 2017. gies for 5G wireless,” IEEE/OSA Journal of Optical Commun. and
[275] S. A. Busari, K. M. S. Huq, S. Mumtaz, L. Dai, and J. Rodriguez, Netw., vol. 8, no. 12, pp. B70–B79, Dec. 2016.
“Millimeter-Wave massive MIMO communication for future wire- [295] M. Al-Hares, P. Assimakopoulos, D. Muench, and N. J. Gomes,
less systems: A survey,” IEEE Communications Surveys & Tutorials, “Scheduling in an Ethernet fronthaul network,” in Proc. IEEE European
vol. 20, no. 2, pp. 836–869, Second Qu. 2018. Conf. on Networks and Communications (EuCNC), 2017, pp. 1–5.
[276] X. Meng, J. Li, D. Zhou, and D. Yang, “5G technology requirements [296] M. K. Al-Hares, P. Assimakopoulos, D. Muench, and N. J. Gomes,
and related test environments for evaluation,” China Communications, “Modeling time aware shaping in an Ethernet fronthaul,” in Proc. IEEE
vol. 13, no. Supplement 2, pp. 42–51, 2016. GLOBECOM, 2017, pp. 1–6.
[277] S. A. Ashraf, I. Aktas, E. Eriksson, K. W. Helmersson, and J. Ansari, [297] P. Assimakopoulos, M. K. Al-Hares, and N. J. Gomes, “Switched
“Ultra-reliable and low-latency communication for wireless factory Ethernet fronthaul architecture for cloud-radio access networks,”
automation: From LTE to 5G,” in Proc. IEEE Int. Conf. Emerging IEEE/OSA Journal of Optical Communications and Networking, vol. 8,
Technologies and Factory Automation, 2016, pp. 1–8. no. 12, pp. B135–B146, Dec. 2016.
[278] J.-B. Seo and V. C. Leung, “Performance modeling and stability of [298] P. Assimakopulos, G. S. Birring, M. K. Al-Hares, and N. J. Gomes,
semi-persistent scheduling with initial random access in LTE,” IEEE “Ethernet-based fronthauling for cloud-radio access networks,” in Proc.
Trans. on Wireless Commun., vol. 11, no. 12, pp. 4446–4456, Dec. IEEE Int. Conf. on Transparent Opt. Netw. (ICTON), 2017, pp. 1–4.
2012. [299] F. Aurzada, M. Scheutzow, M. Reisslein, N. Ghazisaidi, and M. Maier,
“Capacity and delay analysis of next-generation passive optical net-
[279] A. Pizzinat, P. Chanclou, F. Saliou, and T. Diallo, “Things you should
works (NG-PONs),” IEEE Trans. on Commun., vol. 59, no. 5, pp.
know about fronthaul,” IEEE/OSA Journal of Lightwave Technology,
1378–1388, May 2011.
vol. 33, no. 5, pp. 1077–1083, Mar. 2015.
[300] A. Mercian, M. P. McGarry, and M. Reisslein, “Offline and on-
[280] A. Checko, H. L. Christiansen, Y. Yan, L. Scolari, G. Kardaras,
line multi-thread polling in long-reach PONs: A critical evaluation,”
M. S. Berger, and L. Dittmann, “Cloud RAN for mobile networksa
IEEE/OSA Journal of Lightwave Technology, vol. 31, no. 12, pp. 2018–
technology overview,” IEEE Communications Surveys & Tutorials,
2028, Jun. 2013.
vol. 17, no. 1, pp. 405–426, First Qu. 2015.
[301] Y. Gong, X. Wang, M. Malboubi, S. Wang, S. Xu, and C.-N. Chuah,
[281] D. Wubben, P. Rost, J. S. Bartelt, M. Lalam, V. Savin, M. Gorgoglione, “Towards accurate online traffic matrix estimation in software-defined
A. Dekorsy, and G. Fettweis, “Benefits and impact of cloud computing networks,” in Proc. ACM SIGCOMM Symp. on Software Defined Netw.
on 5G signal processing: Flexible centralization through cloud-RAN,” Res., 2015, pp. 1–7.
IEEE Signal Processing Magazine, vol. 31, no. 6, pp. 35–44, Nov.
[302] J. Liu, P. Zhang, H. Wang, and C. Hu, “CounterMap: Towards generic
2014.
traffic statistics collection and query in Software Defined Network,” in
[282] J. Bartelt, P. Rost, D. Wubben, J. Lessmann, B. Melis, and G. Fettweis, Proc. IEEE/ACM Int. Symp. on Quality of Service (IWQoS), 2017, pp.
“Fronthaul and backhaul requirements of flexibly centralized radio 1–5.
access networks,” IEEE Wireless Communications, vol. 22, no. 5, pp. [303] A. Yassine, H. Rahimi, and S. Shirmohammadi, “Software defined
105–111, Oct. 2015. network traffic measurement: Current trends and challenges,” IEEE
[283] I. A. Alimi, A. L. Teixeira, and P. P. Monteiro, “Towards an efficient C- Instrumentation & Measurement Magazine, vol. 18, no. 2, pp. 42–50,
RAN optical fronthaul for the future networks: A tutorial on technolo- Apr. 2015.
gies, requirements, challenges, and solutions,” IEEE Communications [304] Y. Luo and N. Ansari, “Limited sharing with traffic prediction for
Surveys & Tutorials, vol. 20, no. 1, pp. 708–769, First Qu. 2018. dynamic bandwidth allocation and QoS provisioning over Ethernet
[284] S. Bjomstad, D. Chen, and R. Veisllari, “Handling delay in 5G passive optical networks,” OSA Journal of Optical Networking, vol. 4,
ethernet mobile fronthaul networks,” in Proc. IEEE European Conf. no. 9, pp. 561–572, Sep. 2005.
on Networks and Commun. (EuCNC), Jun. 2018, pp. 1–9. [305] Y. Zhu and M. Ma, “IPACT with grant estimation (IPACT-GE) scheme
[285] C.-Y. Chang, N. Nikaein, R. Knopp, T. Spyropoulos, and S. S. Ku- for Ethernet passive optical networks,” IEEE/OSA Journal of Lightwave
mar, “FlexCRAN: A flexible functional split framework over ethernet Techn., vol. 26, no. 14, pp. 2055–2063, Jul. 2008.
fronthaul in cloud-RAN,” in Proc. IEEE ICC, 2017, pp. 1–7. [306] D. Hisano, T. Kobayashi, H. Ou, T. Shimada, H. Uzawa, J. Terada, and
[286] A. Rostami, P. Ohlen, K. Wang, Z. Ghebretensae, B. Skubic, M. Santos, A. Otaka, “TDM-PON for accommodating TDD-based fronthaul and
and A. Vidal, “Orchestration of RAN and transport networks for 5G: secondary services,” IEEE/OSA J. Lightwave Techn., vol. 35, no. 14,
An SDN approach,” IEEE Communications Magazine, vol. 55, no. 4, pp. 2788–2796, Jul. 2017.
pp. 64–70, Apr. 2017. [307] N. Carapellese, A. Pizzinat, M. Tornatore, P. Chanclou, and S. Gosselin,
[287] S. Bidkar, J. Galaro, and T. Pfeiffer, “First demonstration of an “An energy consumption comparison of different mobile backhaul and
ultra-low-latency fronthaul transport over a commercial TDM-PON fronthaul optical access architectures,” in Proc. European Conf. on Opt.
platform,” in Proc. IEEE/OSA Optical Fiber Commun. Conf. and Commun. (ECOC), 2014, pp. 1–3.
Exposition (OFC), 2018, pp. 1–3. [308] Z. Tan, C. Yang, and Z. Wang, “Energy consume analysis for ring-
[288] A. M. Mikaeil, W. Hu, T. Ye, and S. B. Hussain, “Performance topology TWDM-PON front-haul enabled cloud RAN,” IEEE/OSA
evaluation of XG-PON based mobile front-haul transport in cloud- Journal of Lightwave Techn., vol. 35, no. 20, pp. 4526–4534, 2017.
RAN architecture,” IEEE/OSA Journal of Optical Communications and [309] J. Wu, S. Guo, H. Huang, W. Liu, and Y. Xiang, “Information and
Networking, vol. 9, no. 11, pp. 984–994, 2017. communications technologies for sustainable development goals: State-
[289] P. Chanclou, L. A. Neto, K. Grzybowski, Z. Tayq, F. Saliou, and of-the-art, needs and perspectives,” IEEE Communications Surveys &
N. Genay, “Mobile fronthaul architecture and technologies: A RAN Tutorials, vol. 20, no. 3, pp. 2389–2406, Third Qu. 2018.
equipment assessment,” IEEE/OSA Journal of Opt. Commun. and [310] X. Liu, F. Effenberger, N. Chand, L. Zhou, and H. Lin, “Demonstration
Netw., vol. 10, no. 1, pp. A1–A7, 2018. of bandwidth-efficient mobile fronthaul enabling seamless aggregation
[290] J.-I. Kani, J. Terada, K.-I. Suzuki, and A. Otaka, “Solutions for future of 36 E-UTRA-like wireless signals in a single 1.1-GHz wavelength
mobile fronthaul and access-network convergence,” IEEE/OSA Journal channel,” in Proc. OSA Optical Fiber Commun. Conf. and Exhibition
of Lightwave Techn., vol. 35, no. 3, pp. 527–534, Feb. 2017. (OFC), 2015, pp. 1–3.
 56

[311] K.-G. Nguyen, Q.-D. Vu, M. Juntti, and L.-N. Tran, “Energy efficient [333] F. Aurzada, M. Lévesque, M. Maier, and M. Reisslein, “FiWi access
precoding C-RAN downlink with compression at fronthaul,” arXiv networks based on next-generation PON and gigabit-class WLAN
preprint arXiv:1703.05996, 2017. technologies: A capacity and delay analysis,” IEEE/ACM Trans. on
[312] A. Thyagaturu, Z. Alharbi, and M. Reisslein, “R-FFT: Function split Netw., vol. 22, no. 4, pp. 1176–1189, Aug. 2014.
at IFFT/FFT in unified LTE CRAN and cable access network,” IEEE [334] P.-Y. Chen and M. Reisslein, “FiWi network throughput-delay mod-
Transactions on Broadcasting, vol. 64, no. 3, pp. 648–665, Sep. 2018. eling with traffic intensity control and local bandwidth allocation,”
[313] N. J. Gomes, P. Chanclou, P. Turnbull, A. Magee, and V. Jungnickel, Optical Switching and Networking, vol. 28, pp. 8–22, Apr. 2018.
“Fronthaul evolution: from CPRI to Ethernet,” Optical Fiber Technol- [335] J. Liu, H. Guo, H. Nishiyama, H. Ujikawa, K. Suzuki, and N. Kato,
ogy, vol. 26, no. Part A, pp. 50–58, Dec. 2015. “New perspectives on future smart FiWi networks: Scalability, reliabil-
[314] M. Mao, R. Giddings, B. Cao, Y. Xu, M. Wang, and J. Tang, ity, and energy efficiency,” IEEE Communications Surveys & Tutorials,
“DSP-enabled reconfigurable and transparent spectral converters for vol. 18, no. 2, pp. 1045–1072, Second Qu. 2016.
converging optical and mobile fronthaul/backhaul networks,” Optics [336] A. Ahmed and A. Shami, “RPR–EPON–WiMAX hybrid network: A
Express, vol. 25, no. 12, pp. 13 836–13 856, 2017. solution for access and metro networks,” IEEE/OSA Journal of Optical
[315] N. Cvijetic, A. Tanaka, K. Kanonakis, and T. Wang, “SDN-controlled Communications and Networking, vol. 4, no. 3, pp. 173–188, Mar.
topology-reconfigurable optical mobile fronthaul architecture for bidi- 2012.
rectional CoMP and low latency inter-cell D2D in the 5G mobile era,” [337] N. Ghazisaidi, F. Paolucci, and M. Maier, “SuperMAN: Optical-
Optics Express, vol. 22, no. 17, pp. 20 809–20 815, Aug. 2014. wireless integration of RPR and WiMAX,” OSA Journal of Optical
[316] R. Li, R. Wang, N. Halachmi, Q. Zhong, W. Cheng, L. Wang, and Networking, vol. 8, no. 3, pp. 249–271, Mar. 2009.
J. Wang, “X-ethemet: Enabling integrated fronthaul/backhaul archi-
[338] M. Maier, M. Reisslein, and A. Wolisz, “A hybrid MAC protocol for a
tecture in 5G networks,” in Proc. IEEE Int. Conf. Standards for
metro WDM network using multiple free spectral ranges of an arrayed-
Communications and Networking. IEEE, 2017, pp. 121–125.
waveguide grating,” Computer Networks, vol. 41, no. 4, pp. 407–433,
[317] IETF, “Flexible ethernet (FlexE) deep dive - IETF datatracker.”
Mar. 2003.
[Online]. Available: https://datatracker.ietf.org/meeting/98/materials/
slides-98-ccamp-102-flexe-technology-deep-dive/ [339] M. Scheutzow, M. Maier, M. Reisslein, and A. Wolisz, “Wavelength
[318] M. Mezzavilla, M. Zhang, M. Polese, R. Ford, S. Dutta, S. Rangan, reuse for efficient packet-switched transport in an AWG-based metro
and M. Zorzi, “End-to-end simulation of 5G mmWave networks,” IEEE WDM network,” IEEE/OSA Journal of Lightwave Technology, vol. 21,
Communications Surveys & Tutorials, vol. 20, no. 3, pp. 2237–2263, no. 6, p. 1435, Jun. 2003.
Third Qu. 2018. [340] H.-S. Yang, M. Herzog, M. Maier, and M. Reisslein, “Metro WDM
[319] X. Wang, L. Kong, F. Kong, F. Qiu, M. Xia, S. Arnon, and G. Chen, networks: Performance comparison of slotted ring and AWG star net-
“Millimeter wave communication: A comprehensive survey,” IEEE works,” IEEE Journal on Selected Areas in Communications, vol. 22,
Communications Surveys & Tutorials, vol. 20, no. 3, pp. 1616–1653, no. 8, pp. 1460–1473, Oct. 2004.
Third Qu. 2018. [341] C. Cox, An Introduction to LTE: LTE, LTE-Advanced, SAE and 4G
[320] C. Dehos, J. L. González, A. De Domenico, D. Ktenas, and L. Dussopt, Mobile Communications. John Wiley & Sons, Hoboken, NJ, 2012.
“Millimeter-wave access and backhauling: the solution to the exponen- [342] J. Chen and J. Li, “Efficient mobile backhaul architecture offering ultra-
tial data traffic increase in 5G mobile communications systems?” IEEE short latency for handovers,” in Proc. IEEE Int. Conf. Transparent
Communications Magazine, vol. 52, no. 9, pp. 88–95, Sep. 2014. Optical Networks, 2016, pp. 1–1.
[321] K. Nichols, V. Jacobson, A. McGregor, and J. Iyengar, “Controlled [343] V. Yazıcı, U. C. Kozat, and M. O. Sunay, “A new control plane for 5G
delay active queue management,” RFC 8289, Jan. 2018. [Online]. network architecture with a case study on unified handoff, mobility,
Available: https://rfc-editor.org/rfc/rfc8289.txt and routing management,” IEEE Communications Magazine, vol. 52,
[322] M. Mathis, J. Semke, J. Mahdavi, and T. Ott, “The macroscopic be- no. 11, pp. 76–85, Nov. 2014.
havior of the TCP congestion avoidance algorithm,” ACM SIGCOMM [344] E. Wong, E. Grigoreva, L. Wosinska, and C. M. Machuca, “Enhancing
Computer Commun. Rev., vol. 27, no. 3, pp. 67–82, Jul. 1997. the survivability and power savings of 5G transport networks based
[323] D. Xu, Y. Li, X. Chen, J. Li, P. Hui, S. Chen, and J. Crowcroft, “A on DWDM rings,” IEEE/OSA Journal of Optical Commun. and Netw.,
survey of opportunistic offloading,” IEEE Communications Surveys & vol. 9, no. 9, pp. D74–D85, Sep. 2017.
Tutorials, vol. 20, no. 3, pp. 2198–2236, Third Qu. 2018. [345] C.-P. Li, J. Jiang, W. Chen, T. Ji, and J. Smee, “5G ultra-reliable
[324] P.-H. Kuo and A. Mourad, “Millimeter wave for 5G mobile fronthaul and low-latency systems design,” in Proc. IEEE European Conf. on
and backhaul,” in Proc. IEEE European Conf. on Networks and Networks and Commun., 2017, pp. 1–5.
Communications, 2017, pp. 1–5. [346] J. Liu, B. Bai, J. Zhang, and K. B. Letaief, “Cache placement in Fog-
[325] V. Jungnickel, D. Schulz, J. Hilt, C. Alexakis, M. Schlosser, L. Grobe, RANs: From centralized to distributed algorithms,” IEEE Trans. on
A. Paraskevopoulos, R. Freund, B. Siessegger, and G. Kleinpeter, Wireless Commun., vol. 16, no. 11, pp. 7039–7051, Nov. 2017.
“Optical wireless communication for backhaul and access,” in Proc. [347] A. Sengupta, R. Tandon, and O. Simeone, “Fog-aided wireless net-
IEEE European Conf. on Opt. Commun., 2015, pp. 1–3. works for content delivery: Fundamental latency tradeoffs,” IEEE
[326] D. Chitimalla, K. Kondepu, L. Valcarenghi, M. Tornatore, and Trans. on Info. Th., vol. 63, no. 10, pp. 6650–6678, Oct. 2017.
B. Mukherjee, “5G fronthaul-latency and jitter studies of CPRI over
[348] J. Kakar, S. Gherekhloo, Z. H. Awan, and A. Sezgin, “Fundamental
Ethernet,” IEEE/OSA Journal of Optical Communications and Net-
limits on latency in cloud-and cache-aided hetnets,” in Proc. IEEE Int.
working, vol. 9, no. 2, pp. 172–182, Feb. 2017.
Conf. on Communications, 2017, pp. 1–6.
[327] L. Ferrari, N. Karakoc, A. Scaglione, M. Reisslein, and A. Thyagaturu,
“Layered cooperative resource sharing at a wireless SDN backhaul,” in [349] A. Radwan, M. F. Domingues, and J. Rodriguez, “Mobile caching-
Proc. IEEE Int. Conf. on Commun. Workshops (ICC Workshops), Int. enabled small-cells for delay-tolerant e-health apps,” in Proc. IEEE
Workshop on 5G Architecture (5GARCH), 2018, pp. 1–6. Int. Conf. on Communications Workshops, 2017, pp. 103–108.
[328] N. A. Jagadeesan and B. Krishnamachari, “Software-defined network- [350] K. Miyanabe, K. Suto, Z. M. Fadlullah, H. Nishiyama, N. Kato,
ing paradigms in wireless networks: A survey,” ACM Computing H. Ujikawa, and K. i. Suzuki, “A cloud radio access network with
Surveys (CSUR), vol. 47, no. 2, pp. 27:1–27:11, Jan. 2015. power over fiber toward 5G networks: QoE-guaranteed design and
[329] R. Maallawi, N. Agoulmine, B. Radier, and T. B. Meriem, “A com- operation,” IEEE Wireless Commun., vol. 22, no. 4, pp. 58–64, Aug.
prehensive survey on offload techniques and management in wireless 2015.
access and core networks,” IEEE Communications Surveys & Tutorials, [351] M. Olsson, C. Cavdar, P. Frenger, S. Tombaz, D. Sabella, and R. Jantti,
vol. 17, no. 3, pp. 1582–1604, Third Qu. 2015. “5GrEEn: Towards green 5G mobile networks,” in Proc. IEEE Int.
[330] A. S. Thyagaturu, Y. Dashti, and M. Reisslein, “SDN-based smart gate- Conf. on Wireless and Mobile Comp., Netw. and Commun. (WiMob),
ways (Sm-GWs) for multi-operator small cell network management,” 2013, pp. 212–216.
IEEE Transactions on Network and Service Management, vol. 13, no. 4, [352] D. Sabella, A. De Domenico, E. Katranaras, M. A. Imran, M. Di Giro-
pp. 740–753, Dec. 2016. lamo, U. Salim, M. Lalam, K. Samdanis, and A. Maeder, “Energy
[331] R. Chundury, “Mobile broadband backhaul: Addressing the challenge,” efficiency benefits of RAN-as-a-service concept for a cloud-based 5G
Planning Backhaul Networks, Ericsson Review, no. 3, pp. 4–9, 2008. mobile network infrastructure,” IEEE Access, vol. 2, pp. 1586–1597,
[332] P. Ameigeiras, J. J. Ramos-Munoz, L. Schumacher, J. Prados-Garzon, 2014.
J. Navarro-Ortiz, and J. M. Lopez-Soler, “Link-level access cloud [353] N. Abbas, Y. Zhang, A. Taherkordi, and T. Skeie, “Mobile edge
architecture design based on SDN for 5G networks,” IEEE Network, computing: A survey,” IEEE Internet of Things Journal, vol. 5, no. 1,
vol. 29, no. 2, pp. 24–31, 2015. pp. 450–465, Feb. 2018.
 57

[354] I. Ahmad, W. Chen, and K. Chang, “LTE-railway user priority-based [375] B. Yi, X. Wang, K. Li, and M. Huang, “A comprehensive survey of
cooperative resource allocation schemes for coexisting public safety network function virtualization,” Computer Networks, vol. 133, pp.
and railway networks,” IEEE Access, vol. 5, pp. 7985–8000, 2017. 212–262, Mar. 2018.
[355] G. Carvajal, L. Araneda, A. Wolf, M. Figueroa, and S. Fischmeister, [376] M. Alizadeh, A. Kabbani, T. Edsall, B. Prabhakar, A. Vahdat, and
“Integrating dynamic-TDMA communication channels into COTS eth- M. Yasuda, “Less is more: Trading a little bandwidth for ultra-low
ernet networks,” IEEE Transactions on Industrial Informatics, vol. 12, latency in the data center,” in Proc. USENIX Conf. on Networked Sys.
no. 5, pp. 1806–1816, Oct. 2016. Des. and Implem., 2012, pp. 19–19.
[356] W. Chen, S. A. Patel, P. Gaal, H. Xu, and T. Luo, “Techniques for [377] M. Alizadeh, T. Edsall, S. Dharmapurikar, R. Vaidyanathan, K. Chu,
handling channel state information (CSI) in ultra low latency ULL- A. Fingerhut, F. Matus, R. Pan, N. Yadav, and G. Varghese, “CONGA:
LTE,” Dec. 21 2015, US Patent App. 14/977,163. Distributed congestion-aware load balancing for datacenters,” in ACM
[357] D. Choudhury and T. Inoue, “Guest editorial special issue on 5G SIGCOMM Computer Commun. Rev., vol. 44, no. 4, Oct. 2014, pp.
wireless communication systems and technologies,” IEEE Transactions 503–514.
on Microwave Theory and Techniques, vol. 64, no. 7, pp. 2205–2206, [378] S. N. Avci, Z. Li, and F. Liu, “Congestion aware priority flow control
Jul. 2016. in data center networks,” in Proc. IEEE IFIP Networking Conf. and
[358] M. Condoluci, T. Mahmoodi, E. Steinbach, and M. Dohler, “Soft Workshops, May 2016, pp. 126–134.
resource reservation for low-delayed teleoperation over mobile net- [379] T. Berisa and M. Maier, “Low-latency polling for passive optical
works,” IEEE Access, vol. 5, pp. 10 445–10 455, 2017. networks,” IEEE Communications Letters, vol. 17, no. 6, pp. 1288–
[359] G. Durisi, T. Koch, and P. Popovski, “Toward massive, ultrareliable, and 1291, Jun. 2013.
low-latency wireless communication with short packets,” Proceedings [380] X. Cao, I. Popescu, G. Chen, H. Guo, N. Yoshikane, T. Tsuritani, J. Wu,
of the IEEE, vol. 104, no. 9, pp. 1711–1726, Sep. 2016. and I. Morita, “Optimal and dynamic virtual datacenter provisioning
[360] L. Fan, Z. Dong, and P. Yuan, “The capacity of device-to-device over metro-embedded datacenters with holistic SDN orchestration,”
communication underlaying cellular networks with relay links,” IEEE Optical Switching and Networking, vol. 24, pp. 1–11, Apr. 2017.
Access, vol. 5, pp. 16 840–16 846, 2017. [381] I. Fujiwara, M. Koibuchi, H. Matsutani, and H. Casanova, “Swap-And-
[361] B. Lee, S. Park, D. J. Love, H. Ji, and B. Shim, “Packet structure and Randomize: A method for building low-latency HPC interconnects,”
receiver design for low latency wireless communications with ultra- IEEE Trans. on Parallel and Distr. Systems, vol. 26, no. 7, pp. 2051–
short packets,” IEEE Transactions on Communications, vol. 66, no. 2, 2060, July 2015.
pp. 796–807, Feb. 2018. [382] B. Guan, J. Wu, Y. Wang, and S. U. Khan, “CIVSched: A
[362] F. Lu, L. Cheng, M. Xu, J. Wang, S. Shen, and G. K. Chang, communication-aware inter-VM scheduling technique for decreased
“Orthogonal and sparse chirp division multiplexing for MMW fiber- network latency between co-located VMs,” IEEE Transactions on
wireless integrated systems,” IEEE Photonics Technology Letters, Cloud Computing, vol. 2, no. 3, pp. 320–332, Jul. 2014.
vol. 29, no. 16, pp. 1316–1319, Aug. 2017. [383] Z. Guo and Y. Yang, “High-speed multicast scheduling in hybrid
optical packet switches with guaranteed latency,” IEEE Transactions
[363] S. A. Patel, W. Chen, A. Damnjanovic, P. Gaal, M. S. Vajapeyam,
on Computers, vol. 62, no. 10, pp. 1972–1987, Oct. 2013.
and H. Xu, “Traffic data allocations in low latency LTE downlink
[384] ——, “Low-latency multicast scheduling in all-optical interconnects,”
communications,” Oct. 28 2015, US Patent App. 14/925,501.
IEEE Trans. on Commun., vol. 62, no. 4, pp. 1310–1323, Apr. 2014.
[364] J. Pilz, M. Mehlhose, T. Wirth, D. Wieruch, B. Holfeld, and
[385] K. He, W. Qin, Q. Zhang, W. Wu, J. Yang, T. Pan, C. Hu, J. Zhang,
T. Haustein, “A tactile internet demonstration: 1 ms ultra low delay
B. Stephens, A. Akella, and Y. Zhang, “Low latency software rate
for wireless communications towards 5G,” in Proc. IEEE INFOCOM
limiters for cloud networks,” in Proc. ACM Asia-Pacific Workshop on
Wkshps, 2016, pp. 862–863.
Networking (APNet), 2017, pp. 78–84.
[365] B. P. Rimal, D. P. Van, and M. Maier, “Mobile edge computing
[386] M. Khabbaz, K. Shaban, and C. Assi, “Delay-aware flow scheduling in
empowered fiber-wireless access networks in the 5G era,” IEEE Com-
low latency enterprise datacenter networks: Modeling and performance
munications Magazine, vol. 55, no. 2, pp. 192–200, Feb. 2017.
analysis,” IEEE Transactions on Communications, vol. 65, no. 5, pp.
[366] M. Salem, A. Adinoyi, H. Yanikomeroglu, and D. Falconer, “Fair 2078–2090, May 2017.
resource allocation toward ubiquitous coverage in OFDMA-based [387] A. G. Kumbhare, Y. Simmhan, M. Frincu, and V. K. Prasanna,
cellular relay networks with asymmetric traffic,” IEEE Transactions “Reactive resource provisioning heuristics for dynamic dataflows on
on Vehicular Technology, vol. 60, no. 5, pp. 2280–2292, Jun. 2011. cloud infrastructure,” IEEE Transactions on Cloud Computing, vol. 3,
[367] T. K. Vu, C. F. Liu, M. Bennis, M. Debbah, M. Latva-aho, and C. S. no. 2, pp. 105–118, Apr. 2015.
Hong, “Ultra-reliable and low latency communication in mmWave- [388] L. Liu, Z. Zhang, and Y. Yang, “Packet scheduling in a low-latency
enabled massive MIMO networks,” IEEE Communications Letters, optical interconnect with electronic buffers,” IEEE/OSA Journal of
vol. 21, no. 9, pp. 2041–2044, Sep. 2017. Lightwave Technology, vol. 30, no. 12, pp. 1869–1881, Jun. 2012.
[368] J. Y. Wei and R. I. McFarland, “Just-in-time signaling for WDM [389] ——, “In-order packet scheduling in optical switch with wavelength
optical burst switching networks,” IEEE/OSA Journal of Lightwave division multiplexing and electronic buffer,” IEEE Transactions on
Technology, vol. 18, no. 12, pp. 2019–2037, Dec. 2000. Communications, vol. 62, no. 6, pp. 1983–1994, Jun. 2014.
[369] J. Wu, C. Yuen, N. M. Cheung, J. Chen, and C. W. Chen, “Enabling [390] S. Liu, Q. Cheng, M. R. Madarbux, A. Wonfor, R. V. Penty, I. H. White,
adaptive high-frame-rate video streaming in mobile cloud gaming and P. M. Watts, “Low latency optical switch for high performance
applications,” IEEE Transactions on Circuits and Systems for Video computing with minimized processor energy load [invited],” IEEE/OSA
Technology, vol. 25, no. 12, pp. 1988–2001, Dec. 2015. Journal of Optical Communications and Networking, vol. 7, no. 3, pp.
[370] J. Zhang, Y. Ji, S. Jia, H. Li, X. Yu, and X. Wang, “Reconfigurable op- A498–A510, Mar. 2015.
tical mobile fronthaul networks for coordinated multipoint transmission [391] Y. Liu, D. Niu, and B. Li, “Delay-optimized video traffic routing
and reception in 5G,” IEEE/OSA Journal of Optical Communications in software-defined interdatacenter networks,” IEEE Transactions on
and Networking, vol. 9, no. 6, pp. 489–497, Jun. 2017. Multimedia, vol. 18, no. 5, pp. 865–878, May 2016.
[371] I. Afolabi, T. Taleb, K. Samdanis, A. Ksentini, and H. Flinck, “Network [392] F. Liu, J. Guo, X. Huang, and J. C. S. Lui, “eBA: Efficient bandwidth
slicing & softwarization: A survey on principles, enabling technologies guarantee under traffic variability in datacenters,” IEEE/ACM Trans. on
& solutions,” IEEE Communications Surveys & Tutorials, vol. 20, Networking, vol. 25, no. 1, pp. 506–519, Feb. 2017.
no. 3, pp. 2429–2453, Third Qu. 2018. [393] J. W. Lockwood and M. Monga, “Implementing ultra low latency data
[372] A. Blenk, A. Basta, M. Reisslein, and W. Kellerer, “Survey on center services with programmable logic,” in Proc. IEEE Symp. High-
network virtualization hypervisors for software defined networking,” Performance Interconnects, 2015, pp. 68–77.
IEEE Communications Surveys & Tutorials, vol. 18, no. 1, pp. 655– [394] W. Miao, S. Di Lucente, J. Luo, H. Dorren, and N. Calabretta, “Low
685, First Qu. 2016. latency and efficient optical flow control for intra data center networks,”
[373] A. Blenk, A. Basta, J. Zerwas, M. Reisslein, and W. Kellerer, “Control OSA Optics Express, vol. 22, no. 1, pp. 427–434, Jan. 2014.
plane latency with SDN network hypervisors: The cost of virtualiza- [395] W. Miao, F. Agraz, S. Peng, S. Spadaro, G. Bernini, J. Perelló, G. Zer-
tion,” IEEE Transactions on Network and Service Management, vol. 13, vas, R. Nejabati, N. Ciulli, D. Simeonidou et al., “SDN-enabled OPS
no. 3, pp. 366–380, Sep. 2016. with QoS guarantee for reconfigurable virtual data center networks,”
[374] V. Eramo, E. Miucci, M. Ammar, and F. G. Lavacca, “An approach for IEEE/OSA Journal of Optical Communications and Networking, vol. 7,
service function chain routing and virtual function network instance no. 7, pp. 634–643, Jul. 2015.
migration in network function virtualization architectures,” IEEE/ACM [396] W. Miao, F. Yan, and N. Calabretta, “Towards Petabit/s all-optical flat
Trans. on Networking, vol. 25, no. 4, pp. 2008–2025, Aug. 2017. data center networks based on WDM optical cross-connect switches
 58

with flow control,” IEEE/OSA Journal of Lightwave Techn., vol. 34, Akhilesh Thyagaturu is an Engineer at Intel Corpo-
no. 17, pp. 4066–4075, Sep. 2016. ration, Chandler, AZ, USA, and an Adjunct Faculty
[397] J. Perello, S. Spadaro, S. Ricciardi, D. Careglio, S. Peng, R. Nejabati, in the School of Electrical, Computer, and Energy
G. Zervas, D. Simeonidou, A. Predieri, M. Biancani, H. J. S. Dorren, Engineering at Arizona State University (ASU),
S. D. Lucente, J. Luo, N. Calabretta, G. Bernini, N. Ciulli, J. C. Sancho, PLACE Tempe. He received the Ph.D. in electrical engi-
S. Iordache, M. Farreras, Y. Becerra, C. Liou, I. Hussain, Y. Yin, PHOTO neering from Arizona State University, Tempe, in
L. Liu, and R. Proietti, “All-optical packet/circuit switching-based data HERE 2017. He serves as reviewer for various journals
center network for enhanced scalability, latency, and throughput,” IEEE including the IEEE Communications Surveys & Tu-
Network, vol. 27, no. 6, pp. 14–22, Nov. 2013. torials, IEEE Transactions of Network and Service
[398] S. H. S. Rezaei, A. Mazloumi, M. Modarressi, and P. Lotfi-Kamran, Management, and Optical Fiber Technology. He was
“Dynamic resource sharing for high-performance 3-D networks-on- with Qualcomm Technologies Inc., San Diego, CA,
chip,” IEEE Computer Architecture Letters, vol. 15, no. 1, pp. 5–8, USA, as an Engineer from 2013 to 2015.
Jan. 2016.
[399] G. M. Saridis, S. Peng, Y. Yan, A. Aguado, B. Guo, M. Arslan,
C. Jackson, W. Miao, N. Calabretta, F. Agraz et al., “Lightness:
A function-virtualizable software defined data center network with
all-optical circuit/packet switching,” IEEE/OSA Journal of Lightwave Ziyad Alharbi is a researcher at King Abdulaziz
Technology, vol. 34, no. 7, pp. 1618–1627, Apr. 2016. City for Science and Technology (KACST), Riyadh,
[400] A. Shpiner and E. Zahavi, “Race cars vs. trailer trucks: Switch buffers Saudi Arabia. He received his B.Sc. degree in
sizing vs. latency trade-offs in data center networks,” in Proc. IEEE Electrical Engineering from King Fahd University
Symp. on High-Performance Interconnects (HOTI), 2016, pp. 53–59. PLACE of Petroleum and Minerals, Saudi Arabia, and his
[401] B. Stephens, A. L. Cox, A. Singla, J. Carter, C. Dixon, and W. Felter, PHOTO M.S. in Electrical Engineering from Arizona State
“Practical DCB for improved data center networks,” in Proc. IEEE HERE University, Tempe. Currently, he is working towards
INFOCOM, 2014, pp. 1824–1832. his Ph.D. in Electrical Engineering at Arizona State
[402] P. Teymoori, D. Hayes, M. Welzl, and S. Gjessing, “Even lower latency, University. He serves as reviewer for various jour-
even better fairness: Logistic growth congestion control in datacenters,” nals including IEEE Communications Surveys & Tu-
in Proc. IEEE Conf. on Local Computer Netw. (LCN), Nov 2016, pp. torials, Computer Networks, and Optical Switching
10–18. and Networking
[403] W. Wang, Y. Sun, K. Salamatian, and Z. Li, “Adaptive path isolation
for elephant and mice flows by exploiting path diversity in datacenters,”
IEEE Transactions on Network and Service Management, vol. 13, no. 1,
pp. 5–18, Mar. 2016.
[404] L. Wang, X. Wang, M. Tornatore, K. J. Kim, S. M. Kim, D.-U. Kim,
K.-E. Han, and B. Mukherjee, “Priority-aware scheduling for packet- Cuixiang Wang is a lecturer in the school of
switched optical networks in datacenter,” in Proc. Int. Conf. on Optical Information Engineering at Yancheng Institute of
Netw. Design and Modeling (ONDM), May 2017, pp. 1–5. Technology, Yancheng, China. Currently, she is also
[405] E. Wong, M. P. I. Dias, and L. Ruan, “Predictive resource allocation a visiting scholar in the School of Electrical, Com-
PLACE puter, and Energy Engineering at Arizona State
for tactile internet capable passive optical LANs,” IEEE/OSA Journal
PHOTO University (ASU), Tempe. She received her B.Sc.
of Lightwave Technology, vol. 35, no. 13, pp. 2629–2641, Jul. 2017.
HERE degree in Computer Science and Technology from
[406] M. C. Yuang, P. L. Tien, H. Y. Chen, W. Z. Ruan, T. K. Hsu,
S. Zhong, J. Zhu, Y. Chen, and J. Chen, “OPMDC: Architecture design Qufu Normal University, Qufu, China, in 2007 and
and implementation of a new optical pyramid data center network,” her M.S. degree in Computer Software and Theory
IEEE/OSA Journal of Lightwave Technology, vol. 33, no. 10, pp. 2019– from Nanjing University of Posts and Telecommu-
2031, May 2015. nications, Nanjing, China in 2010. Her research is
[407] M. Caporuscio, P.-G. Raverdy, H. Moungla, and V. Issarny, “ubiSOAP: focused on wireless network routing and network simulation.
A service oriented middleware for seamless networking,” in Proc. Int.
Conf. on Service-Oriented Computing, LNCS 5364. Springer, Berlin,
Heidelberg, 2008, pp. 195–209.

Xing Shao is an associate professor in the school

of Information Engineering at Yancheng Institute
of Technology, Yancheng, China. Currently, he is
also a visiting scholar in the School of Electrical,
PLACE Computer, and Energy Engineering at Arizona State
PHOTO University (ASU), Tempe. He received the Ph.D.
HERE in Information Network from Nanjing University of
Posts and Telecommunications, Nanjing, China, in
2013. His research is focused on wireless multihop
networking, routing and network simulation.
Ahmed Nasrallah is currently a researcher funded
by Kuwait University to pursue the Ph.D. degree in
Computer Engineering at Arizona State University,
Tempe. He received his B.Sc. degree in Electrical
PLACE and Computer Engineering from the University of
PHOTO Dayton, Dayton Ohio, and his M.S. degree in Com- Martin Reisslein (S’96-M’98-SM’03-F’14) is a
HERE puter Engineering from Arizona State University. Professor in the School of Electrical, Computer,
His research is focused on communication and mul- and Energy Engineering at Arizona State University
timedia networking. (ASU), Tempe. He received the Ph.D. in systems
PLACE engineering from the University of Pennsylvania in
PHOTO 1998. He currently serves as Associate Editor for
HERE the IEEE Transactions on Mobile Computing, the
IEEE Transactions on Education, and IEEE Access
as well as Computer Networks and Optical Switching
and Networking. He is Associate Editor-in-Chief for
the IEEE Communications Surveys & Tutorials and
chairs the steering committee of the IEEE Transactions on Multimedia.
 59

Hesham ElBakoury is a thirty five year veteran in

the telecommunications and data networking indus-
try with an extensive background and expertise in
the architecture, design, and development of Dis-
PLACE tributed Systems and Broadband Access, Enterprise
PHOTO and Telco Communications Systems. He is a Prin-
HERE cipal Architect in Futurewei focusing on advanced
technology research and standards in the network
research Lab. Prior to joining Futurewei, Mr. ElBak-
oury was Chief Systems Architect in Hitachi-CTA
EPON Access Systems Division, and Chief Systems
Architect in Nortel and Bell-Northern Research where he led the architecture,
design, and development of several very successful Switching/Routing, Secu-
rity and Carrier Ethernet products. In Nortel/BNR, Mr. ElBakoury initiated and
led the Autonomic Network research project in the Enterprise division, and
the Software Design and Code Reuse Project in the Data Networking division.
Mr. ElBakoury has been active in different standard groups, including IEEE
802, IEEE 1904, IETF, ONF, OIF, MEF, the SCTE Energy 2020 program,
and CableLabs where he has been heavily involved in IEEE 802.3/802.1,
DPoE/DPoG, DOCSIS 3.1, Full-Duplex DOCSIS 3.1, SDN/NFV, Distributed
CCAP Architectures, and Business Services projects. He holds an M.S.C.
degree from Waterloo University, ONT, Canada.