Scheduling Real-Time Communication in IEEE 802.

1Qbv
Time Sensitive Networks ∗

Silviu S. Craciunas Ramon Serna Oliver Martin Chmelík Wilfried Steiner

TTTech Computertechnik AG
Schönbrunner Straße 7
1040 Vienna, Austria
{silviu.craciunas, ramon.serna.oliver, martin.chmelik, wilfried.steiner}@tttech.com

ABSTRACT abling mixed-criticality communication allowing real-time

The enhancements being developed by the Time-Sensitive and best-effort traffic to coexist simultaneously within IEEE
Networking Task Group as part of IEEE 802.1 emerge as 802 networks. Quality of service, as introduced in the IEEE
the future of real-time communication over Ethernet net- 802.1BA Audio/Video Bridging (AVB) standard, has raised
works for automotive and industrial application domains. significant interest within the industrial and automotive do-
In particular IEEE 802.1Qbv is key to enabling timeliness mains, but falls short of the industry expectations for hard
guarantees via so-called time-aware shapers. In this paper, real-time applications demanding reliable and fully deter-
we address the computation of fully deterministic schedules ministic communication [2, 25, 32]. Enhancing the features
for 802.1Qbv-compliant multi-hop switched networks. We of AVB with fault-tolerant mechanism allowing strict timing
identify and analyze key functional parameters affecting the guarantees with low latency and jitter is desirable to support
deterministic behaviour of real-time communication under real-time applications over Ethernet.
802.1Qbv and, based on a generalized configuration of these The IEEE 802.1 Time Sensitive Networking (TSN) task
parameters, derive the required constraints for computing group [20] is in the process of standardizing time-sensitive
offline schedules guaranteeing low and bounded jitter and capabilities over IEEE 802 networks by defining a whole
deterministic end-to-end latency for critical communication range of 802.1 sub-standards, covering aspects that range
flows. Furthermore, we discuss several optimization direc- from clock synchronization and frame preemption to re-
tions and concrete configurations exposing trade-offs against dundancy management and scheduled traffic enhancements.
the required computation time. We also show the perfor- Key enablers of real-time communication in TSN networks
mance of our approach via synthetic network workloads on are a network-wide clock reference with bounded precision,
top of different network configurations. known and bounded network latencies (e.g. switch for-
warding times, link delays), as well as isolation of critical
and non-critical traffic classes through a global communica-
1. INTRODUCTION tion schedule. IEEE 802.1ASrev [20] defines the basis for
Ethernet has evolved to be the standard open communi- a network-wide time-synchronization protocol, achieving a
cation mechanism for a wide range of application domains global network clock and basic fault-tolerance by means of a
originally not bound to strict timing requirements. Typical re-election mechanism of the clock master (also called grand-
design and performance criteria for standard Ethernet de- master) in case of failures. Time sensitive communication
ployments are related to best-effort communication, in which takes place via so-called time-sensitive streams (or flows),
maximizing throughput and minimizing average delays are where a stream is defined by a sender (or talker) and one or
primary objectives. In the real-time domain, safety-critical multiple receivers (or listeners). Individual messages belong-
timing aspects have been introduced by means of technolo- ing to a flow are defined using the VLAN identifier (VID)
gies like TTEthernet (SAE AS6802 [21, 34]), PROFINET, of VLAN-tagged frames. Streams can have different priori-
and EtherCAT [27] among others. Recently, the conver- ties defining their traffic class, given by the 3 priority bits of
gent needs of the industrial automation and automotive do- the priority code point (PCP) of the IEEE 802.1Q header.
mains are pushing for the standardization of solutions en- IEEE 802.1Qbv [19] defines a time-based shaper functional-
∗ ity enabling time-triggered communication [22] at the egress
This project has received funding from the European
Union’s Horizon 2020 research and innovation programme ports. A time-aware shaper is essentially a gate enabling or
under grant agreement No 700665 (CITADEL). disabling the transmission of frames for a queue following
the specification of a periodic schedule.
Traditionally, worst-case end-to-end communication la-
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed tencies in non-scheduled networks (i.e., using priority
for profit or commercial advantage and that copies bear this notice and the full cita- queues) have been analyzed through methods like network
tion on the first page. Copyrights for components of this work owned by others than calculus [12, 14] or the trajectory approach [4]. In such ap-
ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or re-
publish, to post on servers or to redistribute to lists, requires prior specific permission proaches, the end-to-end latency of flows is computed based
and/or a fee. Request permissions from [email protected]. on traffic priorities and arrival patterns making composi-
RTNS ’16, October 19-21, 2016, Brest, France tional system design and isolation of time behaviour very
c 2016 ACM. ISBN 978-1-4503-4787-7/16/10. . . $15.00 difficult. With the introduction of 802.1Qbv, the comple-
DOI: http://dx.doi.org/10.1145/2997465.2997470

183
 Queue 1 mine at which time instants a queue is opened and traffic
is forwarded to the egress port and at which time instants
Queue 2 the queue is closed such that any pending traffic remains
Port A
(ingress) buffered. If multiple queues are opened at the same time,
Queue 3
SW Priority Port C the queue priority determines which of the queues is allowed
Engine filter (egress) to forward frames.
Port B We identify two key parameters that affect the timing
(ingress)
properties of network flows in 802.1Qbv-capable networks,
Queue n
namely Device capabilities and Queue configuration.
One important characteristic is the capability of devices,
namely, if they are scheduled or not. If the end-systems are
Figure 1: Simplified view of a 802.1Qbv-capable switch. not scheduled but switches are, frames from the end-nodes
may arrive in any order to the switches. Therefore, the first
switch on the route has to act as a synchronization gate.
mentary approach, which we present in this paper, is now This synchronization is possible if there are enough queues
also supported, where correct schedules with respect to in- to isolate flows arriving simultaneously, allowing thus each
dividual flow requirements are created offline. The major of them to be scheduled deterministically. Complementary,
benefit of the 802.1Qbv extension is that it allows temporal end-systems may be scheduled but switches are not. In such
isolation and compositional system design when flows are a case, switches act as delay elements but cannot control
scheduled end-to-end. the time of forwarding frames. If end-systems and switches
In our previous work [7, 8, 33], we have addressed sev- are scheduled, the scheduler can plan when to send indi-
eral scheduling problems in TTEthernet networks. However, vidual frames of flows such that they arrive with a known
the particularities of TSN demand different solutions than order at the first switch and switches can control forward-
those employed in the past. In this paper, we address the ing times. We denote a network configuration (G(E)) where
scheduling problem arising from the IEEE 802.1Qbv exten- only end-system are scheduled with Ve and one where only
sion on multi-hop fully switched TSN networks, presenting a switches are scheduled with Vs . We denote fully scheduled
method for computing static schedules for 802.1Qbv-capable networks, i.e., all devices are scheduled, with configuration
network devices using Satisfiability Modulo Theories (SMT) Ve+s . Here we focus on deterministic networks with sched-
and, alternatively, Optimization Modulo Theories (OMT) uled end-systems and switches and leave the other corner-
solvers. We first identify key functional parameters affect- cases for a more in-depth analysis in future work.
ing the behaviour of TSN networks and extract a general- The number of queues per egress port, which translates
ized configuration of these parameters for hard real-time net- to the number of different priorities that can be handled,
works (Section 2). In Section 3 we describe the formalization affects the amount of traffic and the real-time properties to
of the network model used throughout the paper. Based on be guaranteed. Another important aspect is how the queues
the formalization and the generic configuration, we adapt a operate. Queues may follow a strict priority policy, meaning
subset of constraints general to deterministic Ethernet from that the gate is always opened and the arbitration is purely
previous work and, additionally, construct TSN-specific con- based on their assigned priority. Using the time-gates, how-
straints that correctly formulate the real-time behaviour of ever, queues can be scheduled following a time-triggered
TSN streams (Section 4). In Sections 5 and 6 we discuss (TT) paradigm, where the priority is overruled by the sched-
the implication of different configurations on the scheduling ule, i.e., the schedule of the timed-gate defines the servicing
problem and present implementation details of our method order, not the priority. If queues are scheduled following
using SMT and OMT solvers together with a discussion on a mutually exclusive pattern, i.e., no two queue gates are
optimization objectives. After the experimental evaluation opened at the same time, the result is a fully deterministic
in Section 7, we survey related work in Section 8 and con- forwarding policy. In this paper we focus on creating sched-
clude the paper in Section 9. ules for the timed-gates of scheduled queues. We denote
the queue configuration by the tuple G(Q) = hℵ, ℵtt , ℵprio i,
2. 802.1Qbv FUNCTIONAL PARAMETERS where ℵ is the total number of queues per port, ℵtt is the
number of queues operating as scheduled (TT) queues, and
Figure 1 depicts a logical representation of a simplified
ℵprio is the remaining number of priority queues. Without
802.1Qbv-capable switch with three ports. The figure de-
loss of generality we assume that in the system configura-
picts a scenario in which network traffic traverses the switch
tion, the scheduled queues always have the highest priorities
from an ingress port to an egress port. Internally, the ports
of all queues, including the remaining priority queues.
are connected to a filter distributing incoming frames to the
Having different queue types opens up the possibility to
associated queue of the egress port based on classification
have two different types of traffic, namely scheduled and
criteria like e.g. the priority code point (PCP) of the IEEE
non-scheduled traffic. Scheduled traffic is assigned to the
802.1Q header. On regular switches with multiple ports, in-
scheduled queues in order to guarantee latency and jitter
coming traffic will be directed to the corresponding egress
requirements. Non-scheduled traffic is isolated from sched-
port by the switching engine. Every port will have a similar
uled traffic within the priority queues of a device. Within
logical composition on egress as the one depicted, including
the priority queues, non-scheduled flows may interfere with
a series of logical queues buffering the respective frames un-
each other, but not with scheduled flows (since the associ-
til their transmission. In the most generic representation,
ated scheduled queues always have higher priority). For the
each 802.1Qbv queue has a timed gate associated to it en-
non-scheduled traffic, worst-case end-to-end latencies in the
abling or disabling the transmission of frames according to
presence of interference have been analyzed through meth-
a predefined static schedule. Hence, scheduled events deter-

184
ods like network calculus [14] or the trajectory approach [4] (listeners). Without loss of generality, we choose to sim-
for AFDX networks while recently, similar analysis methods plify the notation by restricting the number of receivers
have been employed for AVB networks [12]. to one. Enhancing the model to include multicast flows
Scheduled traffic is defined as having requirements on the is a trivial extension without implications on the valid-
maximum end-to-end latency as well as minimal jitter con- ity of the presented method. We denote the input set of
straints, i.e., each periodic instance of the communication flows by S. Similar to [33], a flow si ∈ S from sender
flow has to arrive at the same instant in time. In real-time node va to receiver node vb , routed through the intermedi-
systems, this traffic is said to have high criticality. Non- ary nodes (i.e. switches) v1 , v2 , . . . , vn−1 , vn is expressed as
scheduled traffic cannot be guaranteed with minimal jitter si = [[va , v1 ], [v1 , v2 ], . . . , [vn−1 , vn ], [vn , vb ]]. A flow is char-
and may or may not have requirements on the end-to-end acterized by the tuple hsi .e2e, si .L, si .T i, denoting the max-
latency. In this paper we address the scheduling of high- imum allowed end-to-end latency, the data size in bytes, and
criticality traffic for fully-scheduled networks consisting of the period of the flow, respectively.
802.1Qbv devices with ℵtt ≥ 1. In the model differentiate between a flow and the instance
A system configuration, composed of the 2 parameters of a flow on a link. An instance of a flow si ∈ S routed
[v ,v ]
presented above, is defined by the tuple hG(E), G(Q)i. In through link [va , vb ] ∈ L is denoted by si a b . We denote
Sections 3 and 4 we introduce the 802.1Qbv-specific net- the queue for the given flow instance in the egress port as a
[v ,v ]
work and traffic model and define scheduling constraints variable si a b .p. Note that the queue variable is synony-
for high-criticality traffic, respectively, assuming generic pa- mous to the priority of the flow within the egress port of
rameter configurations for critical communication, namely the respective device. Additionally, we draw an equivalence
{Ve+s , hn, m, n−mi}. Our constraints generate schedules for between the egress port and its respective transmission link.
the m queues while ensuring that critical flows meet their It is important to note that the queue assigned to flow in-
latency and jitter requirements. Later, in Section 6, we dis- stances of the same flow may or may not change along the
cuss several scenarios also including priority queues and the subsequent egress ports2 .
resulting trade-offs based on specific system configurations. Since the data size of a flow is allowed to exceed the Eth-
ernet MTU size in TSN, each flow instance is associated
with a set of frames, each with size less than or equal to
3. NETWORK AND TRAFFIC MODEL the MTU size. Maintaining our notation from [8], we de-
Our work targets multi-hop layer 2 switched Ethernet net- [v ,v ]
note the set of frames fi,ja b of a flow instance si a b by
[v ,v ]
works over full-duplex multi-speed physical links. We model [v ,v ]
the network, similar to [8] and [33], as a directed graph Fi a b . Additionally, the first and last frame of the set,
G(V, L), where the nodes (switches and end-systems) are the ordered by the schedule offset on the link, are expressed
[v ,v ] [v ,v ] [v ,v ]
set of graph vertices (V) and the links between nodes are rep- as Fi a b with fi,1a b and last(Fi a b ), respectively. A
[v ,vb ] [va ,vb ]
resented through the graph edges (L ⊆ V ×V). A full-duplex frame fi,ja ∈ Fi is defined by the tuple
physical link between nodes va ∈ V and vb ∈ V results in two
directional logical links, each denoted by an ordered tuple, [v ,vb ] [v ,vb ] [v ,vb ]
hfi,ja .φ, fi,ja .T, fi,ja .Li,
namely [va , vb ] ∈ L and [vb , va ] ∈ L, respectively. Maintain-
ing the notation from [8], a physical link [va , vb ] is character- [v ,vb ] [v ,vb ]
where fi,ja .φ ∈ [0, fi,ja .T ] is the offset in macroticks
ized by the tuple h[va , vb ].s, [va , vb ].d, [va , vb ].mt, [va , vb ].ci, [v ,v ]
of the frame on link [va , vb ], fi,ja b .T = d [vas,vi b.T].mt e is
where [va , vb ].s is the speed of the link, [va , vb ].d is the propa-
gation delay on the medium, and [va , vb ].mt is the macrotick the period of the flow scaled to the link macrotick, and
[v ,v ] i ×[va ,vb ].s
of the link. We extend the definition with a new parameter, fi,ja b .L = d L[v a ,vb ].mt
e is the transmission duration of
[va , vb ].c, which represents the number of available queues the frame also scaled to the link macrotick [8].
in the device. Since any given egress port is connected to at
most one link, we establish an equivalence between the port
and its associated link. Therefore, we refer to [va , vb ].c as
4. SCHEDULING CONSTRAINTS
the number of egress queues1 present in the respective port In order compute a schedule for the timed gates of the
as a property of the link. We derive the transmission time scheduled queues we define scheduling constraints for the
of a frame between two nodes through the link speed (or frame offset variables (φ) and the flow instance queue vari-
bit rate) which represents the transfer rate over the physical ables (p), such that the correct temporal behaviour on the
network cable. For e.g. on a 1Gbit/sec link an MTU-sized scheduled communication is guaranteed. We differentiate
IEEE 802.1Q Ethernet frame of 1542 bytes would have a between basic constraints for deterministic Ethernet (Sec-
transmission time of 12.336µsec. The propagation delay is tion 4.1) and 802.1Qbv specific constraints (Section 4.2) for-
additional delay on frame arrival resulting from the delay malising the specific behaviour of time sensitive flows.
on the physical medium and the link length. The macrotick
is a design property either given by device constraints or 4.1 Basic Deterministic Ethernet Constraints
system-level design specifications. It specifies the length of In [33], a standard TTEthernet model is used where peri-
a discrete time unit defining the granularity of time events odic flows are limited to one single frame. In [8], the model is
for the given link. enhanced allowing combined scheduling of preemptive tasks
A stream, or flow, is a periodic multicast data transmis- and network flows in a TTEthernet network. The frame
sion from one sender (talker) to one or multiple receivers description and model from [8], where tasks are modelled
as sets of frames, fits well into the multi-frame flow model
1
The proposed value for this parameter is 8 according to
2
802.1Qbv [19], however, we do not put any limitation on Different queue assignment along multi-hop routes can oc-
this number for the scheduling algorithm. cur by means of e.g. VLAN re-tagging.

185
 [vx,va [vx,va [vx,va
] ] ]
] [va,vb] ] [va,vb] ] [va,vb]
[vy,va [vy,va [vy,va
va va va

(a) Interleaving flows in egress queue. (b) Isolation of flows in different queues. (c) Isolation of flow arrival.

Figure 2: Flow interleaving and isolation within an egress port of a 802.1Qbv switch.

that we discuss here. Hence, in this section we briefly reit- The constraint imposes that a frame can only be scheduled
erate the most important constraints from [8], adapted and on a subsequent link [vx , vb ] after the complete reception on
changed to fit our model, and refer the reader to [8, 33] for the previous link [va , vx ], including the propagation delay
a more complete and detailed list of scheduling constraints of the respective link ([va , vx ].d). Note that, as opposed
for deterministic Ethernet. to [8], the constraint is set to individual frames rather than
Frame Constraint. The frame offset of any frame sched- the complete flow instance, to allow initiating forwarding as
uled in the network has to be greater than or equal to 0. Ad- soon as the first frame is fully received without requiring to
ditionally, the entire transmission window (offset plus frame buffer the entire flow.
duration) has to fit within the frame period. Hence, we have End-to-End Constraint. The maximum end-to-end la-
the condition adapted from [8] tency constraint specifies that the difference between the ar-
[v ,v ] [v ,v ]
rival and sending time of a flow has to be less than or equal
∀si ∈ S, ∀[va , vb ] ∈ si , ∀fi,ja b ∈ Fi a b : to the specified maximum. We denote the sending link of

[v ,v ]
 
[v ,v ] [v ,v ] [v ,v ]
 flow si with src(si ) and the last link before the receiving
fi,ja b .φ ≥ 0 ∧ fi,ja b .φ ≤ fi,ja b .T − fi,ja b .L .
node with dest(si ).
Link Constraint. No two frames that are routed ∀si ∈ S : src(si ).mt × fi,1
src(si )
.φ + si .e2e ≥ (3)
through the same physical link in the network can overlap
dest(si ) dest(si )
in the time domain. The constraint adapted from [8] is as dest(si ).mt × (last(Fi ).φ + last(Fi ).L).
follows
[va ,vb ] [va ,vb ]
4.2 802.1Qbv Constraints
∀[va , vb ] ∈ L, ∀Fi , Fj , i 6= j In TTEthernet a static schedule is provided per device
[v ,vb ] [va ,vb ] [v ,vb ] [va ,vb ] port and defines the temporal behaviour of all frames of the
∀fi,ka ∈ Fi , ∀fj,la ∈ Fj ,
scheduled flows whereas in 802.1Qbv the schedule is spec-
∀α ∈ [0, hpji /si .T
− 1], ∀β ∈ [0, hpji /sj .T − 1] : ified for an egress queue, affecting multiple flows with the
same traffic class (priority). Moreover, a TTEthernet stream

[v ,v ] [v ,v ]
fi,ka b .φ + α × fi,ka b .T ≥
 is defined as one frame of at most MTU size whereas flow
[v ,v ] [v ,v ]
fj,la b .φ + β × fj,la b .T + fj,la
[v ,vb ]
.L ∨ (1) in 802.1Qbv can be composed of multiple frames. Hence,
 802.1Qbv-specific constraints need to be specified. We elab-
[v ,v ] [v ,v ]
fj,la b .φ + β × fj,la b .T ≥ orate on these constraints using a simplified example, de-
 picted in Figure 2, where two flows arriving at a switch from
[v ,vb ] [v ,vb ] [v ,vb ] different sources are forwarded via the same egress port, dis-
fi,ka .φ + α × fi,ka .T + fi,ka .L ,
cussing both the case when they are queued in the same
where hpji = lcm(si .T, sj .T ) is the hyperperiod of si and sj . scheduled queue and the case when they use different sched-
Flow Transmission Constraint. The propagation of uled queues.
frames of a flow must follow the sequential order along Egress Interleaving. In the first scenario, depicted in
the routed path of the flow. The network precision, de- Figure 2(a), the two flows arrive from different links at
noted with δ, represents the worst-case difference between roughly the same time. Regardless of the synchronization
the local clocks of any two synchronized (e.g. via the IEEE protocol used, there will always be an amount of synchro-
802.1AS [20] time-synchronization protocol) devices. Please nization error between individual devices that may lead to
note that in the TTEthernet synchronization protocol [35] alternating arrival order of frames during runtime. There-
the synchronization frames typically have the highest pri- fore, the order in which the individual frames are placed in
ority and introduce a shuffling delay to the critical frames. the scheduled queue at runtime is non-deterministic. It is
The IEEE 802.1AS protocol also requires synchronization important to note that the schedule controls the timing of
frames that travel on the same physical links as the critical the events on the egress port, not the order of frames in the
frames but they can have a lower priority than the critical queue. If the timed gate of the respective queue is opened
traffic and hence do not introduce a shuffling delay. first for the time interval needed to transmit 2 frames and
some time later for 3 frames, as depicted in the example,
∀si ∈ S, ∀[va , vx ], [vx , vb ] ∈ si , any combination of the respective frames of both flows may
[v ,vx ] [va ,vx ] [v ,vb ] [vx ,vb ] occur on the egress port. Hence, the timely behaviour of the
∀fi,ja ∈ Fi , ∀fi,jx ∈ Fi :
two flows may oscillate each periodic instance accumulating
[v ,v ]
fi,jx b .φ × [vx , vb ].mt − [va , vx ].d − δ ≥ (2) jitter for the overall end-to-end transmission. In order to
[v ,v ] [v ,v ] guarantee that the given end-to-end latency is always ful-
(fi,ja x .φ + fi,ja x .L) × [va , vx ].mt. filled, the worst-case delay introduced on each egress port

186
along the flow path must be considered in the end-to-end Flow Isolation Constraint. In reality, frame loss can
latency constraint for the flow. However, the worst-case jit- occur, periodic payload size may vary over time, or frames
ter that accumulates with each hop along the route of the might be dropped due to e.g. time-based ingress policing
flow is not desirable in hard real-time systems. Addition- (e.g. IEEE 802.1Qci). In these cases there may be instances
ally, the non-determinism introduced by the interleaving of where a deterministic behaviour on the egress port cannot be
individual frames weakens the temporal isolation of flows guaranteed. Consider the case where two frames from two
since a change in one flow may affect other flows by adding different flows are scheduled to arrive one after the other
interference delay and jitter. at the switch. Both are placed in the desired order in the
One solution to avoid this delay and jitter is hence to same queue. At a later point in time, after both have been
introduce conditions on the computed schedule preventing placed in the queue, the timed gate of the queue is opened
flows from interleaving, thus guaranteeing the isolation of for the first frame (belonging to the first flow) and only at a
flows (or frames) in the transmitting queues. In other words, later point in time for the second frame. If the first frame is
the constraints must either enforce a deterministic order of lost, the frame from the second flow will take its place in the
the flows in queues or place flows in different queues such queue and be transmitted in the time slot originally reserved
that each frame is dispatched deterministically according to for the first flow leading to non-determinism and jitter. To
the schedule of the associated timed gate. In this case, the guarantee isolation in a real network, we first consider iso-
scheduler either enforces two flows arriving during interfer- lating flows in the time domain assuming that they share
ing intervals to be placed in different queues (Figure 2(b)), the same egress queue. Hence, we formulate constraint (4)
or else allow them to be placed in the same queue but ensure to enforce a correct ordering of flows, i.e., if a frame of a
that the intended order and transmission time for all frames given flow has entered a queue, no frame of another flow
of the flows are preserved (Figure 2(c)). may arrive at the queue until all frames of the previous flow
We first assume that the two considered flows arriving at have been fully dispatched. We remind the reader that the
the same device will be queued in the same egress queue first and last frame of a flow, already ordered by their trans-
[v ,v ]
and later relax this assumption and formulate the complete mission time (i.e. scheduled offset), are defined by fi,1a b
802.1Qbv isolation constraint for high-criticality flows en- [va ,vb ]
and last(Fi ), respectively. Hence, we have
abling deterministic behaviour in the time domain.
[v ,v ] [v ,v ]
Let si a b and sj a b be, respectively, the flow instances ∀[va , vb ] ∈ L, ∀si
[va ,vb ] [va ,vb ]
, sj ∈ S, i 6= j,
of si ∈ S and sj ∈ S scheduled on the outgoing link (and
hence egress port) [va , vb ] of device va . We remind the ∀α ∈ [0, hpji /si .T− 1], ∀β ∈ [0, hpji /sj .T − 1] :
reader the equivalence between link [va , vb ] and the respec-

[v ,v ]
last(Fi a b ).φ × [va , vb ].mt + α × si .T + δ ≤
tive egress port and its associated output queues. Flow si
arrives at the device va from some device vx on link [vx , va ].

[v ,v ]
fj,1y a .φ × [vy , va ].mt + β × sj .T + [vy , va ].d ∨ (4)
Similarly, flow sj arrives from another device vy on incoming 
link [vy , va ]. If si and sj would arrive from the same device, [v ,v ]
last(Fj a b ).φ × [va , vb ].mt + α × sj .T + δ ≤
constraint (1) would ensure that the frames of the flows will 
not overlap in the time domain. [v ,v ]
fi,1x a .φ × [vx , va ].mt + β × si .T + [vx , va ].d .
Ideal scenario. Let’s assume a perfect network where no
frame is lost or discarded and all flows are sent constantly The constraint ensures that once a flow has arrived at the
in full size by their sending nodes (i.e. constant payload receiving device [va , vb ], no other flow can arrive at the same
size). In order to guarantee the desired output schedule, it egress port until the first flow has been completely sent on
would suffice to ensure that any two individual frames of the output port. Hence, individual frames of the first arriv-
different flows are not scheduled to arrive at the same time, ing flow will not be interleaved with any frames of another
hence having a deterministic queuing order. The isolation flow until the queue has been completely emptied.
condition in an ideal network is Frame Isolation Constraint. Despite being generally
[va ,vb ] [va ,vb ] faster to solve since not all interleavings of individual frames
∀[va , vb ] ∈ L, ∀si , sj ∈ S, i 6= j,
[v ,v ] [v ,v ] [v ,v ] [va ,vb ]
have to be considered, the flow isolation constraint is restric-
∀fi,ka b ∈ Fi a b , ∀fj,la b ∈ Fj , tive and may decrease the search space for valid schedules.
∀α ∈ [0, hpji /si .T− 1], ∀β ∈ [0, hpji /sj .T
− 1] : This can be easily proven by considering an input where a
 high-rate flow has a period which is equal to or less than
[v ,v ]
fi,kx a .φ × [vx , va ].mt + α × si .T + [vx , va ].d + δ ≤ the combined transmission duration of all frames of a low-
 rate flow. In this case, there is at least one period instance
[v ,v ]
fj,ly a .φ × [vy , va ].mt + β × sj .T + [vy , va ].d ∨ of the high rate flow in which its frames have to interleave

[v ,v ]
with the frames of the low-rate flow. Given the previous
fj,ly a .φ × [vy , va ].mt + β × sj .T + [vy , va ].d + δ ≤ isolation constraint, this would not be schedulable if there
[v ,v ]
 is only one queue per port, i.e., the flows must be placed in
fi,kx a .φ × [vx , va ].mt + α × si .T + [vx , va ].d , different queues. The counterexample can be generalized for
an arbitrary number of queues.
def
where hpji = lcm(si .T, sj .T ) is, as before, the hyperperiod In order to avoid this, we relax the previous constraint to
of flows si and sj . The condition imposes that individual allow frames interleaving between flows in a queue while in
frames of any two flows sharing a queue on device [va , vb ] the same time guaranteeing that the order on the output
are sent from their sending devices ([vx , va ] and [vy , va ]) in port is deterministic. We do this by ensuring that there are
such a way that they would arrive at the respective ports of only frames of one flow in the queue at a time, i.e., frames
the receiving device [va , vb ] in a deterministic order. from another flow may only enter the queue if the already

187
queued frames of the initial flow have been serviced. We
Priority 1
enforce that, if two frames are from different flows, one can
only be scheduled to arrive at the shared queue if the other Priority 2
has already been dispatched from that queue. The frame Priority 3
isolation condition for schedulability is then
[va ,vb ] [va ,vb ] Combined
∀[va , vb ] ∈ L, ∀si , sj ∈ S, i 6= j,
[v ,vb ] [va ,vb ] [v ,vb ] [va ,vb ] BE-queue
∀fi,ka ∈ Fi , ∀fj,la ∈ Fj ,
ρ ρ
∀α ∈ [0, hpji /si .T − 1], ∀β ∈ [0, hpji /sj .T − 1] :

[v ,v ]
fj,la b .φ × [va , vb ].mt + α × sj .T + δ ≤ Figure 3: Example of generating BE-queue schedules.

[v ,v ]
fi,kx a .φ × [vx , va ].mt + β × si .T + [vx , va ].d ∨ (5)
 queue (BE-queue) is reserved for non-scheduled flows, al-
[v ,v ]
fi,ka b .φ × [va , vb ].mt + β × si .T + δ ≤ lowing two alternative approaches. On one hand, leaving
 its timed gate permanently open may result in interference
[v ,v ]
fj,ly a .φ × [vy , va ].mt + α × sj .T + [vy , va ].d , to scheduled traffic due to occasional shuffling caused by
ongoing transmissions originated just before the opening of
For further use, we denote constraint (5) (or, if the flow scheduled queues. This can be included in the schedulability
isolation method is used, constraint (4)) between two flows conditions by extending the duration of individual scheduled
[v ,v ] [v ,v ]
as Φ(si a b , sj a b ). frames assuming that, in the worst case, one non-scheduled
The above constraints ((4) and (5)) apply to frames in the frame of MTU size can shuffle each critical frame. On the
same queue. However, as mentioned before, the scheduler other hand, we propose to generate a schedule for the timed
may choose to place flows in different queues. Hence, the gate of the BE-queue by negating the combined schedule of
complete constraint for minimum jitter scheduling of high- all scheduled queues, i.e., for each gate open event of any
criticality flows is other queue of the port, we create a gate close event for
[v ,v ] [v ,v ] the BE-queue, and vice-versa. If need be, depending of de-
∀[va , vb ] ∈ L, ∀si a b , sj a b ∈ S, vice implementation, gate closing events for the BE-queues
   
[v ,v ] [v ,v ] [v ,v ] [v ,v ]
Φ(si a b , sj a b ) ∨ si a b .p 6= sj a b .p , (6) may be decreased by the duration of MTU-sized frame in
order to guarantee that no unscheduled frame may interfere
[va ,vb ] [va ,vb ] scheduled frames. In Figure 3 we depict a simple example
with si .p ≤ [va , vb ].c and sj .p ≤ [va , vb ].c.
where the BE-queue schedule is generated from the com-
bined schedules of all other queues but with the duration of
5. 802.1QBV CONFIGURATIONS an MTU BE frame (denoted by ρ in the figure) subtracted
The constraints presented above apply to the m scheduled from the gate open events of the combined schedule.
queues of a generic configuration for critical traffic in fully Note that, if all egress ports would have as many queues as
scheduled 802.1Qbv networks, namely {Ve+s , hn, m, n−mi}. there are incoming scheduled flows, each could be assigned
We now briefly discuss the implications of a number of spe- to its own dedicated queue. Hence, the problem is similar to
[v ,v ] [v ,v ]
cial configurations which include non-scheduled traffic and scheduling TTEthernet, since si a b .p 6= sj a b .p is always
different configurations of the scheduled and priority queues. true. In this case, each queue will behave as a “large” buffer,
Configuration 1: {Ve+s , h1, 1, 0i} This case corresponds sufficient to accommodate all frames of each flow instance,
to devices in a network having one single queue per egress guaranteeing isolation between flows through the schedule.
port operated as a scheduled queue. Therefore, condi- Configuration 4: {Ve+s , hn, m, n − mi} This scenario is
[v ,v ] [v ,v ]
tion si a b .p 6= sj a b .p is always false and, hence, high- suitable for high-criticality applications that feature both
criticality flows are scheduled to be completely sequential in scheduled and non-scheduled traffic. The separation in
the time domain. This case amounts to serializing incoming scheduled and non-scheduled queues influences both the so-
traffic that converges on the same egress port reproducing a lution space for high-criticality flows and worst-case latency
similar behaviour to bus systems like, e.g. TTP [23]. of non-scheduled flows for analysis methods like network cal-
Configuration 2: {Ve+s , hn, 1, n − 1i} This is a general- culus [12, 14] or the trajectory approach [4]. If m = 1, we
ization of case 1, in which high-criticality flows are assigned have configuration 2 in which non-scheduled traffic can be
to the scheduled queue while non-scheduled flows use one of optimized at the expense of solution space for high-criticality
the n − 1 available priority queues. The timely behaviour traffic. On the other hand, if m = n − 1, the solution space
of non-scheduled flows depends on queue priorities and ar- for high-criticality traffic is maximized but the pessimism of
rival patterns [12, 14]. This configuration corresponds to the the latency bounds of non-scheduled flows may be higher. In
special case presented in [31] compatible with the urgency- Section 6.1 we discuss optimization criteria for the general
based scheduler approach proposed by the authors. This case of this configuration.
configuration is suitable for legacy AVB systems that need Configuration 5: {Ve+s , hn, 0, ni} This case mimics the
to be augmented with a few additional high-criticality flows behaviour of a standard AVB (IEEE 802.1BA) network in
requiring tighter temporal bounds. which flows are serviced according to the priority assigned to
Configuration 3: {Ve+s , hn, n, 0i} Contrary to case 2, all their traffic class, which is equivalent to setting the timed-
of the queues are designated as scheduled queues but we gates to be permanently opened. End-to-end properties of
assign n − 1 queues for scheduled traffic, increasing the so- the non-scheduled flows are subject to complementary anal-
lution space for critical traffic. The remaining scheduled ysis methods like e.g. [6, 9, 12].

188
6. 802.1QBV SCHEDULER time the provided solution is optimized with respect to cer-
Satisfiability Modulo Theories (SMT) are designed to tain properties of the system. Optimization solvers, like
determine the satisfiability of first-order logical formulas Gurobi [16] or GLPK [15], are explicitly designed to solve
against certain background theories like linear integer arith- constrained optimization problems and can be used to tackle
metic (LA(Z)) or bit-vectors (BV) [3, 29]. On top of check- optimized scheduling (cf. [8] for an example). In recent
ing satisfiability, SMT solvers also provide a model for the years, a new field, called Optimization Modulo Theories
given satisfiable context which represents one solution (out (OMT), has emerged where certain SMT solvers are aug-
of a set of potentially multiple feasible solutions) for the mented with optimization capabilities [5, 24, 30]. A typical
given variables and constraints. NP-complete scheduling optimization objective used in industry is minimizing end-
problems that exhibit combinatorial characteristic and have to-end latency of selected flows. The minimization objec-
arithmetic constraints, like the one addressed in this pa- tive is easily expressed based on constraint (3) from Sec-
per, present a suitable use-case for constraint-satisfaction tion 4.2 and has been discussed in previous work (cf. [8]).
SMT solving in linear arithmetic [1, 11]. Using SMT for Here, we focus on optimization opportunities arising from
solving deterministic network scheduling problems was first the specifics of IEEE 802.1Qbv.
proposed by Steiner in [33]. In the presented method we have seen how scheduled flows
The aim of our scheduling algorithm for 802.1Qbv is to are isolated either in the time domain or in different queues
find values for all individual frame offsets and queue assign- in order to avoid interleaving. Scheduling along the time
ments in each respective egress port of flows routed along domain is constrained by the end-to-end latency require-
the network such that the set of constraints are met. We ments and periods of the set of flows, while the selection of
define both frame offsets and queue assignment indexes as queues is limited by the device capacity and the necessity to
integer variables to the SMT context and generate assertions integrate other traffic classes. We have also seen that non-
(in linear arithmetic) that correspond to the constraints de- scheduled flows, may benefit from a larger number of pri-
fined in the previous sections. These frame offsets represent ority queues in the post-analysis step. On the other hand,
the open (and close events) for the timed-gate of the re- a small number of scheduled queues available to scheduled
spective queue assignment. The transformation from frame flows, reduces the solution space due to the rather stringent
offset and queue index into gate open close events is straight- isolation constraints (cf. constraints (4) and (5)). Hence,
forward, i.e., the offset represents the gate open event for the any pre-assignment of queues to traffic classes introduces a
given queue index and the gate close event is marked by the trade-off between schedulability of high-criticality flows and
duration of the respective frame. timeliness properties and flexibility for non-scheduled traffic.
When it comes to scheduling problems, the scalability The inclusion of optimization, however, opens up alter-
of the SMT approach depends on several key factors, ana- native design space exploration mechanisms. Therefore, we
lyzed in detail in [8]. However, an additional consideration, propose an optimization objective to find the minimal num-
specific to 802.1Qbv, is the interleaving of different flows ber of queues required for scheduled traffic such that a valid
in a given queue. To improve scalability, especially when solution is still feasible (i.e. schedulability is guaranteed).
scheduling large networks, Steiner introduced an incremen- This number represents a design input for the assignment
tal backtracking algorithm in [33] which we also use here in of queues to traffic classes on a per-device basis. The opti-
a modified form. Our approach attempts to schedule one mization condition is easy to express using our approach
flow at a time by adding the flow variables and constraints by having an additional variable ([va , vb ].κ) for each de-
to the SMT context and trying to solve the problem with vice representing the number of required scheduled queues
the added constraints. If a solution is found, the schedule for for high-criticality flows for each egress port. Minimizing
the flow is fixed by asserting the constant value provided by the number of used queues can be specified in different
the SMT model into the context. This repeats until either ways depending on the OMT implementation and design
the complete schedule is found (i.e. all flows are scheduled) goals. We define a global objective to minimize the ac-
or an incremental step is deemed unfeasible by the solver. In crued sumPof the number of queues used per egress port, i.e.,
the latter case, the constraints of the current flow could not minimize [va ,vb ]∈L [va , vb ].κ. Complementary, OMT solvers
be satisfied with the previous context, so the SMT context like Z3 (v4.4.1) offer the possibility to incorporate multi-
is backtracked by removing the last scheduled flow and rein- ple different objectives either optimized independently or
troducing it together with the unfeasible step in question. combined using Pareto fronts or lexicographic priorities [5].
Backtracking repeats as long as the merged step results in an Thus, additionally to the constraints defined in Section 4.1
unfeasible problem. In the worst case, the algorithm sched- and 802.1Qbv constraint (6), we add, ∀[va , vb ] ∈ L
ules all flows in one single step. However, in the average case,
there is a significant performance improvement that can be minimize [va , vb ].κ
substantial especially when network utilization is low. In subject to [va , vb ].κ ≤ [va , vb ].c,
the case of infeasiblity, we save the state of the context with [va ,vb ] [va ,vb ]
si .p ≤ [va , vb ].κ, ∀si ∈ S.
the highest number of solved flows. Hence, even if the whole
set of flows was not schedulable, we still provide a partial In this case the solution given by the solver will not only
solution for a subset of the given flows. return a valid schedule (in terms of frame offsets and queue
assignments per device) but also the minimum number of
6.1 Optimization queues per device/port which are required to fulfil the com-
SMT solvers return the first encountered valid solution, puted schedule for high-criticality flows. The remaining
if one exists, for the given context out of the set of (po- queues are then guaranteed to optimize the trade-off for
tentially) multiple solutions. Scheduling solutions, like the non-scheduled flows. As can be seen, there is no assump-
one presented here, would benefit even more if at the same tion on the value of [va , vb ].c which represents the maximum

189
 4h 4h 4h

30 min 30 min 30 min

1 min 1 min 1 min

runtime

runtime

runtime
1 sec 1 sec 1 sec

P1={10, 20}[ms] P1={10, 20}[ms] P1={10, 20}[ms]

P2={10, 25, 50, 100}[ms] P2={10, 25, 50, 100}[ms] P2={10, 25, 50, 100}[ms]
10 ms P3={5, 10, 200, 500}[ms] 10 ms P3={5, 10, 200, 500}[ms] 10 ms P3={5, 10, 200, 500}[ms]
5(40) 15(144) 25(238) 5(52) 25(310) 50(641) 10(125) 50(738) 100(1490)
Number of flows (Average number of frame instances) Number of flows (Average number of frame instances) Number of flows (Average number of frame instances)

(a) Simple topology (3 ES, 1 SW) (b) Medium topology (5 ES, 2 SW) (c) Complex topology (7 ES, 5 SW)

Figure 4: Runtime with varying number of flows and periods and different network topologies.

number of queues. We can therefore also reformulate the M1 runtime as percentage of M2 runtime

M1 over M2 reduction [%]

Runtime reduction through M1
optimization problem in terms of design space exploration,
100
i.e., given an unschedulable system, what is the minimum
number of queues in each device/port that are necessary in 80
order for the problem to become schedulable. 60
40
7. EXPERIMENTS 20
For the experimental evaluation we use the Z3 v4.4.1
0
solver (64bit) [10] which features algorithms for solving lin-

simp
simp /P1/5
simp /P2/5
simp /P3/5
simp /P1/15
simp /P2/15
simp /P3/15
simp /P1/25
simp /P2/25
med /P3/25
med /P1/5
med /P2/5
med /P3/5
med /P1/25
med /P2/25
med /P3/25
med /P1/50
med /P2/50
com /P3/50
com x/P1/1
com x/P2/1
com x/P3/1
com
com x/P2/5
com x/P3/5
com x/P1/1
com
ear optimization objectives in addition to the normal SMT

ium
ium
ium
ium
ium
ium
ium
ium
ium
ple
ple
ple
plex
ple
ple
ple
plex
plex
le
le
le
le
le
le
le
le
le
functionality [5]. We reproduced additionally several tests

/P1/5

/P2/1 0
/P3/1 0
without optimization with Yices v2.4.2 (64bit) [13] using

0
0
0
0
0
0
0
0
00
quantifier-free linear integer arithmetic (LA(Z)) as the back-
Figure 5: Flow vs. frame isolation runtime comparison.
ground theory. Since we observe similar trends as in the Z3
experiments and a comparison of SMT solvers is outside the
scope of this paper, we omit the evaluation with Yices and in significantly higher runtime of the algorithm. As is to be
concentrate on Z3. All experiments were run on a 64bit expected with NP-complete problems, the runtime increases
4-core3 3.40GHz Intel Core-i7 PC with 4GB memory. exponentially with an increase in the number of flows and
We analyze the scalability and schedulability aspects of frames that have to be scheduled. For e.g., using the com-
our methods over a number of synthetic configurations on plex topology (Figure 4(c)), we schedule 10 flows with 118
top of three predefined network topologies ranging from 3 frame instances and period configuration P 1 in under 1 sec
end-systems connected to one switch to 7 end-systems con- while scheduling 100 flows with 1394 frame instances and
nected through 5 switches (via 1Gbit/s links with a 1µsec with period configuration P 3 takes over 4 hours to solve.
macrotick granularity). We set the time-out value for a run Figure 5 shows the reduction of the runtime when using
to 5 hours. As shown in [8], a higher utilization of the net- the flow isolation approach as a percentage of the runtime
work links results in a more difficult problem set for the of the frame isolation method for each of the above inputs.
SMT solver. Hence, we keep the size of the topologies rela- Using the flow isolation method, we expect on average a re-
tive small in comparison to the number of flows (and frames) duced runtime of the scheduler at the expense of schedulabil-
in order to achieve a higher utilization on the links. ity. We replicating the above experiments using the flow iso-
In the first set of experiments (Figure 4) we show the scal- lation (constraint (4)) method, denoted by M 1 in Figure 5,
ability of the frame isolation method (using the incremental instead of the frame isolation constraint (constraint (5)), de-
backtracking implementation with step size of 1 flow) when noted by M 2. As can be seen, in these experiments the flow
varying the problem set in 3 dimensions, namely in topol- isolation method is always faster than the frame isolation
ogy size, number of flows and flow periods (chosen randomly approach. In order to confirm the trend, we have run addi-
from 3 sets of predefined periods). The configuration we use tional experiments with 381 randomly generated test cases
for the scalability tests is {Ve+s , h8, 8, 0i}, the data size for varying between the 3 topologies and period sets presented
each flow is chosen uniformly between 2 and 8 MTU-sized above and with up to 1000 flows and data sizes from 500
frames, and the senders and receivers of the flows are also up to 8000 bytes. Out of these, 17 reached the time-out
chosen randomly from the set of end-systems in the network. of 5 hours with either M 24 or with both methods and were
The logarithmic y-axis shows the runtime and the x-axis deemed infeasible. For the remaining 336 inputs, M 1 was on
the number of flows and average number of frames scheduled. average 13% faster than M 2 with a median of 8.03%. The
We observe that the period set has a significant impact on accumulated runtime of all successful runs was around 36.7
the scalability, i.e. a more complex relationship between the hours for M 1 and just over 59 hours for M 2, which amounts
periods of the flows, and hence, a higher hyperperiod, results to a 30.73% improvement.
3
Note that the current implementation of Z3 (and Yices)
4
does not take advantage of the multi-core nature of the CPU In some isolated cases, the frame isolation method timed
as it runs in a single core. out while the flow isolation found a solution within minutes.

190
 Flow isolation Optimized Flow isolation 8. RELATED WORK
Frame isolation Optimized Frame isolation
10 min Scheduling critical traffic in deterministic networks using
1 min
SMT solvers was first proposed by Steiner in [33] and ex-
tended in our previous work [7, 8] to include scheduling of

runtime
10 sec
preemptive tasks running on the end-system nodes. Schedul-
1 sec ing traffic with different criticality classes in TTEthernet
while optimizing the end-to-end delay of rate-constrained
100 ms flows has been studied in [36].
Scheduling problems, with and without optimization, for
Flow isolation Frame isolation other proprietary technologies like PROFINET, FlexRay,

queues
3
2
and TTP have been address in [17, 18, 26, 37]. At the other
1 end of the spectrum, methods for determining worst-case
end-to-end latencies for non-scheduled networks (like AFDX
T0
T0
T0
T0
T0
T0
T0
T0
T0
T1
T1
T1
T1
T1
T1
T1
T1
T1
T1
T2
1
2
3
4
5
6
7
8
9
0
1
2
3
4
5
6
7
8
9
0
or AVB) have been developed for example in [4, 12, 14].
More recent approaches [9, 28] compute AVB worst case
Figure 6: Flow vs. frame isolation w/ and w/o optimization. traversal times using network calculus. A new asynchronous
traffic class is introduced in [31] within the context of TSN
Although in some cases the flow isolation method reduces where the goal is to offer compositional analysis capabili-
the execution time significantly and may generally be prefer- ties by maintaining the traffic pattern between egress ports
able to the frame isolation approach (e.g. if schedulability along the route. This approach considers standard AVB pri-
is not likely to be an issue), it brings mostly moderate (lin- ority behaviour since it assumes that the timed gates of the
ear) improvements in runtime, especially in relation to the egress port queues are open at all times.
exponential nature of the problem. Meyer et al. [25] study the interference of time-triggered
For the schedulability tests we generate random test cases communication on AVB traffic in TSN networks when syn-
with either simple, medium or complex topologies and with chronous traffic is added to the credit-based shaper defined
3, 4 or 5 flows. The periods of the flows are chosen randomly in 802.1Qbv. The simulation framework presented in the
between two different orders of magnitude, i.e., one set of pe- paper does not address the scheduling problem but assumes
riods is {1ms, 2ms, 3ms, 5ms, 10ms}, while the others have that an already existing TT schedule is created under the
values from the set {100µsec, 150µsec, 200µsec, 500µsec}. condition that TT frames circumvent the queues of the out-
We scale the number of frames per flow accordingly, i.e. put port and are selected by the credit-based shaper with
flows with periods over 1ms have 10, 20, 40, 60 or 100 a higher priority than AVB streams. This approach as-
frames per stream and high-rate flows have either 1 or 2 sumes non-standard-compliant 802.1Qbv capabilities of de-
frames. Through this, we try to generate the scenario de- vices combining a TTEthernet-like buffered approach with
scribed in Section 4.2 in which frames of different flows the scheduled queue architecture of TSN.
have to interleave if scheduled in the same egress queue. Alderisi et al. [2] introduce a new type of traffic class with
In Figure 6 we show the runtime of the scheduler with and real-time guarantees called Scheduled Traffic (ST) which
without optimization objectives using both flow and frame corresponds to our high-criticality class. Similar to [25], the
isolation methods. As optimization objective we minimize paper considers a strict isolation of the mechanisms handling
the accrued sum of the number of queues used per egress ST and AVB traffic to ensure non-interference. As opposed
port, therefore, aiming at the lowest overall number of re- to our approach, both papers view the scheduled or time-
quired queues per egress port. From all randomly generated triggered traffic class and the AVB class as fundamentally
test cases we selected 20 for which the minimum number of dichotomic and hence the two classes need separate (and dif-
queues required was different between the flow and frame ferent) hardware mechanisms. Additionally, the authors do
isolation methods. In order to obtain a global minimum, we not address the underlying scheduling problem introduced
schedule all flows at once (i.e. without incremental steps) by the timed gates of the priority queues.
when using optimization. For the non-optimized runs we To the best of our knowledge, this is the first study which
use the incremental version of our algorithm. discusses the functional parameters of 802.1Qbv devices im-
As expected, the runtime is generally higher when opti- pacting the capabilities to control the temporal behaviour of
mizing regardless of whether we use flow or frame isolation flows and the resulting scheduling problem concerning the
without optimization. In most cases, we observe that the timed gate functionality of egress ports for high-criticality
frame isolation method requires one single queue per egress traffic.
port for any device to satisfy the schedule while the flow iso-
lation method usually needs more than one queue in several 9. CONCLUSION
ports/devices. This also means that configurations 1 or 2 In this work we addressed the scheduling problem aris-
(defined in Section 5) would not render a feasible solution ing from the IEEE 802.1Qbv extension on multi-hop fully
using the flow isolation method due to the limited amount of switched TSN networks, presenting several methods for com-
queues for high-criticality traffic flows. For some use-cases puting static schedules via Satisfiability Modulo Theories
(T 05, T 07, and T 09), the flow isolation method requires 3 (SMT). We identified key functional parameters affecting
queues for some devices, while the frame isolation remains the behaviour of 802.1Qbv communication and, based on a
at 1 queue per port for all devices. For other cases (T 08, generalized configuration of these parameters for real-time
T 13, T 17 and T 20) the frame isolation method requires a traffic, we derived constraints for creating correct offline
minimum of 2 queues for some ports (still less than the flow schedules guaranteeing low and bounded jitter as well as de-
isolation for the respective use-cases). terministic end-to-end latencies for critical communication

191
flows. Furthermore, we discussed several optimization direc- [19] Institute of Electrical and Electronics Engineers,
tions as well as concrete system configurations which open Inc. 802.1Qbv - Enhancements for Scheduled Traffic. http:
up a number of trade-offs both at runtime as well as in the //www.ieee802.org/1/pages/802.1bv.html, 2016. Draft 3.1.
design phase of the system. To demonstrate our approach, [20] Institute of Electrical and Electronics Engineers,
we have performed an evaluation in terms of scalability and Inc. Time-Sensitive Networking Task Group. http://www.
schedulability via synthetic network workloads and system ieee802.org/1/pages/tsn.html, 2016. retrieved 06-Jul-2016.
configurations. [21] Issuing Committee: As-2d2 Deterministic Ethernet
And Unified Networking. SAE AS6802 Time-Triggered
References Ethernet. http://standards.sae.org/as6802/, 2011. retrieved
20-May-2014.
[1] Abraham, E., and Kremer, G. Satisfiability checking: The-
ory and applications. In Proc. SEFM (2016), vol. 9763 of [22] Kopetz, H., and Bauer, G. The time-triggered architec-
LNCS, Springer International Publishing. ture. Proceedings of the IEEE 91, 1 (2003), 112–126.

[2] Alderisi, G., Patti, G., and Bello, L. L. Introducing [23] Kopetz, H., and Grunsteidl, G. TTP - a time-triggered
support for scheduled traffic over IEEE audio video bridging protocol for fault-tolerant real-time systems. In Proc. 23rd
networks. In Proc. ETFA (2013), IEEE Computer Society. IEEE International Symposium on Fault-Tolerant Comput-
ing (FTCS-23) (June 1993), pp. 524–533.
[3] Barrett, C., Sebastiani, R., Seshia, S., and Tinelli, C.
Satisfiability modulo theories. In Handbook of Satisfiability, [24] Li, Y., Albarghouthi, A., Kincaid, Z., Gurfinkel, A.,
vol. 185. IOS Press, 2009. and Chechik, M. Symbolic optimization with SMT solvers.
SIGPLAN Not. 49, 1 (Jan. 2014).
[4] Bauer, H., Scharbarg, J., and Fraboul, C. Improving
the worst-case delay analysis of an AFDX network using [25] Meyer, P., Steinbach, T., Korf, F., and Schmidt, T.
an optimized trajectory approach. Industrial Informatics, Extending IEEE 802.1 AVB with time-triggered scheduling:
IEEE Transactions on 6, 4 (2010). A simulation study of the coexistence of synchronous and
asynchronous traffic. In Proc. VNC (2013), IEEE Computer
[5] Bjørner, N., Phan, A., and Fleckenstein, L. νz - an Society.
optimizing SMT solver. In Proc. TACAS (2015), Springer.
[26] Pop, P., Eles, P., and Peng, Z. Schedulability-driven com-
[6] Bordoloi, U. D., Aminifar, A., Eles, P., and Peng, Z. munication synthesis for time triggered embedded systems.
Schedulability analysis of ethernet avb switches. In Proc. Real-Time Syst. 26, 3 (2004), 297–325.
RTCSA (2014), IEEE Computer Society. [27] Prytz, G. A performance analysis of EtherCAT and
[7] Craciunas, S. S., and Serna Oliver, R. SMT-based PROFINET IRT. In Proc. ETFA (2008), IEEE Computer
task- and network-level static schedule generation for time- Society.
triggered networked systems. In Proc. RTNS (2014), ACM. [28] Queck, R. Analysis of ethernet AVB for automotive net-
works using network calculus. In Proc. ICVES (2012), IEEE
[8] Craciunas, S. S., and Serna Oliver, R. Combined task-
Computer Society.
and network-level scheduling for distributed time-triggered
systems. Real-Time Systems 52, 2 (2016), 161–200. [29] Sebastiani, R. Lazy satisfiability modulo theories. JSAT 3,
3-4 (2007), 141–224.
[9] De Azua, J. A. R., and Boyer, M. Complete modelling of
AVB in network calculus framework. In Proc. RTNS (2014), [30] Sebastiani, R., and Trentin, P. OptiMathSAT: A Tool
ACM. for Optimization Modulo Theories. In Proc. CAV (2015),
vol. 9206 of LNCS, Springer.
[10] De Moura, L., and Bjørner, N. Z3: An efficient SMT
solver. In Proc. TACAS (2008), Springer-Verlag. [31] Specht, J., and Samii, S. Urgency-based scheduler for
time-sensitive switched ethernet networks. In Proc. ECRTS
[11] De Moura, L., and Bjørner, N. Satisfiability modulo the- (2016), IEEE Computer Society.
ories: Introduction and applications. Commun. ACM 54, 9
(2011), 69–77. [32] Steinbach, T., Lim, H.-T., Korf, F., Schmidt, T.,
Herrscher, D., and Wolisz, A. Tomorrow’s in-car inter-
[12] Diemer, J., Thiele, D., and Ernst, R. Formal worst-case connect? a competitive evaluation of IEEE 802.1 AVB and
timing analysis of ethernet topologies with strict-priority and Time-Triggered Ethernet (AS6802). In Proc. VTC (2012),
AVB switching. In Proc. SIES (2012), IEEE Computer So- IEEE Computer Society.
ciety.
[33] Steiner, W. An evaluation of SMT-based schedule synthe-
[13] Dutertre, B. Yices 2.2. In Proc. CAV (2014), vol. 8559 of sis for time-triggered multi-hop networks. In Proc. RTSS
Lecture Notes in Computer Science, Springer, pp. 737–744. (2010), IEEE Computer Society.

[14] Frances, F., Fraboul, C., and Grieu, J. Using network [34] Steiner, W., Bauer, G., Hall, B., and Paulitsch, M.
calculus to optimize the AFDX network. In Proc. ERTS TTEthernet: Time-Triggered Ethernet. In Time-Triggered
(2006). Communication, R. Obermaisser, Ed. CRC Press, Aug 2011.

[15] GLPK. GNU Linear Programming Kit. http://www.gnu. [35] Steiner, W., and Dutertre, B. The TTEthernet synchro-
org/software/glpk/. retrieved 20-Jul-2016. nisation protocols and their formal verification. Int. J. Crit.
Comput.-Based Syst. 4, 3 (2013).
[16] Gurobi Optimization, I. Gurobi optimizer reference man-
ual, version 6.0, 2014. retrieved 12-Jan-2015. [36] Tamas-Selicean, D., Pop, P., and Steiner, W. Synthesis
of communication schedules for TTEthernet-based mixed-
[17] Hanzalek, Z., Burget, P., and Šucha, P. Profinet IO IRT criticality systems. In Proc. CODES+ISSS (2012), ACM.
message scheduling. In Proc. ECRTS (2009), IEEE.
[37] Zeng, H., Zheng, W., Di Natale, M., Ghosal, A.,
[18] Huang, J., Blech, J. O., Raabe, A., Buckl, C., and Giusto, P., and Sangiovanni-Vincentelli, A. Schedul-
Knoll, A. Static scheduling of a time-triggered network-on- ing the flexray bus using optimization techniques. In Proc.
chip based on SMT solving. In Proc. DATE (2012), IEEE. DAC (2009), ACM.

192