10/13/2008

Real-Time Networks and A Distributed Real-Time System

Distributed Systems
 Topics
 Distributed Real-Time Systems
 Bus-based multi-processor systems
 Real Time Networks
 RT busses e.g. CAN, TTP, TTCAN
 Analysis of Distributed RT Systems
 Message Transmission Analysis
 Response Time Analysis

1 2

Why Distributed Systems ? Dsign of Distributed RTS

 Physically distributed applications - Tasks
(close to physical equipment, e.g. engine control) - Execution time Running
 Modularity (components developed in isolation)
Kernel
- Period system
 Scalability (just add another node in the network) - Deadlines
- Dependences
allocation/


Som e new cars contain > 3 m iles of wire
Clearly inappropriate to connect all pairs of communicating entities w ith their own w ires
scheduling
 Which needs O(n*n) w ires

 Fault tolerance (errors only propagate within sub-part of Functional

system) Design
Challenge
 build complex distributed systems and maintain high reliability
3 aspects: The core for the development

at low cost! - off-line allocation of real time systems

- run-time scheduling
- á priori schedulability analysis
3 4

RT-Networking: Basic Problem RT-Networking: Solutions

Bounded latency: A B

CSMA/CD Token-ring
 Competing traffic (Carrier Sense M ultiple Access / Collision Detection)
Physical or logical ring
 Guarantees Ethernet
Collisions Circulating token
 Hard-RT: Absolute G.
 Soft-RT: Probabilistic G. back-off No collisions
RT-guarantees possible
 Other issues Stochastic behaviour
 Reliability No RT-guarantees
 F. detection & recovery D
 Resource Utilisation
B A B

C
A

C D
5 6

1
 10/13/2008

RT-Networking: Solutions More examples

 Time triggered:
 SAFEbus - airplanes, eg. Boeing
 TTA - cars, eg. Audi, Volkswagen
 FlexRay - cars, eg. BMW, DaimlerChrysler
Time triggered Event triggered  Event triggered
TDMA CSMA/CR (Collision Resolution)  CAN - cars, eg. Volvo, Saab, VW, Ford, GM
(Time Division Multiple Access,e.g. GSM ) (Carrier Sense Multiple Access / Collision Resolution)  Byteflight - cars, BMW
Pre-Scheduled Priority driven  LIN – a cheaper and simple bus protocol
Predictable Dynamic Scheduling
Testable Flexible  Mixtures
Static e.g. CAN  Time-Triggered CAN (TTCAN)
Pri: A>B>C>D
e.g. TTP  TTA extended with events

0 T time 0 time
Response time for C
7 8

CAN: Controller Area Network CAN is Synchronous

 Initiated in the late 70’s to connect a number of processors over a cheaper shared serial
bus  Fundamental requirement: Everyone on the bus sees the current
 From Bosch (mid 80ies) for automotive applications bit before the next bit is sent
 De facto standard for  This is going to permit a very clever arbitration scheme (later)
twisted-pair,
invehicle comm.  Ethernet does NOT have this requirement This is one reason Ethernet
(100 million CAN nodes sold 2000, 300 million sold 2004) optic fibre or coax
bandwidth can be much higher than CAN
 Cost ~$3 / node
 $1 for CAN interface  Time per bit:
 $1 for the transceiver
 Speed of electrical signal propagation 0.1-0.2 m/ns
 $1 for connectors and additional board area
 40 Kbps CAN bus → 25000 ns per bit
 Controllers available
(from Phillips, Intel, NEC, Siemens, etc.)  A bit can travel 2500 m (max bus length 1000~1250 m)
 Shared broadcast bus (one sender/many receivers) (CSMA/CR)  1 Mbps CAN bus → 1000 ns per bit
 CAN permits ev eryone on the bus to talk  A bit can travel 100 m (max bus length 40~50 m)
 Medium speed:  Bandwidth
 Max: 1Mbit/sec; typically used from 35 Kbit/sec up to 500Kbit/sec
 Highly robust (error mechanisms to overcome disturbance on the bus) and  1 Mbps up to 40~50 m
 0.5 Mbps upto 80~100 m
 Real-time guarantees can be made about CAN performanc
 40 Kbps up to ~1000 m
 5 Kbps up to ~10,000 m

9 10

More on CAN CAN Addressing

 Message based with payload size 0-8 bytes  CAN bus can have an arbitrary number of nodes
 Not for bulk data transfer!  Nodes do not have proper addresses
 But perfect for many embedded control applications  Rather, each message has an 11-bit “field identifier”
 In extended mode, identifiers are 29 bits
 Everyone interested in a message type listens for it
 CAN interfaces are usually pretty smart  Works like this: “I’m sending a temperature sensor reading”
 Interrupt only after an entire message is received  Not like this: “I’m sending a message to node 8”
 Filter out unwanted messages in HW – zero CPU load
 Designer should allocate the message identifiers to the
stations (different nodes send different messages!)

 Each node has a queue for messages ordered by

priorities/identifiers

11 12

2
 10/13/2008

CAN Message Types Controller Area Network (CAN)

Frame layout:
 Data frame:
 Frame containing data for transmission

 Remote frame:
 Frame requesting the transmission of a specific identifier

 Error frame:
 Frame transmitted by any node detecting an error

 Overload frame:  Small sized frames (messages)

 Frame to inject a delay between data and/or remote frames if a receiver is  0 to 8 bytes
not ready  Very different from mainstream computing messaging
 Relatively high overhead
 A frame size of more than 100 bits to send just 64 bits

CAN-frame id control data

11 bits 36 bits 0-8 bytes
13 14

Controller Area Network (CAN)

Node 1
The CAN Arbitration Mechanism
Host CAN Vcc +5V
Microcontroller Transceiver
Line Voltage

3.5 V

 Shared broadcast bus

CAN_L
TxD Vdiff Vdiff
Driver

CAN_H

 Bus behaves like a large AND-gate

1.5 V
Vcc +5V

Recessive Dominant Recessive

RxD Logic "1" Logic "0" Logic "1"
- if all nodes sends 1 the bus becomes 1, otherwise 0.
Receiver

Node
2
Node
3
Node
n  A frame is tagged by an identifier
120 R 120 R
 indicates contents of frame D
 also used for arbitration as ”priority” B

 Bit-wise arbitration A C

Each message has unique priority 

Arbitration Field Control Field Data Field ACK
Delimiter

node with message with lowest id wins arbitration
SOF ID part 1 RTR IDE r0 DLC DATA (0-8 by tes) CRC code EOF
 Lowest id = highest priority!
Max. 64 data

 The CAN bus is a priority-based scheduled resource

1 11 1 1 1 4 15 111 7
bits

CAN data frame

15 16

Details on CAN The CAN Arbitration Mechanism

Vill sända
Send ram?
a frame
 When the bus is busy, the stations wait (listening all time) D
 As soon as the bus is idle, all stations who want to send B
enter the arbitration phase (run the arbitration algorithm) Nej A C
Bussen
Bus node A wins
 Transmit the highest priority message, from the most significant ledig?
bit to the least significant one
free? Example: arbitration
Ja
 0 is the highest priority!!
 0: dominant bit (in fact, sending 0 by ”high voltage”)
Node A B C D
 1: recessieve bit Send ut
Lägg
id-bit
ID
bit 00
Priority 001 010 101 011
 It behaves like an AND-gate
 Send and monitor: Läs värdet
Read data on
 Send a 1, but monitor a 0: a collision på bussen Nej
bus
 the protocol says: nodes sending 0’s win, the others back off (monitor and
send) Nej Ja Send Läs värdet Ja Sista
The same
Samma som Läggnext
ut Read data on The same
Samma som The last
 This means: the highest priority message wins, to be transmitted bit bit
nästa på bussen
bus biten?
utlagt?
as sent? utlagt?
as sent? bit?
 E.g. 100, 101, 111 on three stations, 100 will be sent
Nej Ja

Send theresten
Skicka rest of
av ramen
the frame
17 18

3
 10/13/2008

“Idiot” Node Error handling

 What happens if a CAN node goes crazy/haywire  Several types of errors:

and transmits too many high priority frames?  Checksum error, acknowledge error,
 This can make the bus useless bit error, ...
 Assumed not to happen

 Schemes for protecting against this have been  When error is detected by node it sends an
developed but are not commonly deployed error frame
 Most likely this happens very rarely
 starting with 6 dominant bits (000000) in a row
 CAN bus is usually managed by hardware
 tells other nodes that error occurred
 other nodes then also send error frames
 Arbitration restarts when bus is idle

 In effect, error frames are used to resync

protocol engine
19 20

Transmission Errors Details on CAN

 CAN has a mechanism to protect against broken  After the priority transmitted (the arbitration is finished), the
rest of the message is transmitted
hardware: error counters
 A message contains: 0-8 bytes for data and 47bits OH
 The CAN controller in a node counts failed frames and  Priority/identity: 11bits
successful frames  Data field: 0-8 bytes long
 When errors exceed a threshold, the controller gets disconnected  CRC field (checksum, parity bits etc: checking the message has
 ERROR-counter EC not been corrupted, and other ”housekeeping” bits)
 EC:= EC+1 when an error is signalled  Out of the 47, 34 bits are bitstuffed

 EC:= EC-1 when a frame is correctly received  000000 and 111111 are reserved as ”marker” to signal all
 EC > K  the node shuts-off itself (is fail-silent) stations on the bus
 So ”bitstuffing” is needed: whenever 00000 or 11111 appears in a
bitstream, an extra bit of the opposite sign should be added
 E.g. 1111 1000 0111 1000 0111 1 should be
1111 1000 0011 1110 0000 1111 10

21 22

More details on CAN CAN Message Scheduling

 The total number before bitstuffing: 8n+47  Network scheduling is usually non-preemptive
 Non-preemptive scheduling means high-priority sender must wait while
 After bitstuffing: 8n+47+(34+8n-1)/4 
low-priority sends
Short message length keeps this delay small
 Max: 64+47+24=135 bits
 E.g. 1Mbit/sec, 1 bit needs 1 micro seconds
 The max transmission time for one message= 135
micro sec

23 24

4
 10/13/2008

Transmission Delay
A

The CAN-bus Abstraction B

Calculation for CAN

C
D
Response time
Frames
queued in
priority order
Set of messages = M (queued on different nodes)
D
B

A C
Mj = < Tj, Cj > (Mj  M )
Tj = period (time betw een queuing)
Cj = transmission time
Bj = blocking time (waiting for low priority message, bus non-preemptive)
The whole bus + CAN
controllers can be
abstracted as one queue
A
Worst-case waiting/queuing time (before transmission ):

qi = Bi +S j hp(i)  qi/Tj Cj
B
Removed after
C transmission time
hp(i) = frames with priority higher than Pi
Frame in
D Priority queue
transmission Worst-case Response time (delay before delivered) :

Response time Ri = Ci+qi

25 26

Transmission delay analysis End of Story?

 Ci = (number of bits) X (time to transmit 1 bit)

D
B

 Bi = MAX
A C
k  lp(i) (Ci) <= time to transmit 135 bits

 Unfortunately not!
 Worst case: Bi = Ci= 135 micro sec  Non-periodic queuing times causes jitter
(for 1MB/sec CAN)  No global time reference
 Transmission errors (recovery + retransmission)

27 28

Queuing causes jitter Adding Jitter to the Analysis

D
B
C
A New equation for worst-case Transmission Delay:
task_1 RTOS
 Task_3 on node A executes with certain
task_2 period
task_3
task_4  Message mAtoB gets same period as R i = J i +q i + C i
task_3() { task_3 qi = Bi +S j hp(i) (qi+Jj)/TjCj
 Shortest time before send:
while(1) {
read_sensor();
if(...)
// do some work
BCET = C3 min for task_3
else
// do some other work
 Longest time before send: task_3’s worst
send_CAN(m AtoB, prio); case response time = R3
// some more work
sleep_until_next_period();  R3 - C3 min= jitter for message mAtoB
}

29 30

5
 10/13/2008

Analysis of Distr.
Transmission Errors
Systems
 Max number of errors must be bounded
D
B
C
 Fault hypothesis 
A

 Error function E(t) = max time required for error Detect obstacle Initial Send msg Calculate Send msg Inflate
signalling and recovery in any time interval of length t (read sensor) processing on bus action on bus airbag
time

 System wide (end-to-end) timing requirements

 control closed over the entire system
New equation for worst-case transmission delay:  includes sensors, CPUs, controllers, busses, actuators, OS, ...
Ri = J i + q i + Ci  Holistic analysis can be applied!

qi = Bi + E(qi) +S j hp(i) (qi+Jj)/TjCj

31 32

Holistic Scheduling Problem Distributed Systems

 When tasks on a node can both send and receive  Tasks on CPUs are exchanging msgs over CAN
messages we have a holistic scheduling problem
 Tasks are queuing messages
 The equations giving the worst case time for
 Completion times will vary =>
tasks depends on messages arriving at the node
 Jitter (variations in release times) will be inherited
 We cannot apply the processor
scheduling analysis before we send(i)
0 CPU
dest(i)
 Message m(i), queued by a task send(i):
act(i)
get values from the bus  Jm(i)= Rsend(i) – Csend(i)
scheduling analysis m sg(i)

 Task dest(i) is activated by a

CAN
 Similarly: We cannot apply the m sg(j)

bus scheduling analysis before we get values message m(i): 0

send(i)
CPU CPU
dest(i)

from the processor scheduling analysis  Jdest(i) = Rm(i) - Cm(i) m sg(i)

 Solution: Holistic Analysis

CAN

33 34

Distributed Systems
0
send(i) dest(i)
CPU

 Example: m sg(i)

Rsend(i) = Csend(i)+Sj hp(send(i))  (Rsend(i) + Jj)/T j Cj

CAN
Node A: 

 Jm(i)= Rsend(i) - Csend(i)

CAN:  Rm(i) = qm(i) + Jm(i) + Cm(i) Problem with CAN: some of the message
 qm(i) = Bm(i) +S jhp(m(i)) (qm(i) + Jm(j)) /Tm(j) Cm(j) may never get a chance for transmission
Node B:  Jdest(i) = Rm(i) - Cm(i)
 Rdest(i) = wdest(i)+ Jdest(i)
 wdest(i) = Cdest(i)+Sjhp(dest(i)) (wdest(i) + Jj)/Tj  Cj

Cm(i) = Bm(i) = 135 micro sec

35 36

6
 10/13/2008

Event Driven
Peak Load Deadline for
message 9 Other Solutions:
e.g.TTP - the Time Triggered Protocol
Network Message Queue

9 9 9 9
9 8 9
9 9 2 5 8 9
3 6 1 2 5 8 9  Intended for X-by-wire applications
3 6 1 2 5 8 9
CAN
BUS Example: Break-by-wire in car
Time  A lot of features built in into the bus protocol
Message Ready

9
(which must be added on top of the CAN bus)

 Conceptually similar to
3 6 2 1 5 8

Time static cyclic scheduling

2 9

netw ork
8
3 6

Engin Gear-Shift Gear-Box ABS ABS ABS ABS

37 38
e

TTCAN: an example of TTP

TTP - Time Triggered (TDMA)

Node 1 Node 2 Node 3 Node 4

Node 1 Node 2 Node 3 Node 4 Master

All nodes
has identical
message tables Transmission
Column 0 1 2 3 0 1 2 3 0
TDMA round TDMA round

REF1 MSG1 MSG2 MSG3 REF1 MSG MSG5 REF1

4

Basic Cycle (n) Basic Cycle

(n+1) Time
TDMA - slot reserved fram e Nod 1 does not Delay of m essage
for node 1 use its slot. sending due to
synchronization
39 40

TTP - CAN: a comparison Trends for RT networks in Automotives

TTP CAN  Today CAN dominates

 Event triggered
 Time triggered  Time-triggered seems to be the future for X-by-wire: TTP
 No message sending
 Overallocation of
if not neccessary e.g. FlexRay, TTCAN
aperiodic messages
 Jitter due to varying  Future cars will include many different and parallel buses:
 No jitter
system loads  CAN for comfort
 Ultra-reliable systems
 Priority driven  TT for X-by-wire
 Includes distributed  MOST (Media Oriented Systems Transport) for multimedia
 RT-Network
system functionality  etc.
 Clock-synchronization  Some functionality
 Fault-handling added on top
 Membership protocol
 Capacity 10 Mb/sec  Capacity: 1Mb/sec

41 42