Parallel & Distributed Computing

Fall 2022
Dr. Zeshan Iqbal
Lecture 6B: Multicast

Multicast Problem

2
 Other Communication Forms
• Multicast à message sent to a group
of processes
• Broadcast à message sent to all
processes (anywhere)
• Unicast à message sent from one
sender process to one receiver process

Who Uses Multicast?

• A widely-used abstraction by almost all cloud systems
• Storage systems like Cassandra or a database
– Replica servers for a key: Writes/reads to the key are multicast within the replica
group
– All servers: membership information (e.g., heartbeats) is multicast across all servers
in cluster
• Online scoreboards (ESPN, French Open, FIFA World Cup)
– Multicast to group of clients interested in the scores
• Stock Exchanges
– Group is the set of broker computers
– Groups of computers for High frequency Trading
• Air traffic control system
– All controllers need to receive the same updates in the same order

4
 Multicast Ordering
• Determines the meaning of “same order” of multicast delivery
at different processes in the group
• Three popular flavors implemented
by several multicast protocols
1. FIFO ordering
2. Causal ordering
3. Total ordering

1. FIFO ordering
• Multicasts from each sender are received in the order they are
sent, at all receivers
• Don’t worry about multicasts from different senders
• More formally
– If a correct process issues (sends) multicast(g,m) to group g and then
multicast(g,m’), then every correct process that delivers m’ would
already have delivered m.

6
 FIFO Ordering: Example
P1
M1:1 M1:2
Time

P2

M3:1
P3

P4
M1:1 and M1:2 should be received in that order at each receiver
Order of delivery of M3:1 and M1:2 could be different at different receivers

2. Causal Ordering
• Multicasts whose send events are causally related, must be
received in the same causality-obeying order at all receivers
• Formally
– If multicast(g,m) à multicast(g,m’) then any correct process
that delivers m’would already have delivered m.
– (à is Lamport’s happens-before)

8
 Causal Ordering: Example
P1 M1:1
Time
M2:1
P2

M3:1 M3:2
P3

P4 M3:1 à M3:2, and so should be received in that order at each receiver

M1:1 à M3:1, and so should be received in that order at each receiver
M3:1 and M2:1 are concurrent and thus ok to be received in different orders at
different receivers

Causal vs. FIFO

• Causal Ordering => FIFO Ordering
• Why?
– If two multicasts M and M’ are sent by the same process P, and M was sent
before M’, then M à M’
– Then a multicast protocol that implements causal ordering will obey FIFO
ordering since M à M’
• Reverse is not true! FIFO ordering does not imply causal ordering.

10
 Why Causal at All?
• Group = set of your friends on a social network
• A friend sees your message m, and she posts a response (comment)
m’ to it
– If friends receive m’ before m, it wouldn’t make sense
– But if two friends post messages m” and n” concurrently, then they can be
seen in any order at receivers
• A variety of systems implement causal ordering: Social networks,
bulletin boards, comments on websites, etc.

11

3. Total Ordering
• Also known as “Atomic Broadcast”
• Unlike FIFO and causal, this does not pay attention to order of
multicast sending
• Ensures all receivers receive all multicasts in the same order
• Formally
– If a correct process P delivers message m before m’
(independent of the senders), then any other correct process P’
that delivers m’ would already have delivered m.

12
 Total Ordering: Example
P1 M1:1
Time
M2:1
P2

M3:1 M3:2
P3

P4 The order of receipt of multicasts is the same at all processes.

M1:1, then M2:1, then M3:1, then M3:2
May need to delay delivery of some messages

13

Hybrid Variants
• Since FIFO/Causal are orthogonal to Total,
can have hybrid ordering protocols too
– FIFO-total hybrid protocol satisfies both FIFO
and total orders
– Causal-total hybrid protocol satisfies both Causal
and total orders

14
 Implementation?
• That was what ordering is
• But how do we implement each of these orderings?

15

FIFO Multicast: Data Structures

• Each receiver maintains a per-sender sequence number
(integers)
– Processes P1 through PN
– Pi maintains a vector of sequence numbers Pi[1…N]
(initially all zeroes)
– Pi[j] is the latest sequence number Pi has received from
Pj

16
 FIFO Multicast: Updating Rules
• Send multicast at process Pj:
– Set Pj[j] = Pj[j] + 1
– Include new Pj[j] in multicast message as its sequence number
• Receive multicast: If Pi receives a multicast from Pj with
sequence number S in message
– if (S == Pi[j] + 1) then
• deliver message to application
• Set Pi[j] = Pi[j] + 1
– else buffer this multicast until above condition is true

17

FIFO Ordering: Example

P1
[0,0,0,0] Time

P2
[0,0,0,0]

P3
[0,0,0,0]

P4
[0,0,0,0]

18
 [1,0,0,0]
P1
[0,0,0,0] P1, seq: 1 Time

P2
[0,0,0,0] [1,0,0,0]
Deliver!
P3
[0,0,0,0] ?

P4
[1,0,0,0]
[0,0,0,0]
Deliver!
FIFO Ordering: Example

19

[1,0,0,0] [2,0,0,0]
P1
[0,0,0,0] P1, seq: 1 P1, seq: 2 Time

P2
[0,0,0,0] [1,0,0,0]
Deliver!
P3
[0,0,0,0] [0,0,0,0]
Buffer!
P4 [1,0,0,0]
[1,0,0,0]
[0,0,0,0] Deliver this!
Deliver!
Deliver buffered <P1, seq:2>
FIFO Ordering: Example Update [2,0,0,0]

20
 [1,0,0,0] [2,0,0,0]
P1
[0,0,0,0] P1, seq: 1 P1, seq: 2 [2,0,0,0] Time
Deliver!
P2
[0,0,0,0] [1,0,0,0]
Deliver!
P3
[0,0,0,0] [0,0,0,0]
Buffer!
P4 [1,0,0,0]
[1,0,0,0]
[0,0,0,0] Deliver this!
Deliver!
Deliver buffered <P1, seq:2>
FIFO Ordering: Example Update [2,0,0,0]

21

[1,0,0,0] [2,0,0,0] [2,0,1,0]

P1 Deliver!
[0,0,0,0] P1, seq: 1 P1, seq: 2 [2,0,0,0] Time
Deliver! [2,0,1,0]
P2
[0,0,0,0] [1,0,0,0] Deliver!
Deliver!
P3, seq: 1
P3
[0,0,0,0] [0,0,0,0] [2,0,1,0]
Buffer! ?
P4 [1,0,0,0]
[1,0,0,0]
[0,0,0,0] Deliver this!
Deliver!
Deliver buffered <P1, seq:2>
FIFO Ordering: Example Update [2,0,0,0]

22
 [1,0,0,0] [2,0,0,0] [2,0,1,0]
P1 Deliver!
[0,0,0,0] P1, seq: 1 P1, seq: 2 [2,0,0,0] Time
Deliver! [2,0,1,0]
P2 Deliver!
[0,0,0,0] [1,0,0,0]
Deliver! [1,0,1,0]
P3 P3, seq: 1 Deliver!
[0,0,0,0] [0,0,0,0] [2,0,1,0]
[2,0,1,0]
Buffer!
Deliver!
P4 [1,0,0,0]
[1,0,0,0]
[0,0,0,0] Deliver this!
Deliver!
Deliver buffered <P1, seq:2>
FIFO Ordering: Example Update [2,0,0,0]

23

Causal Ordering
• Multicasts whose send events are causally related, must be
received in the same causality-obeying order at all
receivers
• Formally
– If multicast(g,m) à multicast(g,m’) then any correct
process that delivers m’ would already have delivered m.
– (à is Lamport’s happens-before)

24
 Causal Multicast: Datastructures
• Each receiver maintains a vector of per-sender sequence
numbers (integers)
– Similar to FIFO Multicast, but updating rules are different
– Processes P1 through PN
– Pi maintains a vector Pi[1…N] (initially all zeroes)
– Pi[j] is the latest sequence number Pi has received from Pj

25

Causal Multicast: Updating Rules

• Send multicast at process Pj:
– Set Pj[j] = Pj[j] + 1
– Include new entire vector Pj[1…N] in multicast message as its sequence number
• Receive multicast: If Pi receives a multicast from Pj with vector
M[1…N] (= Pj[1…N]) in message, buffer it until both:
1. This message is the next one Pi is expecting from Pj, i.e.,
• M[j] = Pi[j] + 1
2. All multicasts, anywhere in the group, which happened-before M have been
received at Pi, i.e.,
• For all k ≠ j: M[k] ≤ Pi[k]
• i.e., Receiver satisfies causality
3. When above two conditions satisfied, deliver M to application and set Pi[j] = M[j]

26
 [1,0,0,0]
P1
[0,0,0,0] [1,0,0,0] Time

P2
[0,0,0,0]

P3
[0,0,0,0]

P4
[0,0,0,0]
Causal Ordering: Example

27

[1,0,0,0]
P1
[0,0,0,0] Time
[1,1,0,0]
P2
[0,0,0,0] [1,0,0,0]
Deliver!
P3
[0,0,0,0]

P4
[0,0,0,0] [1,0,0,0]
Deliver!
Causal Ordering: Example

28
 [1,0,0,0] [1,1,0,0]
P1 Deliver!
[0,0,0,0] Time
[1,1,0,0]
P2
[0,0,0,0] [1,0,0,0]
Deliver!
P3
[0,0,0,0] Missing 1 from P1
Buffer!
P4
[0,0,0,0] [1,0,0,0]
Deliver!
Causal Ordering: Example

29

[1,1,0,0] Deliver!
[1,0,0,0] Deliver! Receiver satisfies causality
P1
[0,0,0,0] Deliver! Time
[1,1,0,0] Receiver satisfies causality
P2
[0,0,0,0] [1,0,0,0]
Deliver!
P3
[0,0,0,0] Missing 1 from P1
Buffer!
P4
[1,0,0,0] [1,0,0,1]
[0,0,0,0]
Deliver!
Causal Ordering: Example

30
 [1,1,0,0] Deliver!
[1,0,0,0] Deliver! Receiver satisfies causality
P1
[0,0,0,0] Deliver! Time
[1,1,0,0] Receiver satisfies causality
P2
[0,0,0,0] [1,0,0,0]
Deliver!
P3
[0,0,0,0] Missing 1 from P1 Missing 1 from P1
Buffer! Buffer!
P4
[1,0,0,0] [1,0,0,1]
[0,0,0,0]
Deliver!
Causal Ordering: Example

31

[1,1,0,0] Deliver!
[1,0,0,0] Deliver! Receiver satisfies causality
P1
[0,0,0,0] Deliver! Time
[1,1,0,0] Receiver satisfies causality
P2
[0,0,0,0] [1,0,0,0]
Deliver!
P3
[0,0,0,0] Missing 1 from P1 Missing 1 from P1
Buffer! Buffer!
P4
[1,0,0,0] [1,0,0,1] Deliver P1’s multicast
[0,0,0,0] Receiver satisfies causality for buffered multicasts
Deliver! Deliver P2’s buffered multicast
Causal Ordering: Example Deliver P4’s buffered multicast

32
 [1,1,0,0] Deliver!
[1,0,0,0] Deliver! Receiver satisfies causality
P1
[0,0,0,0] Deliver! Time
[1,1,0,0] Receiver satisfies causality
P2
[0,0,0,0] [1,0,0,0]
Deliver! Deliver!
P3
[0,0,0,0] Missing 1 from P1 Missing 1 from P1
Buffer! Buffer!
P4
[1,0,0,0] [1,0,0,1] Deliver P1’s multicast
[0,0,0,0] Receiver satisfies causality for buffered multicasts
Deliver! Deliver P2’s buffered multicast
Causal Ordering: Example
Deliver P4’s buffered multicast

33

Total Ordering
• Ensures all receivers receive all multicasts in the
same order
• Formally
– If a correct process P delivers message m before
m’ (independent of the senders), then any other
correct process P’ that delivers m’ would already
have delivered m.

34
 Sequencer-based Approach
• Special process elected as leader or sequencer
• Send multicast at process Pi:
– Send multicast message M to group and sequencer
• Sequencer:
– Maintains a global sequence number S (initially 0)
– When it receives a multicast message M, it sets S = S + 1, and multicasts
<M, S>
• Receive multicast at process Pi:
– Pi maintains a local received global sequence number Si (initially 0)
– If Pi receives a multicast M from Pj, it buffers it until it both
1. Pi receives <M, S(M)> from sequencer, and
2. Si + 1 = S(M)
• Then deliver it message to application and set Si = Si + 1

35

Summary: Multicast Ordering

• Ordering of multicasts affects correctness

of distributed systems using multicasts
• Three popular ways of implementing
ordering
– FIFO, Causal, Total
• And their implementations
• What about reliability of multicasts?
• What about failures?

36
 Parallel & Distributed Systems
Fall 2021
Dr. Zeshan Iqbal
Lecture 6: Reliable Multicast and Virtual
Synchrony

37

Reliable Multicast

• Reliable multicast loosely says that every process

in the group receives all multicasts
– Reliability is orthogonal to ordering
– Can implement Reliable-FIFO, or Reliable-Causal, or
Reliable-Total, or Reliable-Hybrid protocols
• What about process failures?
• Definition becomes vague

38
 Reliable Multicast (under failures)

• Need all correct (i.e., non-faulty) processes

to receive the same set of multicasts as all
other correct processes
– Faulty processes stop anyway, so we won’t
worry about them

39

Implementing Reliable Multicast

• Let’s assume we have reliable unicast (e.g.,
TCP) available to us
• First-cut: Sender process (of each multicast M)
sequentially sends a reliable unicast message to all
group recipients
• First-cut protocol does not satisfy reliability
– If sender fails, some correct processes might receive
multicast M, while other correct processes might not
receive M

40
 REALLY Implementing Reliable Multicast

• Trick: Have receivers help the sender

– Sender process (of each multicast M)
sequentially sends a reliable unicast
message to all group recipients
– When a receiver receives multicast M, it
also sequentially sends M to all the
group’s processes

41

Virtual Synchrony or View Synchrony

• Attempts to preserve multicast ordering and
reliability in spite of failures
• Combines a membership protocol with a
multicast protocol
• Systems that implemented it (like Isis) have
been used in NYSE, French Air Traffic
Control System, Swiss Stock Exchange

42
 Views
• Each process maintains a membership list
• The membership list is called a View
• An update to the membership list is called a View Change
– Process join, leave, or failure
• Virtual synchrony guarantees that all view changes are delivered
in the same order at all correct processes
– If a correct P1 process receives views, say {P1}, {P1, P2, P3}, {P1, P2}, {P1,
P2, P4} then
– Any other correct process receives the same sequence of view changes (after
it joins the group)
• P2 receives views {P1, P2, P3}, {P1, P2}, {P1, P2, P4}
• Views may be delivered at different physical times at
processes, but they are delivered in the same order

43

VSync Multicasts
• A multicast M is said to be “delivered in a view V at process Pi”
if
– Pi receives view V, and then sometime before Pi receives the next view it
delivers multicast M
• Virtual synchrony ensures that
1. The set of multicasts delivered in a given view is the same set at all
correct processes that were in that view
• What happens in a View, stays in that View
2. The sender of the multicast message also belongs to that view
3. If a process Pi does not deliver a multicast M in view V while other
processes in the view V delivered M in V, then Pi will be forcibly removed
from the next view delivered after V at the other processes

44
 View{P1,P2,P3,P4} View{P1,P2,P3}
P1
M1
Time
View{P1,P2,P3,P4} M2 View{P1,P2,P3}
P2

View{P1,P2,P3,P4} View{P1,P2,P3}
P3

View{P1,P2,P3,P4} M3
P4
Crash
Satisfies virtual synchrony

45

View{P1,P2,P3,P4} View{P1,P2,P3}
P1
M1
Time
View{P1,P2,P3,P4} M2 View{P1,P2,P3}
P2

View{P1,P2,P3,P4} View{P1,P2,P3}
P3

View{P1,P2,P3,P4} M3
P4
Crash
Does not satisfy virtual synchrony

46
 View{P1,P2,P3,P4} View{P1,P2}
P1
M1
Time
View{P1,P2,P3,P4} M2 View{P1,P2}
P2

View{P1,P2,P3,P4}
P3

View{P1,P2,P3,P4} M3
P4
Crash
Satisfies virtual synchrony

47

View{P1,P2,P3,P4} View{P1,P2,P3}
P1
M1
Time
View{P1,P2,P3,P4} M2 View{P1,P2,P3}
P2

View{P1,P2,P3,P4} View{P1,P2,P3}
P3

View{P1,P2,P3,P4} M3
P4
Crash
Does not satisfy virtual synchrony

48
 View{P1,P2,P3,P4} View{P1,P2,P3}
P1
M1
Time
View{P1,P2,P3,P4} M2 (not delivered at P2) View{P1,P2,P3}
P2

View{P1,P2,P3,P4} View{P1,P2,P3}
P3

View{P1,P2,P3,P4} M3
P4
Crash
Satisfies virtual synchrony

49

View{P1,P2,P3,P4} View{P1,P2,P3}
P1
M1
Time
View{P1,P2,P3,P4} M2 View{P1,P2,P3}
P2

View{P1,P2,P3,P4} View{P1,P2,P3}
P3

View{P1,P2,P3,P4} M3
P4
Crash
Does not satisfy virtual synchrony

50
 View{P1,P2,P3,P4} View{P1,P2,P3}
P1
M1
Time
View{P1,P2,P3,P4} M2 View{P1,P2,P3}
P2

View{P1,P2,P3,P4} View{P1,P2,P3}
P3

View{P1,P2,P3,P4} M3
P4
Crash
Does not satisfy virtual synchrony

51

View{P1,P2,P3,P4} View{P1,P2,P3}
P1
M1
Time
View{P1,P2,P3,P4} M2 View{P1,P2,P3}
P2

View{P1,P2,P3,P4} View{P1,P2,P3}
P3

View{P1,P2,P3,P4} M3
P4
Crash
Satisfies virtual synchrony

52
 What about Multicast Ordering?
• Again, orthogonal to virtual synchrony
• The set of multicasts delivered in a view can be ordered
either
– FIFO
– Or Causally
– Or Totally
– Or using a hybrid scheme

53

About that name

• Called “virtual synchrony” since in spite of running on an
asynchronous network, it gives the appearance of a
synchronous network underneath that obeys the same
ordering at all processes
• So can this virtually synchronous system be used to
implement consensus?
• No! VSync groups susceptible to partitioning
– E.g., due to inaccurate failure detections

54
 View{P1,P2,P3,P4} View{P1}
P1
M1
Time
View{P1,P2,P3,P4} M2 View{P2, P3}
P2

View{P1,P2,P3,P4} View{P2,P3}
P3

View{P1,P2,P3,P4} M3
P4
Crash
Partitioning in View synchronous systems

55

Summary

• Multicast an important building block for cloud

computing systems
• Depending on application need, can implement
– Ordering
– Reliability
– Virtual synchrony

56