Programming Distributed

Systems
Mae Milano

Is HARD
1
2
3
4
5
6
Destructive Data Races Coordination Overhead Consistency Errors
7
We’ve built a cloud, but
we don’t know how to
program it yet.

8
 Building Programming Languages for
Distributed Systems

Destructive Data Races Coordination Overhead Consistency Errors

9
 But first…

WHAT IS A DATA WHAT WHAT ARE

RACE? COORDINATION CONSISTENCY
OVERHEAD? ERRORS?

10
 Data Race: two or more threads
simultaneously access memory, and [at
least] one of them is writing

11
 Split Increment
x = x + 1 x = x + 1

Data Race: two or more threads

simultaneously access memory, and [at
least] one of them is writing
12
 Split Increment
6x = x
5 + 1 6x = x
5 + 1

Destructive Data Race: a data race

6 which causes program errors
5 5
5
6

Data Race: two or more threads

simultaneously access memory, and [at
least] one of them is writing
13
 Transactions
x = x + 1 x = x + 1

Transaction: a unit of code that

appears to execute all at once

Runtime overhead!
Expensive for
single-machine
concurrency

14
 But first…

WHAT IS A DATA WHAT WHAT ARE

RACE? COORDINATION CONSISTENCY
OVERHEAD? ERRORS?

15
 X = 7?

[Memory] Consistency in Distributed Storage Systems

16
 X←7
X = 7? X = 7?

[Memory] Consistency in Distributed Storage Systems

17
 X←7 Hard to
The isle of Paxos
Implement
Efficiently!
write x
X←7
X=7

read x

Linearizability: Strong Consistency

18
 But first…

WHAT IS A DATA WHAT WHAT ARE

RACE? COORDINATION CONSISTENCY
OVERHEAD? ERRORS?

19
 The isle of Paxos

Linearizability: Strong Consistency

20
 (Quite Literally)

The speed of light is slow

We have outgrown Paxos
Replicas will (temporarily) diverge
21
 This is hard to
program against!

X←7
X←7 X = 15
TARDiS TARDiS TARDiS
Database Database Database

Weak Consistency
22
TARDiS: A Branch-and-Merge Approach To Weak Consistency
WHAT IS A DATA WHAT WHAT ARE
RACE? COORDINATION CONSISTENCY
OVERHEAD? ERRORS? 24
 Consider: an online game service
fine with weak needs strong
consistency consistency

RECORD WINS AND PERFORM ONLINE RUNS TOURNAMENTS

LOSSES MATCHMAKING

26
 Milano&Myers,
PLDI 2018

MixT: Safely programming

with weak consistency
Allow as much weak consistency as the application tolerates

27
 wins: player ↦ int losses: player ↦ int rank: int ↦ player

weak transaction strong transaction

match(player w, player l){ invite_to_tourney {
wins[w]++; list<player> best;
Recording losses[l]++; for (i : 1..10){
} best.add(rank[i])
wins and }
Running return best;
Tournaments }

28
 wins: player ↦ int losses: player ↦ int rank: int ↦ player

weak transaction strong transaction

match(player w, player l){ invite_to_tourney {
wins[w]++; list<player> best;
Recording losses[l]++; for (i : 1..10){
} best.add(rank[i])
wins and }
Running return best;
Tournaments }

Wins, rank
losses

Weakly-consistent storage Strongly-consistent 29storage

Composing consistency: populating rank
transaction calculate_rank {
wins: player ↦ int
list<float, player> win_loss_ratios;

for (p : all_players){
win_loss_ratios.add(wins[p]/losses[p],p); losses: player ↦ int
}

sort(win_loss_ratios);
rank: int ↦ player

rank.assign_from(table(win_loss_ratios).values);
}
(gets a sorted list of players)
30
 weak transaction strong transaction
Mixing match(player w, player l){ invite_to_tourney{…
Consistency wins[w]++; }

Across losses[l]++;
transaction
}
Transactions calculate_rank{…}

Wins, rank
losses

Weakly-consistent storage Strongly-consistent 31storage

Mixing
Consistency invite_to_tourney

Across
Transactions
calculate_rank

32
Strong outputs must stay strong
Formally: strong observations are not influenced by weak operations

33
 weak transaction strong transaction
Mixing match(player w, player l){ invite_to_tourney{…
Consistency wins[w]++; }

Across losses[l]++;
transaction
}
Transactions MixT recognizes
calculate_rank{…}

improper influence
at compile time

Wins, rank
losses

Weakly-consistent storage Strongly-consistent 34storage

Eventual Consistency (Weak)

Enforces

Linearizability (Strong)

37
A Lattice of Consistency Improper influence
guaranteed when:
a less consistent read
Eventual Consistency influences a more consistent
write
Enforces

… Causal Consistency …
Enforces

Linearizability
Viottie & Vukolić. CSUR 2016

38
We can prevent [this kind of]
improper influence via an
information-flow type system
Commonly used in security to protect
data integrity or confidentiality

39
 Composing consistency: populating rank
transaction calculate_rank { Contradiction!
list<float,
< player>
> win_loss_ratios; Two labels
inferred for rank
for (p : all_players){
win_loss_ratios.add(wins[p]/losses[p],p);
weak influence }
on label: wins/losses
assign_from sort(win_loss_ratios); as weak

rank.assign_from(table(win_loss_ratios).values);
( ( ) )
}
label: rank as
strong 40
Fixing the transaction
transaction calculate_rank {
list<float, player> win_loss_ratios;

for (p : all_players){
win_loss_ratios.add(wins[p]/losses[p]},p);
}
}
sort(win_loss_ratios);

rank.assign_from(table(win_loss_ratios).values);
}

41
 Fixing the transaction
transaction calculate_rank { safe by dynamic
list<float, player> win_loss_ratios; enforcement!
Not equivalent to
declassification
for (p : all_players){
dynamic upgrade
win_loss_ratios.add(
for read quorum_read{ wins[p]/losses[p]},p);
}
consistency
}
sort(win_loss_ratios);

rank.assign_from(table(win_loss_ratios).values);
}

42
 MixT allows
weak mail
delivery here

weak transaction strong transaction

Mixing match(player w, player l){ invite_to_tourney{…
Consistency wins[w]++; }

Across losses[l]++;
transaction
}
Transactions calculate_rank{…}

Wins, rank
losses

Weakly-consistent storage Strongly-consistent 43storage

By preventing improper influence
via information-flow, MixT yields
reasonable semantics for mixed-
consistency transactions.
A strong observer will only witness strong behavior.

44
A system, not just a language
Deep embedding compatible with
in C++ existing databases

supports multi-
~15,000 line total
consistency
implementation
transactions

Code: https://mpmilano.github.io/MixT/
Paper: https://languagesforsyste.ms/publication/mixt
45
 When are weakly-consistent
annotations warranted?
What programs are always safe
under weak consistency?

48
 Building Programming Languages for
Distributed Systems

PLDI ’18, VLDB ‘22, OOPSLA ‘22,

PLDI 2022 TOCS ’19, SoCC ’17, AiSys ‘19 CIDR ’21, SNAPL ‘19

Destructive Data Races Coordination Overhead Consistency Errors

49
 Building Programming Languages for
Distributed Systems

PLDI ’18, VLDB ‘22, OOPSLA ‘22,

PLDI 2022 TOCS ’19, SoCC ’17, AiSys ‘19 CIDR ’21, SNAPL ‘19

Destructive Data Races Coordination Overhead Consistency Errors

50
Reliable observations
for building correct
code with weak
consistency

53
Reliable Observations
• Form guarantees about distributed state
• More restricted mutations allow more general observations

Wins[p] ≥ 15

No concurrent mutations can violate this

54
 Reliable Observations
Monotonic object: Threshold observation:
mutations are inflationary comparisons with constants
with respect to some order. are stable predicates

grow monotonically

increment-only Single-writer Multi-writer

Constants grow-only sets
counters registers Registers

More Observations More Mutations55

Programming monotonically
Name is
Boaty
• If all shared objects only grow… McBoatface

• And we only observe

thresholds… Player won has
• Or stable characteristics… at least this
many matches
• Our program can be correct
under weak consistency

Bonus: avoid
coordination!

56
Can we build something useful with
monotonicity?
Yes! many core systems components are
monotonic!

57
Derecho: powered by monotonicity

STATE-MACHINE 10X FASTER PAPERS, POWERING

REPLICATION THAN TUTORIALS IN CASCADE AND
VIA ATOMIC ZOOKEEPER TOCS, EUROSYS, DERECHODDS
MULTICAST SOSP

59
 Derecho:
TOCS 2019

Monotonic:
SSS 2023

Monotonicity
powers Derecho

60
Stable message delivery

A B

61
 A shared state table (SST) Custom DSL guarantees all
programs are monotonic
Cell contains counters or sets
supports monotonic update,
monotonic functions,
threshold reads X Y Z

A
B
C
62
 Stable message delivery

A
Received
from A
Received
from B
Received
from C B
A 0 0 0

B 0 0 0

C 0 0 0

C
63
 Stable message delivery

A
Received
from A
Received
from B
Received
from C B
A 1 0 0

B 0 0 0

C 0 0 0

C
64
 Stable message delivery

A
Received
from A
Received
from B
Received
from C B
A 1 0 0

B 0 0 0

C 0 0 0

C
65
 Stable message delivery

A
Received
from A
Received
from B
Received
from C B
A 1 0 0

B 1 0 0

C 1 0 0

min(col) ≥ 1
is monotonic
Can stably deduce C
message received!
66
 Stable message delivery

A
Received
from A
Received
from B
Received
from C B
A 1 0 0

B 1 0 0

C 1 0 0

C
67
 Received Received Received Received Received Received
from A from B from C from A from B from C

A 1 0 0
A 1 0 0

B 1 0 0
B 1 0 0

AC 1 0 0
C 1 0 0 B

Received Received Received

from A from B from C

A 1 0 0

C B 1 0 0

C 1 0 0

68
 Received Received Received Received Received Received
from A from B from C from A from B from C

A 0 0 0
A 0 0 0

B 0 0 0
B 0 0 0

AC 0 0 0
C 0 0 0 B

Received Received Received

from A from B from C

A 0 0 0

C B 0 0 0

C 0 0 0

69
 Received Received Received Received Received Received
from A from B from C from A from B from C

A 1 0 0
A 0 0 0

B 0 0 0
B 0 0 0

AC 0 0 0
C 0 0 0 B

Received Received Received

from A from B from C

A 0 0 0

C B 0 0 0

C 0 0 0

70
 Received Received Received Received Received Received
from A from B from C from A from B from C

A 1 0 0
A 0 0 0

B 0 0 0
B 0 0 0

AC 0 0 0
C 0 0 0 B

Received Received Received

from A from B from C

A 0 0 0

C B 0 0 0

C 0 0 0

71
 Received Received Received Received Received Received
from A from B from C from A from B from C

A 1 0 0
A 0 0 0

B 0 0 0
B 1 0 0

AC 0 0 0
C 0 0 0 B

Received Received Received

from A from B from C

A 0 0 0
blocking until
C B 0 0 0 min(col) ≥ 1

C 1 0 0

72
 Received Received Received Received Received Received
from A from B from C from A from B from C

A 1 0 0
A 0 0 0

B 0 0 0
B 1 0 0

AC 0 0 0
C 0 0 0 B

Received Received Received

from A from B from C

A 0 0 0

C B 0 0 0

C 1 0 0

73
 Received Received Received Received Received Received
from A from B from C from A from B from C

A 1 0 0
A 1 0 0

B 1 0 0
B 1 0 0

AC 1 0 0
C 1 0 0 B

Received Received Received

from A from B from C

A 1 0 0
min(col) ≥ 1
C B 1 0 0

C 1 0 0

74
We built strongly-consistent consensus
atop the weak consistency.
Monotonicity provides convergence and
race freedom entirely for free.

75
Monotonicity builds better systems
Monotonic programs Monotonic
are correct under languages fit
weak consistency systems primitives

At the core of a best-

Used to build a new
in-class system:
consensus protocol
Derecho

Project: https://derecho-project.github.io/

77
Not everything
is monotonic!

78
 Reminder: Races
6x = x
5 + 1 6x = x
5 + 1

Transaction: a unit of code that

6 appears to execute all at once
5
5
6

Runtime overhead!
Expensive for
single-machine
concurrency

Data Race: two or more threads

simultaneously access memory, and [at
least] one of them is writing
81
 Building Programming Languages for
Distributed Systems

PLDI ’18, VLDB ‘22, OOPSLA ‘22,

PLDI 2022 TOCS ’19, SoCC ’17, AiSys ‘19 CIDR ’21, SNAPL ‘19

Destructive Data Races Coordination Overhead Consistency Errors

82
 Building Programming Languages for
Distributed Systems

PLDI ’18, VLDB ‘22, OOPSLA ‘22,

PLDI 2022 TOCS ’19, SoCC ’17, AiSys ‘19 CIDR ’21, SNAPL ‘19

Destructive Data Races Coordination Overhead Consistency Errors

83
 Milano, Turcotti, Myers:
PLDI 2022

A Flexible Type System for

Fearless Concurrency
Eliminating all destructive data races at compile time,
without complex annotations or excessive runtime
overhead

85
 Mutual exclusion:
one thread at a time
86
 Mutual exclusion:
one thread at a time
87
Challenge: safely releasing locks

Mutual exclusion:
one thread at a time
88
Challenge: safely releasing locks
Need to track references
to locked objects!

Mutual exclusion:
one thread at a time
93
Rust: allow only trees!

94
Rust: allow only trees!
At most one reference to
each locked object

Ownership: exclusive
ability to access an object
96
Hard* to write in Rust!

*without RC<RefCell>, Ghost State, unsafe, etc

97
 Implementing a doubly-linked list
struct Node<T>{ struct Node<T>{
next: Option<RC<RefCell<Node<T>>>> next,
prev: Option<RC<RefCell<Node<T>>>> prev, Node<T> next;
payload : T
}; Node<T> prev;
Adds runtime overhead
iso T payload;
}
Uses unsafe internally

98
 Challenge: path confusion
When two syntactically distinct paths might point to the same place
(impossible in trees)

102
 class ListNode<T>{
Circular Doubly-Linked List ListNode<T> next;
ListNode<T> hd = list.head; ListNode<T> prev;
ListNode<T> snd = hd.next; iso T payload;
send(thread1,hd.payload);
send(thread2,snd.payload); }
this_thread.exit(); hd snd
next next next next

seems ok…

prev prev prev prev

thread1 thread2
payload payload payload

104
 class ListNode<T>{
Circular Doubly-Linked List ListNode<T> next;
ListNode<T> hd = list.head; ListNode<T> prev;
ListNode<T> snd = hd.next; iso T payload;
send(thread1,hd.payload);
send(thread2,snd.payload); }
this_thread.exit();

next

prev

thread1 thread2
payload

105
 class ListNode<T>{
Circular Doubly-Linked List ListNode<T> next;
ListNode<T> hd = list.head; ListNode<T> prev;
ListNode<T> snd = hd.next; iso T payload;
send(thread1,hd.payload); wrongly assumed
send(thread2,snd.payload); hd and snd distinct }
this_thread.exit();
snd
next

hd

prev

thread1 thread2
payload

106
 class ListNode<T>{
Modelling path confusion in types ListNode<T> next;
ListNode<T> prev;
regions are iso T payload;
shared as a unit: }
no concurrency
hd snd
within a region* Iso keyword
next next next next
establishes
statically-enforced
boundaries
prev prev prev prev

thread1 thread2
payload payload payload

107
 class ListNode<T>{
Accessing multiple children ListNode<T> next;
if distinct(hd,snd)){ ListNode<T> prev;
send(t1,hd.payload); iso T payload;
send(t2,snd.payload); hd snd }
}

payload

110
 class ListNode<T>{
Accessing multiple children ListNode<T> next;
if distinct(hd,snd)){ ListNode<T> prev;
send(t1,hd.payload); iso T payload;
send(t2,snd.payload); hd snd }
}

payload payload

111
 class ListNode<T>{
Splitting the List ListNode<T> next;
if disconnected(hd,snd)){ ListNode<T> prev;
send(t1,hd); iso T payload;
send(t2,snd); hd snd }
}

payload

112
 class ListNode<T>{
Splitting the List ListNode<T> next;
if disconnected(hd,snd)){ ListNode<T> prev;
send(t1,hd); iso T payload;
hd snd }
send(t2,snd);
}

payload payload

113
A full-featured list library
List<T> newList(); T? pop(List<T>) -> T?

void swapTails(List<T>, List<T>); T? removeTail(List<T>);

void concatenate(List<T>, List<T> snd) void push_tail(List<T>, T elem)

consumes snd consumes elem

T? removeNth(List<T>, int); void push(List<T>, T elem)

consumes elem
void insertNth(List<T>, T elem, int)
consumes elem

114
 Fearless Concurrency Works

Provably eliminate thousands of lines,

destructive data including previously-
races impossible examples

Avoid dynamic
Proposed for Swift!
overheads

https://forums.swift.org/t/pitch-safely-sending-non-sendable-values-across-isolation-domains
115
 Building Programming Languages for
Distributed Systems

PLDI ’18, VLDB ‘22, OOPSLA ‘22,

PLDI 2022 TOCS ’19, SoCC ’17, AiSys ‘19 CIDR ’21, SNAPL ‘19

Destructive Data Races Coordination Overhead Consistency Errors

117
Thank you!

118