6.

888:
Lecture 3
Data Center Congestion Control
Mohammad Alizadeh

Spring 2016

1
 100Kbps–100Mbps links Transport
~100ms latency inside the DC
INTERNET

Fabric

10–40Gbps links
~10–100μs latency

Servers
 Transport
inside the DC
INTERNET

Fabric

Interconnect for distributed compute workloads

data- map-
web
Servers app cache HPC monitoring
base reduce
 What’s Different About DC Transport?
Network characteristics
– Very high link speeds (Gb/s); very low latency (microseconds)

Application characteristics
– Large-scale distributed computation

Challenging traffic patterns

– Diverse mix of mice & elephants
– Incast

Cheap switches
– Single-chip shared-memory devices; shallow buffers
4
 Data Center Workloads
Mice & Elephants

Short messages
(e.g., query, coordination) Low Latency

Large flows
(e.g., data update, backup) High Throughput
 Incast
Worker 1 • Synchronized fan-in congestion

Aggregator
Worker 2

Worker 3

RTOmin = 300 ms

Worker 4 TCP timeout

 Vasudevan et al. (SIGCOMM’09)

 Incast in Bing
MLA Query Completion Time (ms)

Requests are jittered over 10ms window.

Jittering trades of median for high percentiles
Jittering switched off around 8:30 am.
7
 DC Transport Requirements

1. Low Latency
– Short messages, queries

2. High Throughput
– Continuous data updates, backups

3. High Burst Tolerance

– Incast

The challenge is to achieve these together

8
High Throughput Low Latency

Baseline fabric latency (propagation + switching): 10 microseconds

High Throughput Low Latency

Baseline fabric latency (propagation + switching): 10 microseconds

High throughput requires buffering for rate mismatches

… but this adds significant queuing latency
Data Center TCP
 TCP in the Data Center

TCP [Jacobsen et al.’88] is widely used in the data center

– More than 99% of the traffic

Operators work around TCP problems

‒ Ad-hoc, inefficient, often expensive solutions
‒ TCP is deeply ingrained in applications

Practical deployment is hard

 keep it simple!
 Review: The TCP Algorithm
Sender 1 Additive Increase:
W  W+1 per round-trip time
Multiplicative Decrease:
W  W/2 per drop or ECN mark

ECN Mark (1 bit)

Window Size (Rate)

Receiver

Time

Sender 2

ECN = Explicit Congestion Notification

 TCP Buffer Requirement

Bandwidth-delay product rule of thumb:

– A single flow needs C×RTT buffers for 100% Throughput.

B < C×RTT B ≥ C×RTT

Buffer Size

B
B
Throughput

100% 100%
 Reducing Buffer Requirements
Appenzeller et al. (SIGCOMM ‘04):
– Large # of flows: is enough.

Window Size
(Rate)

Buffer Size

Throughput
100%

15
 Reducing Buffer Requirements
Appenzeller et al. (SIGCOMM ‘04):
– Large # of flows: is enough

Can’t rely on stat-mux benefit in the DC.

– Measurements show typically only 1-2 large flows at each server

Key Observation:
Low variance in sending rate  Small buffers suffice

16
 DCTCP: Main Idea
Extract multi-bit feedback from single-bit stream of ECN marks
– Reduce window size based on fraction of marked packets.

ECN Marks TCP DCTCP

1011110111 Cut window by 50% Cut window by 40%

0000000001 Cut window by 50% Cut window by 5%

TCP DCTCP
Window Size (Bytes)
Window Size (Bytes)

Time (sec) Time (sec)

 DCTCP: Algorithm
Switch side: B Mark K Don’t
Mark
– Mark packets when Queue Length > K.

Sender side:
– Maintain running average of fraction of packets marked (α).

# of marked ACKs
each RTT : F     (1 g)  gF
Total # of ACKs

 Adaptive window decreases: 

W  (1 )W
 2
– Note: decrease factor between 1 and 2.
 DCTCP vs TCP
Experiment: 2 flows (Win 7 stack), Broadcom 1Gbps Switch

700
Buffer is mostly empty
(Packets)

600
Queue Length(KBytes)

500
400
300
200 DCTCPTCP, 2 flowsIncast by creating a
700

mitigates
Queue Length (Packets)

600
500
400

DCTCP, 2 flows
300
200 TCP

ECN Marking Thresh = 30KB

DCTCP

large buffer headroom

100

100
0
0
Time (seconds)

0
70 0
0
Que Length(Packets)

60 0
50 0

Time (seconds)
40 0
30 0
20 0 T CP , 2 f low s
DC TC P , 2 flo w s
10 0
0
0
Tim e (s ec on ds )
 Why it Works
1. Low Latency
 Small buffer occupancies → low queuing delay

2. High Throughput
 ECN averaging → smooth rate adjustments, low variance

3. High Burst Tolerance

 Large buffer headroom → bursts fit
 Aggressive marking → sources react before packets are
dropped

21
DCTCP Deployments

21
Discussion

22
 What You Said?
Austin: “The paper's performance comparison to RED
seems arbitrary, perhaps RED had traction at the time?
Or just convenient as the switches were capable of
implementing it?”

23
 Evaluation
Implemented in Windows stack.
Real hardware, 1Gbps and 10Gbps experiments
– 90 server testbed
– Broadcom Triumph 48 1G ports – 4MB shared memory
– Cisco Cat4948 48 1G ports – 16MB shared memory
– Broadcom Scorpion 24 10G ports – 4MB shared memory

Numerous micro-benchmarks
– Throughput and Queue Length – Fairness and Convergence
– Multi-hop – Incast
– Queue Buildup – Static vs Dynamic Buffer Mgmt
– Buffer Pressure

Bing cluster benchmark

Bing Benchmark (baseline)
Background Flows Query Flows

25
 Bing Benchmark (scaled 10x)
Incast
Deep buffers fix
Completion Time (ms)

incast, but increase

latency
DCTCP good for both
incast & latency

Query Traffic Short messages

(Incast bursts) (Delay-sensitive)
 What You Said
Amy: “I find it unsatisfying that the details of many
congestion control protocols (such at these) are so
complicated! ... can we create a parameter-less
congestion control protocol that is similar in behavior to
DCTCP or TIMELY?”

Hongzi: “Is there a general guideline to tune the

parameters, like alpha, beta, delta, N, T_low, T_high, in
the system?”

27
 A bit of Analysis
B K
How much buffering does DCTCP need for
100% throughput?
 Need to quantify queue size oscillations (Stability).

Packets sent in this

Window Size RTT are marked.

W*+1
W*

(W*+1)(1-α/2)

# of pkts in last RTT of Period

a=
# of pkts in Period Time
22
 A bit of Analysis
B K
How small can queues be without loss of
throughput?
 Need to quantify queue size oscillations (Stability).

for TCP:
K > (1/7) C x RTT
K > C x RTT

What assumptions does the

model make?
22
 What You Said
Anurag: “In both the papers, one of the difference I saw
from TCP was that these protocols don’t have the “slow
start” phase, where the rate grows exponentially
starting from 1 packet/RTT.”

30
 Convergence Time
DCTCP takes at most ~40% more RTTs than TCP
– “Analysis of DCTCP: Stability, Convergence, and Fairness,” SIGMETRICS 2011

Intuition: DCTCP makes smaller adjustments than TCP, but makes

them much more frequently

TCP DCTCP

31
 TIMELY

 Slides by Radhika Mittal (Berkeley)

Qualities of RTT
• Fine-grained and informative

• Quick response time

• No switch support needed

• End-to-end metric

• Works seamlessly with QoS

RTT correlates with queuing delay
 What You Said
Ravi: “The first thing that struck me while reading these
papers was how different their approaches were. DCTCP even
states that delay-based protocols are "susceptible to noise in
the very low latency environment of data centers" and that
"the accurate measurement of such small increases in
queuing delay is a daunting task". Then, I noticed that there
is a 5 year gap between these two papers… “

Arman: “They had to resort to extraordinary measures to

ensure that the timestamps accurately reflect the time at
which a packet was put on wire…”
35
Accurate RTT Measurement
Hardware Assisted RTT Measurement
Hardware Timestamps
– mitigate noise in measurements

Hardware Acknowledgements
– avoid processing overhead
Hardware vs Software Timestamps

Kernel Timestamps introduce significant noise in RTT

measurements compared to HW Timestamps.
Impact of RTT Noise

Throughput degrades with increasing noise in RTT.

Precise RTT measurement is crucial.
TIMELY Framework
Overview
Data

RTT RTT Rate Rate

Measurement Computation Pacing Engine
Engine Engine

Timestamps
Paced
Data
RTT Measurement Engine

Serialization Delay RTT

tsend tcompletion
SENDER

Propagation &
Queuing Delay

RECEIVER
HW ack

RTT = tcompletion – tsend – Serialization Delay

Algorithm Overview

Gradient-based
Increase / Decrease
Algorithm Overview

Gradient-based
Increase / Decrease

gradient = 0
RTT

Time
Algorithm Overview

Gradient-based
Increase / Decrease

gradient > 0
RTT

Time
Algorithm Overview

Gradient-based
Increase / Decrease

gradient < 0
RTT

Time
Algorithm Overview

Gradient-based
Increase / Decrease
RTT

Time
Algorithm Overview

Gradient-based
Increase / Decrease

To navigate the
throughput-latency
tradeoff and
ensure stability.
 Why Does Gradient Help Stability?
Source

e(t) =RTT (t) - RTT0

Source

e(t) + ke'(t)

Feedback higher order derivatives

Observe not only error, but change in error – “anticipate” future state
49
 What You Said
Arman: “I also think that deducing the queue length
from the gradient model could lead to miscalculations.
For example, consider an Incast scenario, where many
senders transmit simultaneously through the same
path. Noting that every packet will see a long, yet
steady, RTT, they will compute a near-zero gradient and
hence the congestion will continue.”

50
Algorithm Overview

Additive Gradient-based Multiplicativ

Increase Increase / Decrease e Decrease
Tlow Thigh

To navigate the
To keep tail
Better Burst throughput-latency
latency within
Tolerance tradeoff and
acceptable limits.
ensure stability.
Discussion

52
Implementation Set-up
TIMELY is implemented in the context of RDMA.
– RDMA write and read primitives used to invoke NIC
services.

Priority Flow Control is enabled in the network fabric.

– RDMA transport in the NIC is sensitive to packet drops.
– PFC sends out pause frames to ensure lossless network.
“Congestion Spreading” in Lossless Networks

SE

P
SE
PAU

A
U
PA

U
PA
US

SE
E
S E

PAUSE
PAUSE
U
PA PA
US
E
E

SE
PAUS

PAU
54
TIMELY vs PFC

55
TIMELY vs PFC

56
 What You Said
Amy: “I was surprised to see that TIMELY performed so
much better than DCTCP. Did the lack of an OS-bypass
for DCTCP impact performance? I wish that the authors
had offered an explanation for this result.”

57
Next time: Load Balancing

58
59