Scribd
Upload
49 views40 pages

Multi-Resource Fair Queueing For Packet Processing:, Vyas Sekar, Matei Zaharia, Ion Stoica

The document proposes Dominant Resource Fair Queueing (DRFQ), a generalization of fair queueing to schedule packets that consume multiple resources. DRFQ provides allocations in time that achieve the Dominant Resource Fairness (DRF) allocation properties of equalizing dominant shares across flows. DRFQ extends the concept of virtual time to multiple resources to provide both memoryless scheduling and dovetailing of resource consumption over packets in a flow. The paper analyzes why existing single-resource policies violate strategy-proofness and share guarantees in multi-resource environments, and evaluates an implementation of DRFQ in Click.

Uploaded by

bhaskar.jain20021814
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
49 views40 pages

Multi-Resource Fair Queueing For Packet Processing:, Vyas Sekar, Matei Zaharia, Ion Stoica

The document proposes Dominant Resource Fair Queueing (DRFQ), a generalization of fair queueing to schedule packets that consume multiple resources. DRFQ provides allocations in time that achieve the Dominant Resource Fairness (DRF) allocation properties of equalizing dominant shares across flows. DRFQ extends the concept of virtual time to multiple resources to provide both memoryless scheduling and dovetailing of resource consumption over packets in a flow. The paper analyzes why existing single-resource policies violate strategy-proofness and share guarantees in multi-resource environments, and evaluates an implementation of DRFQ in Click.

Uploaded by

bhaskar.jain20021814
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 40

DRFQ: Multi-Resource Fair Queueing for Packet Processing

Ali Ghodsi1,3, Vyas Sekar2, Matei Zaharia1, Ion Stoica1


1UC Berkeley, 2Intel ISTC/Stony

Brook, 3KTH

Increasing Network Complexity


Packet processing becoming evermore sophisticated
Software Defined Networking (SDN) Middleboxes Software Routers (e.g. RouteBricks) Hardware Acceleration (e.g. SSLShader)

Data plane no longer merely forwarding


WAN optimization Caching IDS VPN
2

Motivation
Flows increasingly have heterogeneous resource consumption
Intrusion detection bottlenecking on CPU Small packets bottleneck memory-bandwidth Unprocessed large packets bottleneck on link bw

Scheduling based on a single resource insufficient

Problem
How to schedule packets from different flows, when packets consume multiple resources?

How to generalize fair queueing to multiple resources?


4

Contribution
Allocation in Space Single-Resource Fairness Multi-Resource Fairness Max-Min Fairness DRF Allocation in Time Fair Queueing DRFQ

Generalize Virtual Time to Multiple Resources


5

Outline
Analysis of Natural Policies DRF allocations in Space DRFQ: DRF allocations in Time Implementation/Evaluation

Desirable Multi-Resource Properties


Share guarantee:
Each flow can get 1/n of at least one resource

Strategy-proofness:
A flow shouldnt be able to finish faster by increasing the resources required to process it.

Violation of Share Guarantee


Example of FQ applied to a
Two resources CPU and NIC, used serially Two flows with profiles <2 s,1 s> and <1 s,1 s> FQ based on NIC alternates one packet from each flow
CPU NIC time 0 1 2 p1 p1 p1 3 100% 33% 50% 66% 33% 0% CPU NIC
8

p2 p1 4 5

p2 p2 6 p2

p3

p3 p3 p3 9

p4

flow 1 flow 2

7 33%

10

11

flow 1 flow 2

Share Guarantee Violated by Single Resource FQ

Violation of Strategy-Proofness
Bottleneck fairness
Determine which resource is bottlenecked Apply FQ to that resource

Example with Bottleneck Fairness


2 resources (CPU, NIC), 3 flows <10,1>, <10,14>, <10,14> CPU bottlenecked and split equally
100% 50%
33%

100% 50%
48%

flow 1

flow 2
33%

flow 3

0%

CPU

NIC

0%

CPU

NIC

Flow 1 changes to <10,7>. NIC bottlenecked and split equally

Bottleneck Fairness Violates Strategy-Proofness

Natural Policy
Per-Resource Fairness (PRF)
Have a buffer between each resource Apply fair queueing to each resource

Per-Resource Fairness not strategy-proof


2 resources, 2 flows <4,1>, <1,2>
100% 50%
57% 43% 86%

100% 50%

33%

66%

Flow 1 Flow 2

0%

14%

66%

33%

CPU

NIC

0%

CPU

NIC
11

Flow 1 changes demand to <4,2>

Problems with Per-Resource Fairness


PRF violates strategy-proofness
Can be manipulated by wasting resources

PRF requires per-resource queues


Problematic for parallel resource consumption e.g. CPU and memory consumption in a module

12

Why care about strategy-proofness?


Lack of strategy-proofness encourages wastage
Decreasing goodput of the system

Networking applications especially savvy


Peer-to-peer apps manipulate to get more resources

Trivially guaranteed for single resource fairness


But not for multi-resource fairness
13

Summary of Policies
Policy Fair Queueing a Single Resource Bottleneck Fairness Per-Resource Fairness Dominant Resource Fairness X X X Share Guarantee StrategyProofness

14

Outline
Analysis of Natural Policies DRF allocations in Space DRFQ: DRF allocations in Time Implementation/Evaluation

15

Dominant Resource Fairness


DRF originally in the cloud computing context
Satisfies share guarantee Satisfies strategy-proofness

16

DRF Allocations
Dominant resource of a user is the resource she is allocated most of
Dominant share is the users share of her dominant resource

DRF: apply max-min fairness to dominant shares


Equalize the dominant share of all users Total resources: User 1 demand: User 2 demand:
100%

<16 CPUs, 16 GB mem> <3 CPU, 1 GB mem> dom res: CPU <1 CPU, 4 GB mem> dom res: mem
3 CPUs 12 GB 66% User 1 User 2

50% 66% 0% 12 CPUs CPU 4 GB mem


17

Allocations in Space vs Time


DRF provides allocations in space
Given 1000 CPUs and 1 TB mem, how much to allocate to each user

DRFQ provides DRF allocations in time


Multiplex packets to achieve DRF allocations over time

18

Outline
Analysis of Natural Policies DRF allocations in Space DRFQ: DRF allocations in Time Implementation/Evaluation

19

Determining Packet Resource Consumption


A-priori packet link usage known in FQ
Packet size divided by throughput of link

Packet processing time a-priori unknown for multi-resources


Depends on the modules that process it

Leverage Start-time Fair Queueing (SFQ)


Schedules based on virtual start time of packets Start time of packet p independent of resource consumption of packet p
20

Memoryless Requirement
Virtual Clock simulates flows with dedicated 1/n link
Attach start and finish tags according to dedicated link Serve packet with smallest finish tag (work conserving)

Problem
During light load a flow might get more than 1/n That flow experiences long delays when new flows start

Requirement: memoryless scheduling


A flows share of resources should be independent of its share in the past 21

Memoryless through Virtual Time


Virtual time to track amount service received
Virtual Time V(t) A unit of virtual time always corresponds to same amount of service
60 40 20

Example with 2 flows


Time 0: one backlogged flow Time 20: two backlogged flows

20

40

60

80

Schedule the packets according to V(t)

Real Time

Assign virtual start/finish time when packet arrives

Dove-tailing Requirement
FQ: flow size should determine service, not packet size
Flow with 10 1kb packets gets same service as 5 2kb packets

Want flow processing time, not packet processing time


Example: give same service to these flows: Flow 1: p1 <1,2>, p2 <2,1>, p3 <1,2>, p4 <2,1>, Flow 2: p1 <3,3>, p2 <3,3>, p3<3,3>, p4 <3,3>,

Requirement: dove-tailing
Packet processing times should be independent of how resource consumption is distributed in a flow
23

Tradeoff
Dovetailing and memoryless property at odds
Dovetailing needs to remember past consumption

DRFQ developed in three steps


Memoryless DRFQ: uses a single virtual time Dovetailing DRFQ : use virtual time per resource DRFQ: generalizes both, tradeoff between memoryless and dovetailing
24

Memoryless DRFQ
Attach a virtual start and finish time to every packet
S(p) and F(p) of packet p

Computing virtual finish time F(P)


1. F(p) = S(p) + maxi{ p_time(p, i) } p_time(p, i) = { processing time of p on resource i }

Computing virtual start time, S(p)


2. S(p) = max( F(p-1), C(t) ) C(t) = { max start time of currently serviced packet }

Service the packet with minimum virtual start time


25

Memoryless DRFQ example


Two flows become backlogged at time 0
Flow 1 alternates <1,2> and <2,1> packet processing Flow 2 uses <3,3> packet processing time

1. F(p) = S(p) + maxi{ p_time(p, i) } 2. S(p) = max( F(p-1), C(t) )


Flow 1 P5 S: 8 F: 10 Flow 1 P4 S: 6 F: 8 Flow 1 P3 S: 4 F: 6 Flow 2 P3 S: 6 F: 9 Flow 1 P2 S: 2 F: 4 Flow 2 P2 S: 3 F: 6 Flow 1 P1 S: 0 F: 2 Flow 2 P1 S: 0 F: 3
27

Flow 1 gets worse service than Flow 2

Dovetailing DRFQ
Keep track of start and finish time per resource
Use max start time per resource for scheduling Dovetail by keeping track of all resource usage

28

Dovetailing DRFQ example


Two flows become backlogged at time 0
Flow 1 alternates <1,2> and <2,1> packet processing Flow 2 uses <3,3> per packet
Flow 1 S1: 6 F1: 7 S2: 6 F2: 8 Flow 1 S1: 4 F1: 6 S2: 5 F2: 6 Flow 1 S1: 3 F1: 4 S2: 3 F2: 5 Flow 2 S1: 6 F1: 9 S2: 6 F2: 9 Flow 1 S1: 1 F1: 3 S2: 2 F2: 3 Flow 2 S1: 3 F1: 6 S2: 3 F2: 6 Flow 1 S1: 0 F1: 1 S2: 0 F2: 2 Flow 2 S1: 0 F1: 3 S2: 0 F2: 3
29

Dovetailing ensures both flows get same service

DRFQ algorithm
DRFQ bounds dovetailing to processing time
Dovetail up to processing time units Memoryless beyond

DRFQ is a generalization
When =0 then DRFQ=memoryless DRFQ When = then DRFQ=dovetailing DRFQ

Set to a few packets worth of processing


30

Outline
Analysis of Natural Policies DRF allocations in Space DRFQ: DRF allocations in Time Implementation/Evaluation

31

Isolation Experiment
DRFQ Implementation in Click
2 elephants: 40K/sec basic, 40K/sec IPSec 2 mice: 1/sec basic, 0.5/sec basic

Non-backlogged flows isolated from backlogged flows32

Simulating Bottleneck Fairness


2 flows and 2 res. <CPU, NIC>
Demands <1,6> and <7,1> bottleneck unclear

Especially bad for TCP and video/audio traffic


33

Summary
Packet processing becoming evermore sophisticated
Consume multiple resources

Natural policies not suitable


Per-Resource Fairness (PRF) not strategy-proof Bottleneck Fairness doesnt provide isolation

Proposed Dominant Resource Fair Queueing (DRFQ)


Generalization of FQ to multiple resources Generalizes virtual time to multiple resources Provides tradeoff between memoryless and dovetailing Provides share-guarantee (isolation) and strategyproofness
34

35

36

Overhead
350 MB trace run through our Click implementation Evaluate overhead of two modules
Intrusion Detection, 2% overhead Flow monitoring, 4% overhead

37

Determining Resource Consumption


Resource consumption obvious in routers
Packet size divided by link rate

Generalize consumption to processing time


Normalized time a resource takes to process packet

Normalized processing time


e.g. 1 core takes 20s to service a packet, on a quad-core the packet processing time is 5s Packet processing time packet service time
38

Module Consumption Estimation


Linear estimation of processing time
For module m and resource r as function of packet size

R2 > 0.90 for most modules


39

Simulating Bottleneck Fairness


2 flows and 2 res. <CPU, NIC>
Demands <1,6> and <7,1> bottleneck unclear

CPU bottleneck: 7<1,6> + <7,1> = <14,43> NIC bottleneck: <1,6> + 6<7,1> = <43, 12> Periodically oscillates the bottleneck

40

TCP and oscillations


Implemented Bottleneck Fairness in Click
20 ms artificial link delay added to simulate WAN Bottleneck determined every 300 ms 1 BW-bound flow and 1 CPU-bound flow

Oscillations in Bottleneck degrade performance of TCP


41

Multi-Resource Consumption Contexts


Different modules within a middlebox
E.g. Bro modules for HTTP, FTP, telnet

Different apps on a consolidated middlebox


Different applications consume different resources

Other contexts
VM scheduling in hypervisors Requests to a shared service (e.g. HDFS)
42

You might also like