Scribd
Upload
80 views13 pages

2012 LPC Networking Qdisc Fastabend

This document discusses challenges with using queuing disciplines and traffic classification with multiqueue network devices. It proposes using a multiqueue-aware queuing discipline called mqprio that maps traffic to queues based on priority. Additional enhancements are suggested, such as a pre-enqueue hook for filtering and actions, and developing lockless queuing disciplines to improve performance. Overall, the goal is to better leverage multiqueue hardware for traffic management and quality of service using software techniques.

Uploaded by

John
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
80 views13 pages

2012 LPC Networking Qdisc Fastabend

This document discusses challenges with using queuing disciplines and traffic classification with multiqueue network devices. It proposes using a multiqueue-aware queuing discipline called mqprio that maps traffic to queues based on priority. Additional enhancements are suggested, such as a pre-enqueue hook for filtering and actions, and developing lockless queuing disciplines to improve performance. Overall, the goal is to better leverage multiqueue hardware for traffic management and quality of service using software techniques.

Uploaded by

John
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 13

Motivation

• 10Gbps NICs generally available


– With many queues maybe >100

• 40Gbps and faster is coming soon!


– With even more queues

• Open-vSwitch

• multiqueue (sch_mq) scheduler is fast… but


– traffic classification with multiqueue devices?
– leverage hardware attributes to help QOS?
– can we do SW QOS?
sch_mq
dev_queue_xmit(skb)

skb_tx_hash()
queue_mapping strategy: xps, jhash

skb.queue_mapping
qdisc_enqueue

q.lock q.lock q.lock q.lock

pfifo_fast pfifo_fast pfifo_fast pfifo_fast

qdisc_dequeue
txq.xmit_lock txq.xmit_lock txq.xmit_lock txq.xmit_lock

tx_ring tx_ring tx_ring tx_ring


sch_mq
dev_queue_xmit(skb)

skb_tx_hash()
queue_mapping strategy: xps, jhash

skb.queue_mapping
qdisc_enqueue

q.lock q.lock q.lock q.lock

pfifo_fast pfifo_fast pfifo_fast pfifo_fast

qdisc_dequeue
txq.xmit_lock txq.xmit_lock txq.xmit_lock txq.xmit_lock

tx_ring tx_ring tx_ring tx_ring


With a mechanism to map skb to queue(s) we could…
– use correct qdisc for the traffic type
– tune hardware TX rings for type of traffic
– implement hardware QOS per queue(s)
• ETS (802.1Q), Link strict (802.1Q), rate limits, etc.
HTB qdisc would also work but does not utilize multiqueue HW
sch_mqprio
dev_queue_xmit(skb)

skb_tx_hash() strategy: map skb priority to queue sets

qdisc_enqueue

q.lock q.lock q.lock q.lock

Use correct qdisc for the traffic type.


TC0 TC1 TC2

pfifo_fast pfifo_fast pfifo_fast pfifo_fast

qdisc_dequeue
txq.xmit_lock txq.xmit_lock txq.xmit_lock txq.xmit_lock

Configure tx rings to match traffic type


possibly via DCBnl for ETS, low latency or
tx_ring tx_ring tx_ring tx_ring
through sysfs for BQL, etc.
Benchmark tests for Qdisc
– Intel Xeon X5570 @ 2.93Ghz -- quad-core 2 sockets + HT
– Pktgen injecting packets into vlan devices -- 16 threads

Pktgen Qdisc Benchmarks


14000000

12000000

10000000

mq
8000000
mqprio
pps

6000000 fq_codel
pfifo_fast
4000000

2000000

0
64 128 256 512 1024
Pkt Size
Cut through Latency
• Prio + HTB latency lots of jitter Figure1: mqprio HW ETS with link strict
– Avg Latency 6.8ms

• Hardware QOS with mqprio qdisc


– Avg Latency 0.066ms

• TODO: isolate hardware benefit

Figure2: Prio qdisc with HTB

Prio Qdisc
1:0

1:1 1:2 1:3


Pfifo Fast Pfifo Fast HTB
Strict Prio Strict Prio
PHB Configurable
PHB
Precedence 6 rate
Precedence 7

3:1 3:2 3:3 3:4 3:5 3:6


HTB Leaf HTB Leaf HTB Leaf HTB Leaf HTB Leaf HTB Leaf
PHB PHB PHB PHB PHB PHB
Precedence 5 Precedence 4 Precedence 3 Precedence 2 Precedence 1 Precedence 0
• Problems/Limitations
– sch_mqprio only allows mapping via priority. Good enough?
– filters/actions run under qdisc
– Hard to use qdiscs with gobal state tbf, htb, etc.

• Add pre_enqueue() hook before enqueue()


– Allows filters/actions to be run before qdisc enqueue
– Use skb_edit and queue_mapping actions to control mqprio and mq
flow to queue maps.

• How?
– RCU tcf_proto, classifiers, actions (uses call_rcu_bh)
– Stats per cpu
Example
#tc qdisc add dev eth3 root mqprio num_tc 3 map 0 0 0 0 1 1 1 2 queues 2@0 2@2 1@4
#tc qdisc add dev eth3 parent 8002:4 fq_codel
#tc qdisc add dev eth3 parent 8002:5 fq_codel
# tc filter add dev eth3 parent 0: prio 1 protocol ip u32 \
match ip dport 118 0xffff action skbedit queue_mapping 4

MqPrio Qdisc
1:0

8001:1 8001:2 8001:3


Mqprio Mqprio Mqprio
TC-0 TC-1
TC-2

8001:4 8001:5 8001:6 8001:7 8001:8


fq_codel fq_codel pfifo_fast pfifo_fast pfifo_fast
priority 0,1,2,3 priority 4,5,6 priority 4,5,6 priority 7
priority 0,1,2,3
Lockless Qdisc’s
• add qdisc skb list lockless variants
• add flag to allow qdisc to run lockless
• enable lockless qdiscs (ingress, fq_codel, tbf, …)
14000000

12000000

10000000

8000000 mq
multiq_ll
6000000
multiq
4000000

2000000

0
64 128 256 512 1024

Or should locking qdisc be attached to mq/mqprio?


Summary
• Problem:
– Queuing disciplines and traffic classification do not work well with
multiqueue devices

• Solution:
– Write multiqueue aware queuing disciplines e.g. mqprio
– stack qdiscs under mq aware qdiscs

• Other solutions?
– use lockless classifiers and actions
– Hardware offloaded QOS
Questions/Comments?

git://github.com/jrfastab/

You might also like