Hardware accelerating Linux network

functions
Roopa Prabhu, Wilson Kok

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

Agenda
● Recap: offload models, offload drivers
● Introduction to switch asic hardware
● L2 offload to switch ASIC
○ Mac Learning, ageing
○ stp handling
○ igmp snooping
○ vxlan
● L3 offload to switch ASIC

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

Offload models ... ● Single consistent netlink based
UAPI

● Single kernel offload API to

rtnetlink api: offload to variety of hardware
bridge vlan add (nics, switch asics, ..)
bridge fdb add

Rtnetlink API PATH

Offload API path

kernel kernel
FDB
FDB (in sync with hw)
bridge
bridge bridge

port1 port2 port3 port4

port1 port2 portn

port1 port2 portn

port1 port1 port2
port1 port2 NIC1
switch asic
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

FDB CPU MEM

NIC1 FDB
NIC2
The bigger
tc OVSdb

picture... iproute2
mstpd
bridge
nftables
snmpd
quagga lldpd
brctl
bird

user swp1 swpN

kernel Bonds Bridges VXLAN

hw driver

Routing Bridge Netfilter

ARP Tables tc
Tables FDB/MDB Tables

kernel

HW
Routing Bridge
ARP Tables acls CPU MEM
Tables FDB/MDB

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

HW offload driver (kernel) switch ports
RTnetlink API

switchdev
mstp routing offload API
daemon
br0

swp2 swpN
user swp1
kernel
netdev_ops {
.ndo_fdb_add/del
.ndo_fib_add/del
}
FIB
hw driver

Bridge br0
FDB/MDB

kernel

HW
HW
Routing Bridge
ARP Tables acls CPU CPU ASIC MEM
MEM
Tables FDB/MDB

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

 switch ports
HW offload driver (user space) rtnetlink API

RtNetlink
mstp routing rtnetlink notifications
daemon listener
br0
hw driver
swp2 swpN
user swp1
kernel

FIB

Bridge br0
FDB/MDB

kernel

HW
HW
Routing Bridge
ARP Tables acls CPU CPU ASIC MEM
MEM
Tables FDB/MDB

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

switch hardware
switch driver:

● Creates netdevs for front

panel ports
swp1 swp2 swp3
swpn ● Port netdevs only see traffic
kernel forwarded to the CPU port

● Sets hardware offload flag

switch
driver
NETIF_F_HW_SWITCH_OFFLOAD
on netdevs

switch hardware
netdevs for each front
panel ports
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada cpu port

1 2 3 n front panel ports

ip link show switch ports
# ip link show 55: swp53: <BROADCAST,MULTICAST> mtu 1500
1: lo: <LOOPBACK> mtu 16436 qdisc noqueue state qdisc noop state DOWN mode DEFAULT qlen 500
DOWN mode DEFAULT link/ether 00:e0:ec:27:4e:f7 brd ff:ff:ff:ff:ff:ff
link/loopback 00:00:00:00:00:00 brd 00:00:00:00: 56: swp54s0: <BROADCAST,MULTICAST> mtu 1500
00:00 qdisc noop state DOWN mode DEFAULT qlen 500
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> link/ether 00:e0:ec:27:4e:fb brd ff:ff:ff:ff:ff:ff
mtu 1500 qdisc mq state UP mode DEFAULT qlen 1000
57: swp54s1: <BROADCAST,MULTICAST> mtu 1500
link/ether 00:e0:ec:27:4e:b6 brd ff:ff:ff:ff:ff:ff qdisc noop state DOWN mode DEFAULT qlen 500
3: swp1: <BROADCAST,MULTICAST,UP,LOWER_UP> link/ether 00:e0:ec:27:4e:fc brd ff:ff:ff:ff:ff:ff
mtu 1500 qdisc pfifo_fast state UP mode DEFAULT
qlen 500 58: swp54s2: <BROADCAST,MULTICAST> mtu 1500
qdisc noop state DOWN mode DEFAULT qlen 500
link/ether 44:38:39:00:27:ac brd ff:ff:ff:ff:ff:ff
link/ether 00:e0:ec:27:4e:fd brd ff:ff:ff:ff:ff:ff
4: swp2: <BROADCAST,MULTICAST> mtu 9000 qdisc
pfifo_fast state DOWN mode DEFAULT qlen 500 59: swp54s3: <BROADCAST,MULTICAST> mtu 1500
qdisc noop state DOWN mode DEFAULT qlen 500
link/ether 00:e0:ec:27:4e:b8 brd ff:ff:ff:ff:ff:ff
link/ether 00:e0:ec:27:4e:fe brd ff:ff:ff:ff:ff:ff

[snip]
switch ports
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

management port
ethtool on switch port
$ethtool swp1
Settings for swp1: Transceiver: external
Supported ports: [ FIBRE ] Auto-negotiation: off
Supported link modes: 1000baseT/Full Current message level: 0x00000000
(0)
10000baseT/Full
Link detected: yes
Supported pause frame use: Symmetric
Receive-only
Supports auto-negotiation: Yes
Advertised link modes: 1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

PHYAD: 0
Creating a hardware accelerated Linux bridge
device
# ip link add br0 type bridge

# ip link set dev swp1 master br0

# ip link set dev swp2 master br0

# bridge vlan add vid 10-20 dev swp1

# bridge vlan add vid 20-30 dev swp2

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

Bonds as bridge ports
rtnetlink api: ● switch ASICS support
bridge vlan add Link aggregation
bridge fdb add

● bonding driver LAG

config is offloaded to the
switch ASIC

● fdb and vlan offloads go

kernel
through the bonding
FDB (in sync with hw)
driver
bridge bonding driver
bridge
bond0

port1 port2 portn-1 portn

port1 port2 portn-1 portn LAG

bond0 (portn-1, NIC1
switch asic rtnetlink API
portn
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

FDB CPU MEM

switchdev
offload API
Bridging hardware offload: packet path
kernel
known unicast (transit)
bridge
BUM*
system generated/
destined to system
swp1 swp2

switch asic

VLAN
swp1 swp2
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada
Bridging hardware offload: packet path

● Known unicast traffic not destined to system is

forwarded only in hardware
● BUM traffic is forwarded in hardware plus a copy MAY
be sent to kernel
● BUM traffic in kernel should not be forwarded again
(duplicate copies from hardware and software)

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

Bridging hardware offload: fdb learn
br0

user rtnetlink swp1 swp2 swpN

kernel
fdb add/update switch driver
00:11:22:33:44:55
00:11:22:33:44:55 vlan 10
br0 intf_id 9876
swp2
Bridge br0 notification
FDB/MDB
hw events: learn/move

kernel

HW
CPU ASIC MEM

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

Bridging hardware offload: learning in HW
● Turn off learning in bridge driver
● switch driver listens to learn notifications from hardware
● converts hardware interface id and vlan to kernel ifindex of bridge
port (and vlan) and bridge
● sends netlink fdb update to kernel (userspace driver) or calls bridge
driver learn sync switchdev API (kernel driver)

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

Bridging hardware offload: kernel ageing

br0

user rtnetlink swp1 swp2 swpN

kernel
fdb update
switch driver
fdb delete

Bridge br0 fdb delete

FDB/MDB
get fdb hit status

kernel

HW
CPU ASIC MEM

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

Bridging hardware offload: hardware ageing

br0

user rtnetlink swp1 swp2 swpN

kernel

switch driver
fdb delete

Bridge br0 fdb delete

FDB/MDB

kernel

HW
CPU ASIC MEM

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

Bridging hardware offload: ageing
Bridge driver very seldom sees packets with hardware offload. FDB
age is not up to date.
Hardware ageing
● bridge driver should not do ageing if hardware is doing it
● fdb show will need to get age from hardware during ‘show’, or need
periodic age update from switch driver
Kernel ageing
● definitely need periodic age update from switch driver

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

STP offload
STP
● bridge driver maintains STP states (either kernel STP or
userspace STP)
● bridge driver communicates STP states to switch driver
using switchdev offload API
● OR a switch driver in userspace can listen to STP state
notifications to update HW state

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

 IGMP snooping offload
kernel
bridge
dev bridge port swp1 grp 224.1.2.3 temp

router ports on bridge: swp2 report

swp1 swp2

switch asic

query
data

swp1 swp2

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada 224.1.2.3 Query
Join 224.1.2.3
IGMP snooping offload
● switch driver configures hardware to send IGMP reports
and queries to software
● bridge driver maintains IGMP group membership
● in some cases the reports or queries need to be re-
forwarded in the kernel

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

 VXLAN offload - hardware vtep

swp3
172.16.21.150
MAC Destination
lo: 172.16.20.103 vxlan100
macC 172.16.21.150
macC
20.0.0.2 unknown 172.16.22.125

MAC Interface bridge

macA swp1

macB swp2

macC vxlan100
swp1 swp2

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

macA macB
20.0.0.3 20.0.0.5
VXLAN offload - hardware vtep
Model
● VXLAN link as bridge port
○ bridging between local ports
○ VXLAN tunneling for remote MACs
● BUM traffic handling
○ multicast
○ using off-system replicator
■ could have a list of redundant replicators, need to choose ONE out of
the list of remote dests (per flow or per vni etc.)
○ self replication
■ vtep sends to a list of remote vteps, need to choose ALL of the list of
remote dests

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

VXLAN offload - ovsdb integration
Agent to translate ovsdb schema objects to kernel constructs.

OVSDB Linux kernel

logical switch vxlan link + bridge

physical switch tunnel_ip vxlan link local ip

logical port binding bridge member port, vlan

unicast remote mac + physical locator bridge fdb (mac, vlan, dst <remote ip>)

mcast remote mac “unknown” + physical vxlan link default dest

locator list

unicast local mac + physical locator bridge fdb (mac, vlan, local dev)

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

l3 offloads
iproute rtnetlink API path
ip route add 1.1.1.1/32
nexthop via Quagga/Bird
192.168.200.3 nexthop offload API path
via 192.168.200.4

Network
manager

arping for
unresolved
nexthop
user swp1 swp2 swpN

kernel

switch driver
FIB neigh table

kernel

HW
Routing Tables Neigh tables CPU ASIC MEM

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

l3 hardware offload
● Routes via routing daemons go to the kernel
● Unresolved next hops, point to CPU in HW
● switch driver tries to resolve them by probes
(arping)
● Refresh neigh entries for pkts routed through
hardware (hit bit provided by hardware)
Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada