10 Gb Ethernet
Mark Wagner
Senior Software Engineer, Red Hat
10 Gb Ethernet - Overview
This presentation is about 10Gb Ethernet performance, tuning and
functionality in RHEL5.X
RHEL4.X
also contain support for 10GbE cards
Needed to limit our focus based on resources for summit
This talk will cover:
Platform characteristics to look for
Tools that I find useful to debug or tune
Tuning steps
Performance of 10GbE networking in:
A virtualized
MRG
environment
Messaging, Realtime, Grid
Some Quick Disclaimers
Test results based on two different platforms
Cards supplied by three different vendors
Chelsio, Intel, Neterion
Red Hat supports all of devices used for this presentation
Intel, AMD
We do not recommend one over the other
Testing based on performance mode
Maximize a particular thing at the expense of other things
Not recommended for production
Don't assume settings shown will work for you without some tweaks
Take Aways
Hopefully, you will be able to leave this talk with:
An understanding of the tools available to help you evaluate
your network performance
An understanding of 10GbE performance under RHEL5
Use this talk as suggestions of things to try
My testing based on local network wide area network will
be different
Do not assume all setting will work for you without some
tweaks
Take Aways - continued
Read the vendors Release Notes, tuning guides, etc
Visit their website
Install and read the source
/usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18.x86_64/Documentation/networking
Use the Internet as starting place for the knobs to turn
Do not assume every setting on the Internet will work for
you, often times different websites conflict with their advice
Internet
search linux tcp_window_scaling performance
some sites say to set it to 0, other sites set it to 1
Let's run a test to see what works best in *my*
testbed....
AQuickExample
Internetsearchforlinuxtcp_window_scalingperformance
willshowsomesitessaytosetitto0otherssaysetitto
1
[[email protected]]#sysctlwnet.ipv4.tcp_window_scaling=0
[[email protected]]#./netperfP1l30H192.168.10.100
RecvSendSend
SocketSocketMessageElapsed
SizeSizeSizeTimeThroughput
bytesbytesbytessecs.10^6bits/sec
87380163841638430.002106.40
[[email protected]]#sysctlwnet.ipv4.tcp_window_scaling=1
[[email protected]]#./netperfP1l30H192.168.10.100
RecvSendSend
SocketSocketMessageElapsed
SizeSizeSizeTimeThroughput
bytesbytesbytessecs.10^6bits/sec
87380163841638430.015054.68
Platform Features
Multiple processors
Fast memory
All my testing has been done on PCIe
PCI-X not really fast enough for full-duplex
Look for width of 8 lanes (8x)
PCIe
is typically 250 Gb/sec per lane
Look for support of MSI-X interrupts
Server
OS (RHEL does this :)
Driver
Tools
Monitor / debug tools
mpstat
reveals per cpu stats, Hard/Soft Interrupt usage
vmstat
vm page info, context switch, total ints/s, cpu
netstat
per nic status, errors, statistics at driver level
lspci
list the devices on pci, indepth driver flags
oprofile
system level profiling, kernel/driver code
modinfo list information about drivers, version, options
sar
collect, report, save system activity information
Many others available- iptraf, wireshark, etc
Sample use for some of these embedded in talk
Tools (cont)
Tuning tools
ethtool
View and change Ethernet card settings
sysctl
View and set /proc/sys settings
ifconfig
View and set ethX variables
setpci
View and set pci bus params for device
netperf
Can run a bunch of different network tests
/proc
OS info, place for changing device tunables
ethtool
Works mostly at the HW level
ethtool -S provides HW level stats
Counters
since boot time, create scripts to calculate diffs
ethtool -c - Interrupt coalescing
ethtool -g - provides ring buffer information
ethtool -k - provides hw assist information
ethtool -i - provides the driver information
ethtoolcinterruptcoalesce
[root@perf10~]#ethtoolceth2
Coalesceparametersforeth2:
AdaptiveRX:offTX:off
statsblockusecs:0
sampleinterval:0
pktratelow:0
pktratehigh:0
rxusecs:5
rxframes:0
rxusecsirq:0
rxframesirq:0
txusecs:0
txframes:0
txusecsirq:0
txframesirq:0
<truncated>
ethtoolgHWRingBuffers
[root@perf10~]#ethtoolgeth2
Ringparametersforeth2:
Presetmaximums:
RX:16384
RXMini:0
RXJumbo:16384
TX:16384
Currenthardwaresettings:
RX:1024
RXMini:1024
RXJumbo:512
TX:1024
Typically these numbers correspond the number of buffers, not the
size of the buffer
With some NICs creating more buffers decreases the size of each
buffer which could add overhead
ethtoolkHWOffloadSettings
[root@perf10~]#ethtoolketh2
Offloadparametersforeth2:
Cannotgetdeviceudplargesendoffloadsettings:Operation
notsupported
rxchecksumming:on
txchecksumming:on
scattergather:on
tcpsegmentationoffload:on
udpfragmentationoffload:off
genericsegmentationoffload:off
These provide the ability to offload the CPU for calculating the
checksums, etc.
ethtoolidriverinformation
[root@perf10~]#ethtoolieth2
driver:cxgb3
version:1.0ko
firmwareversion:T5.0.0TP1.1.0
businfo:0000:06:00.0
[root@perf10~]#ethtoolieth3
driver:ixgbe
version:1.1.18
<truncated>
[root@dhcp47154~]#ethtoolieth2
driver:Neterion(ed.notes2io)
version:2.0.25.1
<truncated>
sysctl
sysctl is a mechanism to view and control the entries under
the /proc/sys tree
sysctl -a
- lists all variables
sysctl -q
- queries a variable
sysctl -w
- writes a variable
When setting values, spaces are not allowed
sysctlwnet.ipv4.conf.lo.arp_filter=0
Setting a variable via sysctl on the command line is not
persistent The change is only valid until the next reboot
Write entries into the /etc/sysctl.conf file to have them
applied at boot time
Some Important settings for sysctl
Already showed tcp_window_scaling issue
By default, Linux networking not tuned for max performance,
more for reliability
Buffers are especially not tuned for local 10GbE traffic
Remember that Linux autotunes buffers for connections
Don't forget UDP !
Try via command line
Look at documentation in /usr/src
When you are happy with the results, add to /etc/sysctl.conf
/usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18.x86_64/Documentation/networking
Some Important settings for sysctl
net.ipv4.tcp_window_scaling - toggles window scaling
Misc TCP protocol
net.ipv4.tcp_timestamps - toggles TCP timestamp support
net.ipv4.tcp_sack - toggles SACK (Selective ACK) support
TCP Memory Allocations - min/pressure/max
net.ipv4.tcp_rmem - TCP read buffer - in bytes
overriden
by core.rmem_max
net.ipv4.tcp_wmem - TCP write buffer - in bytes
overridden
net.ipv4.tcp_mem - TCP buffer space
measured
by core/wmem_max
in pages, not bytes !
Some Important settings for sysctl
CORE memory settings
net.core.rmem_max - max size of rx socket buffer
net.core.wmem_max -max size of tx socket buffer
net.core.rmem_default - default rx size of socket buffer
net.core.wmem_default - default tx size of socket buffer
net.core.optmem_max - maximum amount of option memory
buffers
net.core.netdev_max_backlog how many unprocessed rx packets
before kernel starts to drop them
These settings also impact UDP
netperf
http://netperf.org
Feature Rich
Read documentation
Default test is TCP_STREAM uses send() call
TCP_SENDFILE uses sendfile() call much less copying
TCP_RR Request / Response tests
UDP_STREAM
Many others
Know what you are testing
Linux has several automatic features that may cause
unanticipated side effects
Message delivery - Linux does its best to get message from A to B
Packet
may get from A to B via different path than you think
Check
arp_filter settings - sysctl -a | grep arp_filter
Automatic buffer sizing
Be
explicit if it matters to you
Control your network route :
Check arp_filter settings with sysctl
sysctl -a | grep arp_filter
A setting
of 0 says uses any path
If more than one path between machines, set arp_filter=1
Look for increasing interrupt counts in /proc/interrupt or
increasing counters via ifconfig or netstat
LabSwitch
1GbE
10GbE
Know what you are testing - Hardware
Did the PCIe bus get negotiated correctly?
Use
Did the interrupts come up as expected
MSI-X can make a big difference
On
lspci
some cards its not on by default
Several vendors have information on changing the default PCI-E
settings via setpci
Read
the Release Notes / README !
lspcivalidateyourslotsettingforeachNIC
lspcivvs09:00.0
09:00.0Ethernetcontroller:10GbESinglePortProtocolEngineEthernetAdapter
<truncated>
Capabilities:[58]ExpressEndpointIRQ0
Device:Supported:MaxPayload4096bytes,PhantFunc0,ExtTag+
Device:LatencyL0s<64ns,L1<1us
Device:AtnBtnAtnIndPwrInd
Device:Errors:CorrectableNonFatalFatalUnsupported
Device:RlxdOrd+ExtTagPhantFuncAuxPwrNoSnoop+
Device:MaxPayload128bytes,MaxReadReq512bytes
Link:SupportedSpeed2.5Gb/s,Widthx8,ASPML0sL1,Port0
Link:LatencyL0sunlimited,L1unlimited
Link:ASPMDisabledRCB64bytesCommClkExtSynch
Link:Speed2.5Gb/s,Widthx4
Vectortable:BAR=4offset=00000000
Some General System Tuning Guidelines
To maximize network throughput lets
Disable irqbalance
service irqbalance stop
chkconfig irqbalance off
Disable cpuspeed
default gov=ondemand, set governer to performance
Use affinity to maximize what WE want
Process
affinity
Use taskset or
Interrupt affinity
MRGs Tuna
grep eth2 /proc/interrupts
echo 80 > /proc/irq/177/smp_affinity
Performance Tuning Outline
IRQ Affinity / Processor Affinity - No magic formula
experiment to get the best results
Interrupt coalescing
* My * experience is that chip architectures play a big role
Try to match TX and RX on same socket / data caches
sysctl.conf
Increase/decrease memory parameter for network
Driver Setting
NAPI if driver supports
HW ring buffers
TSO, UFO, GSO
Actual Tuning Example
You just got those new 10GbE cards that you told the CIO
would greatly improve performance
You plug them in and run a quick netperf to verify your choice
NewBoards,firstRun
#./netperfP1l60H192.168.10.10
RecvSendSend
SocketSocketMessageElapsed
SizeSizeSizeTimeThroughput
bytesbytesbytessecs.10^6bits/sec
87380163841638460.005012.24
Hmm, about 5 Gb/sec, half of what the CIO is expecting
Lets see where the bottleneck is :
NewBoards,firstrunmpstatPALL5
Transmit
CPU%sys%iowait%irq%soft%steal%idleintr/s
all2.170.000.351.170.0096.2312182.00
017.400.002.809.200.0070.0012182.00
10.000.000.000.000.00100.000.00
20.000.000.000.000.00100.000.00
30.000.000.000.000.00100.000.00
40.000.000.000.000.00100.000.00
50.000.000.000.000.00100.000.00
60.000.000.000.000.00100.000.00
70.000.000.000.000.00100.000.00
Receive
CPU%sys%iowait%irq%soft%steal%idleintr/s
all4.860.000.077.560.0087.4910680.90
038.900.000.6060.400.000.0010680.90
10.000.000.000.000.00100.000.00
20.000.000.000.000.00100.000.00
30.000.000.000.000.00100.000.00
40.000.000.000.000.00100.000.00
50.000.000.000.000.00100.000.00
60.000.000.000.000.00100.000.00
70.000.000.000.000.00100.000.00
Tuning Identify the Bottlenecks
Run mpstatPALL5 while running netperf
Review of output shows core0 on Receive side is pegged
So lets try to set some IRQ affinity
grep for the NIC in /proc/interrupts
Echo the desired value into /proc/irq/XXX/smp_affinity
Does
NOT persist across reboots
Setting IRQ Affinity
CPU cores designated by bitmap
cat/proc/cpuinfo to determine how the BIOS presented
the CPUs to the system
Some go Socket0, core0, socket1, core0
Others go Socket0, core0, socket0, core1
Understand the layout of L2 cache in relationship to the cores
Remember these values do not persistent across reboots!
Set IRQ affinity
echo80>/proc/irq/192/smp_affinity
Use TUNA
KnowYourCPUcorelayout
#cat/proc/cpuinfo
processor:0
physicalid:0
coreid:0
processor:1
physicalid:1
coreid:0
processor:2
physicalid:0
coreid:1
processor:3
physicalid:1
coreid:1
processor:4
physicalid:0
coreid:2
processor:5
physicalid:1
coreid:2
processor:6
physicalid:0
coreid:3
processor:7
physicalid:1
coreid:3
Socket0
Socket1
SettingIRQAffinity
Now lets move the interrupts, remember your core mapping is important
Note that the separate irq for TX and RX
Transmit
#grepeth2/proc/interrupts
CPU0CPU1CPU2CPU3CPU4CPU5CPU6CPU7
74:3603450000000PCIMSIXeth2tx0
82:6479600000000PCIMSIXeth2rx0
90:00000000PCIMSIXeth2lsc
#echo40>/proc/irq/74/smp_affinity
#echo80>/proc/irq/82/smp_affinity
Receive
#grepeth2/proc/interrupts
CPU0CPU1CPU2CPU3CPU4CPU5CPU6CPU7
194:64770000000PCIMSIXeth2
202:57954050000000PCIMSIXeth2(queue0)
#echo40>/proc/irq/194/smp_affinity
#echo80>/proc/irq/202/smp_affinity
TuningRun2,IRAAffinity
#./netperfP1l30H192.168.10.10
RecvSendSend
SocketSocketMessageElapsed
SizeSizeSizeTimeThroughput
bytesbytesbytessecs.10^6bits/sec
87380163841638430.005149.89
OK, so throughput is up slightly, lets look for bottlenecks
Run2mpstatPALL5outputs
Transmit
CPU%sys%iowait%irq%soft%steal%idleintr/s
all2.350.000.431.430.0095.7512387.80
00.000.000.000.000.00100.001018.00
10.000.000.000.000.00100.000.00
20.000.000.000.000.00100.000.00
30.000.000.000.000.00100.000.00
40.000.000.000.000.00100.000.00
50.000.002.802.000.0095.204003.80
618.600.000.609.400.0070.807366.40
70.000.000.000.000.00100.000.00
Receive
CPU%sys%iowait%irq%soft%steal%idleintr/s
all4.670.000.077.750.0087.4910989.79
00.000.000.000.000.00100.001018.52
10.000.000.000.000.00100.000.00
20.000.000.000.000.00100.000.00
30.000.000.000.000.00100.000.00
40.000.000.000.000.00100.000.00
50.000.000.000.000.00100.000.00
60.000.000.000.000.00100.000.00
737.360.000.6061.940.000.009971.17
Run 2 review data next steps
We moved interrupts and reran the test,
saw a very slight improvement in throughput,
not
really expecting much...yet
Looking at the mpstat output we can see that we are still
bottlenecked on the receive side.
Lets now try to add some process affinity to the mix
Remember core mappings
Try to get netperf to run on a sister core to the interrupts
Typically use taskset or TUNA for this, can also handle
programatically
netperf
also has a built in mechanism
Run3AddProcessAffinity
#./netperfP1l30H192.168.10.10T5,5
RecvSendSend
SocketSocketMessageElapsed
SizeSizeSizeTimeThroughput
bytesbytesbytessecs.10^6bits/sec
87380163841638430.004927.19
Throughput is down slightly , lets see where the bottleneck is now
Run3Bottlenecks
Transmit
CPU%sys%iowait%irq%soft%steal%idleintr/s
all4.350.000.3516.000.0079.2511272.55
00.000.000.000.000.00100.001020.44
10.000.000.000.000.00100.000.00
20.000.000.000.000.00100.000.00
30.000.000.000.000.00100.000.00
40.000.000.000.000.00100.000.00
534.730.002.5962.080.000.004011.82
60.000.000.4065.600.0034.006240.28
70.000.000.000.000.00100.000.00
Receive
CPU%sys%iowait%irq%soft%steal%idleintr/s
all3.850.000.1017.620.0078.3917679.00
00.000.000.000.000.00100.001016.80
10.000.000.000.000.00100.000.00
20.000.000.000.000.00100.000.00
30.000.000.000.000.00100.000.00
40.000.000.000.000.00100.000.00
530.900.000.0063.700.005.100.00
60.000.000.000.000.00100.000.00
70.000.000.8077.300.0021.9016661.90
Run3, Analysis + next steps
By adding process affinity things have changed
The bottleneck is now on the transmit side
core5 on TX is 100%, core5 on RX side is handling the load
Try moving process affinities around (already done)
Change the code (if you can)
Default netperf method uses send() which copies data around
Try TCP_SENDFILE which use the sendfile() system call
Try bigger MTU
Currently at 1500
Run4Changesend()tosendfile()
#./netperfP1l30H192.168.10.10T5,5tTCP_SENDFILEF
/data.file
RecvSendSend
SocketSocketMessageElapsed
SizeSizeSizeTimeThroughput
bytesbytesbytessecs.10^6bits/sec
87380163841638430.006689.77
Run4mpstatoutputsendfileoption
Transmit
CPU%sys%iowait%irq%soft%steal%idleintr/s
all1.550.000.387.300.0090.7012645.00
00.000.000.000.000.00100.001018.20
10.000.000.000.000.00100.000.00
20.000.000.000.000.00100.000.00
30.000.000.000.000.00100.000.00
40.000.000.000.000.00100.000.00
512.380.002.2014.170.0070.663973.40
60.000.000.8044.200.0055.007653.20
70.000.000.000.000.00100.000.00
Receive
CPU%sys%iowait%irq%soft%steal%idleintr/s
all5.730.310.0418.490.0075.006050.75
00.202.500.100.100.0095.201051.65
10.000.000.000.000.00100.000.00
20.100.000.000.000.0099.900.00
30.300.000.000.000.0098.200.00
40.000.000.000.000.00100.000.00
545.250.000.0052.750.001.900.00
60.000.000.000.000.00100.002.70
70.000.000.4095.010.004.594996.60
Tuning - Identifying the Bottlenecks
Wow, our core with the netperf process on TX went from 0%
idle to 70% idle by switching the system call
Overall the system reclaimed 10%
Nice jump in throughput, but we are still not near 10Gb
Let's try a larger MTU
ifconfigeth2mtu9000up
Note
this causes a temporary drop in the connection
Run5KickupMTU=9000
#./netperfP1l30H192.168.10.10T5,5tTCP_SENDFILEF
/data.file
RecvSendSend
SocketSocketMessageElapsed
SizeSizeSizeTimeThroughput
bytesbytesbytessecs.10^6bits/sec
87380163841638430.009888.66
OK, now we can show this to the CIO
Run5KickupMTU=9000
TX
CPU%sys%iowait%irq%soft%steal%idleintr/s
all1.400.000.324.270.0093.9013025.80
00.000.000.000.000.00100.001015.00
10.000.000.000.000.00100.000.00
20.000.000.000.000.00100.000.00
30.000.000.000.000.00100.000.00
40.000.000.000.000.00100.000.00
511.000.002.207.400.0078.804003.80
60.000.000.4026.600.0073.008007.20
70.000.000.000.000.00100.000.00
RX
CPU%sys%iowait%irq%soft%steal%idleintr/s
all6.630.000.395.400.0087.1069932.10
00.000.000.000.000.00100.001017.80
10.000.000.000.000.00100.000.00
20.000.000.000.000.00100.000.00
30.000.000.000.000.00100.000.00
40.000.000.000.000.00100.000.00
553.000.000.0020.200.0022.900.00
60.000.000.000.000.00100.000.00
70.000.003.1023.020.0073.8768914.20
Features - Multi-queue
RHEL5 has support for several vendors RX multi-queue
Typically enabled via script or module parameters
Still no TX multi-queue (that I know of at least)
RX Multi-queue big gainer when there are multiple applications
using the network
Use
affinity for queues and match queue to task (taskset)
Potential advantage with single version of netperf if you have
slower CPUs / memory
SingleRXQueueMultiplenetperf
1016.95192.168.10.37
819.41192.168.10.12
898.93192.168.10.17
961.87192.168.10.16
3696
CPU%sys%iowait%irq%soft%steal%idleintr/s
all3.900.000.0023.630.0072.441054.80
00.000.000.00100.000.000.001054.80
10.200.000.000.200.0099.600.00
28.200.000.0019.000.0072.800.00
38.220.000.0024.050.0067.740.00
40.000.000.000.000.00100.000.00
57.400.000.0024.800.0067.800.00
67.210.000.0021.240.0071.540.00
70.000.000.000.000.00100.000.00
MultipleRXQueues,multiplenetperfs
1382.25192.168.10.37
2127.18192.168.10.17
1726.71192.168.10.16
1986.31192.168.10.12
7171
CPU%sys%iowait%irq%soft%steal%idleintr/s
all6.550.000.1842.840.0050.4428648.00
02.400.000.006.600.0091.001015.40
111.450.000.4083.130.005.029825.00
22.000.000.2097.800.000.002851.80
30.000.000.000.000.00100.000.00
410.600.000.0036.200.0053.200.00
50.000.000.000.000.00100.000.00
611.220.000.0035.870.0052.910.00
714.770.000.8083.030.001.2014955.80
Interruptdistributionw/MultiQueue
[]#grepeth2/proc/interrupts
CPU0CPU1CPU2CPU3CPU4CPU5CPU6CPU7
130:50000000eth2
138:2411032798000000eth2q0
146:10182180300000eth2q1
154:10052425180000eth2q2
162:10001849812000eth2q3
170:10000730195000eth2q4
178:10000084269400eth2q5
186:10000001809018eth2q6
MRG and 10GbE
A different ballgame
Latency often trumps throughput
Messaging tends to leverage UDP and small packets
But most people want both
Predictability is important
New Tuning Tools w/ RH MRG
MRG Tuning using the TUNA dynamically control
Device IRQ properties
CPU affinity / parent and threads
Scheduling policy
New Tuning Tools w/ RH MRG
MRG Tuning using the TUNA dynamically control
Process affinity / parent and threads
Scheduling policy
TuningNetworkAppsMessages/sec
10 Gbit Nics Stoakley 2.67 to Bensley 3.0 Ghz
Tuning enet gains +25% in Ave Latency,
RT kernel reduced peak latency but smoother how much?
RedHatMRGPerformanceAMQPMess/s
Intel8cpu/16gb,10Gbenet
Messages/sec(32bytesize)
600000
500000
400000
300000
200000
100000
0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
Samples(MillionMessage/sample)
rhel52_base
rhel52_tuned
rhelrealtime_tune
Latency
The interrupt coalescing settings are vital
ethtool -c eth3 to read , -C to set
Rx-usecs
tells how often to service interrupt
NAPI may help as you can handle multiple interrupts
Also look at using TCP_NODELAY options
May help with latency but hurt throughput
Nagles Algorithm tries to fill packets in order to avoid overhead of
sending
LowerRXLatencywithethtoolC
#ethtoolceth6
Coalesceparametersforeth6:
<truncate>
rxusecs:125
rxframes:0
rxusecsirq:0
rxframesirq:0
#./netperfH192.168.10.12tTCP_RR
Local/Remote
SocketSizeRequestResp.ElapsedTrans.
SendRecvSizeSizeTimeRate
bytesBytesbytesbytessecs.persec
16384873801110.008000.27
Lower rx-usecs on the receiver and rerun
#ethtoolCeth6rxusecs100
#./netperfH192.168.10.12tTCP_RR
16384873801110.0010009.83
10GbE in a Virtual World
Single guest w/ 1Gb interface Good
Single guest w/ 10 Gb interface - Better than 1Gb
Several copies needed for security
cpu/memory BW limits 10Gbit performance
Better than 1Gb network but not wire speed.
Same speed as using a dummy network
Use RHEL5.2 nic limit
10GbEScalingw/XenonRHEL5.2
10GbE in a Virtual World - cont
Going forward:
PCI_passthru looks promising for performance
Guest
has direct control over NIC
Network throughput for fully-virt/HVM guests will be limited
to 100Mb/s. It can be improved by installing the para-virt
drivers for fully-virt/HVM RHEL/Windows guests
kvm has virtio work going on to improve performance
Wrap up
There are lots of knobs to use, the trick is finding them and
learning how to use them
Learn to use the tools to help investigate issues
Full-Duplex (20Gb/sec) is possible under RHEL5
Questions ?
Credits
The following helped me along the way in preparing this
presentation, to them a special thank you
Andy Gospo
Don Dutile
D John Shakshober
JanMark Holzer
