Cisco ACI Nexus 9000 Hardware

www.lumoscloud.com
[email protected]
Agenda
 Nexus 9000 At A Glance
 Switch on Chips (ASICs)
 Nexus 9500
 Nexus 9300
 Interface Options
Nexus 9000 Line At A Glance
 Nexus 9000 Design Principles
 Power
Decrease power per port
 Performance
Line rate 40/100 G ports
 Price
Decrease cost per port
 Programmability
Open REST APIs
 Port Density
Concentrating forwarding capacity
in smaller space
Nexus 9000 Operating Modes

ACI Fabric MP-BGP EVPN Fabric (NX-OS)

Traditional “Core/Agg/Access”
(NX-OS)
 Nexus 9000 At-A-Glance

9500 Series 9300 Series 9200 Series

ACI and NX-OS Mode ACI and NX-OS Mode NX-OS Mode
Modular Chassis Switches Fixed and Modular Fixed
ToR/EoR
Cisco ACI Switch Roles
Nexus 9500
Nexus 9336PQ
(baby spine)
Spine Nexus 9364C
(junior spine)
40G or 100G

Leaf

Nexus 9300
APIC APIC APIC APIC Cluster
Switch on Chip (ASICs)
 Switch on Chip (SoC) Basics
 Heart of modern DC
switches
 Often referred to as “ASIC”
or “silicon” This line card has four SoCs,
 ToR comprised on a single sitting under the heat sinks
SoC typically
 Chassis have one or more
SoCs in line cards and in
fabric modules
 Anatomy of a SoC
Divided into 4 sections, each with Example: Broadcom Trident 2
32 x 10G SerDes and 240G
interconnect (25% oversubscribed)
128 x 10 GE SerDes interfaces
(Serializer/Deserializer)

240G 240G
interconnect interconnect
10 interfaces combined to form
240G 240G
interconnect
a CFP2 100 GE interface
interconnect

12 MB shared
buffer 4 interfaces combined to form
a native 40 GE interface
 Anatomy of a SoC
Divided into 2 slices, no
oversubscription (line rate for all Example: Cisco LSE
packet sizes)
72 x 25 GE SerDes interfaces
40 MB total buffer (2x20 MB)
20 MB Slice
Buffer
240G
interconnect
2 interfaces combined to form
240G 240G
interconnect
a native 50 GE interface
interconnect

20 MB Slice
Buffer
4 interfaces combined to form
a native 40 or 100 GE interface
Cisco Spine Switch on Chips (SoC)
2014 2016 2017
Chip Name ASE ASE2 S6400
SerDes Speed 10 Gig 25 Gig 25 Gig
# SerDes 144 144 256

Mode ACI Spine ACI/NX-OS ACI/NX-OS

Only
Buffer 23 MB 30MB 40MB

Size 28 nm 16 nm 16 nm
 Cisco Leaf Switch on Chips (SoC)
2014 2015 2016 2017 2018
Chip Name ALE* ALE2/2+* LSE LS1800FX LS3600FX2
SerDes 10 Gig 10 Gig 25 Gig 25 Gig 25 Gig
Speed
# SerDes 96 48 72 72 144
Mode ACI/NX- ACI/NX-OS ACI/NX-OS ACI/NX-OS ACI/NX-OS
OS
Buffer 40 MB 25 MB 40 MB 40 MB 40 MB
Size 28 nm 28 nm 16 nm 16 nm 16 nm

*ALE and ALE2/2+ require NFE (Broadcom Trident

2), LSE, LS1800FX, and LS3600FX2 do not
 ToR Fixed Anatomy

Nexus 93180YC-EX
CPU LSE

48 x 1/10/25G 6 x 40/100G
1G mgmt
SFP28 QSFP28
Cisco ACI Fabric Modules: Gen 2 (9504 with FM-E example)
Fabric Module 1 Fabric Module 2 Fabric Module 3 Fabric Module 4

ASE2 ASE2 ASE2 ASE2

8 x 100 Gbit =
800 Gbit

LSE LSE LSE LSE LSE LSE LSE LSE LSE LSE LSE LSE LSE LSE LSE LSE
Line Card 1 Line Card 2 Line Card 3 Line Card 4
800 Gbit x 4
32 x QSFP28 100G
FM= 3.2 Tbit
Cisco ACI Fabric Modules: Gen 2 (9504 with FM-E example)
Fabric Module 1 Fabric Module 2 Fabric Module 3 Fabric Module 4

ASE2 ASE2 ASE2 ASE2

8 x 100 Gbit =
800 Gbit

LSE LSE LSE LSE LSE LSE LSE LSE LSE LSE LSE LSE LSE LSE LSE LSE
Line Card 1 Line Card 2 Line Card 3 Line Card 4
800 Gbit x 4
32 x QSFP28 100G
FM= 3.2 Tbit
Nexus 9500
 Cisco Nexus 9500 At A Glance
Cisco Nexus® 9500 Fabric Bandwidth:
16-Slot 3.6 Tbps/ slot

8-Slot
Cloud Deployment Options:
ACI Multi-Site, Telemetry, Cloud
4-Slot Scale
21 RU

13 RU

7 RU
Multi-Generation
Investment Protection:
Nexus 9516 Nexus 9508 Nexus 9504 No Mid-plane, Power Supply Headroom for
Payload Slots 16 8 4 100/400G and integrated optics/ encryption
Capacity (Full Duplex) 57.6 Tbps 28.8 Tbps 14.4 Tbps
100G Density 576 288 144
50G Density 1152 576 288
40G Density 576 288 144
10/25G Density 2304 1152 576

2nd Generation of Insieme ASICs – 16nm

Technology
True Front-To-Back Airflow

No Mid Plane and Perforated Faceplate for

True Front-to-Back Cooling
 Save Cooling and Power
 Same chassis for multiple generations of line
cards and fabric modules
 Nexus 9504/9508/9516 Components
Interchangeable Chassis Size Specific
 Supervisor modules  Chassis
 System controllers  Fabric modules
 Power supplies  Fan modules
 Line Cards
 Supervisor Function Bifurcation
Splitting functions into
two separate units
grants better scale, All-in-one supervisor design (Nexus 7000 Sup 2 shown)
faster reboots, etc.

Supervisor System Controller

 Routing protocols (OSPF, ISIS, etc.)  Overall chassis control
 STP  OOB network
 Control plane functions  Fan speed
 CLI, NXAPI  Temperature monitoring
 Nexus 9000 Power Supplies

N9K-PUV-3000W-B AC/DC universal power supply

N9K-PAC-3000W-B AC power supply
 High voltage AC or DC
 Single 20A input (220V)
 108-305 VAC
 192-400 VDC
 S-Grid safety cables

90% efficiency or better

N9K-PDC-3000W-B DC power supply Redundant (N+1, N+N)
 -48 to -60V DC Port side air intake
Nexus 9500 ACI Line Cards
 Nexus 9500 Fabric Modules
Merchant/Merchant+
9500 Series
• Cost-effective ports
• Basic functionality
• VXLAN routing (merchant+
1st Gen only)
• ACI Spine/NX-OS
• 10/40 Gigabit
9504 9508 9516 N9K-C950X-FM
Multiple Upgrade Paths to 10/25/40/50/100 Gigabit

Cloudscale
2nd Gen Merchant Ultra-Deep Buffers
• Line-rate all ports/all
speeds • Cost effective • Large inter-DC
ports buffers
• Advanced functionality
• NX-OS only • VXLAN routing
• Large intra-DC buffers Large forwarding
• Common •
• VXLAN routing table
chipset
• Large forwarding table (Broadcom) • NX-OS
• ACI Spine/NX-OS
N9K-C950X-FM-S N9K-C950X-FM-R
N9K-C950X-FM-E
 ACI Capable Fabric Modules
(6) x N9K-C95XX-FM (4/5) x N9K-C95XX-FM-E

X9732C-EX ACI Spine X9732C-FX ACI Spine

X9736PQ ACI Spine

1st Generation X9736C-FX ACI Spine

 C950X-FM (1st Gen/Broadcom T2)
 Supports ACI C9504-FM
Spine line card NFE
(1 x NFE)
and NX-OS line
cards (must be C9508-FM
one or the NFE NFE
(2 x NFE)
other)
 40 Gigabit line NFE NFE C9516-FM
cards require 6 NFE NFE (4 x NFE)
fabric modules
 Up to 1.44 Tbps
per line card
NFE = Broadcom Trident 2 SoC
 C950X-FM-E (Cisco ASE2)
 NX-OS or ACI mode
 Requires 4 FM-E ASE2
C9504-FM-E
modules (1 x ASE2)
 Up to 3.2 Tbps per
line card C9508-FM-E
ASE2 ASE2
 Supports (2 x ASE2)
1/10/25/40/50/100
Gigabit ASE2 ASE2 C9516-FM-E
 9504/8-FM-E, flows ASE2 ASE2 (4 x ASE2)
can be 100 Gbps
 9516-FM-E, flows
are limited to 50
Gbps
ASE2 = Cisco SoC (Sugarbowl)
 C950X-FM-E2 (Cisco S6400)
 NX-OS or ACI
mode
C9508-FM-E2
 Requires 4 FM- S6400
(1 x S6400)
E2 modules
 Up to 3.2 Tbps
per line card S6400 S6400
C9516-FM-E2
 Supports (2 x S6400)
1/10/25/40/50/
100 Gigabit
 Flows can be
100 Gbps
 Only for 9508
and 9516
X9732C-EX 100G Cloud Scale Line Card

 160 MB buffer
 32 x 100 GE ports
 ACI mode or NX-OS mode
 LSE-based
 X9736/32C-FX 100G Cloud Scale Line Card

 160 MB buffer
 X9736C-FX: 36 x 100 GE ports
 X9732C-FX: 32 x 100 GE ports
 ACI mode or NX-OS mode
 LS1800FX-based (LSE2)
 Line rate MACSEC on all ports
 CloudSec encryption on 8 ports
 ACI 9500 Spine Line Card Overview
Series X9736PQ X9732C-EX X9732C-FX X9736C-FX
ASIC ASE x 2 LSE x 4 LS1800FX x 4 LS1800FX x 4
Buffer 42 MB 160 MB 160 MB 160 MB
Interfaces 36 x 40 Gbit 32 x 100 Gbit 32 x 100 Gbit 36 x 100 Gbit
Breakouts No Yes Yes Yes
Supported FMs N9K-C95XX-FM N9K-C95XX-FM-E/E2 N9K-C95XX-FM-E/2 N9K-C95XX-FM-E/2
Telemetry No Yes Yes Yes
ACI/NXOS ACI Only ACI/NX-OS ACI/NX-OS ACI/NX-OS
Multi-Pod Support Yes Yes Yes Yes
Multi-Site Support No Yes Yes Yes
MACSec/CloudSec No No Yes Yes
Nexus 9300
 9300 Spines

Nexus 9336PQ “Baby Spine” Nexus 9364C “Junior Spine”

 36 X 40 GE QSPF+ • 64 x 100 GE QSFP28 (no break-out)
 ACI Spine only • ACI and NX-OS mode supported
 23 MB buffer • 40 MB buffer
 Supports multi- • Supports 40 GE leaf connectivity
pod/stretched fabric • Supports multi-pod, stretched, and multi-site
 No multi-site support • Ports can run 10*/40/50/100 GE [*QSA
required] but can’t be broken out
 Nexus 9300 Series ACI Leafs
 ToR/EoR Fixed and GEM modular
switches
 100m/1/10/25/40/50/100 Gigabit
Ethernet interfaces available
 Universal ports (native Fibre Channel
or Ethernet) on FX series
 FCoE (NPV) in NXOS mode, FX and EX
for ACI
 Port to port encryption (MACSEC) on
FX series
 All Cisco ACI capable
 9300 ACI Leaf Overview
Series Nexus Nexus 9300- Nexus 9300-EX Nexus 9300-FX Nexux 9300-FX2
9300 E
ASIC ALE/ALE2 ALE2+ and T2 LSE LS1800FX LS3600FX2
and T2
Security 4,096 (ALE) 32,768 61,440 61,440 61,440
TCAM 32,768 (128K in 3.2**)
(ALE2)
Buffer 52/37 MB 37 MB 40 MB 40 MB 40 MB
IP-based No Yes Yes Yes Yes
EPG
Telemetry No No Yes Yes Yes
FCoE NPV No No Yes Yes Yes
Native FC No No No Yes Yes
MACSec No No No Yes Yes
 Nexus 9396/93128
EOS/EOL EOS/EOL
Announced Announced
Nexus 9396PX Nexus 9396TX Nexus 93128TX
 48 1/10 GE (SFP)  48 100m/1GE/10GE  96 100m/1GE/10GE
interfaces interfaces (10GBASE-T) interfaces (10GBASE-T)
 1 Uplink module slot  1 Uplink module slot  1 Uplink module slot
Plus 1 GEM module

EOS/EOL
Announced
M12PQ M6PQ M6PQ-E
 ALE based  ALE2 based  ALE2+ based
 4,000 Policy CAM  32,768 Policy CAM  32,768 Policy CAM
 IP-based EPG
Nexus 9300 Series (ALE2/T2)
• TX: 48 x 100 Mbit, 1/10 Gigabit Ethernet
Nexus 9396PX/TX Switch • PX: 48 x 1/10 Gigabit Ethernet
(with M6PQ GEM) • 6 x 40 Gigabit Ethernet Uplinks

• TX: 96 x 100 Mbit, 1/10 Gigabit Ethernet

Nexus 93128TX Switch • 6 x 40 Gigabit Ethernet Uplinks

(with M6PQ GEM)

• TX: 48 x 100 Mbit, 1/10 Gigabit Ethernet

• PX: 48 x 1/10 Gigabit Ethernet
Nexus 9372PX/TX Switch • 6 x 40 Gigabit Ethernet Uplinks

• 96 x 100 Mbit, 1/10 Gigabit Ethernet

Nexus 93120TX Switch • 6 x 40 Gigabit Ethernet Uplinks

• 26 x 40 Gigabit Ethernet host facing

Nexus 9332PQ Switch • 6 x 40 Gigabit Ethernet uplinks

37 MB shared buffer 40,000 Policy CAM entries

FCoE NPV (NX-OS)
Nexus 9300-E Series (ALE2+/T2)
Nexus 9396PX/TX Switch • TX: 48 x 100 Mbit, 1/10 Gigabit Ethernet
• PX: 48 x 1/10 Gigabit Ethernet
(with M6PQ-E GEM) • 6 x 40 Gigabit Uplinks

• TX: 96 x 100 Mbit, 1/10 Gigabit Ethernet

Nexus 93128TX Switch • 6 x 40 Gigabit Uplinks
(with M6PQ-E GEM)

• TX: 48 x 100 Mbit, 1/10 Gigabit Ethernet

Nexus 9372PX-E/TX-E Switch • PX: 48 x 1/10 Gigabit Ethernet
• 6 x 40 Gigabit Uplinks

37 MB shared buffer
• 25 MB ALE2
ALE2+ (-E models) brings IP-
• 12 MB NFE based EPGs
 Nexus 9300-EX Series (LSE)
Cisco Nexus 93180YC-EX
• 48 x 1/10/25 Gigabit Interfaces
• 6 x 100 Gigabit ACI uplinks
• 40 MB buffer
• Cisco LSE ASIC
Cisco Tetration Analytics Support (EX)
Cisco Nexus 93108TC-EX FCoE NPV Support in ACI/NX-OS mode
• 48 x 100 Mbps or 1/10 Gigabit Interfaces
• 6 x 100 Gigabit ACI uplinks
• 40 MB buffer
• Cisco LSE ASIC

Cisco Nexus 93180LC-EX

• 24 x 40/50 Gbps QSFP+ Host-Facing Ports
• 6 x 40/100 Gbps QSFP+ Uplinks
• 40 MB buffer
• Cisco LSE ASIC
Nexus 9300-FX Series (LS1800FX)
Cisco Nexus 93180YC-FX
• 48 x host facing configurable universal ports
• 1/10/25 Gigabit Ethernet/FCoE
• 8/16/32 GFC native Fibre Channel
• 6 x 100 Gigabit Ethernet
• ACI uplinks
• FCoE uplinks
Cisco Nexus 93108TC-FX
• 48 x 100 Mbps or 1/10 Gigabit Ethernet Interfaces
• 6 x 100 Gigabit Ethernet uplinks
• ACI uplinks
• FCoE uplinks

NX-OS and ACI Mode Supported

40 MB Buffer
Cisco Tetration Analytics Support
Line-rate 802.1ae MACSEC on all ports
Nexus 9300-FX2 Series (LS3600FX2)
Cisco Nexus 9336C-FX2
• 36 x QSFP28
• 10/25/40/100 Gigabit Ethernet
• 3.6 Tbps
• Line rate > 170 byte packets

Cisco Nexus 93240YC-FX2

• 48 x 1/10/25 Gigabit Ethernet Interfaces
• 18 x 40/100 Gigabit Ethernet uplinks
• 1:1 oversubscription capability

NX-OS and ACI Mode Supported

40 MB Buffer
Cisco Tetration Analytics Support and SSX
Line-rate (> 170 bytes)
802.1ae MACSEC on all ports
 Nexus 93240YC-FX2

Potential for same uplink

bandwidth as downlink
bandwidth (1:1) 12 x 40/100 GE interfaces
(2.4 Tbps)

• 40 MB buffer
• 1 x 1 GE mgmt.
48 x 1/10/25 GE Host • 1 x 10 GE SFP+ mgmt.
interfaces (2.4 Tbps) • 4.8 Tbps total
• Tetration, SSX
Nexus 9336C-FX2

• 36 x 10/25/40/100 GE interfaces • NX-OS/ACI mode capable

(uplink/downlink) • Wire-rate MACSEC on all ports
• Supports 4 x 10/25 GE breakout • 1 x 1 GE mgmt.
• Supports FEX • 1 x 10 GE SFP+ mgmt.
• 40 MB buffer • Tetration, SSX Telemetry Support
 93180LC-EX vs 9336C-FX2
Ports 1-24 40/50 GE to hosts Ports 25, 27, 29-32 uplinks

93180LC-EX
• 24 40/50 GE ports (host)
• 6 100 GE ports (uplink) Ports 26,28 not used
• 3.6 Tbps total
All ports can run 100 GE (or 40, or 4 x 10/25 breakout)

9336C-FX2
• 36 ports 40/100 GE (uplink/downlink)
• 7.2 Tbps total
Nexus 9348GC-FXP

• 48 100m/1 GE BASE-T ports • Local switching (not a FEX)

• 4 x 10/25 GE SFP28 ports • NX-OS/ACI mode capable
• 2 x 40/100 GE QSFP28 ports • Based on LS1800FX (LSE2)
• 40 MB buffer • Wire-rate MACSEC on all ports
• Tetration support
INTERFACE OPTIONS
 Optical Interface Options

Quad-Lane
Single-Lane • 1G SFP (via QSA)
• 1G SFP • 10G SFP+, Twinax, AOC (via QSA)
• 10G SFP+, Twinax, AOC • 25G SFP28, Twinax, AOC (via SLIC)
• 25G SFP28, Twinax, AOC • 50G Twinax, AOC (via SLIC)
• 100G QSFP28, Twinax, AOC
Multiple Speeds with QSFP28 interface
QSFP28 has 4 serial electrical links. So
no 3x40G or 10x10G breakout QSFP+ • 1x40G
• 4x10G

QSFP28 • 1x100G
• 4x25G
• 2x50G

QSA with
• 1x10G
SFP • 1x1G
Support for 40G Optics QSFP+

100m, MMF 10km, SMF 40km, SMF

SR4 QSFP+ LR4 QSFP+ ER4 QSFP+

2km, SMF 10m, Copper

WSP-Q40GLR4L 4 X 10G AOC QSF+ S
QSFP+
 Support for 40G Optics
QSFP+ Fiber Connectors Distance

QSFP-40G-SR4 MMF MPO 100m

QSFP-40G-SR4-S MMF MPO 150m

QSFP-40G-CSR4 MMF MPO 400m

QSFP-40GE-LR4 SMF LCpair 10km

QSFP-40G-LR4 SMF LCpair 10km

QSFP-40G-ER4 SMF LCpair 40km

WSP-Q40GLR4L SMF LCpair 2km

QSFP-40G-LR4-S SMF LCpair 10km

 Support for 100G Optics

QSFP28 Fiber Connectors Distance

SR4 MMF MPO-MTP12 Upto100m

LR4 SMF LCpair Upto10km

CWDM4 SMF LCpair Upto2km

CU1/2/3/5m Copper Build-inQSFP28 Upto5m

AOC
Copper Build-inQSFP28 Upto20m
1/2/3/5/10/15/20m
What about 25G?
25 Gbps Ethernet & Standards
Consortium IEEE

Distance Passive: 1,2,3 meter Passive: 1,2,3,5 meter

Optics: SR

Deployment Within Rack Across Rack

Supporting Platform N9200, N9300-EX Nexus 9300-FX

N3200

25G NIC ( Verified ) Mellanox None available yet

25G NIC (Ongoing Qlogic, BRCM

Testing)
 Cisco QSFP 40/100 Gbps BiDi
• Uses OM3/OM4 multi-mode
fibre (normally 10 or 25 Gbps) to
run 40 or 100 Gbps links
• Plugs into QSFP+/QSFP28 ports
• Cost effective way to upgrade
bandwidth without running new
cable
 Cisco 40/100G BiDi
Switch with Switch with
40/100G Port 40/100G Port
• Standard QSFP form factor
Dual Rate 100G Link Dual Rate • Leverage existing multi-
40/100G BiDi 40/100G BiDi
mode fibre infrastructure
for 100G/40G
Switch with
40G Port • 40/100G BiDi priced the
Dual Rate 40G Link same as 100G SR4
40G BiDi
40/100G BiDi

40G Link Dual Rate

40G BiDi
40/100G BiDi
40/100G Bidi Distance Limits

Mode MMF Type Reach (meters) Reach (feet)

40G OM3 100m 328 ft
OM4 150m 492 ft
100G OM3 70m 229 ft
OM4 100m 328 ft
25 GIGABIT CONSIDERATIONS
 25 Gigabit Standards
Ethernet Consortium IEEE 802.3by
• Up to 3 meters • Reed-Solomon FEC is
required for distances
greater than 3 meters,
which is part of the IEEE
standard
Series Consortium 25 GE IEEE 25 GE
-EX Yes No
-FX Yes Yes
ANALYITCS
Streaming Statistics Export (SSX)
• Streams statistics and other ASIC- • User defines streaming parameters
level data (which stats, how often, which
• Direct export to ASIC (no switch collector)
CPU involvement) • S6400 / LS3600FX2

S6400 / LS3600FX2
Stats
SSX Tables
Registers

Lookup
Parser Pipeline