Broadcom J2/Ramon White Boxes components in

Distributed Disaggregated Chassis Routing System

AT&T
Peng WB HW Group
6/5/2019
AT&T - OCP
DRAFT1- Draft1 - 6/4/2019
Large WAN SP-Class Routers Use Cases

Access Layer

Aggregation Layer

Edge Layer

Core Layer

PE P-Core PE
(L2-VPN) (L3-VPN)

P-Core P-Core
PE
PE (L2-VPN)
(Internet) P-Core

PE PE
(L3-VPN) (Internet)

AT&T - OCP
Draft1 - 6/4/2019
Traditional Modular Chassis Design for Large WAN SP-Class Routers

Physical Chassis View Functional Block View

Timing Modules

RPModules

Ports Switch
Modules Fabric
Modules

Other Considerations:
Mechanical and Physical Challenge System Timing Distribution and Synchronization
➢ Port Density ➢
Via Electrical PCB traces
➢ Cabling Density RP to LC Control and Management path via
➢ Scale Out Growth ➢
internal Ethernet Switched path or can be via the
➢ Power Density internal Switch Fabric or other methods ➔ Depend
➢ Cooling Density on technology and implementation. AT&T - OCP
Draft1 - 6/4/2019
 Disaggregated Distributed Chassis Approach for Large WAN SP-Class Routers

Physical Pizza Box View

Internal Ethernet
Control/System
Cabinet3 Management
Externalized as
Optical/Copper
Timing?? Connections to external
LC Module LC Module LC Module LC Module
2 3 Ethernet SW
0 1
Ethernet Functional Block View
Cabinet2 Timing ??
SW1

Fabric InterConnect Fabric Fabric Fabric Fabric RP1 Routing System- NOS runs on internal RP Modules
Externalized as Optical Module 0 Module 1 Module 2 Module 3
RP0 Ethernet SW0
RP0 RP1
Ethernet SW1
Routing System
Ethernet NOS runs on
Cabinet4
LC Module LC Module LC Module LC Module SW0 Externalized RPs
4 5 6 7 Which can be COTS
Switch Fabric/BackPlane

Backplane
Connectors
Cabinet1
LC LC LC
CPU CPU CPU

ports ports ports

Relax Mechanical and Physical Challenge

➢ Realize each Functional Sub-block of the traditional

system into a Physical White box to serve that function.

➢ NOS needs to deal with distributed nature of DDC.

AT&T - OCP
Draft1 - 6/4/2019
Broadcom J2 and Ramon White Box Components For DDC

❑ WB1: 2 RU [40x100G QSFP28] I/0 Ports

+ [13 QSFP-DD 400G] for Fabric Ports –
Need to support 4x10G Breakout mode.

❑ WB2: 2 RU [10x400G QSFP-DD] I/0 Ports

+ [13 QSFP-DD 400G] Cell Fabric
Connections

❑ WB3: 2 RU [48x400G QSFP-DD] Cell

Fabric Connections

❑ WB4: 1 RU [ 24x400G QSFP-DD] Cell

Fabric Connections.

AT&T - OCP
Draft1 - 6/4/2019
DDC J2/Ramon WB Elements: [ WB1, WB2, WB3, WB4]

WB3: (48x400G QSFP-DD)

WB4: (24x400G QSFP-DD)

Chassis Beacon Chassis Beacon

Craft
FABRIC PORTS
2RU Fabric Ports Beacon FABRIC PORTS
1RU Fabric Ports Beacon (24 x QSFP-DD)
Craft (48 x QSFP-DD)

EIA-19”
EIA-19”

Cell Fabric Connections (CFC): Chassis Beacon/ID = 2 Digit 7 Segment LED

Distance LIMIT = 100 meters
DAC Option
• 400G DAC Limit (1-3m) CFC LED Colors { Green, Amber/Red, Blue}
Optical options Blue for Fabric Port Beaconing
• 400G AOC
• 400G Transceiver + Fibers

WB2: (10x400G QSFPDD) WB1 (40x100G QSFP28)

Fabric Ports Beacon Chassis Beacon

Chassis Beacon
Sliding Adjustment Fabric Ports Beacon
Post Depth [26-30 inches] I/O Ports
2RU I/O Ports FABRIC PORTS
(20 x QSFP28)
Craft

(10 x QSFP-DD) (13 x QSFP-DD) 2RU FABRIC PORTS

Craft
I/O Ports (13 x QSFP-DD)

Flush Mount (20 x QSFP28)

EIA-19”
EIA-19”
AT&T - OCP
Draft1 - 6/4/2019
Major Requirements
o The White Boxes in a DDC Routing system are designed to meet TP76200/TP76450 Level 3
requirements operating in a Central Office/Telco environment. Must pass homologation
requirements for use in Most of the World (MOW).
• Physical Dimension: 1RU or 2RU, EIA-19”, Max-Depth = 30 in.
• Front Flush Mount, Rear Mount Sliding adjustable For Post Depth from [26-30in]
• Front to Back Air flow – (Port to Power Direction)
• Hot-swappable Fans and PSU modules. 1+1 Redundant/ Loadsharing AC or DC PSU – High Efficiency - 80Plus Gold or
better.
• Circuitry to support 1588V2 and SyncE, T-TC, T-OC, T-BC support within each WB. ** (DDC Synch). Target Class B.
• OnBoard BMC with dual flash for recoverable remote field upgrade of BMC. BMC is an optional factory installable option.
• Intel x86 CPU with TPM (No Coin Cell Battery) and dual flash for recoverable remote field upgrade. Factory Option for
different CPU models. Broadwell DE {Model1 8 cores, Model2 6 cores, Model3 4 cores}
• Broadcom [J2 + HBM Deep buffer] + [OP2 for high route Scale] designed to support line rate forwarding on all ports at
published minimum packet size.
• Broadcom Ramon for Fabric Whitebox.

AT&T - OCP
Draft1 - 6/4/2019
DDC WBx mounting – Allowing clearance for Cabling & Power
Top

Front Back
DC Fuse

Min Min
6” 8”
AC
PDU/

2RU DDC WBx Vert

Flush
DC
Mounting Fuse
~28” for rear Post
Range {26”-30”}
30”
Protrude
NOTE: Cabinet Illustration is to provide a Mounting

physical mounting context for the application of

these WB in AT&T which do not necessarily apply US- 42” or 48” MOW-1100mm
to other customers of service providers.
Side View
➢ The key requirement is that the rear
mounting needs to be adjustable.
AT&T - OCP
Draft1 - 6/4/2019
High-Level Block Diagram: WB1 [40x100G] I/O + [13x400G] CFC
Silk Screen Numbering Convention:
Within a grouping
1. {0,1,2,…i}
FANi FAN2 FAN1 FAN0 2. Top to bottom
3. Left to right
FAN Control Board
I/O Ports in Group 0 support 4x10G break out
I/O Ports Numbering Silk Screen
PSU1 PSU0 Dual Flash CPLD/FPGA/Other
Dual Flash Glue Logic
BMC LPC,USB, DRAM 0 2 4 6 8 10 12 14 16 18
PCI I2C, MDIO, GPIO
X86 CPU Traces no show for readability 1 3 5 7 9 11 13 15 17 19
I-210 SSD

CS TPM*
CS 10G-KR PCI 20 22 24 26 28 30 32 34 36 38
Enable
PSU Control Board Disable 21 23 25 27 29 31 33 35 37 39
J2-BCM 88690 OP2
BCM16K

Detect PHY SERDES

Fabric Ports Numbering

RJ45 Micro Reverse GearBoxes Retimers* • Retimers used as
Serial USB SFP+ SERDES 0 2 4 6 8 10 12
Necessary for SI
13 QSFP-DD 1 3 5 7 9 11
RJ45 40 QSFP28 I/O Ports 2-digit
USB SFP+ Fabric Ports Optical
OOB Optical Carrier & LED 7 LED
Carrier & LED

Example of Associated Port Blocked when a

Port is operating as 4x10G
7LED 0 2 4 6 8 10 12 14 16 18
Craft
Console
20 x 100G QSFP28 I/O Group 0 13x400G
QSFP-DD Fabric Ports 1 3 5 7 9 11 13 15 17 19
OOB/SFP+ Group 0
Section
20 x 100G QSFP28 I/O Group 1
20 22 24 X
26 28 30 32 34 36 38
X
21 23 25 27 29 31 33 35 37 39

AT&T - OCP
Draft1 - 6/4/2019
High-Level Block Diagram: WB2 [10x400G] I/O + [13x400G] CFC

Silk Screen Numbering Convention:

Within a grouping
1. {0,1,2,…i}
FAN0 FAN0 FAN0 FAN0 2. Top to bottom
3. Left to right
FAN Control Board

I/O Ports Numbering Silk Screen

PSU PSU Dual Flash CPLD/FPGA/Other
Dual Flash Glue Logic 0 2 4 6 8
DRAM
BMC LPC,USB,
PCI 1 3 5 7 9
I2C, MDIO, GPIO
I-210
X86 CPU SSD Traces no show for readability

CS TPM*
CS 10G-KR
Enable PCI

PSU Control Board Disable

J2-BCM 88690 OP2

BCM16K Fabric Ports Numbering

Detect PHY 0 2 4 6 8 10 12
SERDES • Retimers used as 1 3 5 7 9 11
RJ45 Micro Retimers* Necessary for SI
Serial USB SFP+
13 QSFP-DD
RJ45 10-QSFPDD 2-digit
USB SFP+ Fabric Ports Optical
OOB I/O Ports Optical Carrier & LED 7 LED
Carrier & LED

7LED
Craft
Console 10 x 400G QSFP-DD 13x400G
OOB/SFP+ QSFP-DD Fabric Ports
Section I/O Group 0 Group 0

AT&T - OCP
Draft1 - 6/4/2019
High-Level Block Diagram: WB3 [48x400G] CFC

Silk Screen Numbering Convention:

Within a grouping
1. {0,1,2,…i}
2. Top to bottom
FAN0 FAN0 FAN0 FAN0 3. Left to right

FAN Control Board

Fabric Ports Numbering Silk Screen

PSU PSU CPLD/FPGA/Other

0 2 4 6 8 10 12 14 16 18 20 22
Dual Flash
Dual Flash Glue Logic 1 3 5 7 9 11 13 15 17 19 21 23
DRAM
BMC LPC,USB,
PCI I2C, MDIO, GPIO
I-210
X86 CPU SSD Traces no show for readability
24 26 28 30 32 34 36 38 40 42 44 46
CS TPM*
CS 10G-KR PCI 25 27 29 31 33 35 37 39 41 43 45 47
Enable
PCI-SW
PSU Control Board Disable

Ramon Ramon
Detect PHY

RJ45 Micro Retimers* • Retimers used as

Serial SFP+ Retimers*
USB SERDES SERDES Necessary for SI

RJ45 24 QSFP-DD Fabric Ports Optical Carrier & LED

USB SFP+ 2-digit
OOB 24 QSFP-DD Fabric Ports Optical Carrier & LED 7 LED

7LED
Craft
Console
24 x 400G QSFP-DD Fabric Ports Group 0
OOB/SFP+
Section
24 x 400G QSFP-DD Fabric Ports Group 1
AT&T - OCP
Draft1 - 6/4/2019
High-Level Block Diagram: WB4 [24x400G] CFC

Silk Screen Numbering Convention:

Within a grouping
FANi FAN2 FAN1 FAN0 1. {0,1,2,…i}
2. Top to bottom
FAN Control Board 3. Left to right

PSU1 PSU0 Dual Flash CPLD/FPGA/Other Fabric Ports Numbering Silk Screen
Dual Flash Glue Logic
DRAM
BMC LPC,USB,
PCI
0 2 4 6 8 10 12 14 16 18 20 22
I2C, MDIO, GPIO
I-210
X86 CPU SSD Traces no show for readability 1 3 5 7 9 11 13 15 17 19 21 23
CS TPM*
CS 10G-KR PCI
Enable
PSU Control Board Disable

Ramon
Detect PHY

SERDES
RJ45 Micro • Retimers used as
Serial USB SFP+
Retimers* Necessary for SI
USB RJ45 2-digit
SFP+
OOB 24 QSFP-DD Fabric Ports Optical Carrier & ED 7 LED

Craft
7LED
Console
OOB/SFP+ 24 x 400G QSFP-DD Fabric Ports Group 0
Section

AT&T - OCP
Draft1 - 6/4/2019
Craft/Console Ports
Craft Type QTY Purpose
Micro USB Serial 1 Console
RJ-45 USB Serial 1 Console
USB Port 1 USB data access
RJ-45 10M/100M/1G 1 Ethernet OOB Powered on Standby Power rail
10G SFP+ ports 2 For Internal Management Communication for the Modules in a large DDC-RS system

Console:
Only one Serial input can be active for the Console. Micro USB will have higher priority than RJ45 when something is connected to Micro USB such as a terminal from a
technician. Otherwise, the default is RJ45 which would be connecting to a terminal server by default. The Serial console needs to support default selectable between the
BMC or the X86 CPU. (customer-provisioned choice).

OOB Management:
The RJ-45 OOB Ethernet management port needs to be operational even when the system is in the shutdown mode. As such it needs to be designed using the standby
power rail. It also needs to provide simultaneous connectivity to the X86 CPU and the BMC. The Intel I-210 NIC is specified to use for the RJ-45 OOB Ethernet Management.
Design should allow to shutdown Ethernet OOB access to the BMC as needed.

USB port:
USB port can be used to provide access to external USB drive for initial system setup or system rebuild. However, for security reasons, once the system is up and running
under the control of the NOS, the hardware/firmware of the box needs to provide a mechanism to turn off access to this external USB port. This lock needs to persist even
after a cold boot, warm boot, other reset mechanism. To unlock this would require authentication to access the utility to set the registers to unlock it.

2x10G SFP+:
These two 10G SFP+ ports provide connections to the X86 Host CPU. This will be used for communication with the Route-Engine Controller of the DDC-RS when this is one
component of the larger DDC-RS.

AT&T - OCP
Draft1 - 6/4/2019
Notes on Cell Fabric Connections (CFC)
❖ 1 Cell Fabric Connection = 400G QSFP-DD Optics + Fiber (100m Max) +
400G QSFP-DD Optics

❖ 1 Cell Fabric Connection = 400G QSFP-DD DAC (1-3m Max)

❖ 1 Cell Fabric Connection = 400G QSFP-DD AOC (100m Max)

Fiber (100m Max)

400G QSFP-DD Optic 400G QSFP-DD Optic

Traffic over Fabric Cell Connections are Fixed sized Cells

As opposed to the I/O Port traffic which are Ethernet Note:
variable length Packetized format.
100m Max is a Broadcom
An ingress Ethernet Packet from one I/O port on WBx Fabric Latency Constraint
needs to egress an I/O port on WBy, it is broken up into a
number of cells and sprayed over multiple Fabric
interfaces toward the Egress WBy where it is reassembled
before egressing an I/O port on WBy.

AT&T - OCP
Draft1 - 6/4/2019
 LEGEND:
Sample Illustrations for a 16T Cluster Configuration: 2-Serial Console to AT&T TS I/O- I/O Ports
2x1G to Lights Out LAN CFC – Cell Fabric Connections
2x1/10G to AT&T NM LAN
DDC- Disaggregated Distributed Chassis

100G
Optics
RP-0 One “DDC Router”
100G QSFP 100G Internal Mgmt Connections
Eth-SW-0 Optic 2 per RE Controller
48x1/10G SFP
RP-1
100G 1G Internal Mgmt Connections
+ 6x100G QSFP 100G QSFP Optics 1 per WBx
Optic 10G Internal Mgmt Connections
Eth-SW-1 WB3- 48x400G QSFP-DD CFC Or 2 WB4 for 2 per WBx
48x1/10G SFP Fabric Redundancy
+ 6x100G QSFP

WB1: 40x100G QSFP I/O WB1: 40x100G QSFP I/O WB1: 40x100G QSFP I/O
WB2: 10x400G QSFP-DD I/O + 13x400G QSFP-DD CFC + 13x400G QSFP-DD CFC
+ 13x400G QSFP-DD CFC + 13x400G QSFP-DD CFC

100G I/O Ports from DDC Router

400G I/O Ports from DDC Router

100G I/O ports from

DDC Router

Notes on WB1: Can support 10G I/O needs for I/O Ports for MOW
Supports up to 80x10G. Give up 2x100G ports to get 4x10G ports
AT&T - OCP
Draft1 - 6/4/2019
Sample Illustration for a Larger Cluster Configuration:
Different Sized “DDC Router”
Illustrating ALL 13 CFC from WB2: 10x400G QSFP-DD I/O
WB3- 48x400G QSFP-DD CFC
each deployed WB Connected + 13x400G QSFP-DD CFC
to Fabric WB

WB3- 48x400G QSFP-DD CFC WB2: 10x400G QSFP-DD I/O

RP-0 + 13x400G QSFP-DD CFC

WB2: 10x400G QSFP-DD I/O

+ 13x400G QSFP-DD CFC WB3- 48x400G QSFP-DD CFC RP-1
WB2: 10x400G QSFP-DD I/O
+ 13x400G QSFP-DD CFC
Eth-SW2
WB1: 40x100G QSFP I/O WB3- 48x400G QSFP-DD CFC 48x1/10G SFP
+ 13x400G QSFP-DD CFC + 6x100G QSFP
WB2: 10x400G QSFP-DD I/O
+ 13x400G QSFP-DD CFC
Eth-SW1
48x1/10G SFP
WB3- 48x400G QSFP-DD CFC + 6x100G QSFP

WB2: 10x400G QSFP-DD I/O

+ 13x400G QSFP-DD CFC WB1: 40x100G QSFP I/O
+ 13x400G QSFP-DD CFC
WB3- 48x400G QSFP-DD CFC
2x400G QSFP-DD CFC

WB1: 40x100G QSFP I/O WB1: 40x100G QSFP I/O

+ 13x400G QSFP-DD CFC WB3- 48x400G QSFP-DD CFC + 13x400G QSFP-DD CFC

AT&T - OCP
Draft1 - 6/4/2019
• Q&A:

• Next session: Timing & Synchronization across DDC

AT&T - OCP
Draft1 - 6/4/2019
• Q&A:

• Future Discussion:
Timing & Synchronization across DDC

AT&T - OCP
Draft1 - 6/4/2019
• Q&A:

• Next session: Timing & Synchronization across DDC

AT&T - OCP
Draft1 - 6/4/2019
 AT&T - OCP
Draft1 - 6/4/2019