BCN-B Changes in HW architecture

Aug 24, 2012

Tomi Mnnikk

For internal use only

1
Nokia Siemens Networks

Presentation / Author / Date

Why BNC-B is needed?

A: 8-box full-mesh module-to-module connectivity
A connection from
one box to all other
boxes requires 7
10GE interfaces.
Two 10GE interfaces
are still required for
external connectivity,
e.g. to site router.

BCN-A has 6 10GE interfaces

=> 3 more 10GE interfaces are needed.
For internal use only
2
Nokia Siemens Networks

Presentation / Author / Date

BCN-B Summary of main changes

Changes to the Ethernet switching subsystem
The number of 10GE ports is increased from 6 to 9 to
enable interconnecting 8 BCN modules in full-mesh

configuration
provide two 10GE ports for customer use
A trace port is added for debugging
As the result of above changes the Ethernet switching
subsystem impacted by the following limitations
Add-in cards slot #1 has only one 10GE interface
AMC slot #1 no longer has a base interface
The number of external 1GE interfaces is reduced from
16 to 10
Changes to internal and external port numbering is kept at
minimum for the smallest possible SW impact
Power feed of add-in cards is increased from 90 W to 120 W
for better support of future CPU generations/technologies

For internal use only

3
Nokia Siemens Networks

Presentation / Author / Date

PCI

0/17
0/18
0/19
0/20
1GigE

SFP 22
SFP 21
SFP 20
SFP 19

0/28

0/19
0/20

AMC 2

0/27
0/26

0/17
0/18

0/15
0/16

0/13
0/14

0/13

0/11

SFP 16

0/10

0/8

SFP 13

0/7

SFP 12

0/6

SFP 11

0/5

SFP 10

0/4

SFP 9

0/3

SFP 8

0/2

SFP 7

0/1

xaui0
xaui1

xaui0
xaui1
Add-in Card 4

0/11
0/12

xaui0
xaui1
Add-in Card 5

0/9

SFP 14

xaui0
xaui1

Add-in Card 3

0/14

SFP 17

xaui0
xaui1

Add-in Card 2

0/15

SFP 18

MGT

Add-in Card 1

0/16

0/12

SFP 15

AMC 1

0/25

0/25
0/26

0/4
0/3

0/9
0/10

0/7
0/8

BCM56512
0/5
0/6

xaui0
xaui1
Add-in Card 6
xaui0
xaui1
Add-in Card 7
xaui0
xaui1
Add-in Card 8

UART
PCIe#0
mgmt0
USB
IPMB-L
PCIe#1
UART
PCIe#0
mgmt0
USB
IPMB-L

UART
PCIe#0
mgmt0
USB
IPMB-L

UART
PCIe#0
mgmt0
USB
IPMB-L

UART
PCIe#0
mgmt0
USB
IPMB-L

dtl0,dtl1

CPU

0/21
0/22
0/23
0/24

mgt0
mgt0.800

BCN-A HW
(current)

Local
Management
Processor

MAC

LAN1

MAC

LAN2

UART
PCIe#0
mgmt0
USB
IPMB-L

UART
PCIe#0
mgmt0
USB
IPMB-L

UART
PCIe#0
mgmt0
USB
IPMB-L
PCIe#1

BCM56820
0/2
0/1
0/24
0/23

SFP+ 6

SFI/XAUI

SFP+ 4

SFI/XAUI

0/22
0/21
For internal use only
4
Nokia Siemens Networks

SFP+ 5
SFP+ 3
SFP+ 2

SFI/XAUI

SFP+ 1

PCI

1GigE

SFP 22
SFP 21
SFP 20
SFP 19

AMC 1

CPU

0/25

0/19
0/26

P9
P10

P1

AMC 2

P8

P0

0/27

P9

0/17
0/18

P11

0/15
0/16

0/13
0/14

0/13

0/11

SFP 16

0/10

0/11
0/12

BCM56820

0/9

SFP 14

0/8

SFP 13

0/7

0/25
0/26

0/4
0/3

0/9
0/10

SFP+ 11

SFI/XAUI

xaui0
xaui1

0/5
0/6

0/24
Trace

0/28

0/23
0/22
0/21

For internal use only

5
Nokia Siemens Networks

0/20

Local
Management
BCN-B the xaui1
Processor

UART
PCIe#0
mgmt0
USB
IPMB-L

UART
PCIe#0
mgmt0
USB
IPMB-L

xaui0
xaui1

xaui0
xaui1

xaui0
xaui1
Add-in Card 8

0/2
0/1

UART
PCIe#0
mgmt0
USB
IPMB-L

UART
PCIe#0
mgmt0
USB
IPMB-L

Add-in Card 7

0/28
0/27

UART
PCIe#0
mgmt0
USB
IPMB-L
PCIe#1

In
interface
Add-in Card 3of slot#1 can only operate in
1000BASE-X serdes mode
xaui0
xaui1
=> The correctMAC
interface LAN1
mode
Add-in Card 4
has to be selected for Octeons
xaui0
cards using QLM_MODELAN2
and
xaui1
MAC
QLM_SPD pins
Add-in Card 5

Add-in Card 6
0/7
0/8

SFP+ 12

xaui0
xaui1
Add-in Card 2

P10

0/14BCM56514

SFP 17

xaui0
xaui1
Add-in Card 1

P0

0/15

0/12

MGT

P8

0/16

SFP 18

SFP 15

P1

dtl0,dtl1

0/21
0/22
0/23
0/24
0/6
0/17
0/18
0/19
0/20
0/5

mgt0
mgt0.800

BCN-B HW
(new)

UART
PCIe#0
mgmt0
USB
IPMB-L

UART
PCIe#0
mgmt0
USB
IPMB-L

UART
PCIe#0
mgmt0
USB
IPMB-L
PCIe#1

BCM56820:
XG0-XG23 <-> 0/1 - 0/24
GE0-GE3 <-> 028 - 0/25
BCN56514:
GE1-GE24 <-> 0/1 - 0/24
XG1-XG4 <-> 0/25 - 0/28

SFP+ 6

SFI/XAUI

SFP+ 5
SFP+ 4

SFI/XAUI

SFP+ 3
SFP+ 2

SFI/XAUI
SFI/XAUI

SFP+ 1
SFP+ 0

BCN-B front panel Ethernet port arrangement and

switch port mapping

SFP+0 SFP+2 SFP+4 SFP+6

0/20
0/22
0/24
0/2

SFP+12 SFP14 SFP16 SFP18 SFP20 SFP22

0/14
0/28
0/8
0/10
0/12
0/16

12

14

16

18

20

22

11

13

15

17

19

21

Trace SFP+1 SFP+3 SFP+5

0/28
0/21
0/23
0/1

Main switch

For internal use only

6
Nokia Siemens Networks

Presentation / Author / Date

SFP+11 SFP13 SFP15 SFP17 SFP19 SFP21

0/13
0/27
0/7
0/9
0/11
0/15

Extension switch

SLOT#1 XAUI1 interface control

Current support (manual control)
BOC-A, BMPP2-A
DIP switch is used to set correct interface mode to Octeon pins
(QLM_MODE[1:0], QLM_SPD[3:0])

BMPP2-B
Jumpers are used to set correct interface mode to Octeon pins
(QLM_MODE[1:0], QLM_SPD[3:0])

Planned support (SW control)

MMC detects slot# and automatically configures interface XAUI1 of
Octeon in to 1G mode (1000BASE-X serdes), when placed in slot#1.
When placed in other slots, XAUI1 is configured to 10G mode (XAUI)

For internal use only

7
Nokia Siemens Networks

Presentation / Author / Date

XAUI1 interface control: SW control on BMPP2-A

OCTEON

MMC
SEL

QLM_REF_CLK

100 MHz
156.25 MHz
MMC_QML_CLK_SEL

1
1
I/O
I/O

Required
signals for
SW control

VCMC
For internal use only
8
Nokia Siemens Networks

Presentation / Author / Date

GA[2:0]

I/O
IPMB-L

QLM_SPD0
QLM_SPD1
QLM_SPD2
QLM_SPD3

I/O

Set Card Config

QLM_MODE0
QLM_MODE1

Add-in card
System board
3-bit slot address

HW revision information
Info now available at VCMCs i/o (was at LMPs i/o)
Both Major and minor version information available, e.g. SW
is aware of BOM changes
MMC
SCL
SDA

For internal use only

9
Nokia Siemens Networks

Presentation / Author / Date

Other, smaller improvements

Powerup sequence improvements
Use of power-on HW reset for critical controls, such as card

activate/deactivate, MMC enable, MMC reset

SFP/SFP+ module interface reliability
Separated management bus with recovery mechanism a
faulty module can no longer jam the entire I2C subsystem of
LMP
Short circuit protection (poly-switch) for module power feed
a faulty module can no longer cause current trip to the 3V3
bus of the mother board
Overvoltage protection for LMPs serial port using a TVS
diode + series resistor)
Separate I2C management bus for PSUs
Ground contacts are added to the PSU control cable
connector (future option for EMI-improved cable)

For internal use only

10
Nokia Siemens Networks

Presentation / Author / Date

Mechanical changes
Front plate connector layout
addition of 3 10GE/1GE interfaces (SFP+0, SFP+11,

SFP+12)
addition of Trace port
removal of 4 1GE ports (SFP7, SFP8, SFP9, SFP10)
One fixing hole of mother board moved
also support pillar in the chassis is moved
=> mother board/ chassis versions can not be mixed!!
Inlet sensor assembly
To increase inlet air temperature measurement accuracy
and improve the operation of the cooling algorithm, the inlet
temperature sensors are moved away from the mother
board in to inlet air flow using a PCB + cable assembly
For internal use only
11
Nokia Siemens Networks

Presentation / Author / Date

Mechanical changes, cont.

Power Supply Unit inlet air duct
Target is to secure adequate amount of cooling air for the PSU by

separating a portion of the chassis inlet air opening for PSUs only (to be
verified by thermal tests)

New Heat sinks

Some of the heat sinks are modified for thermal and manufacturability
improvements

For internal use only

12
Nokia Siemens Networks

Presentation / Author / Date

BCN-B Front panel layout

For internal use only

13
Nokia Siemens Networks

Presentation / Author / Date

BCN-B front panel printing modifications

For internal use only

14
Nokia Siemens Networks

Presentation / Author / Date

Inlet air temperature sensor assembly

Sensors are located on a small PCB placed in inlet air flow
3 sensors are used for redundancy and SW compatibility

For internal use only

15
Nokia Siemens Networks

Presentation / Author / Date

PSU air guide to provide separated inlet

For internal use only

16
Nokia Siemens Networks

Presentation / Author / Date

BNC-B Mother board assembly

10G PHY
heat sinks
(3 pcs) are
slightly
higher.

Main switch heat sink

width increased for
improved cooling
performance.

One PCB
fixing hole
is moved
to make
room for
SFPs

For internal use only

17
Nokia Siemens Networks

LMP heat sink,

modified for better
manufacturability

Presentation / Author / Date

Motherboard layout, top view

BCM
56820

LSI
SAS
1064E

BCM
53212

PEX
8648

MPC
8545
BCM
8742

BCM
8742

BCM
56514

BCM
8726

For internal use only

18
Nokia Siemens Networks

Presentation / Author / Date

Thermal improvements
More accurate thermal control due to inlet sensor relocation
Heat sinks of some critical mother board components are

increased
Main switch base area doubled: 45x45 mm -> 60x70 mm
10G PHYs base area increased by 90 %: 18.5x16.5 mm
-> 24x24 mm, height doubled: 8.5 mm -> 17 mm
LMP a new design for better manufacturability, base area
increased by 80 %: 37.5x37.5 mm -> 50x50 mm
Higher rpm system fans taken in to use (3700 -> 4000 rpm)
AMC air flow is lowered (12000 -> 10000 rpm)
-> add-in card air flow is improved
Separated inlet air channel for the PSUs
To be verified by thermal testing. Decision to add air guide will be done based on test
results.

For internal use only

19
Nokia Siemens Networks

Presentation / Author / Date

Power feed improvements

Power feed capacity for add-in cards is increased from 90 W

to 120 W
The entire power feed path from site power feed to add-in
cards was analysed (current/power capacity, voltage drop)
Current capacity of copper layer areas and vias are
enforced in the lower right hand corner (PSU feed point) and
add-in card connectors
The number of power feed pins in the add-in card
connectors interface is increased from 9 to 12
Per slot circuit breaker trip point is increased
Power supply cage and output cable set need no
modifications
Load of a single PSU is at maximum, when 8 x 120 W add-in
cards are in use. When two PSUs are used, load of a single
PSU is ~50% (load sharing)
Note! Thermal testing of PSUs is required in order to
verify their performance under all environmental

For internal use only

20
Nokia Siemens Networks

Presentation / Author / Date

Cooling of higher power (> 90 W) add-in cards

NSNs site requirements for temperature
Recommended: +20+30 C
Normal: +5+40 C
Exceptional: -5+50 C

All BCN products to date comply with these specs

New add-in cards consuming > 90 W will have to be studied

case by case
Relaxed thermal specs may have to be considered

Full capacity is provided but ambient temperatures exceeding +40 C

are not allowed
If a specific temperature threshold (e.g. inlet air, component Tc) is
exceeded, capacity is reduced (e.g. throttling, speed step)

For internal use only

21
Nokia Siemens Networks

Presentation / Author / Date