MC-LAG

ACTIVE-ACTIVE V1.1

Peter Lundqvist, Ericsson Alliance

[email protected]
August 2013
LEGAL DISCLAIMER
This statement of product direction sets forth Juniper Networks' current
intention, and is subject to change at any time without notice. No
purchases are contingent upon Juniper Networks delivering any feature
or functionality depicted on this statement.

2 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

 BASICS
3 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net
ACRONYMS

What is “DC” regards networking in its basics ? WAN

Well a bunch of Blade/Virtual servers connected
with Switched Ethernet. WAN/Core

Acronyms used to describe the Nodes:

 WAN Nodes = Core layer, mostly dedicated routers
connectd the DC to a Core
 Spine Nodes = Distribution layer connects L2
Spine/
switches. The Spine layer is normally both L3 and
Distribution
L2 and is the aggregation layer between Core and
Access
MC-LAG
 Leaf Nodes = Access layer, smaller switches that MC-LAG
connect the Hosts in the DC. Most often L2 only and
are named as TOT (Top of rack) switches LAG LAG
Leaf/
Access

4 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

TRAFFIC PATTERN

Traffic pattern acronyms: WAN

 North-South
 East-West traffic WAN/Core

In its basic
 North-South are in/out of the Datacenter
 East-West are server to server traffic
Spine/
Distribution

MC-LAG
MC-LAG

LAG LAG
Leaf/
Access

5 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

LAG
Link Aggregation (LAG) is a way to increase WAN
bandwidth between nodes.
In brief it’s a bundle of physical Interfaces that WAN/Core

forms one logical interface. LAG are p2p by

its architecture
LAG can be either on L2 or L3 layer
Spine/
LAG are defined by 802.1ad Distribution

LAG LAG
Leaf/
Access

6 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

MC-LAG IN A NUTSHELL
Multi-Chassis Link Aggregation Group (MC-LAG)
provide LAG interfaces across dual homed chassis on WAN
the leaf layer
There are NO Virtual-chassi (VC) or similar running, WAN/Core
the leaf routers in a MC-LAG run individual
controlplanes
MC-LAG provides node level redundancy, multi-homing
support and a loop free Layer2 network without the
need for running any form of Spanning Tree Protocol ICL/
ICCP
Spine/
(STP, RSTP, MSTP etc…) Distribution

It uses Inter-Chassis Control Protocol (ICCP) to

exchange control information between MC-LAG nodes MC-LAG
MC-LAG
The Client node is unaware of the MC-LAG, it just
terminates physical links in a Link Aggregation Group LAG LAG
Leaf/
(LAG) Access
MC-LAG can be Active/Passive or Active/Active, this
presentation focus on the Active/Active scenario.

7 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

 MC-LAG ACTIVE/ACTIVE THEORY
8 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net
L2 CHALLENGES WITH DUALHOME LAG ?

Core Core
Link Link

Node 1 Node 2
ICL

ICC
P
MC-LAG

Single- LAG Single-

homed 1 homed 2

Access
Switch

Traffic received on Node 1 from MC-LAG could be flooded over the

Inter Chassi Link (ICL) to reach Node 2. Once it reaches Node 2, it
could be flooded back again to MC-LAG = Loop !

9 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

L2 CHALLENGES WITH DUALHOME LAG…

Core Core
Link Link

Node 1 Node 2
ICL

ICC
P
MC-LAG

Single- LAG Single-

homed 1 homed 2

Access
Switch

Another scenario is with traffic received on Single-homed 1 to be

flooded to MC-LAG and ICL towards Node 2. Once Node 2 receives
such traffic from ICL, it could be again flooded to MC-LAG = Traffic
duplication !

10 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

MC-LAG L2 FORWARDING RULES OVERVIEW
Core Core
ICCP exchanges L2 forwarding Link Link
information Node 2
Node 1 ICL
ICL protect connectivity in case of
ICC
failure of all links P
MC-LAG

Single- LAG Single-

homed 1 homed 2

Access
Following L2 forwarding rules apply: Switch

 Packets received on local MC-LAG/Single-homed link will follow L2 FIB forwarding

 MC-LAG routers will only forward traffic to the ICL link in case of Single-homed src or
Core/MC-LAG interfaces are down
 MC-LAG routers receiving packet on ICL will only forward traffic to Local links
(Core/single-homed). MC-LAG will only be used in case of MC-LAG is down on MC-
LAG peer router. ICCP notify peer about MC-LAG status

11 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

MC-LAG L3 OVERVIEW
Integrated Routing and Bridging (IRB) MC-LAG and IRB

 Provides integrated L2 and L3 forwarding

on a single interface
 Shared IP/MAC address across 2*chassis
 Essential for VM mobility
 Single ARP table across 2*chassis

ICCP is used to sync ARP and IRB MAC

address information
If VRRP is desired:
 VRRP enhancement on backup to provide
Active/Active forwarding Hitless VM mobility
Thanks to shared IRB IP/MAC address
 VRRP needed to sync MAC address across Load-balancing of traffic between
MC-LAG members
IRB peers

12 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

MC-LAG ACTIVE/ACTIVE SUMMARY
Available since JUNOS 11.1 (VRRP) and 11.2 (MAC-Sync IRB)
Supports both nodes in Active (forwarding) state
MC-LAG Active/Active supported with both L2 and L3
 VRRP Active/Active
 MAC-Sync IRB Active/Active
 ARP state-sync with MC-LAG Active/Active
 L2 forwarding info state-sync MC-LAG Active/Active

13 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

MC-LAG IGMP SNOOPING ENHANCEMENTS
Available with JUNOS 11.2
Configured on the bridge-domain associated with MC-Links
Expects PIM on the IRB interface – only one Router will be the
DR
 Reports to be replicated to the peer router, forwarding to be done
only by one

14 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

MC-LAG CAVEATS
Requires Trio/Cassis chipset based MPCs
Does not support the following
 VPLS Active/Active (VPLS demands Active/Passive)
 Spanning Tree
 For L2 forwarding, Bridge Domains of type vlan-id all, none and
tags
 LSYS support on non-default LSYS instances

ISSU support: 11.2

15 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

 PROTOCOL THEORY
LACP, ICCP, ICL…
16 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net
LAG/LACP
Link Aggregation (LAG) is a way to increase bandwidth between
nodes. In brief it’s a bundle of physical Interfaces that forms one
logical interface. LAG are p2p by its architecture
Link Aggregation Control Protocol (LACP) are commonly used
with LAG. LAG/LACP are defined by 802.1ad.
LACP main functions are
 Automatic addition and deletion of individual links to the aggregate
bundle without user intervention
 Link monitoring to check whether both ends of the bundle are
connected to the correct group

17 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

 LACP MESSAGES
[…]
Sep 24 15:06:54 lacpd_ppm_start_periodic_send: iflidx = 386
Sep 24 15:06:54 lacpd_ppm_start_periodic_send: queued request on ppm_program_thread
Sep 24 15:06:54 lacpd_kick_ppm_job: kick starting ppm job
Sep 24 15:06:54 lacpd_ppm_job: start
Sep 24 15:06:54 lacpd_ppm_job: start periodic send, iflidx = 386
Sep 24 15:06:54 Transmitting LACP pdu - interface xe-0/0/0
Sep 24 15:06:54 lacp subtype 0x1 lacp version number 0x1
Sep 24 15:06:54 first tlv type 0x1 actor info len 0x14
Sep 24 15:06:54 actor sys priority 0x7f actor sys 00:00:00:00:00:01
Sep 24 15:06:54 actor key 0x1 actor port priority 0x7f actor port 0x8001 actor state 0x3f
Sep 24 15:06:54 second tlv type 0x2 partner info len 0x14
Sep 24 15:06:54 partner sys priority 0x7f partner sys b0:c6:9a:d8:c9:00
Sep 24 15:06:54 partner key 0x1 partner port priority 0x7f partner port 0x2 partner state 0x3f
Sep 24 15:06:54 third tlv type 0x3 collector info len 0x10 collector max del 0x0 fourth tlv type 0x0 terminator len 0
Sep 24 15:06:54 lacpd_ppm_rmt_program_send: Programming ppmd for ifl index: 386
Sep 24 15:06:54 lacpd_ppm_job: port xe-0/0/0 PPM_DEQ_REQ
[…]

lunkan@mx_2> show lacp interfaces ae1

Aggregated interface: ae1
LACP state: Role Exp Def Dist Col Syn Aggr Timeout Activity
xe-0/0/0 Actor No No Yes Yes Yes Yes Fast Active
xe-0/0/0 Partner No No Yes Yes Yes Yes Fast Active
LACP protocol: Receive State Transmit State Mux State
xe-0/0/0 Current Fast periodic Collecting distributing

18 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

INTER-CHASSIS CONTROL PROTOCOL (ICCP)
The most important part of a MC-LAG is ICCP.
Without ICCP, MC-LAG would be just a LAG without any ability to
terminate a LAG from a Leaf node into 2 separate Spine nodes without
exessive usage of Loop detection protocols like Spanning-tree etc…
ICCP session exchange controlplane information between the MC-LAG
pairs:
Syncronisation of L2 FIB (MAC addresses)
ARP and VRRP state information
Etc…

19 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

ICCPD
ICCP is a TCP session between Spine pairs in a MC-LAG, and usually
loopback<>loopback addressing
WAN
ICCPD is a new daemon
lunkan@mx_1> show system processes extensive | grep iccp
1396 root 1 96 0 6116K 2660K select 0:00 0.00% iccpd

[…]
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|M|x|x|x| Ver | Message Type |Message Length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|Message Length | MSG-ID | 1.5.1.1 1.5.1.2
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ ICC
| Redundancy Group Id | P
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
[…]
MC-LAG
MC-LAG

20 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

ICCP TCP SESSION
lunkan@mx_1> show system connections inet extensive
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address
tcp4 0 0 1.5.1.1.33012 1.5.1.2.54842 WAN
ESTABLISHED
sndsbcc: 0 sndsbmbcnt: 0 sndsbmbmax: 4194304
sndsblowat: 2048 sndsbhiwat: 524288
rcvsbcc: 0 rcvsbmbcnt: 0 rcvsbmbmax: 4194304
rcvsblowat: 1 rcvsbhiwat: 524288
proc id: 0 proc name:
iss: 4011561344 sndup: 4011712781
snduna: 4011712883 sndnxt: 4011712883 sndwnd: 131070
sndmax: 4011712883 sndcwnd: 8192 sndssthresh: 1073725440
irs: 1413551117 rcvup: 1413551686
rcvnxt: 1413551686 rcvadv: 1413682756 rcvwnd: 524288
rtt: 0 srtt: 3203 rttv: 9
rxtcur: 1200 rxtshift: 0 rtseq: 4011712781 1.5.1.1 1.5.1.2
rttmin: 1000 mss: 4096 ICC
[…]
P
lunkan@mx_1> show iccp detail
Redundancy Group Information for peer 1.5.1.2 MC-LAG
TCP Connection : Established MC-LAG
Liveliness Detection : Up
Redundancy Group ID Status
1 Up

Client Application: lacpd

Redundancy Group IDs Joined: 1

Client Application: l2ald_iccpd_client

Redundancy Group IDs Joined: 1

21 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

ICCP UPDATE MC-LAG PAIR WITH NEW L2 ENTRY
[edit]
lunkan@mx_1# run show bridge mac-table
WAN
MAC flags (S -static MAC, D -dynamic MAC, L -locally learned
SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC)

Routing instance : default-switch

Bridging domain : mclag_1, VLAN : 11
MAC MAC Logical
address flags interface
00:07:e9:13:a6:3f DL ae1.0

[edit]
lunkan@mx_2# run show bridge mac-table DR
DL
MAC flags (S -static MAC, D -dynamic MAC, L -locally learned ICC
SE -Statistics enabled, NM -Non configured MAC, R -Remote PE MAC) P
Routing instance : default-switch
Bridging domain : mclag_1, VLAN : 11 MC-LAG
MAC MAC Logical MC-LAG
address flags interface
00:07:e9:13:a6:3f DR ae1.0

00:07:e9:13:a6:3f
1.5.0.222

22 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

INTER CHASSIS LINK (ICL)
The ICL is playing a crucial role in Active/Active scenarios
 Loop-avoidance interaction
 Providing L2 connectivity for single-homed nodes
 Required for multicast over MC-LAG example VRRP, ISIS
hello/advertisment

C- C-
Link Link
N1 ICC N2
P
ICL
MC-LAG

Single- LAG Single-

homed 1 homed 2

Access
Switch

23 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

 MC-LAG IRB/VRRP EXAMPLE
24 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net
ICP/ICCP LAG
[…]
ge-1/3/9 {
gigether-options {
802.3ad ae0;
[…]
ge-1/3/10 {
gigether-options {
802.3ad ae0;
[…]
ae0 {
description "ICL/ICCP LAG between MCLAG peers";
flexible-vlan-tagging;
mtu 4472;
encapsulation flexible-ethernet-services;
aggregated-ether-options { ge-1/3/9
minimum-links 1;
ae0
link-speed 1g;
lacp { ge-1/3/10
active;
periodic fast;
[…]
unit 0 { ARP, VRRP hello etc…
description ICL;
encapsulation vlan-bridge; MC-LAG
vlan-id 11;
family bridge;
}
unit 1 { ICCP messages etc…
description ICCP;
vlan-id 111;
family inet {
address 1.5.5.5/30; 00:07:e9:13:a6:3f
1.5.0.222
}
family iso;
[…]
25 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net
ICCP/ICL OAM
lunkan@mx_1# show interfaces
lo0 {
unit 0 {
family inet {
address 1.5.1.1/32;
}
family iso {
address 49.0000.0010.0500.1001.00;
[…]

lunkan@mx_2# show protocols

iccp {
local-ip-addr 1.5.1.2; lo0 lo0
peer 1.5.1.1 { 1.5.1.1/32 1.5.1.2/32
redundancy-group-id-list 1; IP OAM ICCP session ge-1/3/9
liveness-detection {
ae0
minimum-interval 100;
multiplier 3; ge-1/3/10
[…]
oam {
ethernet {
connectivity-fault-management { Eth OAM ICL Link
maintenance-domain ICL {
level 0; MC-LAG
maintenance-association iclx {
continuity-check {
interval 100ms;
}
mep 2 {
interface ae0.0;
direction down;
auto-discovery; 00:07:e9:13:a6:3f
[…] 1.5.0.222

26 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

MC-LAG NODE 1
[…]
interfaces {
xe-0/0/0 {
gigether-options {
802.3ad ae1;
[…]
ae1 {
description MCLAG;
flexible-vlan-tagging;
encapsulation flexible-ethernet-services;
aggregated-ether-options {
lacp {
lo0 lo0
active; 1.5.1.2/32
1.5.1.1/32
system-id 00:00:00:00:00:01;
System-id/admin-key needs to ge-1/3/9
admin-key 1;
be the Same on both peers
} ae0
mc-ae {
Mc-lag-id/redundancy-group ge-1/3/10
mc-ae-id 1;
redundancy-group 1; Needs to be the same on both peers
xe-0/0/0
chassis-id 0; xe-0/0/0
mode active-active;
status-control active;
events {
iccp-peer-down { MC-LAG
ae1

force-icl-down; VID 11
[…]
unit 0 {
encapsulation vlan-bridge;
vlan-id 11;
multi-chassis-protection 1.5.1.2 {
interface ae0.0; 00:07:e9:13:a6:3f
1.5.0.222

27 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

MC-LAG NODE 2
[…]
interfaces {
xe-0/0/0 {
gigether-options {
802.3ad ae1;
[…]
ae1 {
description MCLAG;
flexible-vlan-tagging;
encapsulation flexible-ethernet-services;
aggregated-ether-options {
lacp {
lo0 lo0
active; 1.5.1.2/32
1.5.1.1/32
system-id 00:00:00:00:00:01;
System-id/admin-key needs to ge-1/3/9
admin-key 1;
be the Same on both peers
} ae0
mc-ae {
Mc-lag-id/redundancy-group ge-1/3/10
mc-ae-id 1;
redundancy-group 1; Needs to be the same on both peers
xe-0/0/0
chassis-id 1; xe-0/0/0
mode active-active;
status-control active;
events {
iccp-peer-down { MC-LAG
ae1

force-icl-down; VID 11
[…]
unit 0 {
encapsulation vlan-bridge;
vlan-id 11;
multi-chassis-protection 1.5.1.1 {
interface ae0.0;
00:07:e9:13:a6:3f
1.5.0.222

28 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

MC-LAG STATE
[edit]
lunkan@mx_1# run show interfaces mc-ae

[edit]
lunkan@mx80_1# run show interfaces mc-ae
Member Link : ae1
Current State Machine's State: mcae active state
Local Status : active
Local State : up
Peer Status : active
Peer State : up
Logical Interface : ae1.0 lo0 lo0
Topology Type : bridge 1.5.1.1/32 1.5.1.2/32
Local State : up ge-1/3/9
Peer State : up
ae0
Peer Ip/MCP/State : 1.5.1.2 ae0.0 up
ge-1/3/10
[edit]
lunkan@mx_2# run show interfaces mc-ae xe-0/0/0
Member Link : ae1 xe-0/0/0
Current State Machine's State: mcae active state
Local Status : active
Local State : up MC-LAG
ae1
Peer Status : active
Peer State : up
Logical Interface : ae1.0
Topology Type : bridge
Local State : up
Peer State : up
Peer Ip/MCP/State : 1.5.1.1 ae0.0 up
00:07:e9:13:a6:3f
1.5.0.222

29 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

MC-LAG STATE…
lunkan@mx_2> show interfaces ae1
Physical interface: ae1 (MC-AE-1, active), Enabled, Physical link is Up
Interface index: 129, SNMP ifIndex: 567
Description: MCLAG
Link-level type: Flexible-Ethernet, MTU: 1522, Speed: 10Gbps, BPDU Error: None,
MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
Flow control: Disabled, Minimum links needed: 1, Minimum bandwidth needed: 0
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x0
Current address: a8:d0:e5:5d:37:c1, Hardware address: a8:d0:e5:5d:37:c1
Last flapped : 2013-10-01 20:51:08 CEST (00:16:40 ago)
Input rate : 880 bps (1 pps) lo0 lo0
Output rate : 1040 bps (1 pps) 1.5.1.1/32 1.5.1.2/32

ge-1/3/9

ae0
lunkan@mx_1> show interfaces ae1
Physical interface: ae1 (MC-AE-1, active), Enabled, Physical link is Up ge-1/3/10
Interface index: 129, SNMP ifIndex: 608
Description: MCLAG xe-0/0/0 xe-0/0/0
Link-level type: Flexible-Ethernet, MTU: 1522, Speed: 10Gbps,
BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled,
Source filtering: Disabled, Flow control: Disabled, Minimum links needed: 1,
Minimum bandwidth needed: 0 MC-LAG
ae1
Device flags : Present Running
Interface flags: SNMP-Traps Internal: 0x0
Current address: a8:d0:e5:5d:2d:c1, Hardware address: a8:d0:e5:5d:2d:c1
Last flapped : 2013-10-01 20:51:33 CEST (00:18:17 ago)
Input rate : 880 bps (1 pps)
Output rate : 0 bps (0 pps)

00:07:e9:13:a6:3f
1.5.0.222

30 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

BRIDGE/IRB AND VRRP LOGIC NODE 1
interfaces {
[…]
irb {
unit 11 {
family inet {
address 1.5.0.1/24 {
vrrp-group 1 {
virtual-address 1.5.0.254;
priority 100;
fast-interval 100;
preempt;
ge-1/0/0
accept-data; lo0 lo0
} 1.5.1.1/32 1.5.1.2/32
} ge-1/3/9
inactive: address 1.5.0.253/24;
} xe-0/0/0 ae0
family iso; ge-1/3/10
[…]
bridge-domains { xe-0/0/0 VIP 1.5.0.254
mclag_1 { VID 11
xe-0/0/0
L3
vlan-id 11;
Forwarding
inactive: service-id 1;
inactive: mcae-mac-synchronize; MC-LAG
ae1
interface ae0.0; L2
interface ae1.0; Forwarding
routing-interface irb.11;
}
Selective
}
L2 Forwarding
switch-options {
service-id 11;
} 00:07:e9:13:a6:3f
Service-id configured in global stanza since MC-LAG interfaces in the main instance. 1.5.0.222
If Bridging to be run in instance then service-id to be configured there.

31 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

BRIDGE/IRB AND VRRP LOGIC NODE 2
interfaces {
[…]
irb {
unit 11 {
family inet {
address 1.5.0.2/24 {
vrrp-group 1 {
virtual-address 1.5.0.254;
priority 100;
fast-interval 100;
preempt;
accept-data; ge-1/0/0
lo0 lo0
} 1.5.1.1/32 1.5.1.2/32
}
ge-1/3/9
inactive: address 1.5.0.253/24;
} xe-0/0/0 ae0
family iso;
ge-1/3/10
[…]
bridge-domains {
mclag_1 {
xe-0/0/0 VIP 1.5.0.254
xe-0/0/0
vlan-id 11; L3 VID 11
inactive: service-id 1; Forwarding
inactive: mcae-mac-synchronize;
interface ae0.0;
MC-LAG
ae1
L2
interface ae1.0; Forwarding
routing-interface irb.11;
}
} Selective
switch-options { L2 Forwarding
service-id 11;
}
00:07:e9:13:a6:3f
Service-id configured in global stanza since MC-LAG interfaces is in the main instance. 1.5.0.222
If Bridging to be run in instance then service-id to be configured there.

32 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

THIS SERVICE-ID THING
The service-id object at bridge domain level overrides any global
switch-options config for the bridge domain
Both MC-LAG nodes share SAME service-id for given bridge-domain

Service-id used to synchronize applications like IGMP, ARP, MAC

learning for a given service across MC-LAG peers.
[…]
bridge-domains {
mclag_1 {
vlan-id 11;
service-id 1;
inactive: mcae-mac-synchronize;
interface ae0.0;
interface ae1.0;
routing-interface irb.11;
}
}
switch-options {
service-id 11;
}
[…]

33 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

BASIC VRRP, ONLY ONE RESPOND TO TRAFFIC
[edit]
lunkan@mx_2# run show vrrp
Interface State Group VR state VR Mode Timer Type Address
ae0.99 up 1 master Active A 0.420 lcl 1.2.3.2
vip 1.2.3.254

lunkan@mx_2# run show route forwarding-table family inet extensive | find 1.2.3.254

Destination: 1.2.3.254/32
Route type: interface
Route reference: 0 Route interface-index: 0
Flags: sent to PFE
Nexthop: 1.2.3.254
Next-hop type: local Index: 1410 Reference: 2
[…]

[edit]
lunkan@mx_1# run show vrrp
Interface State Group VR state VR Mode Timer Type Address
ae0.99 up 1 backup Active D 3.013 lcl 1.2.3.1
vip 1.2.3.254
mas 1.2.3.2
[edit]
lunkan@mx_1# run show route forwarding-table family inet extensive | find 1.2.3.254

Destination: 1.2.3.254/32
Route type: destination
Route reference: 0 Route interface-index: 329
Flags: sent to PFE
Nexthop: 0:0:5e:0:1:1
Next-hop type: unicast Index: 642 Reference: 1
Next-hop interface: ae0.99

34 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

MC-LAG VRRP, BOTH NODES ACTIVE
lunkan@mx_2# run show vrrp
Interface State Group VR state VR Mode Timer Type Address
irb.11 up 1 master Active A 0.059 lcl 1.5.0.2
vip 1.5.0.254

lunkan@mx_2# run show route forwarding-table family inet extensive | find 1.5.0.254

Destination: 1.5.0.254/32
Route type: interface
Route reference: 0 Route interface-index: 0
Flags: sent to PFE
Nexthop: 1.5.0.254
Next-hop type: local Index: 1350 Reference: 2
[…]

lunkan@mx_1# run show vrrp

Interface State Group VR state VR Mode Timer Type Address
irb.11 up 1 backup Active D 0.264 lcl 1.5.0.1
vip 1.5.0.254

lunkan@mx_1# run show route forwarding-table family inet extensive | find 1.5.0.254

Destination: 1.5.0.254/32
Route type: interface
Route reference: 0 Route interface-index: 0
Flags: sent to PFE
Nexthop: 1.5.0.254
Next-hop type: local Index: 608 Reference: 2

35 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

L3 ROUTING LOGIC
[…]
protocols { L3 Adjency
isis {
level 1 disable;
interface ge-1/0/0.0 {
point-to-point; L3 Passive
}
interface ae0.1 {
point-to-point;
}
inactive: interface all;
interface irb.11 { ge-1/0/0
passive; lo0 lo0
1.5.1.1/32 1.5.1.2/32
}
interface lo0.0; ge-1/3/9
} xe-0/0/0 ae0.1
[…]
ge-1/3/10

[edit] xe-0/0/0 VIP 1.5.0.254

lunkan@mx_1# run show isis adjacency xe-0/0/0
VID 11
Interface System L State Hold ( secs) SNPA
ae0.1 mx80_2 2 Up 23
ge-1/0/0.0 mx-re0 2 Up 20 MC-LAG
ae1.11

[edit]
lunkan@mx_1# run show isis interface
IS-IS interface database:
Interface L CirID Level 1 DR Level 2 DR L1/L2 Metric
ae0.1 2 0x1 Disabled Point to Point 10/10
ge-1/0/0.0 2 0x1 Disabled Point to Point 10/10
irb.11 0 0x1 Passive Passive 10/10 00:07:e9:13:a6:3f
lo0.0 0 0x1 Passive Passive 0/0 1.5.0.222

36 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

 TEST MC-LAG VRRP
37 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net
TRAFFIC TEST
1.5.4.111
oskar# iperf -s -u -p 55555
------------------------------------------- -----------------
Server listening on UDP port 55555
Receiving 1470 byte datagrams
UDP buffer size: 41.1 KByte (default)
------------------------------------------- -----------------
[ 3] local 1.5.4.111 port 55555 connected with 1.5.0.222 port 50161
[ 3] 0.0-10.0 sec 29.8 MBytes 25.0 Mbit s/sec 0.059 ms 20/21277 ( 0.094%)
[ 3] 0.0-10.0 sec 1 datagrams received o ut-of-order

[lunkan@#localhost ~]$ sudo iperf -c 1.5.4. 111 -u -p 55555 -b 25m

------------------------------------------- -----------------
Client connecting to 1.5.4.111, UDP port 55 555
Sending 1470 byte datagrams Src 1.5.0.222
UDP buffer size: 176 KByte (default)
------------------------------------------- ----------------- Dst 1.5.4.111
[ 3] local 1.5.0.222 port 50161 connected with 1.5.4.111 port 55555
[ ID] Interval Transfer Bandwidth
UDP Dstport 55555
[ 3] 0.0-10.0 sec 29.8 MBytes 25.0 Mbit s/sec
[ 3] Sent 21278 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 29.8 MBytes 25.0 Mbit s/sec 0.059 ms 20/21277 (0.094%) ge-1/0/0
[ 3] 0.0-10.0 sec 1 datagrams received o ut-of-order
ge-1/0/0
[…]

ae0
oskar# iperf -s -u -p 66666
------------------------------------------- -----------------
Server listening on UDP port 1130
Receiving 1470 byte datagrams
UDP buffer size: 41.1 KByte (default) xe-0/0/0 xe-0/0/0
------------------------------------------- -----------------
[ 3] local 1.5.4.111 port 1130 connected w ith 1.5.0.222 port 55244
[ 3] 0.0-10.0 sec 29.8 MBytes 25.0 Mbit s/sec 0.022 ms 3/21277 ( 0.014%)
[ 3] 0.0-10.0 sec 1 datagrams received o ut-of-order ae1 ae1
[lunkan@#localhost ~]$ iperf -c 1.5.4.111 - u -p 66666 -b 25m
------------------------------------------- -----------------
Client connecting to 1.5.4.111, UDP port 11 30
Sending 1470 byte datagrams MC-LAG
UDP buffer size: 176 KByte (default) Src 1.5.0.222
------------------------------------------- -----------------
[ 3] local 1.5.0.222 port 55244 connected with 1.5.4.111 port 1130 Dst 1.5.4.111
[ ID] Interval Transfer Bandwidth UDP Dstport 66666
[ 3] 0.0-10.0 sec 29.8 MBytes 25.0 Mbit s/sec
[ 3] Sent 21278 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 29.8 MBytes 25.0 Mbit s/sec 0.022 ms 3/21277 (0.014%)
[ 3] 0.0-10.0 sec 1 datagrams received o ut-of-order
[…]
00:07:e9:13:a6:3f
1.5.0.222

38 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

TRAFFIC TEST…
1.5.4.111
lunkan@mx_1> show interfaces ae1.0 extensive
Logical interface ae1.0 (Index 332) (SNMP ifIndex 617) (Generation 141)
Flags: SNMP-Traps 0x0 VLAN-Tag [ 0x8100.11 ] Encapsulation: VLAN-Bridge
Statistics Packets pps Bytes bps
Bundle:
Input : 21280 0 32259028 0
Output: 5 0 1746 0
Link:
xe-0/0/0.0
Input : 21280 0 32259028 0
Output: 5 0 1746 0
Marker Statistics: Marker Rx Resp Tx Unknown Rx Illegal Rx
xe-0/0/0.0 0 0 0 0
Protocol bridge, MTU: 1522, Generation: 159, Route table: 3 ge-1/0/0
ge-1/0/0
[…]

ae0
lunkan@mx_2> show interfaces ae1.0 extensive
Logical interface ae1.0 (Index 383) (SNMP ifIndex 572) (Generation 192)
Flags: SNMP-Traps 0x0 VLAN-Tag [ 0x8100.11 ] Encapsulation: VLAN-Bridge
Statistics Packets pps Bytes bps xe-0/0/0 xe-0/0/0
Bundle:
Input : 21280 0 32260480 0
ae1 ae1
Output: 812 11 80264 5664
Link:
xe-0/0/0.0
Input : 21280 0 32260480 0 MC-LAG
Output: 812 11 80264 5664
Marker Statistics: Marker Rx Resp Tx Unknown Rx Illegal Rx
xe-0/0/0.0 0 0 0 0
Protocol bridge, MTU: 1522, Generation: 294, Route table: 3
[…]

00:07:e9:13:a6:3f
1.5.0.222

39 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

 MC-LAG IRB/MAC-SYNC EXAMPLE
40 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net
MC-LAG IRB MAC-SYNC
A common L3 Default Gateway hosts that
is shared between MC-LAG nodes
 Same IRB IP for both MC-LAG nodes
 Lowest MAC address will be the Default
Gateway MAC address ge-1/0/0
lo0
lo0

 The MC-LAG node with higher IRB MAC 1.5.1.1/32 1.5.1.2/32

ge-1/3/9

will install the peer MC-LAG node as its xe-0/0/0 ae0

MAC address ge-1/3/10

xe-0/0/0
 MAC-SYNCs gain is that it takes away Defaultgateway
1.5.0.253
xe-0/0/0

the possible scale challenge with massive VID 11

MC-LAG
ae1
volume of VRRP states

41 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

ICP/ICCP LAG
[…]
ge-1/3/9 {
gigether-options {
802.3ad ae0;
[…]
ge-1/3/10 {
gigether-options {
802.3ad ae0;
[…]
ae0 {
description "ICL/ICCP LAG between MCLAG peers";
flexible-vlan-tagging;
mtu 4472;
encapsulation flexible-ethernet-services;
aggregated-ether-options { ge-1/3/9
minimum-links 1;
ae0
link-speed 1g;
lacp { ge-1/3/10
active;
periodic fast;
[…]
unit 0 {
description ICL;
encapsulation vlan-bridge; MC-LAG
vlan-id 11;
family bridge;
}
unit 1 {
description ICCP;
vlan-id 111;
family inet {
address 1.5.5.5/30;
}
family iso;
[…]
42 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net
ICL/ICCP LOGIC/OAM
lunkan@mx_1# show interfaces
lo0 {
unit 0 {
family inet {
address 1.5.1.1/32;
}
family iso {
address 49.0000.0010.0500.1001.00;
[…]

lunkan@mx_2# show protocols

iccp {
local-ip-addr 1.5.1.2; lo0 lo0
peer 1.5.1.1 { 1.5.1.1/32 1.5.1.2/32
redundancy-group-id-list 1; ge-1/3/9
liveness-detection {
ae0
minimum-interval 100;
multiplier 3; ge-1/3/10
[…]
oam {
ethernet {
connectivity-fault-management {
maintenance-domain ICL {
level 0; MC-LAG
maintenance-association iclx {
continuity-check {
interval 100ms;
}
mep 2 {
interface ae0.0;
direction down;
auto-discovery;
[…]

43 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

MC-LAG NODE 1
[…]
interfaces {
xe-0/0/0 {
gigether-options {
802.3ad ae1;
[…]
ae1 {
description MCLAG;
flexible-vlan-tagging;
encapsulation flexible-ethernet-services;
aggregated-ether-options {
lacp {
lo0 lo0
active; 1.5.1.2/32
1.5.1.1/32
system-id 00:00:00:00:00:01;
admin-key 1; ge-1/3/9
} ae0
mc-ae {
ge-1/3/10
mc-ae-id 1;
redundancy-group 1;
xe-0/0/0
chassis-id 0; xe-0/0/0
mode active-active;
status-control active;
events {
iccp-peer-down { MC-LAG
ae1

force-icl-down; VID 11
[…]
unit 0 {
encapsulation vlan-bridge;
vlan-id 11;
multi-chassis-protection 1.5.1.2 {
interface ae0.0;

44 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

MC-LAG NODE 2
[…]
interfaces {
xe-0/0/0 {
gigether-options {
802.3ad ae1;
[…]
ae1 {
description MCLAG;
flexible-vlan-tagging;
encapsulation flexible-ethernet-services;
aggregated-ether-options {
lacp {
lo0 lo0
active; 1.5.1.2/32
1.5.1.1/32
system-id 00:00:00:00:00:01;
admin-key 1; ge-1/3/9
} ae0
mc-ae {
ge-1/3/10
mc-ae-id 1;
redundancy-group 1;
xe-0/0/0
chassis-id 1; xe-0/0/0
mode active-active;
status-control active;
events {
iccp-peer-down { MC-LAG
ae1

force-icl-down; VID 11
[…]
unit 0 {
encapsulation vlan-bridge;
vlan-id 11;
multi-chassis-protection 1.5.1.1 {
interface ae0.0;

45 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

MC-LAG STATE
[edit]
lunkan@mx_1# run show interfaces mc-ae

[edit]
lunkan@mx80_1# run show interfaces mc-ae
Member Link : ae1
Current State Machine's State: mcae active state
Local Status : active
Local State : up
Peer Status : active
Peer State : up
Logical Interface : ae1.0 lo0 lo0
Topology Type : bridge 1.5.1.1/32 1.5.1.2/32
Local State : up ge-1/3/9
Peer State : up
ae0
Peer Ip/MCP/State : 1.5.1.2 ae0.0 up
ge-1/3/10
[edit]
lunkan@mx_2# run show interfaces mc-ae xe-0/0/0
Member Link : ae1 xe-0/0/0
Current State Machine's State: mcae active state
Local Status : active
Local State : up MC-LAG
ae1
Peer Status : active
Peer State : up
Logical Interface : ae1.0
Topology Type : bridge
Local State : up
Peer State : up
Peer Ip/MCP/State : 1.5.1.1 ae0.0 up

46 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

BRIDGE/IRB LOGIC NODE 1
interfaces {
[…]
irb {
unit 11 {
family inet {
address 1.5.0.253/24;
}
family iso;
[…]
bridge-domains {
mclag_1 {
ge-1/0/0
vlan-id 11; lo0 lo0
inactive: service-id 1; 1.5.1.1/32 1.5.1.2/32
mcae-mac-synchronize; ge-1/3/9
interface ae0.0;
interface ae1.0; xe-0/0/0 ae0
routing-interface irb.11; ge-1/3/10
}
} xe-0/0/0 Defaultgateway
switch-options { 1.5.0.253
xe-0/0/0
L3
service-id 11; VID 11
Forwarding
}
MC-LAG
ae1
L2
Forwarding

Selective
L2 Forwarding

47 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

BRIDGE/IRB LOGIC NODE 2
interfaces {
[…]
irb {
unit 11 {
family inet {
address 1.5.0.253/24;
}
family iso;
[…]
bridge-domains {
mclag_1 {
ge-1/0/0
vlan-id 11; lo0 lo0
inactive: service-id 1; 1.5.1.1/32 1.5.1.2/32
mcae-mac-synchronize; ge-1/3/9
interface ae0.0;
interface ae1.0; xe-0/0/0 ae0
routing-interface irb.11; ge-1/3/10
}
} xe-0/0/0 Defaultgateway
switch-options { 1.5.0.253
xe-0/0/0
L3
service-id 11; VID 11
Forwarding
}
MC-LAG
ae1
L2
Forwarding

Selective
L2 Forwarding

48 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

 L3 ROUTING LOGIC
[…]
protocols { L3 Adjency
isis {
level 1 disable;
interface ge-1/0/0.0 {
L3 Passive
point-to-point;
}
interface ae0.1 {
point-to-point;
}
inactive: interface all;
interface irb.11 { ge-1/0/0
lo0 lo0
passive; 1.5.1.2/32
1.5.1.1/32
}
interface lo0.0; ge-1/3/9
} xe-0/0/0 ae0.1
[…]
ge-1/3/10
[edit]
lunkan@mx_1# run show isis adjacency xe-0/0/0 VIP 1.5.0.254
xe-0/0/0
Interface System L State Hold (secs) SNPA VID 11
ae0.1 mx80_2 2 Up 23
ge-1/0/0.0 mx-re0 2 Up 20
MC-LAG
ae1.11
[edit]
lunkan@mx_1# run show isis interface
IS-IS interface database:
Interface L CirID Level 1 DR Level 2 DR L1/L2 Metric
ae0.1 2 0x1 Disabled Point to Point 10/10
ge-1/0/0.0 2 0x1 Disabled Point to Point 10/10
irb.11 0 0x1 Passive Passive 10/10
lo0.0 0 0x1 Passive Passive 0/0

49 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

 TEST MC-LAG MAC-SYNC
50 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net
TRAFFIC TEST
1.5.4.111
[…]
oskar# iperf -s -p 5000 -u
------------------------------------------- -----------------
Server listening on UDP port 5000
Receiving 1470 byte datagrams
UDP buffer size: 41.1 KByte (default)
------------------------------------------- -----------------
[ 3] local 1.5.4.111 port 5000 connected w ith 1.5.0.222 port 48874
[ 3] 0.0-10.0 sec 11.9 MBytes 10.0 Mbit s/sec 0.029 ms 0/ 8504 ( 0%)
[ 3] 0.0-10.0 sec 1 datagrams received o ut-of-order

[lunkan@#localhost ~]$ iperf -c 1.5.4.111 - u -p 5000 -b 10m

------------------------------------------- -----------------
Client connecting to 1.5.4.111, UDP port 50 00 Src 1.5.0.222
Sending 1470 byte datagrams
UDP buffer size: 176 KByte (default) Dst 1.5.4.111
------------------------------------------- -----------------
[ 3] local 1.5.0.222 port 48874 connected with 1.5.4.111 port 5000
UDP Dstport 5000
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 11.9 MBytes 10.0 Mbit s/sec
[ 3] Sent 8505 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 11.9 MBytes 10.0 Mbit s/sec 0.029 ms 0/ 8504 (0%)
[ 3] 0.0-10.0 sec 1 datagrams received o ut-of-order
[…]
ae0
[…]
oskar# iperf -s -u -p 50000
------------------------------------------- -----------------
Server listening on UDP port 50000
Receiving 1470 byte datagrams xe-0/0/0
UDP buffer size: 41.1 KByte (default) xe-0/0/0
------------------------------------------- -----------------
[ 3] local 1.5.4.111 port 50000 connected with 1.5.0.222 port 44426
[ 3] 0.0-10.0 sec 11.9 MBytes 10.0 Mbit s/sec 0.019 ms 0/ 8504 ( 0%) ae1 ae1
[ 3] 0.0-10.0 sec 1 datagrams received o ut-of-order

[lunkan@#localhost ~]$ sudo iperf -c 1.5.4. 111 -u -p 50000 -b 10m

------------------------------------------- -----------------
Client connecting to 1.5.4.111, UDP port 50 000 MC-LAG
Sending 1470 byte datagrams Src 1.5.0.222
UDP buffer size: 176 KByte (default)
------------------------------------------- ----------------- Dst 1.5.4.111
[ 3] local 1.5.0.222 port 44426 connected with 1.5.4.111 port 50000 UDP Dstport 50000
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 11.9 MBytes 10.0 Mbit s/sec
[ 3] Sent 8505 datagrams
[ 3] Server Report:
[ 3] 0.0-10.0 sec 11.9 MBytes 10.0 Mbit s/sec 0.018 ms 0/ 8504 (0%)
[ 3] 0.0-10.0 sec 1 datagrams received o ut-of-order
[…] 00:07:e9:13:a6:3f
1.5.0.222

51 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net

TRAFFIC TEST…
1.5.4.111
lunkan@mx_2> show interfaces ae1.0 extensive
Logical interface ae1.0 (Index 426) (SNMP ifIndex 572) (Generation 239)
Flags: SNMP-Traps 0x0 VLAN-Tag [ 0x8100.11 ] Encapsulation: VLAN-Bridge
Statistics Packets pps Bytes bps
Bundle:
Input : 8507 0 12895160 0
Output: 10 0 3556 0
Link:
xe-0/0/0.0
Input : 8507 0 12895160 0
Output: 10 0 3556 0
Marker Statistics: Marker Rx Resp Tx Unknown Rx Illegal Rx
xe-0/0/0.0 0 0 0 0
Protocol bridge, MTU: 1522, Generation: 386, Route table: 3[…]

ae0
lunkan@mx_1> show interfaces ae1.0 extensive
Logical interface ae1.0 (Index 332) (SNMP ifIndex 617) (Generation 141)
Flags: SNMP-Traps 0x0 VLAN-Tag [ 0x8100.11 ] Encapsulation: VLAN-Bridge
Statistics Packets pps Bytes bps xe-0/0/0
Bundle: xe-0/0/0
Input : 8506 0 12895096 0
ae1 ae1
Output: 8 0 624 0
Link:
xe-0/0/0.0
Input : 8506 0 12895096 0 MC-LAG
Output: 8 0 624 0
Marker Statistics: Marker Rx Resp Tx Unknown Rx Illegal Rx
xe-0/0/0.0 0 0 0 0
Protocol bridge, MTU: 1522, Generation: 159, Route table: 3

00:07:e9:13:a6:3f
1.5.0.222

52 Copyright © 2010 Juniper Netw orks, Inc. w ww.juniper.net