How to setup an ACI Multi-

Site with single Pod and

Multi-Pod

Max Ardica – Principal Engineer - DCNBU

Ramses Smeyers – Principal Consulting Engineer - CX

BRKACI-2291
Cisco Webex Teams

Questions?
Use Cisco Webex Teams to chat
with the speaker after the session

How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Agenda

• Brief Multi-Pod/Multi-Site Review and Positioning

• Prerequisites
• Hardware Inspection and Installation
• Installing the First Site
• Expanding the Single Pod into a Multi-Pod Fabric
• Simplified Tenant Management through MSO
• Adding the DR Site on MSO
• MSO Additional Functionalities

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 4
Brief Multi-Pod /
Multi-Site Review and
Positioning

BRKACI-
2291
Multi-Pod or Multi-Site?

That is the question…

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
And the answer is…

BOTH!

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Systems View (How do these things relate)
Change and Fault Isolation

Active Workloads Layer 3 Active Workloads

Layer 2 & Layer 3 Inter Region Layer 2 & Layer 3

Fabric Change/Fault Domain Fabric Change/Fault Domain Fabric Change/Fault Domain Fabric Change/Fault Domain

Application Policy Change Domain Application Policy Change Domain

Common Namespace (IP, DNS, Active Directory…)

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
Typical Requirement
Creation of Two Independent Fabrics/AZs

Multi-Pod Fabric ‘A’ (Region 1)

‘Classic’ Active/Active (L2 and L3)

Pod ‘1.A’ Pod ‘2.A’

L3 Only ACI Multi-Site L3 Only

Multi-Pod Fabric ‘B’ (Region 2)

‘Classic’ Active/Active (L2 and L3)

Pod ‘1.B’Application Pod ‘2.B’

workloads deployed
across regions
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
 ACI Multi-Pod For More Information on
ACI Multi-Pod:
Overview BRKACI-2003
VXLAN
Inter-Pod
Pod ‘A’ Network
Pod ‘n’

MP-BGP - EVPN

…
Up to 50 msec RTT

APIC Cluster
IS-IS, COOP, MP-BGP IS-IS, COOP, MP-BGP

Availability Zone

• Multiple ACI Pods connected by an IP Inter-Pod L3 • Forwarding control plane (IS-IS, COOP) fault
network, each Pod consists of leaf and spine nodes isolation
• Up to 50 msec RTT supported between Pods • Data Plane VXLAN encapsulation between Pods
• Managed by a single APIC Cluster • End-to-end policy enforcement
• Single Management and Policy Domain
BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 10
 ACI Multi-Pod
Most Common Use Cases

 Need to scale up a single ACI fabric above

Pod
200 leaf nodes supported in a single Pod
Inter-Pod
 Handling 3-tiers physical cabling layout Leaf Nodes Network
(for example traditional N7K/N5K/N2K
deployments)
Spine Nodes

 True Active/Active DC deployments

Pod 1 Pod 2
Single VMM domain across DCs (stretched ESXi
Metro Cluster, vSphere HA/FT, DRS initiated
workload mobility,…)
Deployment of Active/Standby or Active/Acive
clustered network services (FWs, SLBs) across DCs APIC Cluster
DB Web/App Web/App
Application clustering (L2 BUM extension across
Pods)

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
 ACI Multi-Site VXLAN
Overview Inter-Site
Network

MP-BGP - EVPN
Multi-Site
Orchestrator

Site 1 Site 2
REST
GUI
API Availability Zone ‘B’
Availability Zone ‘A’
Region 1

• Separate ACI Fabrics with independent APIC clusters • MP-BGP EVPN control plane between sites
• No latency limitation between Fabrics • Data Plane VXLAN encapsulation across
• ACI Multi-Site Orchestrator pushes cross-fabric sites
configuration to multiple APIC clusters providing • End-to-end policy definition and
scoping of all configuration changes enforcement
BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
 ACI Multi-Site
Most Common Use Cases

• Scale-up model to build a • Data Centre Interconnect (DCI) • ACI Multi-Cloud

very large intra-DC network Integration between on-prem and
Extend connectivity and policy
(above 400 leaf nodes) public clouds
between ‘loosely coupled’ DC sites
Disaster Recovery and IP mobility use
cases

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Prerequisites

BRKACI-
2291
Prerequisites For More Information on starting
an ACI fabric from scratch:
BRKACI-2004

• Before starting, you should have:

• For each APIC a routable IP addresses for OOB mgmt and CIMC
• Functional NTP server
• Serial number of all leaf and spine nodes
• Optionally but recommended:
• 1 IP per leaf and spine for OOB
• SCP / FTP / HTTP server (software)
• Console / serial server
• Infrastructure VLAN / VTEP pool
• vCenter IP address and credentials

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Hardware
Inspection and
Installation
BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Our Setup for Today (High Level View)

Single external network used

for IPN and ISN

IPN/ISN

Site1-Pod 1 Site1-Pod 2 Site2-Pod 1

WAN

Site 1 Site 2

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
Our Setup for Today (Detailed View)
Bru ACI Fabric POD 37 & 38

bdsol-aci38-multisite1

BDSOL-ACI37-SPINE1 BDSOL-ACI37-SPINE2 BDSOL-ACI37-SPINE3 BDSOL-ACI37-SPINE4

BDSOL-ACI38-SPINE1

BDSOL-ACI37-LEAF1

BDSOL-ACI37-APIC1

BDSOL-ACI37-APIC2 BDSOL-ACI38-APIC2

bdsol-aci37-router2

BDSOL-ACI37-APIC3 BDSOL-ACI38-APIC3

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
Installing the First Site
Site1-Pod1 Configuration
Installing the First Site
Site1-Pod1: Initial Fabric Setup (Already Done)

• APIC initial configuration (S1P1-APIC1) [only the 1st one for now]
• 1st leaf discovery
• Spines discovery
• 2nd leaf discovery
• S1P1-APIC2 configuration
• Verification
• OOB mgmt  IPs for leaf and spine nodes

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
Installing the First Site
Site1-Pod1 Fabric

Site1-Pod 1
S1P1-Spine201 S1P1-Spine202
vCenter
Server 1

S1P1-Leaf101 S1P1-Leaf102

ESXi Cluster

192.168.1.1 S1P1-APIC1 S1P1-APIC2

.101
WAN
192.168.200.100/30
.102

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Installing the First Site
Parameters for the APIC Initial Setup Script

S1P1-APIC1 S1P1-APIC2

Fabric name Fabric1 Fabric1

Fabric ID 1 1

Active controllers 3 3

Pod ID 1 1

Controller ID 1 2

TEP Pool 10.0.0.0/16 10.0.0.0/16

Infra VLAN 3937 3937

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Installing the First Site
Site1-Pod1: Remaining Configuration Steps to Do

• NTP configuration
• Route Reflector for intra-BGP VPNv4 sessions
• VMM integration
• Tenant configuration with ‘Ecommerce’ running application
• ‘Ecommerce’ app connectivity verification
• L3Out creation and external connectivity verification

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Demo 1
NTP, VMM, L3Out Configuration and Pod Verification
Expanding the
Single Pod into a
Multi-Pod Fabric
 Expanding the Single Pod into a Multi-Pod Fabric
Adding the IPN and Site1-Pod2

• Step 1: setup the Inter-Pod Network (IPN)

• Step 2: create the Multi-Pod fabric using the APIC Wizard
 Add Site1-Pod1
 Add Site1-Pod2
 Discovery of Pod2’s leaf and spines nodes
• Step 3: S1P2-APIC3 in Pod2 joins the APIC cluster
• Step 4: extend ‘Ecommerce’ Tenant to Pod2 (L3Out, ESXi host, access policies)
• Verification Steps:
• Verify that the existing tenant configuration is extended into the Multi-Pod fabric
• Verify East-West and North-South connectivity

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
Expanding the Single Pod into a Multi-Pod Fabric
Adding the IPN and Site1-Pod2
BDSOL-ACI37-Multipod1 BDSOL-ACI37-Multipod3

IPN
BDSOL-ACI37-Multipod4
BDSOL-ACI37-Multipod2

Site1-Pod 1
Site1-Pod2
S1P1-Spine201 S1P1-Spine202 S1P2-Spine401 S1P2-Spine402

vCenter
Server 1

S1P1-Leaf101 S1P1-Leaf102 S1P2-Leaf301 S1P2-Leaf302

BDSOL-ACI37-APIC2
BDSOL-ACI37-APIC3
BDSOL-ACI37-APIC1

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
 Expanding the Single Pod into a Multi-Pod Fabric
Step 1: Setup the Inter-Pod Network (IPN)
Pod1 External TEP-Pool Pod2 External TEP-Pool
172.16.2.0/24
S1P1-Spine201
172.16.1.0/24
OSPF Area 0 S1P2-Spine401
.2 IPN1 IPN3, .18

.1
Primary RP Backup RP
PIM .17
.10 .26
.101 .102

1/33 1/33
.21
.5
1/35 1/48 1/48 1/36
TEP Pool: PIM PIM
TEP Pool:
10.0.0.0/16 .9 1/35 1/36 .25 10.1.0.0/16
1/34 1/34
S1P1-Spine202 .6 .22 S1P2-Spine402
.13 .109 .110 .29

1/48 PIM 1/48

.14 .30
IPN2 IPN4

Site1-Pod1 Site1-Pod2
IPN Infra Address Space: 172.16.101.0/24

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
Demo 2
Setup the Inter-Pod Network
Expanding the Single Pod into a Multi-Pod Fabric
Step 2: Create the Multi-Pod Fabric Using the APIC Wizard and import Pod2
Spine and Leaf Nodes

Nodes automatically discovered in Site1-Pod2 that

need to be added to the APIC fabric membership table

Node ID Pod ID Name S/N

301 2 S1P2-Leaf301 FDO224702ET

302 2 S1P2-Leaf302 FDO223007J4

401 2 S1P2-Spine401 FDO22472FCV

402 2 S1P2-Spine402 FDO22391NP2

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
Demo 3
Create the Multi-Pod Fabric Using the APIC Wizard
 Expanding the Single Pod into a Multi-Pod Fabric
Step 3: S1P2-APIC3 in Pod2 Joins the APIC Cluster

Pod2 uses TEP Pool

S1P1-APIC1 S1P1-APIC2 S1P2-APIC3 10.1.0.0/16 but
Fabric name Fabric1 Fabric1 Fabric1 S1P2-APIC3 resides
in TEP Pool of Pod1
Fabric ID 1 1 1

Active controllers 3 3 3

Pod ID 1 1 2

Controller ID 1 2 3

TEP Pool 10.0.0.0/16 10.0.0.0/16 10.0.0.0/16

Infra VLAN 3937 3937 3937

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Demo 4
S1P2-APIC3 in Pod2 Joins the APIC Cluster
 Expanding the Single Pod into a Multi-Pod Fabric
Step 4: Extend ‘Ecommerce’ Tenant to Pod2 (L3Out, ESXi Host to VDS, etc.)

Site1-Pod1 Site1-Pod2

1/11 Site1-L3Out
1/11 1/11
1/17 .1 .5 1/19 1/11 .9 .13 1/19
1/17

.14
.2 .6 Stretched ESXi Cluster .10
1/17 1/19 1/17 1/19

.101 .105

Connectivity to the 1/47 WAN 1/47

.106
WAN network is .102

pre-provisioned 1/7 1/9

Ecommerce Tenant External Address Space
1/1 192.168.200.0/24
192.168.100.0/24

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
Demo 5
Extend ‘Ecommerce’ tenant configuration to Pod2
Simplified Tenant
Management
through MSO
Simplified Tenant Management through MSO
Configuration Steps

• Initial setup of MSO

• Adding the Multi-Pod fabric as first site on MSO
• Importing existing ‘Ecommerce’ tenant configuration on MSO

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
 ACI Multi-Site
BGP Inter-Site Peers

Inter-Site
Network
• EVPN-RID, O-UTEP and O-MTEP addresses
Anycast VTEP Addresses:
O-UTEP & O-MTEP
are assigned from the Multi-Site
Orchestrator and must be routable across
the ISN

EVPN-RID 4 • Inter-site communication always happens

encapsulating traffic to one of the Anycast
EVPN-RID 1
EVPN-RID 2 EVPN-RID 3 TEP address (O-UTEP for L3/L3 unicast
forwarding, O-MTEP for BUM forwarding)

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
 Adding the DR Site on MSO
Assign Routable TEP Addresses and BGP EVPN Router-IDs

O-UTEP-S1P1: 172.16.100.101
O-UTEP-S1P2: 172.16.100.102
O-MTEP-S1: 172.16.100.100 IPN/ISN
BGP Speaker 1: 172.16.100.201
BGP Speaker 2: 172.16.100.202
Site1-Pod 1 Site1-Pod 2
BGP Speaker 2
O-UTEP-S1P1 O-UTEP-S1P2
O-MTEP-S1

BGP Speaker 1
Site1-L3Out

WAN

Site 1

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
Schema Design
One Template per Site, plus a ‘Stretched’ Template

Schema Site 1

ANP1 Site 1 Template

(Tenant1)
EPG1 EPG2 BD1 BD2

ANP1 Site 2 Template Site 2

(Tenant1)
EPG3 EPG4 BD3 BD4

ANP1 Site 3 Template

(Tenant1)
EPG5 EPG6 BD5 BD6 Site 3

ANP1 VRF
BD7 C1 C2
EPG7
Contracts

Stretched Template (Tenant1)

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Demo 6
Simplified Tenant Management through MSO
Adding the DR Site
on MSO
Adding the DR Site on MSO
Configuration Steps

• Installing the DR fabric (already done)

• Adding the DR fabric as a second site on MSO (assign routable TEP
addresses and BGP EVPN Router-IDs)
• Verifying IPN connectivity
• Extending the tenant ‘Ecommerce’ to the DR site
• Create access policies, VMM and a local L3Out in the DR fabric
• Extending the existing ‘Ecommerce’ tenant configuration to the DR site
• Verify external connectivity

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Adding the DR Site on MSO
Site2-Pod1 Fabric

Site2-Pod1
S2P1-Spine201 S2P1-Spine202

S2P1-Leaf101 S2P1-Leaf102

ESXi Cluster 2

BDSOL-ACI38-APIC1 BDSOL-ACI38-APIC2 BDSOL-ACI38-APIC3

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
Adding the DR Site on MSO
Parameters for the APIC Initial Setup Script (Already Done)

S2P1-APIC1 S2P1-APIC2 S2P1-APIC3

Fabric name Fabric2 Fabric2 Fabric2 Recommended

to use non
Fabric ID 1 1 1 overlapping with
existing sites.
Active controllers 3 3 3

Pod ID 1 1 1

Controller ID 1 2 3

TEP Pool 10.2.0.0/16 10.2.0.0/16 10.2.0.0/16

Infra VLAN 3937 3937 3937

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
 Adding the DR Site on MSO
Assign Routable TEP Addresses and BGP EVPN Router-IDs

O-UTEP-S1P1: 172.16.100.101 O-UTEP-S1P1: 172.16.200.101

O-UTEP-S1P2: 172.16.100.102 O-MTEP-S1: 172.16.200.100
BGP Speaker 1: 172.16.200.201
O-MTEP-S1: 172.16.100.100 IPN/ISN BGP Speaker 2: 172.16.200.202
BGP Speaker 1: 172.16.100.201
BGP Speaker 2: 172.16.100.202 BGP Speaker 2
Site1-Pod 1 Site1-Pod 2 Site2-Pod 1
BGP Speaker 2
O-UTEP-S1P1 O-UTEP-S1P2 O-UTEP-S1P1
O-MTEP-S1 O-MTEP-S2

BGP Speaker 1 BGP Speaker 1

Site1-L3Out Site2-L3Out

WAN

Site 1 Site 2

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 47
Adding the DR Site on MSO
Verifying IPN connectivity
IPN1 IPN3
S2P1-Leaf201

1/48 .9
.1 1/48
.2
IPN2 IPN4
IPN Site2 Infra Address Space:
172.16.102.0/24
.13
TEP Pool:
1/48
.5 1/48
10.2.0.0/16
.10 1/3
1/2 .2 1/1
.14
.1
1/4
.6 .18 .17 .6
1/5 .5
1/5
S2P1-Leaf202
WAN Infra Address Space: IPN5
172.16.110.0/24
Site2-Pod1

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
 Adding the DR Site on MSO
Create a Local L3Out in the DR Site
Site2-Pod1

Site2-L3Out

1/17 .17 .21 1/19

.22
.18
Ecommerce Tenant External Address Space 1/17 1/19
192.168.200.0/24
.109 1/47

.110 1/11

1/1
192.168.100.0/24

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
Demo 7
Adding the DR Site on MSO
MSO Additional
Functionalities
MSO Additional Functionalities

• End host connectivity verification

• Host route advertisement (inbound traffic optimization)
• Enabling CloudSec encryption between sites

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
ACI Multi-Site
CloudSec Encryption for VXLAN Traffic
Encrypted Fabric to Fabric Traffic
[GCM-AES-256-XPN (64-bit PN)])
CloudSec = “TEP-to-TEP MACSec”

VTEP IP MACSec VXLAN Tenant Packet

VTEP Information
in Clear Text
Inter-Site Network

MP-BGP - EVPN

Multi-Site
Orchestrator

Supported from ACI 4.0(1) release for FX line cards and 9332C/9364C platforms

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
Demo 8
MSO Additional Functionalities
Complete your
online session
survey • Please complete your session survey
after each session. Your feedback
is very important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live t-shirt.
• All surveys can be taken in the Cisco Events
Mobile App or by logging in to the Content
Catalog on ciscolive.com/emea.

Cisco Live sessions will be available for viewing on

demand after the event at ciscolive.com.

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Cisco Webex Teams

Questions?
Use Cisco Webex Teams to chat
with the speaker after the session

How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 56
Continue your education

Demos in the
Walk-in labs
Cisco campus

Meet the engineer

Related sessions
1:1 meetings

BRKACI-2291 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Thank you