ACI Multi-Site Architecture

and Deployment

Max Ardica, Distinguished Engineer

@maxardica

BRKDCN-2980
Cisco Webex App

Questions?
Use Cisco Webex App to chat
with the speaker after the session

How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install the Webex App or go directly to the Webex space Enter your personal notes here

4 Enter messages/questions in the Webex space

Webex spaces will be moderated

until February 24, 2023.

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
Session Objectives

• At the end of the session, the participants should be able to:

✓ Articulatethe different deployment options to interconnect Cisco ACI networks
(Multi-Pod and Multi-Site) and when to choose one vs. the other
✓ Understand the functionalities and specific design considerations associated to the
ACI Multi-Site architecture

• Initial assumption:
✓ Theaudience already has a good knowledge of ACI main concepts (Tenant, BD,
EPG, L2Out, L3Out, etc.)

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
Early Access.
Yes, please.
 How we can help
Cisco Application Centric Infrastructure Services
Strategy and Solution Discovery Optimization and Monitoring Support
Accelerate your data Discover and document business and
technical requirements. Drive automation and
Unleash the full power of data
center solutions through advanced

center transformation infrastructure as code approach network management

with Cisco CX Assessments

Assess the readiness of your
data center environment
Knowledge Transfer
Receive consulting support to drive your
automation and DevOps priorities

Gain faster results by quickly transforming Design, Implement, Migrate Learning | Certifications
Accelerate success throughout your network
and boosting the agility and automation of lifecycle with automation, micro-
Empower your workforce with efficiency
and innovation
your data center operations with CX segmentation, and security

expertise — so you can focus on the

outcomes that matter most

Optimize your Use and Protect your Accelerate

network adopt business multicloud

You don’t have to do it alone.

Visit CX Data Center For more insight, visit the Cisco CX stand in the World of
to learn more Solutions for Lightning Talks and Demos

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
 • Introduction
• Nexus Dashboard Orchestrator (NDO)
Architecture
• Provisioning Policies on NDO

Agenda • Inter-Site Connectivity Deployment

Considerations
• ACI Multi-Site Control and Data Plane
• Connecting to the External L3 Domain
• Network Services Integration (Stretch Goal)

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
Introduction
 ACI Architectural Options
Fabric and Policy Domain Evolution
ACI Single Pod Fabric
ACI Multi-Site

ISN
Fabric ‘A’ Fabric ‘n’
MP-BGP - EVPN

1
ACI Multi-Pod Fabric
IPN
Pod ‘A’ Pod ‘n’
4
2 MP-BGP - EVPN

Single Fabric, Single Multiple Fabrics, Multiple

…
Controller Domain Controller Domains
APIC Cluster

5
ACI Hybrid Cloud/Multi-Cloud
3
ACI Remote Leaf

IPN

Remote Leaf
Location

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
Multi-Pod or
Multi-Site?

That is the
question…

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
And the answer is…

BOTH!

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
Systems View (How do these things relate)
Change and Network Fault Domain Isolation
Active Workloads Layer 3 Active Workloads
Layer 2 & Layer 3 Inter Region Layer 2 & Layer 3

Nexus Dashboard
Orchestrator

Multi-Pod Fabric ‘A’ (Region 1) Multi-Pod Fabric ‘B’ (Region 2)

Fabric NetworkFault Domain Fabric Network Fault Domain Fabric Network Fault Domain Fabric Network Fault Domain

Pod A.1 (AZ 1) Pod A.2 (AZ 2)

Application1 Pod B.1 (AZ 1) Pod B.2 (AZ 2)
Application 2
workloads deployed workloads deployed
Application Policy Change Domain Application Policy Change Domain
across Pods (AZs) across Pods (AZs)

Common Namespace (IP, DNS, Active Directory…)

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
Multi-Pod + Multi-Site
Satisfying Conflicting Requirements (A/A DCs and DR)
Data Center 1 Data Center 2

Multi-Pod Fabric ‘A’

(Region 1)
‘Classic’ Active/Active (L2 and L3)

Pod ‘1.A’ AZ1 Pod ‘2.A’ AZ2

L3 Only Nexus Dashboard L3 Only

Orchestrator

Multi-Pod Fabric ‘B’

(Region 2)
‘Classic’ Active/Active (L2 and L3)

Pod ‘1.B’ AZ1 Pod ‘2.B’ AZ2

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 16
But wait! Couldn’t I
deploy Multi-Site
also to handle more
typical Multi-Pod
use cases?

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
Multi-Site for Active/Active Application
Deployments?
Multi-Site for Active/Active Application Deployments
• ACI Multi-Site allows to extend connectivity and
policies between separate APIC domains
• Layer 3 only across sites
• Layer 2 with and without BUM flooding

• Keep in mind some specific considerations before

deploying Multi-Site for “classic” Active/Active
application deployments (i.e. same application
components deployed across sites)
• Loss of change and network fault domain isolation
across separate ACI domains
• Creation of separate VMM domains by design (loss of
intra-cluster functionalities like DRS, vSphere FT/HA, …)
• Specific service node insertion deployment
considerations (use of separate service nodes per
fabric, limited support for service nodes clustering
across sites, no support for vzAny + PBR, …)

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
 ACI Multi-Site
The Ideal Architecture for “Loosely Coupled” DCs
2 Inter-Site VXLAN

Network 3

MP-BGP - EVPN 3
Nexus Dashboard
Orchestrator

1
4 4
L3 L3
Site 1 5 5 Site 2
REST
GUI
API Region 2
Region 1

• Separate ACI Fabrics with independent APIC clusters • MP-BGP EVPN control plane between sites
• No latency limitation between Fabrics • Data Plane VXLAN encapsulation across sites
• ACI Multi-Site Orchestrator pushes cross-fabric configuration to • End-to-end policy definition and enforcement
multiple APIC clusters providing scoping of all configuration changes

Want to know how to provision Multi-Pod and Multi-Site from scratch?

BRKDCN-2980 Come to Cisco
© 2023 BRKDCN-2919 (Thu Cisco
and/or its affiliates. All rights reserved. @ 8.30
Public am)
19
 ACI Multi-Site Architecture
Most Common Use Cases
• Compartmentation/Scale • Data Center Interconnect (DCI)
Building Multiple Fabrics inside a single Data Center Extend connectivity/policy between ‘loosely coupled’ DC sites
Disaster Recovery and IP mobility use cases

Optimized and controlled L2/L3 connectivity (including optimized/controlled

BUM forwarding), scale out total number of leaf nodes (SP use case)

• Hybrid-Cloud and Multi-Cloud • SP 5G Telco DC/Cloud*

Integration between on-prem and public clouds Centralized DC Orchestration for “Autonomous Fabrics”
(AWS, Azure, GCP)
Nexus Dashboard Optional SR-MPLS/MPLS Handoff on Border Leaf nodes
Orchestrator Nexus Dashboard
Orchestrator

*May also apply to Enterprise deployments

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
Nexus Dashboard
Orchestrator (NDO)
Architecture
Original Multi-Site Orchestrator Option
VM Based MSO Cluster (OVA), Now EoL/EoS

• Supported from the beginning (MSO release 1.0(1))

• Each Cisco Multi-Site Orchestrator node is
Multi-Site Orchestrator Cluster packaged in a VMware vSphere virtual appliance
(OVA)
MSO Node1 MSO Node2 MSO Node3 • For high availability, you should deploy each Cisco
ESXi Hypervisor ESXi Hypervisor ESXi Hypervisor Multi-Site Orchestrator virtual machine on its own
VMware ESXi host
IP network • Requirements for MSO Release 1.2(x) and above:
VMware ESXi 6.0 or later

Site 1 Site 2
….. Site n
Minimum of eight virtual CPUs (vCPUs), 48 Gbps of
memory, and 100 GB of disk space
• MSO 3.1(1) last supported release with this form factor, now
EoL/EoS

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
Cisco Nexus Dashboard Powering automation
Simple to Automate, Simple to Unified agile platform Cisco Nexus
Dashboard

Consume
Cisco Nexus
Dashboard

Insights Fabric Discovery

Fabric Controller
Orchestrator

Data Broker Consume all services in one place SAN Controller

Private cloud Public cloud Custom/third-

party

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
Cisco Nexus Dashboard
Deployment Evolution

Shipping Shipping

Physical Cisco ND Virtual/Cloud Cisco

Platform Cluster ND Platform Cluster

ND virtual cluster supported on ND cloud cluster supported

ESXi and KVM hypervisors for AWS and Azure
Spec: 16 vCPUs, 64Gb ram and 500Gb disk

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
Cisco Multi-Site Orchestrator has become Cisco
Nexus Dashboard Orchestrator

Cisco Nexus
Cisco Multi-Site
Dashboard
Orchestrator
Orchestrator

Up to release 3.1(1) From release 3.2(1)

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
Nexus Dashboard Orchestrator
What NDO Release to Choose?

Recommended Releases per Scenario

Current Release Target Release

1 MSO/NDO 1.1.x to 3.7.1j NDO 3.7(2) (Shipping)

2 NDO 3.7(2) NDO 4.1(2) (Q2CY23)*

3 None - Greenfield NDO 4.1(1) (Shipping)

*NDO 4.1(2) will support one-click GUI upgrade from NDO 3.x releases

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
 Migrating the MSO Cluster to Cisco NDO
• All MSO releases are officially End-of-Life (EOL)
• Customer should (and must) migrate from MSO to NDO
• NDO 3.7(2) release is the recommended target release for this migration

Original MSO Cluster Cisco Nexus Dashboard Orchestrator 3.7(2)

Rollback config to
3 the backup file
Note
This procedure is
Export Import Take care of Templates with only supported for
backup 1 2 backup 4 drifts and perform the “DB SW upgrades but not
cleanup” procedure
for SW downgrades

Backup file

https://www.cisco.com/c/en/us/td/docs/dcn/ndo/3x/deployment/cisco-nexus-dashboard-orchestrator-deployment-guide-371/ndo-deploy-migrate-37x.html

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
Provisioning
Policies on NDO
 Cisco Nexus Dashboard

Supporting Different Types of Policies Orchestrator 4.0(1)

Application Management Policies

Used to define tenant policies
(Application Network Profiles,
EPGs, BDs, VRFs, etc.)

Fabric Management Policies

Used to define fabric access
policies, interface and monitoring
policies

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 30
 Cisco Nexus Dashboard

Provisioning Policies on NDO Orchestrator 4.0(1)

Multiple Template Types

Optimized New in New in New in New in New in

in 4.0(1) 4.0(1) 4.0(1) 4.0(1) 4.0(1) 4.1(1)

EPGs, BDs, Route-Maps, VLAN pools,

VRFs, DHCP-Relay, Physical Domains, Interfaces, SyncE, SPAN Tenant and Full L3Out
Contracts, etc. QoS… NTP, etc. MACsec, etc. Access Sessions Configuration

Application Tenant Policy Fabric Fabric Resources Monitoring Policies L3Out Template
Template Template Policies Template Policies Template Template

Benefits

Simplify Single Pane of Glass

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
 Provisioning Policies on NDO
Why do we “Templatize” the Configuration?

Features
NDO 3.4(1) NDO 3.4(1) NDO 3.4(1) NDO 3.4(1) NDO 3.6(1)
Template versioning Template deployment Change Detach templates Configuration drift
and rollback plan visibility control workflow from Sites reconciliation workflow

Support rollback of template Shows preview of what NDO NDO workflow that synchronizes
New personas for Configuration is not
from newer to older version-id is going to provisioning and merges any config changes
management and removed from the
to each site made in APIC or NDFC domains
Label a template as Golden provisioning of configuration APIC/NDFC domains

Better visibility to reduce More structured

Granular roll back of Simplify the understanding and
errors and seize the deployments Ease of use
templates specific reconciliation of config drifts between
impact of a which enables for migration
configuration NDO and APIC/NDFC
template’s deployment increased flexibility

Benefits
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
Provisioning Policies on NDO
Template-Level Operational Enhancements

For more information and demonstrations of all those NDO template-

level operational enhancements:
• Template Versioning
https://video.cisco.com/video/6277140235001

• Template Deployment Plan Visibility

https://video.cisco.com/video/6277137504001

• Change Control Workflow

https://video.cisco.com/video/6277140011001

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
Provisioning
Application
Policies on NDO
 Cisco Nexus Dashboard

Cisco Nexus Dashboard Orchestrator Orchestrator 4.0(1)

Application Templates
Cisco Nexus
Dashboard Orchestrator

Application
Template

Multi-Site Template (Site-Local

Autonomous Template
or Stretched policies)

Typical Enterprise Deployment Typical Service Provider Use Case

Inter-Site Network

Up to 14 Fabrics Connected to the ISN (VXLAN EVPN Communication) Up to 100 Autonomous Fabrics (no ISN and VXLAN EVPN Connectivity)

Different Template Deployment Modes Can Apply to the Same Set of Fabrics
BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 Application Templates
Multi-Site Templates
Schema
Tenant1 Tenant1
Site Local Stretched
▪ Application Template = ACI policy definition Template Template
(ANP, EPGs, BDs, VRFs, etc.)
▪ Schema = container of Application
Templates sharing a common use-case t0 t1 t1

As a typical use case, a schema can (and should)

be dedicated to a Tenant
▪ The template is the atomic unit of change
for policies
A Multi-Site template associated to a single site Site 1 Site 2
can be pushed only to that site
EFFECTIVE EFFECTIVE
A Multi-Site template associated to multiple sites POLICY POLICY

is concurrently pushed to all those sites

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Best Practices for Multi-Site Templates
One Template per Site, plus Two Templates for “Stretched Objects”
Schema (dedicated to Tenant1)

ANP1 Site 1 Template

Site 1
EPG1 EPG2 BD1 BD2 L3Out-S1*

ANP1 Site 2 Template

EPG3 EPG4 BD3 BD4 L3Out-S2

Site 2

ANP1 Stretched Template 1 (EPGs, BDs, …)

EPG5 EPG6 BD5 BD6 Ext-EPG

ANP1 Stretched Template 2 (VRFs, contracts, …)

EPG7 BD7 VRF C1 SG

*L3Out defined in a separate “L3Out Template” from NDO 4.1(1)

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 37
 Application Templates
Autonomous Templates
• Autonomous templates can also be Schema
associated to one or more fabrics Autonomous Tenant1
Template
• Differently than for Multi-Site templates,
the deployment of an Autonomous
template to different sites won’t cause the
“stretching” of configuration objects (VRFs, t0 t1
BDs, EPGs,…)
• NDO performs a “configuration replication”
function to multiple sites
• Autonomous Templates can be deployed
to different fabrics at different points in Site 1 Site 2
time*
EFFECTIVE EFFECTIVE
• Other template types behave as POLICY POLICY

Application Autonomous templates

*Roadmap feature planned for CY23

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
Inter-Site Connectivity
Deployment
Considerations
Inter-Site Network (ISN) Functional Requirements
Sub-interfaces
(VLAN tag 4) Inter-Site Network

MP-BGP - EVPN

Nexus Dashboard
Orchestrator

• Not managed by APIC or NDO, must be independently configured (day-0 configuration)

• IP topology can be arbitrary, not mandatory to connect all the spine nodes to the ISN
• ISN main functional requirements:
✓ OSPF/BGP* to peer with the spine nodes and exchange TEP address reachability
Must use sub-interfaces (with VLAN tag 4) toward the spines
✓ No multicast requirement for BUM traffic forwarding across sites
✓ Increased end-to-end MTU support (at least 50/54 extra Bytes)

*Requires ACI 5.2(1) and NDO 3.5(1) BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
 Inter-Site Connectivity
Frequently Asked Questions

▪ Any network device capable of routing traffic and supporting

What platforms can or should I packets with increased MTU size can be deployed in the ISN
1 deploy in the ISN? ▪ Need sub-interfaces support for the ISN devices directly
connected to the spines

▪ No, ingress replications is performed by the ACI spine

Do I need to run L3 multicast nodes to forward BUM traffic across sites
2
inside the ISN? ▪ This function is only required for the BDs that are stretched
across sites with BUM flooding enabled

▪ No, the only officially supported configuration consists in

Can I use a Layer 2 only
3 deploying the ISN nodes as L3 network devices (particularly
infrastructure as ISN? the ISN devices connected to the spines)

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
 Inter-Site Connectivity
Frequently Asked Questions (2)

▪ No, the network providing ISN services for Multi-Site could

Do I need to deploy a dedicated also be used for other functions
4 infrastructure as ISN? ▪ It is recommended (but not mandatory) to use a dedicated
VRF for providing ISN connectivity

▪ No, the bandwidth required between sites mostly depends

Is there a minimum bandwidth I
5 on the amount of east-west connectivity expected between
should deploy between sites? sites

▪ No, from ACI release 5.2(1) and NDO release 3.5(1) we

Is OSPF the only protocol
introduced support also for BGP peering between the
6 supported to peer with the ISN spines and the first L3 hop ISN devices
network?

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
ACI Multi-Site
Control and Data
Plane
Namespace
Normalization
and Shadow
Objects
ACI Multi-Site
Network and Identity Extended between Fabrics
Network information carried across Identity information carried across
Fabrics (Availability Zones) Fabrics (Availability Zones)

VTEP IP VNID Class-ID Tenant Packet

No Multicast Requirement
in Backbone, Head-End
Replication (HER) for any
Inter-Site Network Layer 2 BUM traffic)

MP-BGP - EVPN

Nexus Dashboard
Orchestrator

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
 ACI Multi-Site
Inter-Site Policies and Spines’ Translation Tables
• Inter-Site policies defined on the ACI Site 1 Spines Translation Table Site 2 Spines Translation Table
Nexus Dashboard Orchestrator are
pushed to the respective APIC domains ISN
• End-to-end policy consistency
• Creation of ‘Shadow’ EPGs to locally VNID → 15434256
VNID → 16678781
represent the policies Class-ID: 49153 Class-ID: 32770

• Inter-site communication requires the

installation of translation table entries
on the spines (namespace
normalization) EP1 EP2
Site 2
• Translation entries are populated in Site 1
different cases: EP1
C
EP2 EP1
EPG
C
EP2
EPG EPG EPG
• Stretched EPGs/BDs VRF VNID: 16678781 VRF VNID: 16678781 VRF VNID: 15434256 VRF VNID: 15434256
BD VNID: 13543235 BD VNID: 15434518 BD VNID: 13762843 BD VNID: 12753426
• Creation of a contract between not Class-ID: 49153 Class-ID: 31564
‘Shadow’
Class-ID: 32770 Class-ID: 36784

stretched EPGs EPGs

• Preferred Group or vzAny deployments

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
Underlay and
Overlay Control
Plane
Considerations
 ACI Multi-Site
BGP Inter-Site Peers
• Spines connected to the Inter-Site Network perform
two main functions:
Inter-Site
Network 1. Establishment of MP-BGP EVPN peerings with spines in
remote sites
Anycast VTEP Addresses:
O-UTEP & O-MTEP ▪ One dedicated Control Plane address (EVPN-RID) is
assigned to each spine running MP-BGP EVPN
2. Forwarding of inter-sites data-plane traffic
▪ Anycast Overlay Unicast TEP (O-UTEP): assigned to all the
EVPN-RID 4 spines connected to the ISN and used to source and
receive L2/L3 unicast traffic
EVPN-RID 1
EVPN-RID 2 EVPN-RID 3 ▪ Anycast Overlay Multicast TEP (O-MTEP): assigned to all
the spines connected to the ISN and used to receive L2
BUM traffic

• EVPN-RID, O-UTEP and O-MTEP addresses are

assigned from the Nexus Dashboard Orchestrator and
must be routable across the ISN

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
ACI Multi-Site
IP Network Routing Table
Exchanging TEP Information across Sites
O-UTEP A, O-MTEP A
EVPN-RID S1-S4
• OSPF or BGP peering between O-UTEP B, O-MTEP B
Filter out the
spines and Inter-Site network advertisement of internal
EVPN-RID S5-S8
TEP pools into the ISN
• Mandates the use of L3 sub-interfaces
(with VLAN 4 tag) between the spines
and the ISN Inter-Site
OSPF/BGP Network OSPF/BGP
• Exchange of External Spine TEP
addresses (EVPN-RID, O-UTEP S5 S6 S7 S8
S1 S2 S3 S4
and O-MTEP) across sites IS-IS to OSPF/BGP
mutual redistribution
Internal TEP Pool information not TEP Pool 1 TEP Pool 2
needed to establish inter-site
communication (should be filtered out
on the first-hop ISN router)
Nexus Dashboard
Orchestrator
Use of overlapping internal TEP Pools Site 1 Site 2
across sites is fully supported
Leaf Routing Table Leaf Routing Table
IP Prefix Next-Hop IP Prefix Next-Hop
O-UTEP B, Site1-S1, Site1-S2, Site1-S3, O-UTEP A, Site2-S5, Site2-S6, Site2-S7,
O-MTEP B, Site1-S4 O-MTEP A, Site2-S8
EVPN-RID S5-S8 EVPN-RID S1-S4
BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
 ACI Multi-Site
Inter-Site MP-BGP EVPN Control Plane

S3-S4 Table S5-S8 Table

• MP-BGP EVPN used to communicate EP1 Leaf 1 EP2 Leaf 4
Endpoint (EP) information across Sites O-UTEP B
MP-BGP EVPN
EP2 EP1 O-UTEP A
MP-iBGP or MP-EBGP peering options
supported
Remote host route entries (EVPN Type-2) Inter-Site
are associated to the remote site Anycast Network
O-UTEP address O-UTEP A O-UTEP B
• Automatic filtering of endpoint S1 S2 S3 S4 S5 S6 S7 S8
Nexus Dashboard
information across Sites Orchestrator

Host routes are exchanged across sites only COOP COOP

if there is a cross-site contract requiring
communication between endpoints
EP2
EP1
Site 1 Site 2

Define and push inter-site policy

EP1 EP2
C
EPG EPG
BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 59
Data Plane
Communication
across Sites
ACI Multi-Site Policy information
carried across Pods

Inter-Sites Unicast
VTEP IP VNID Class-ID Tenant Packet
Data Plane
Leaf 4
EP1
EP2 O-UTEP B 3 Inter-Site
EP2
EP1
S2-L4-TEP
O-UTEP A
Network

Site 1 Site 2
O-UTEP A VXLAN Inter-Site unicast traffic
sourced from O-UTEP A and
S1 S2 S3 S4 destined to O-UTEP B S5 S6 S7 S8

EP1 e1/3 2
Nexus Dashboard
Orchestrator
4 EP2
EP1
e1/1
O-UTEP A

EP2 O-UTEP B
5 10.10.10.0/24 Proxy B

1
EP1 EP1 EP2 EP2
C 20.20.20.20
10.10.10.10 EPG EPG

6
2 3 4

O-UTEP B O-UTEP B S2-L4-TEP

1 S1-L4-TEP O-UTEP A 6
O-UTEP A

20.20.20.20 20.20.20.20 20.20.20.20 20.20.20.20

= VXLAN Encap/Decap
20.20.20.20
10.10.10.10 10.10.10.10 10.10.10.10 10.10.10.10 10.10.10.10

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
ACI Multi-Site
L3 Only across Sites (“Autonomous Sites”)

Routing across sites via the WAN backbone

“Autonomous ACI Fabrics” (no ISN)

L3Out L3Out

Need to apply a contract

Need to apply a contract WAN between Ext-EPG
between internal EPG and
Ext-EPG associated to the associated to the L3Out in
L3Out in Fabric 1 Fabric 2 and internal EPG
Mandates the use of a multi-VRF
capable backbone network (VRF-Lite,
MPLS-VPN, etc.) to extend multiple
VRFs across fabrics
BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
Simplify Policy
Application
Preferred Group
and vzAny
ACI Multi-Site
Simplify Policy Enforcement: Preferred Groups
Contract required to
Multi-Site Preferred Group communicate with EPG(s)
external to the Preferred Group

App DB
C1 Non-PG
EPG
C2
Free
Web
communication

• ”VRF unenforced” not supported with Multi-Site

• Multi-Site Preferred Group configuration from the Multi-Site Orchestrator is supported from MSO 2.0(2)
release
• Creates ‘shadow’ EPGs and translation table entries ‘under the hood’ to allow ‘free’ inter-site communication
• 250 Preferred Groups supported as MSO release 2.2(3), 1000 from MSO release 2.2(4)
• Typically desired in legacy to ACI migration scenarios

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
 Simplify Policy Enforcement
Preferred Groups for E-W and N-S Flows

• Adding internal EPGs and External EPGs

(associated to L3Outs) to the Preferred Group
Inter Site
Network allows to enable free east-west and north-
south connectivity
Site 1 Site 2
• When adding the Ext-EPG to the Preferred
Group:
• Can’t use 0.0.0.0/0 for classification, needs more
L3Out
specific prefixes
L3Out
Site 1 Site 2
Ext-EPG
EP1 EP2 Ext-EPG
• As workaround it is possible to use 0.0.0.0/1 and
128.0.0.0/1 to achieve the same result
• Must ensure Ext-EPG is a stretched object
Multi-Site Preferred Group

EPG1 EPG2 • Intersite L3Out not supported if the Ext-EPG is

Ext-EPG
part of a Preferred Group (as of NDO 4.1(1))
On NDO

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
Simplify Policy Enforcement
vzAny Support
What is vzAny? Logical object representing all the EPGs in a VRF

Use case 1: Many-to-One communication Use case 2: Enable free communication

(Shared Services) inside a VRF
vzAny (VRF1)
vzAny (VRF1) vzAny (VRF1)
VRF1 or
VRF-Shared
EPG1 EPG2 EPG1 EPG2 EPG1 EPG2
C1 Shared
C P EPG C C1 P
Permit-Any
Permit-Any
EPG3 Ext-EPG
No current Ext-EPG
Service-Graph No current
support in NDO* Service-Graph
support in NDO*

• Multiple EPGs part of a specific VRF1 consume the • vzAny provides and consumes a contract with an associated
services provided by a shared EPG (part of VRF1 or “Permit-any” filter
of a VRF-shared)
• Use ACI fabric only for network connectivity without policy
• VRF-shared can be part of the same tenant or of a enforcement
different tenant
• Equivalent to “VRF unenforced”

*Planned for NDO 4.2(1) release BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
ACI Multi-Site and vzAny
Enable Inter-Site Free Communication Inside a VRF

Inter Site
Network

Site1 Site2

Ext-EPG Ext-EPG
L3Out-Site1 L3Out-Site2

EPG1 EPG2

• Proper translation entries are created on the spines of both fabrics to enable east-west
communication
• Supported also for connecting to the external Layer 3 domain
• vzAny + PBR support for any-to-any communication planned for a future NDO release

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
Connecting to
the External L3
Domain
Different Types
of L3Outs
Connecting to the External Layer 3 Domain
‘Traditional’ IP-Based L3Outs (Recommended Option)

Client

L3Out WAN Edge WAN

Routers

• Connecting to WAN Edge routers from Border Leaf nodes

• VRF-Lite hand-off for extending L3 multi-tenancy outside
the ACI fabric
Border Leafs ▪ Up to 800 L3Outs/VRFs currently supported on the same BL
nodes pair
• Support for host routes advertisement out of the ACI Fabric
from ACI release 4.0(1)
▪ Enabled at the BD level
• Support for L3 Multicast and Shared L3Out

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
Connecting to the External Layer 3 Domain NDO 4.0(2)
Release
SR-MPLS/MPLS Hand-Off on the BL Nodes
NCS5500, NCS540/ 560 or
ASR9K with ACI 5.0(1) release Client

WAN Edge WAN

MPLS tagged Routers
traffic

• Connecting to WAN Edge routers from Border Leaf nodes

• Typically connected directly
• Possible to connect through a Segment Routing enabled network
• Single control plane session (MP-BGP EVPN) for all tenant VRFs
Border Leafs
• BGP EVPN address family to carry DC prefixes, MPLS label for VRF
(VPN label) and color community
• MPLS tagged traffic between the BL nodes and the WAN Edge
routers
FX, FX2, FX3, • Couple of current limitations for Multi-Site deployments
GX Leaf models
• No support for host-based route advertisement
• No current support for Layer 3 Multicast communication

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
Deploying
External EPG(s)
Associated to
the L3Out
ACI Multi-Site and L3Out
Stretching or Not Stretching the Ext-EPG?

• The Ext-EPG can be defined in a template

Inter-Site Network associated to multiple sites (stretched object)
The Ext-EPG must then be mapped to the local
L3Outs in the “site level” section of the template
configuration
L3Outs remain independent objects defined in
each site
• Recommended when the L3Outs in the
Ext-EPG separate sites provide access to a common
L3Out Site 1 L3Out Site 2 set of external resources (as the WAN)
BD-Red BD-Green Simplifies the policy definition and external traffic
WAN classification
Still allows to apply route-map polices on each
L3Out (since we have independent APIC domains)

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
ACI Multi-Site and L3Out
Stretching or Not Stretching the Ext-EPG?

Inter-Site Network
• Separate Ext-EPGs can be defined in
templates mapped to separate sites (non
stretched objects)
Each Ext-EPG can be mapped to the local L3Out
in the “global” or “site level” section of the
template configuration

Ext-EPG Ext-EPG • Allows to apply different policies to each Ext-

L3Out Site 1 L3Out Site 2 EPGs at different time
BD-Red BD-Green • Can still use the same 0.0.0.0/0 network
WAN
configuration for classification on both sites
• May require enablement of Intersite L3Out

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
Solving
Asymmetric
Routing Issues
with the External
Network
 ACI Multi-Site and L3Out
Typical Deployment of Perimeter FWs

Inter-Site Network

Site 1 Site 2

Web-EPG C1 Ext-EPG

L3Out L3Out
Site 1 Site 2
10.10.10.10 IP Subnet 10.10.10.11
IP Subnet Active/Standby
10.10.10.0/24 Active/Standby
10.10.10.0/24

Traffic dropped
because of lack of
state in the FW

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
 Solving Asymmetric Routing Issues ACI 4.0(1)
Release
Use of Host-Routes Advertisement
Inter-Site Network

Site 1 Site 2

Web-EPG C1 Ext-EPG

L3Out L3Out
Site 1 Site 2
10.10.10.10 Host routes 10.10.10.11
10.10.10.10/32 Active/Standby Active/Standby Host routes
10.10.10.11/32
*Alternative could be
running an overlay solution
Host-routes
(LISP, GRE, etc.) injected into the
Enabled on MSO
at the BD level in
WAN*
each site
• Ingress optimization requires host-routes advertisement on the L3Out
• Native support on ACI Border Leaf nodes available from ACI release 4.0(1)
• Not currently supported for SR-MPLS L3Outs
BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
Intersite L3Out
Support
Problem Statement
Behavior before ACI Release 4.2(1)

Supported Design
✓ Not Supported Design
❌
Inter-Site Network Inter-Site Network
X

L3Out L3Out L3Out

Site 1 Site 2 Site 1

WAN, Mainframes, WAN, WAN

Mainframes,
FW/SLB, etc… FW/SLB, etc…

Note: the same consideration applies to both IP-Based L3Outs and SR-MPLS L3Outs
BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
ACI Multi-Site and L3Out
Support of Intersite L3Out

• Starting with ACI Release 4.2(1) it is possible for

Inter-Site Network endpoints in a site to send traffic to resources (WAN,
Mainframes, FWs/SLBs, etc.) accessible via a remote
L3Out connection
MP-BGP VPNv4/VPNv6
• External prefixes are exchanged across sites via MP-
BGP VPNV4/VPNv6 sessions between spines
• Traffic will be directly encapsulated to the TEP of the
L3Out
Site 1
remote BL nodes
• The BL nodes will get assigned an address part of an
WAN, Mainframes,
WAN additional (configurable) prefix that must be routable across
FW/SLB, etc…
the ISN

• Same solution will also support transit routing across

sites (L3Out to L3Out)

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
 ACI Multi-Site and Intersite L3Out ACI 4.2(1)
Release
Supported Scenarios

Inter-Site Network • Inter-Site Network

L3Out • L3Ou • L3Ou

Site 1 t Site t Site
1 2

WAN, Mainframes,
WAN •WAN,WAN
• • WAN,
FW/SLB, etc… Mainframes, Mainframes,
FW/SLB, etc… FW/SLB, etc…

• Endpoint to remote L3Out communication (intra-VRF) • Inter-site transit routing (intra-VRF)

• Endpoint to remote L3Out communication (inter-VRF) • Inter-site transit routing (inter-VRF)

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
Network
Services
Integration
Integration
Models
ACI Multi-Site and Network Services Deployment options fully
supported with ACI Multi-Pod

Integration Models
ISN

• Active and Standby pair deployed across Pods

• Limited supported options

Active Standby

ISN
• Active/Active FW cluster nodes stretched across Sites
(single logical FW)
• Limited supported options

Active/Active Cluster

Active Active

ISN
• Typical deployment model for ACI Multi-Site, each fabric
leverages a dedicated service node function
• Use of PBR to avoid creating asymmetric paths through
stateful devices (FWs, LBs, etc.) for both North-South and
East-West communication
Active/Standby Active/Standby BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 106
 Use of Service Graph and Policy Based Redirection
Resilient Service Node Deployment in Each Site
PBR redirection only supported to a local service function, hence it is important to deploy such function in a resilient way

Active/Standby Cluster Active/Active Cluster Independent Active Nodes

Site1 Site1 Site1

L3 Mode L3 Mode L3 Mode L3 Mode L3 Mode Active/Standby

Active/Standby Cluster Active/Active Cluster Active Node 1 Active Node 2 Node 3

• The Active/Standby pair represents a • The Active/Active cluster represents a • Each Active node represent a unique
single MAC/IP entry in the PBR policy single MAC/IP entry in the PBR policy MAC/IP entry in the PBR policy
• Spanned Ether-Channel Mode supported • Use of Symmetric PBR to ensure each flow
with Cisco ASA/FTD platforms is handled by the same Active node in both
All ASA/FTD nodes must be connected to directions
the same leaf nodes pair
BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 108
Use of Service
Graph and PBR
North-South and
East-West
 Use of Service Graph and Policy Based Redirection
North-South Communication – Inbound Traffic

Inter Site
Network

Site1 Site2
Compute leaf
always applies
the PBR policy Compute leaf
always applies
EPG EPG the PBR policy
Ext C Web

Consumer Provider
(Provider) (Consumer)

L3Out-Site1 L3Out-Site2
10.10.10.10 10.10.10.11
L3 Mode L3 Mode
Active/Standby Active/Standby

• Inbound traffic can enter any site when destined to a stretched subnet (if ingress optimization is not deployed or
possible)
• PBR policy is always applied on the compute leaf node where the destination endpoint is connected
Requires the VRF to have the default policies for enforcement preference and direction
Ext-EPG and Web EPG can indifferently be provider or consumer of the contract

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 113
 Use of Service Graph and Policy Based Redirection
North-South Communication – Outbound Traffic

Inter Site
Network

Site1 Site2
Compute leaf
always applies
the PBR policy Compute leaf
always applies
EPG EPG the PBR policy
Ext C Web

L3Out-Site1 L3Out-Site2
L3Out-Site2
10.10.10.10 10.10.10.11
L3 Mode L3 Mode
Active/Standby Active/Standby

• PBR policy always applied on the same compute leaf where it was applied for inbound traffic
• Ensures the same service node is selected for both legs of the flow
• Different L3Outs can be used for inbound and outbound directions of the same flow

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 114
 Use of Service Graph and Policy Based Redirection
East-West Communication (2)

Inter Site
Network

Site1 Site2

Provider leaf
always applies
the PBR policy
EPG EPG
Web C App

Provider Consumer

L3 Mode
EPG L3 Mode Active/Standby EPG
Active/Standby
Web App

• EPGs can be locally defined or stretched across sites and can be part of the same VRF or in
different VRFs (and/or Tenants)
• PBR policy is always applied on the leaf switch where the Provider endpoint is connected
• The Provider leaf always redirects traffic to a local service node
• Mandates to configure an IP Selector under the Consumer EPG
BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 115
 Use of Service Graph and Policy Based Redirection
East-West Communication (2)

Inter Site
Network

Site1 Site2

Provider leaf
always applies Consumer leaf
the PBR policy does not apply
EPG EPG
Web C App the PBR policy

Provider Consumer

L3 Mode
EPG L3 Mode Active/Standby EPG
Active/Standby
Web App

▪ The Consumer leaf must not apply PBR policy to ensure proper traffic stitching to the FW
node that has built connection state
▪ Ensures both legs of the flow are handled by the same service node

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 116
 Orchestrator Radar

ACI Multi-Site and PBR Enhancements (2HCY23)

Future Supported Use Cases

Any-to-Any Many-to-One Transit Intersite L3Out

• Support only for single service node • Support for one service node only • Redirect intersite transit routing
insertion (if the Provider is the Ext-EPG) or traffic flows
two service nodes (if the Provider • Traffic is redirected via both local
• Distributed deployment model
is a regular EPG) and remote service node
(traffic is redirected via both local
and remote service node) • Intra-VRF only • Support only for single service
• Works for both “network centric” • Traffic redirected only through the node insertion
and “app centric” designs service node on the provider’s site • Intra-VRF and inter-VRF
• Works for both “network centric”
and “app centric” designs
BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 117
ACI Multi-Site
Where to Go for More Information

✓ ACI Multi-Pod White Paper

http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-
737855.html?cachemode=refresh
✓ ACI Multi-Pod Configuration Paper
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739714.html
✓ ACI Multi-Pod and Service Node Integration White Paper
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739571.html
✓ ACI Multi-Site White Paper
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739609.html
✓ Cisco Multi-Site Deployment Guide for ACI Fabrics
https://www.cisco.com/c/en/us/td/docs/dcn/whitepapers/cisco-multi-site-deployment-guide-for-aci-fabrics.html
✓ ACI Multi-Site and Service Node Integration White Paper
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-743107.html
✓ ACI Multi-Site Training Sessions
https://www.cisco.com/c/en/us/solutions/data-center/learning.html#~nexus-dashboard

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 118
Complete your Session Survey
• Please complete your session survey
after each session. Your feedback
is important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (open from Thursday) to
receive your Cisco Live t-shirt.
• All surveys can be taken in the Cisco Events Mobile App or
by logging in to the Session Catalog and clicking the
"Attendee Dashboard” at
https://www.ciscolive.com/emea/learn/sessions/session-catalog.html

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 119
Continue
Agenda Your Education

Visit the Cisco Showcase for related demos.

Book your one-on-one Meet the Engineer meeting.

Attend any of the related sessions at the DevNet,

Capture the Flag, and Walk-in Labs zones.

Visit the On-Demand Library for more sessions

at ciscolive.com/on-demand.

BRKDCN-2980 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 120
Thank you