Routing L3 to External Networks

www.lumoscloud.com
[email protected]
Agenda
 Overview of Layer 3 Connectivity
 Shared L3 Out
Overview of Layer 3
Connectivity
What is an L3out

VRF

BD BD
10.1.1.1/24 10.1.2.1/24

EPG EPG

Servers Contract
Servers
What is an L3out

VRF

BD L3out
10.1.1.1/24
External
External EPG Router
EPG (L3out network)

Servers 123.1.1.0/24
Contract
123.456.789.0/24
......
ACI Routing Keys
• Re-using EPG-Contract model
• IP address based classification
• L3extSubnet != Route
• No endpoint learning on L3out
• MB-BGP is used for redistribution within the fabric
• Route-map is the key for cross protocol redistribution
– Pervasive (static) L3out
– Different protocols
– Same protocol, cross border leafs
 Connecting ACI via Layer 3
A subnet is routed internally as
host routes. Externally, a subnet Leafs connected to outside
is summarized to its configured networks are referred to as
netmask to external devices. “border leafs”. Any leaf can
be a border leaf.

Route learned:
2.2.2.0/24
VM: 2.2.2.10
Inside the fabric
Supported routing
protocols (ACI 3.0) Outside the fabric
• OSPF (v2/v3) Router External routes are distributed
• EIGRP internally via MP-BGP and route
• eBGP reflectors
• iBGP
• Static routes
Router Connections: ECMP and vPC

Multiple links ECMP over

with ECMP multiple routers

Layer 3 vPC is
Router Router supported Router
 Checklist for Layer 3 Connectivity
• Bridge Domain subnet is
marked as “advertised
externally”
• Create Routed Outside (OSPF,
e/iBGP, EIGRP, Static)
• Assign Router IDs to any border
leafs (for dynamic protocols)
• Configure border leaf interface
profiles (SVI, L3, L3 sub-
interface)
• Create an L3 External EPG
Mark Subnet for Routing Externally
Tenant -> (Tenant Name) ->
Networking -> Bridge
Domain -> Subnets

Each subnet created under any Bridge Domain

will be configured as either private to a VRF, or
have the ability to be advertised. Selecting
“Advertised” only provides the ability to be
routed. Other steps must be completed as well.
L3out
• L3out profile – VRF/ Domain/ Protocol
– Node profile – Border leaf/ static route Routing
• Interface profile – interface level
– EPG
• Subnet Contract
• Contract Policy enforcement
Configuration
- Next hop connectivity
- Configure Subnet(s) - Peer established
4 - Advertised externally - Route Learned
- Associate with the L3out
- Enable unicast routing 6 VRF

BD L3out
10.1.1.1/24
L3 Network 1 External
Or Router
EPG 2
External EPG

123.1.1.0/24
Contract
123.456.789.0/24
3 ......
Layer 3 Connectivity Topology
Leaf 101 and 102 will
both need a router ID
each for each VRF

Each interface will leaf101 leaf102

need to be configured
via SVI, routed, or
routed sub-interface
Router1 Router2
 Layer 3 Example: Routed Sub-Interface
Tenant: Coke
VRF: Coke-VRF
Router-ID: 10.254.0.101 Router-ID: 10.254.0.102 Protocol: OSPF
leaf101 leaf102
E1/1.10: 10.201.0.1 E1/1.10: 10.202.0.1

Router1 Router2 GigabitEthernet 1/1.10: 10.202.0.2

GigabitEthernet 1/1.10: 10.201.0.2

 Layer 3 vPC Example: SVI

Tenant: Coke
Router-ID: 10.254.0.101 Router-ID: 10.254.0.102
VRF: Coke-VRF
Protocol: OSPF
Primary IP: 10.202.0.1 leaf101 leaf102
Primary IP: 10.202.0.1
Secondary IP: 10.202.0.2 Secondary IP: 10.202.0.2

vPC

Router1 Port Channel 10: 10.202.0.3

 Create Routed Outside
Tenant -> (Tenant Name) ->
Networking -> External
Routed Networks

Select routing
protocol

Select VRF and External

Routed Domain
ACI L3 Connection to External Network – Route Peering
Interface Options
• L3 port
• SVI
• Sub-interface
L3 Connection Scale
• 400 VRFs per leaf
• ALE1/ALE2
• 10,000 IPv4 LPMs
• 6,000 IPv6 LPMs
• LSE/LS1800FX
• 20,000 IPv4 LPMs
• 10,000 IPv6 LMPs
• 1K LPM entries to derive EPG for
external subnets
ACI L3 Packet Forwarding – Inside and Outside
2A 2B

If the destination IP address is outside, If the destination IP address is

ACI fabric forward to TEP the nearest inside the tenant space, ACI
external router is attached to forwards to the TEP where the
(best route if multiple external attached) destination endpoint is attached

Routes learned from peering

routers are marked as 1
‘outside’
Frame arrives from an endpoint
that is inside tenant space

Single Data Plane with Two Control Planes

Which ‘forwarding space’ is used to forward packet is determined by which IP network it is in and where is it going
• Inside networks are those associated with tenants and their bridge domains (BD’s)
• Outside networks are those associated with the outside routes for each of those tenants
ACI Layer 3 – Route Distribution
BGP Route Reflector
BGP Peering between Leaves and Route
Reflector(s) located in spine

MP-BGP
Route Redistribution between internal
BGP and ‘outside’ occurs on border
leaves
External Tenant and Infra Routes
exchanged with external routers

Fabric leverages MP-BGP for distributing external routes, “outside EPG’s” to leaf switches
• Border leaf switch can peer with external networks and redistribute routing information about external networks into the internal
MP-BGP
o OSPF, Static, iBGP (FCS)
o MP-BGP w EVPN AF, EIGRP, IS-IS, OSPFv3 (Post FCS)
• Only “Public Subnet”(under Bridge Domain configuration) are announced to external network
ACI Layer 3 – BGP Route Reflector Configuration
ACI Layer 3 – OSPFv2 Peering Consideration
BGP Route Reflector interface Ethernet1/1.1000
encapsulation dot1q 1000
vrf member Tenant2
ip address 200.200.200.2/30
ip ospf network point-to-point
ip router ospf 1 area 0.0.0.1
router ospf 1
vrf Tenant2
area 0.0.0.1 nssa
default-information originate always
Sub-interface vrf Tenant3
area 0.0.0.1 nssa
OSPF peering in NSSA area default-information

ACI fabric is not able to be a transit network until ACI v1.1

• Must use non-backbone OSPF area; must use NSSA area
• VRF-lite for tenant routes separation. One OSPFv2 adjacency per tenant or use static routes. OSPF or static routes may be required for
iBGP peer address reachability
• Inside ACI, routes learnt via OSPF are redistributed to BGP and distributed to leaf nodes
• Tenant public subnet is redistributed to OSPF NSSA area in border leaf
ACI Layer 3 Outside – iBGP Peering Consideration
BGP Route Reflector
Support iBGP – one AS number for the fabric
• Customer edge router iBGP peering with
ACI fabric Border Leaf.
AS 300
• VRF-lite for tenant L3 separation. One
iBGP peering per tenant
• Customer Edge Router is needed for:
o Large routing table iBGP

o Or greater VRF support Customer Edge Router

o Or WAN features and more
eBGP
sophisticated BGP policy
Tenant Router Tenant Router

AS 100 Tenant A AS 200 Tenant B

Internal EPG to External EPG
Forwarding and Policy Lookup contract

For L3 outside connection, external EPG is derived from subnet

• Supports multiple external EPGs; external EPG1 could be remote branch or another
DC. External EPG2 could be Internet
4. Apply policy based on
• Different policy for different external EPGs
source, destination EPG and
configured contract LEAF 6
LEAF 1 2. External LPM table lookup External LPM Table
External LPM Table with destination IP. Find border 100.1.1.0/24 ExtEPG1
100.1.1.0/24 Leaf 6 leaf VTEP IP
200.1.1.0/24 ExtEPG2
200.1.1.0/24 Leaf 6

3. Derive destination EPG by

Global Station Table Leaf 1 checking destination IP
10.1.3.35/32 Leaf 3

* Proxy A WEB ExtEPG1

100.1.1.0/24
Local Station Table 10.1.3.11 EPG 10.1.3.35
100.1.2.0/24
10.1.3.11/32 Port 9
ExtEPG2
1. Derive source EPG and set 200.1.1.0/24
source EPG in VXLAN header 200.1.2.0/24
 External EPG to Internal EPG
Forwarding and Policy Lookup
LEAF 6
• For L3 outside connection, external EPG is derived from subnet 1. Check source IP against External LPM Table
External EPG mapping 100.1.1.0/24 ExtEPG1
• Support multiple external EPGs. External EPG1 could be remote table, derive source EPG.
branch or another DC. External EPG2 could be Internet 200.1.1.0/24 ExtEPG2
Mark source EPG in the
• Different policy for different external EPGs VXLAN header
Global Station Table
2. Check destination IP 10.1.3.35/32 Leaf 3
3. Apply policy at ingress in case against GST. Derive 10.1.3.11/32 Leaf 1
there is hit in the GST. If miss apply destination EPG. Send to * Proxy A
policy at egress leaf. egress leaf. If missed send to Local Station Table
spine for proxy

Leaf 6

ExtEPG1
WEB 100.1.1.0/24
100.1.2.0/24
10.1.3.11 EPG 10.1.3.3
5 ExtEPG2
200.1.1.0/24
200.1.2.0/24
ACI L3 Outside Connection – IP Multicast Traffic
ACI supports IGMP snooping and L2 bridging for IP
multicast traffic
• L2 multicast bridging within Bridge
Domain based on IGMP snooping entries.
• Need external PIM router for L3 routing
across Bridge Domain boundary
• L2 outside connection to the external PIM
router for source and receiver bridge
domain
EPG
Bridge
EPG 1 2
Domain
Bridge Domain 1 2
PIM Router
ACI Transit Routing
Transit Routing
VRF

redistributing subnet-A

L3out-1 L3out-2

External EPG External EPG

(L3out network) (L3out network)
OSPF OSPF
Area 0 Area 1

Subnet-A Contract Subnet-B

Transit Routing
VRF

redistributing subnet-B

L3out-1 L3out-2

External EPG External EPG

(L3out network) (L3out network)
OSPF OSPF
Area 0 Area 1

Subnet-A Contract Subnet-B

Transit Routing Checkboxes

Advertise this
transit route to
external routers.

Use this prefix for the Leak this external

external EPG route to other VRF
classifier. (Checked instances. NEW
by default)

Select aggregate
route import and Use this prefix for the NEW
export if using 0/0 external EPG classifier
network, or use across VRF instances
specific routes
 Transit Routing VRF
redistributing Area 0 and Area 1

L3out-1 L3out-2

External EPG External EPG

(L3out network) (L3out network)
OSPF OSPF
Area 0 Area 1

Contract
External Router 1 (Area 0): External Router 2 (Area 1):
•Loopback0: 2.2.2.2/32 •Loopback0: 3.3.3.3/32
•Loopback1: 4.4.4.4/32 •Loopback1: 5.5.5.5/32
•Transits: 192.168.0.0/29 •Transits: 172.16.0.0/29
 Transit Routing Example

Use this prefix for the

external EPG
classifier. (Checked
by default)

Select aggregate
route import and
export if using 0/0
network, or use
External Router 1 (Area 0):
specific routes
External Router 2 (Area 1):
•Loopback0: 2.2.2.2/32 •Loopback0: 3.3.3.3/32
•Loopback1: 4.4.4.4/32 •Loopback1: 5.5.5.5/32
•Transits: 192.168.0.0/29 •Transits: 172.16.0.0/29
Transit Routing Example
No routes being learned
across fabric from
router-2

But, full OSPF adjacency

Use this prefix for the
seen… external EPG
classifier. (Checked
by default)
Only the loopbacks of the ACI
Select aggregate
route import and Fabric (Area 0) are seen. ACI
export if using 0/0
network, or use
blocks transit routes between
specific routes different L3Outs unless permitted
by policy via an OSPF area filter-
list (to verify ssh to the border
leaf and run “show route-map”).
 Transit Routing Example

Use this prefix for the

external EPG
classifier. (Checked
by default)

In the L3Out navigation pane select Now the Import and Export options
the “Route Control Enforcement:” are selectable from the L3Out EPG…
import checkbox…
Transit Routing Example

Use this prefix for the

external EPG
classifier. (Checked
There is still have no change in the routing table.
by default)

Select aggregate
Did you notice that we are
route import and using a catch all 0.0.0.0/0?
export if using 0/0
This would require usortousealso select the aggregate export
network,
and import features on the subnet/network epg we have
specific routes

created for Area 0 and Area 1:

Transit Routing Example

Now that the aggregate Import

Use this prefix for the
and Aggregate Export options are
external EPG checked…
classifier. (Checked
by default)

Select aggregate
route import and
export if using 0/0
network, or use
specific routes
 Transit Routing Example

Use this prefix for the

external EPG
classifier. (Checked
by default)

Select aggregate
route import and
export if using 0/0
network, or use
specific routes

Routes are now being learned across fabric from router-2

Transit Routing Example
We can choose to do specific
routes instead of aggregates…

Use this prefix for the

external EPG
classifier. (Checked
by default)

Select aggregate
route import and
export if using 0/0
network, or use
specific routes

Notice only the 2.2.2.2/32 is now being learned

…But what are these checkboxes actually doing?

Use this prefix for the

external EPG
classifier. (Checked
by default)

Select aggregate
Prior to checking Import/Export features
route import and
export if using 0/0
network, or use
specific routes
After to checking Import/Export features
These checkboxes are updating route-maps and prefix-lists within ACI
Shared L3 Out
 Shared Layer 3 Out
Options for Connecting Tenants to Network
• VRF-Lite (each Tenant with its own routing sessions)
• Shared Common-owned VRF, Bridge Domains in
Common Tenant, Shared L3 Out
• Shared Common-owned VRF, Bridge Domains in each
Tenant, Shared L3 Out
• Individual Tenant VRFs and Bridge Domains, Leaked to
Common Tenant, Shared L3 Out
 Sharing VRF and L3Out Among Tenants
Bridge Domain, Subnet, and L3Out Under Tenant Common
Dynamic protocol
Static route

VRF
Tenant-Pepsi Tenant-Coke Tenant-Common
L3Out
Web APP DB Web APP DB
C C BD-Coke BD-Pepsi
C C 192.168.102.1/24 192.168.101.1/24

• No overlapping IP addresses among tenants, VRF instances shared among tenants, and traffic isolation through contract
• Bridge domain and subnet and L3Out defined under tenant common
• EPG, contract, and application profile under individual tenants
• Dynamic routing protocol with external routers
 Sharing VRF and L3Out Among Tenants
Bridge Domain and Subnet Under Tenant
Static route now dynamic protocol with
Release 11.1(3x) or later

VRF
Tenant-Pepsi Tenant-Coke Tenant-Common

BD-Pepsi BD-Coke
192.168.101.1/24 192.168.102.1/24
L3Out
Web APP DB Web APP DB
C C C C

• L3Out defined under tenant common

• Bridge domain, EPG, contract, and application profile under individual tenants
• Static or dynamic routing protocol between border leaf and external router
 Access Shared Services Located Outside
Design Prior to Cisco ACI 1.2(x) Release
VRF 1 VRF 2
VRF route leak
20.20.20.0/24
• Each tenant has its own VRF instance (due to
External EPG External EPG overlapping IP addresses). Tenants need to access
20.20.20.0/24 20.20.20.0/24 services and resources reachable through L3Out.
• VRF-lite is used between the border leaf and
Tenant-Pepsi Tenant-Coke
external router.
• Each tenant has its own L3Out connection.
VRF 1 L3Out VRF 2 L3Out
• External router has VRF route leak.
BD-Pepsi BD-Coke
192.168.101.1/24 192.168.102.1/24 • Each tenant needs to define an external EPG and
Web APP DB Web APP DB
contract to access the shared services.
C C C C
 Sharing L3Out Across the VRF Instance
With Brazos (Release 11.2)

Tenant-Pepsi Tenant-Coke Tenant-Common Dynamic and static

VRF1 VRF2 VRF3

BD-Pepsi BD-Coke
192.168.101.1/24 192.168.102.1/24
L3Out
External EPG 1
Web APP DB Web APP DB (shared services)
C C C C

• Each tenant needs its own VRF instance.

• L3Out is under tenant common. All tenants have the same L3Out connection.
• Each tenant can access the shared services provided by workloads in external EPGs.
• Address L3Out and external EPG scale.
 Sharing L3Out Across VRF Instances
Route Propagation
MP-BGP with VRF route leak
20.20.20.0/24
Tenant-Pepsi Tenant-Coke Tenant-Common 192.168.101.0/24
192.168.102.0/24
VRF1 VRF2 VRF3
20.20.20.0/24 20.20.20.0/24
BD-Pepsi BD-Coke
192.168.101.1/24 192.168.102.1/24
(public and shared) (public and shared)
Border Leaf L3Out
External EPG 1
Web APP DB Web APP DB 20.20.20.0/24
C C C C 20.20.20.0/24
192.168.101.0/24
192.168.102.0/24

• External routes are learned by the border leaf. The border leaf leaks routes to the tenant VRF instance through multiprotocol
Border Gateway Protocol (MP-BGP).
• Tenant subnets that are marked as public and shared are leaked to the tenant common VRF instance and advertised to
external routers.
 Sharing L3Out Across VRF Instances
with Cisco ACI 1.2(x)
Tenant 1
VRF1
External
EPG
EPG 1 (Consumer)
(Provider or L3Out 1
Consumer) Tenant-Common
VRF-Common
External EPG
(Shared
Tenant 2 L3Out Shared Service Provider)

VRF2
External
EPG
EPG 2 (Consumer)
(Provider or L3Out 2 Consumer Provider
Consumer)

Consumer Provider
or Provider or Consumer

• Shared service provider is an external EPG.

• Shared service provider can be in any tenants.
 Shared Service with L3Out Across
VRF Instances
Tenant 1
VRF1
External
EPG
EPG 1 (Consumer)
(Shared Service L3Out 1
Consumer) Tenant 3
VRF 3
Shared Service
EPG (Provider) External EPG 3
Tenant 2
L3Out 3
VRF2
External
EPG
EPG 2 (Consumer)
(Shared Service L3Out 2 Consumer Provider
Consumer)

Consumer Provider
or Provider or Consumer

• Shared service provider is tenant EPG.

• External EPGs of different tenant and VRF access to shared services.
 Configure Tenant Subnets to Leak
The leakage is triggered by establishing a contract relationship with an external EPG under a different VRF instance.
Bridge domain subnet configuration
Consumer Provider
Tenant EPG

C External EPG
Under Shared L3Out

Do not advertise subnet to

L3Out in its own VRF
instance.

Advertise subnet to L3Out

of its own VRF instance.

Leak subnet to VRF instance

NEW
in which provider EPG
resides.

*Note: Screen shot was taken with Brazos EFT code

 Configure External Subnets to Leak
The leakage is triggered by establishing a contract relationship with a tenant EPG under a different VRF instance.
External EPG subnet configuration

Advertise this transit

route to external routers.

Use this prefix for the Leak this external route to

external EPG classifier. other VRF instances. NEW

Use this prefix for the

external EPG classifier
across VRF instances NEW
(mask out VRF ID; see the
next slide).
 Sharing L3Out Across VRF Instances
Sharing External EPG Classifier across VRF Instances

VRF_ID Prefix Class_ID

VRF1 192.100.1.0/24 1111

VRF1 192.100.2.0/24 1111
VRF2 192.200.1.0/24 2222 This is the shared external
VRF2 192.200.2.0/24 2222 EPG prefix classifier. Mask
out the VRF ID. All tenants
* 20.20.20.0/24 3333 use the same copy of
* 30.30.30.0/24 3333 classifiers to derive the
class ID for the shared
VRF1 0.0.0.0/0 1112
services external EPG.
VRF2 0.0.0.0/0 22223

• Normally policy is applied on the border leaf for traffic leaving the fabric.
• With a shared L3Out solution, policy for traffic destined for a shared service EPG is applied on ingress on the leaf itself.
• The same external EPG definition is used for all tenants for shared services.
 Sharing L3Out Across VRF Instances
Sample Design
Tenant-SharedService Tenant-Consumer
VRF-Service VRF-Consumer
BD-Service BD-Consumer
40.40.40.1/24 50.50.50.1/24
(public and shared) (public and shared)

EPG-Storage
L3Out- EPG-Consumer L3Out-
40.40.40.1/24
SharedService Consumer
(public+shared)
External EPG
External EPG
50.50.51.0/24
0.0.0.0 50.50.52.0/24

• EPG-Storage provides shared service.

• EPGs from other tenants and external EPGs of multiple tenants can access the shared service.
• See the note on the slide for the results of route leaking.
 Sharing L3Out Across VRF Instances
Sample Design
Tenant-SharedService Tenant-Consumer
VRF-Service VRF-Consumer
BD-Service BD-Consumer
40.40.40.1/24 50.50.50.1/24
(public and shared) (public and shared)

EPG-Storage
L3Out- 40.40.40.1/24 L3Out-
EPG-Consumer
SharedService (public Consumer
and shared)
External EPG
External EPG
50.50.51.0/24
0.0.0.0 50.50.52.0/24

• Provider: The same subnet needs to be configured under EPG and bridge domain. Mark the subnet as “public” and “shared.”
• Consumer: The subnet is required only under the bridge domain. Mark the subnet as “public” and “shared.”
• Subnet of external EPG under Tenant-Consumer: Select all three options.
 Shared Service Provider Bridge Domain and EPG
Configuration

The subnet is configured

under the bridge domain.

The same subnet is

configured under the EPG:
required when the EPG is
the provider for shared
services (across VRF
instances).

• Subnet Under Bridge Domain: For advertising routes to L3Out and leaking routes to other VRF instances
• Subnet Under EPG: Required for policy reasons
Shared Service Consumer Configuration

Set the subnet under the bridge domain to be shared.

 External EPG Configuration

• The subnet under the external EPG needs to be marked with Shared Route Control and Shared Security Import Subnet.
• After the Shared Security Import Subnet option is selected, the subnet is used as the external EPG classifier for all VRF instances. When defining
the external EPG in such way from multiple VRF instances, make sure there are no overlapping subnets.