Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

First Published: 2015-12-08

Last Modified: 2019-07-29

Americas Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 527-0883
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS,
INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH
THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY,
CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of
the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS.
CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT
LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS
HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network
topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional
and coincidental.

All printed copies and duplicate soft copies of this document are considered uncontrolled. See the current online version for the latest version.

Cisco has more than 200 offices worldwide. Addresses and phone numbers are listed on the Cisco website at www.cisco.com/go/offices.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: www.cisco.com
go trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any
other company. (1721R)
© 2015–2019 Cisco Systems, Inc. All rights reserved.
Please send general FSF & GNU inquiries to [email protected]. There are also other ways to contact the FSF. Please send broken links and other corrections or suggestions to [email protected].
Please see the Translations README for information on coordinating and submitting translations of this article.

Copyright © 2007, 2009, 2011 Free Software Foundation, Inc. Verbatim copying and distribution of this entire article are permitted worldwide, without royalty, in any medium, provided
this notice, and the copyright notice, are preserved. Updated: Date: 2011/06/28 02:44:32
© 2015–2019 Cisco Systems, Inc. All rights reserved.
 CONTENTS

Full Cisco Trademarks with Software License ?

PREFACE Preface xxi

Audience xxi
New and Changed Information xxi
Document Conventions xxxii
Related Documentation xxxiv
Documentation Feedback xxxiv

CHAPTER 1 Using the APIC CLI 1

Accessing the NX-OS Style CLI 1

Using the NX-OS Style CLI for APIC 2
Differences in Usage from NX-OS 5
Mixing the NX-OS Style CLI and the APIC GUI 5
About the Modes of Configuring Layer 3 External Connectivity 6

CHAPTER 2 Configuring Fabric and Interfaces 9

Fabric and Interface Configuration 9

Graceful Insertion and Removal (GIR) Mode 10
Removing a Switch to Maintenance Mode Using the CLI 11
Inserting a Switch to Operation Mode Using CLI 11
Configuring Physical Ports in Leaf Nodes and FEX Devices Using the NX-OS CLI 11
Configuring Port Channels in Leaf Nodes and FEX Devices Using the NX-OS CLI 14
Configuring Virtual Port Channels in Leaf Nodes and FEX Devices Using the NX-OS CLI 20
Configuring FEX Connections Using Profiles with the NX-OS Style CLI 25
Reflective Relay (802.1Qbg) 26

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

v
 Contents

Enabling Reflective Relay Using the NX-OS CLI 27

Configuring Policy Groups for Interfaces 28
Configuring Overrides for Interfaces 31
About Forwarding Error Correction 33
Configuring FEC Using NX-OS Style CLI 33

CHAPTER 3 Cisco ACI Smart Licensing 35

About Smart Licensing 35

Smart Licensing Usage Guidelines and Limitations 36

Pre-Registration Verifications 36
Verification Checklist for CSSM Configurations 36
Verification Checklist for Smart Licensing and APIC Configurations 36
Registering for Smart Licensing Using the CLI 36

CHAPTER 4 Configuring APIC High Availability 39

About Cold Standby for APIC Cluster 39

Switching Over Active APIC with Standby APIC Using CLI 40

CHAPTER 5 Configuring Tenants 41

Creating a Tenant, VRF, and Bridge Domain 41
Additional Bridge Domain Configuration 44
Configuring an Enforced Bridge Domain 45
Configuring an Enforced Bridge Domain 46
Configuring an Enforced Bridge Domain Using the NX-OS Style CLI 47
Creating an Application Endpoint Group 48
Configuring Legacy Forwarding Mode in the Bridge Domain 51
Configuring Contracts 52
Contract Inheritance 56
About Contract Inheritance 56
Configuring Application or uSeg EPG Contract Inheritance Using the NX-OS Style CLI 57
Configuring L2Out EPG Contract Inheritance Using the NX-OS Style CLI 61
Configuring External L3Out EPG Contract Inheritance Using the NX-OS Style CLI 63
Configuring Contract Preferred Groups 65
About Contract Preferred Groups 65

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

vi
 Contents

Configuring Contract Preferred Groups Using the NX-OS Style CLI 67

Exporting a Contract to Another Tenant 68
Configuring Contract or Subject Exceptions 70
Configuring Contract or Subject Exceptions for Contracts 70
Configure a Contract or Subject Exception Using the NX-OS Style CLI 71
Creating Quota Management 72
About APIC Quota Management Configuration 72
Creating a Quota Management Configuration Using the NX-OS Style CLI 72

CHAPTER 6 Configuring Layer 2 External Connectivity 75

Configuring Layer 2 External Connectivity 75

Configuring VLAN Domains 79
About VLAN Domains 79
Basic VLAN Domain Configuration 80
Advanced VLAN Domain Configuration 81
Associating a VLAN Domain to a Port 82
Associating a VLAN Domain to a Port-Channel 83
Associating a VLAN Domain to a Template Policy-Group 84
Associating a VLAN Domain to a Template Port-Channel 85
Associating a VLAN Domain to a Virtual Port-Channel 85
Configuring Q-in-Q Encapsulation Mapping for EPGs 86
Q-in-Q Encapsulation Mapping for EPGs 86
Mapping EPGs to Q-in-Q Encapsulated Leaf Interfaces Using the NX-OS Style CLI 87
Support Fibre Channel over Ethernet Traffic on the ACI Fabric 88
Supporting Fibre Channel over Ethernet Traffic on the ACI Fabric 88

FCoE NX-OS Style CLI Configuration 91

Configuring FCoE Connectivity Without Policies or Profiles Using the NX-OS Style CLI 91
Configuring FCoE Connectivity With Policies and Profiles Using the NX-OS Style CLI 94
Configuring FCoE Over FEX Using NX-OS Style CLI 98
Verifying FCoE Configuration Using the NX-OS Style CLI 100
Undeploying FCoE Elements Using the NX-OS Style CLI 101
Fibre Channel NPV 102
Fibre Channel Connectivity Overview 102
Fibre Channel N-Port Virtualization Guidelines and Limitations 103

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

vii
Contents

Configuring FC Connectivity Without Policies or Profiles Using the NX-OS CLI 104
Configuring FC Connectivity With Policies or Profiles Using the NX-OS CLI 106
Configuring 802.1Q Tunnels 108
About ACI 802.1Q Tunnels 108
Configuring 802.1Q Tunnels Using the NX-OS Style CLI 110
Example: Configuring an 802.1Q Tunnel Using Ports with the NX-OS Style CLI 111
Example: Configuring an 802.1Q Tunnel Using Port-Channels with the NX-OS Style CLI 112
Example: Configuring an 802.1Q Tunnel Using Virtual Port-Channels with the NX-OS Style
CLI 113
Configuring Dynamic Breakout Ports 113
Configuration of Dynamic Breakout Ports 113
Configuring Dynamic Breakout Ports Using the NX-OS Style CLI 114
Configuring Port Profiles 118
Configuring Port Profiles 118
Port Profile Configuration Summary 120
Configuring a Port Profile Using the NX-OS Style CLI 122
Verifying Port Profile Configuration and Conversion Using the NX-OS Style CLI 123
Microsegmentation on Virtual Switches 124
Configuring Microsegmentation on Virtual Switches 124
Configuring Microsegmentation with Cisco ACI Using the NX-OS-Style CLI 125
Configuring Microsegmentation on Bare-Metal 127

Using Microsegmentation with Network-based Attributes on Bare Metal 127

Configuring a Network-Based Microsegmented EPG in a Bare-Metal Environment Using the NX-OS
Style CLI 127
Configuring Layer 2 IGMP Snoop Multicast 129
About Cisco APIC and IGMP Snooping 129
Enabling IGMP Snooping Static Port Groups 130
Configuring and Assigning an IGMP Snooping Policy to a Bridge Domain using the NX-OS Style
CLI 130
Enabling IGMP Snooping and Multicast on Static Ports in the NX-OS Style CLI 132
Enabling IGMP Snoop Access Groups 133
Enabling Group Access to IGMP Snooping and Multicast using the NX-OS Style CLI 133
Deploying an EPG on a Specific Port with APIC Using the NX-OS Style CLI 135
Configuring Port Security 136
About Port Security and ACI 136

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

viii
 Contents

Port Security Guidelines and Restrictions 137

Port Security at Port Level 137

Configuring a Port Security Policy Group Template 137

Configuring Port Security on an Interface Using a Template 139
Configuring Port Security on an Interface Using Overrides 140
802.1x Port and Node Authentication 141
802.1x Port and Node Authentication 141
Configuring a Port Authentication Policy 141
Configuring a Node Authentication Policy 142
Configuring Proxy ARP 144
About Proxy ARP 144
Guidelines and Limitations 149
Configuring Proxy ARP Using the Cisco NX-OS Style CLI 149
Configuring Flood in Encapsulation 151
Configuring Traffic Storm Control 152
About Traffic Storm Control 152
Storm Control Guidelines 152
Configuring a Traffic Storm Control Policy Using the NX-OS Style CLI 154
Configuring MACsec 155
About MACsec 155
Guidelines and Limitations for MACsec 156
Configuring MACsec Using the NX-OS Style CLI 158

CHAPTER 7 Configuring Layer 3 External Connectivity 161

About the Modes of Configuring Layer 3 External Connectivity 161

Configuring Layer 3 External Connectivity 163
Routed Connectivity to External Networks 163
About Routed Connectivity to Outside Networks 163
Layer 3 Out for Routed Connectivity to External Networks 164
Guidelines for Routed Connectivity to Outside Networks 165
External Layer 3 Outside Connection Types 167
Configuring a Layer 3 Outside for Tenant Networks Using the NX-OS Style CLI 169
NX-OS Style CLI Example: L3Out Prerequisites 173
NX-OS Style CLI Example: L3Out 173

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

ix
Contents

Layer 3 Routed and Sub-Interface Port Channels 175

About Layer 3 Port Channels 175
Configuring a Layer 3 Routed Port-Channel Using the NX-OS CLI 175
Configuring a Layer 3 Sub-Interface Port-Channel Using the NX-OS CLI 177
Adding Ports to the Layer 3 Port-Channel Using the NX-OS CLI 180
Layer 3 Out to Layer 3 Out Inter-VRF Leaking 181
Configuring Shared Layer 3 Out Inter-VRF Leaking Using the NX-OS Style CLI - Named Example
182

Configuring Shared Layer 3 Out Inter-VRF Leaking Using the NX-OS Style CLI - Implicit Example
183

About SVI External Encapsulation Scope 185

Encapsulation Scope Syntax 187
Configuring SVI Interface Encapsulation Scope Using NX-OS Style CLI 187
About SVI Auto State 188

Guidelines and Limitations for SVI Auto State Behavior 189

Configuring SVI Auto State Using NX-OS Style CLI 189
Configuring an Interface and Static Route 190

OSPF Configuration 193

Configuring OSPF 193
Creating OSPF VRF and Interface Templates 196
BGP Configuration 200
Configuring BGP 200
Creating BGP Address Family and Timer Templates 201
Configuring BGP Address Family and Timers 202
Configuring a BGP Neighbor 204
Configuring a Per VRF Per Node BGP Timer Policy Using the NX-OS Style CLI 208
Configuring BGP Max Path 209
Configuring AS Path Prepend 210
Configuring AS Path Prepend Using the NX-OS Style CLI 211
Route Distribution Into BGP 212
Configuring a Route-Profile with Tenant Scope 212
Configuring a Redistribute Route-Profile 213
Configuring BGP Route Dampening 214
EIGRP Configuration 217

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

x
 Contents

Creating EIGRP VRF and Interface Templates 217

Configuring EIGRP Address Family and Counters 219
Configuring an EIGRP Interface 221
Configuring Route-Maps 224
Configuring Templates 224
About Route Profiles 224
Configuring a Tenant-Scoped Route Profile 224
Configuring a VRF-Scoped Route Profile 226
Creating a Route-Map 228
Configuring Route-Maps in Routing Protocols 232
Configuring an Export Map (Inter-VRF Route Leak) 233
Configuring Bi-Directional Route Forwarding (BFD) 234
About BFD 234
Configuring BFD Globally 235
Configuring BFD Globally on Leaf Switch Using the NX-OS Style CLI 237
Configuring BFD Globally on Spine Switch Using the NX-OS Style CLI 238
Overriding Global BFD Settings 239
Configuring BFD Interface Override Policy 239
Applying the BFD Interface Override Policy to Interfaces 242
Enabling BFD on Consumer Protocols 244
Enabling BFD on the BGP Consumer Protocol 244
Enabling BFD on the EIGRP Consumer Protocol 246
Enabling BFD on the OSPF Consumer Protocol 246
Enabling BFD on the Static Route Consumer Protocol 247
Configuring BFD Consumer Protocols Using the NX-OS Style CLI 248
Configuring Layer 3 Multicast 249
Layer 3 Multicast 249
Guidelines and Restrictions for Configuring Layer 3 Multicast 250
Configuration Steps for Layer 3 Multicast 252
Configuring PIM Options for Layer 3 Multicast 252
Configuring IGMP Options on the VRF for Layer 3 Multicast 255
Configuring an L3 Out for Layer 3 Multicast 259
Example: Configuring Layer 3 Multicast 263
Configuring External-L3 EPGs 264

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xi
Contents

Configuring Layer 3 External Connectivity Using the Named Mode 266

Creating a Named L3Out 266
Configuring Layer 3 Interfaces for a Named L3Out 268
Configuring Route Maps for a Named L3Out 270
Configuring Routing Protocols for a Named L3Out 273
Configuring BGP for a Named L3Out 273
Configuring OSPF for a Named L3Out 274
Configuring EIGRP for a Named L3Out 277
Configuring External-L3 EPGs for a Named L3Out 279
IPv6 Neighbor Discovery 280
Neighbor Discovery 280
Configuring a Tenant, VRF, and Bridge Domain with IPv6 Neighbor Discovery on the Bridge Domain
Using the NX-OS Style CLI 281
Guidelines and Limitations 282
Configuring an IPv6 Neighbor Discovery Interface Policy with RA on a Layer 3 Interface Using the
NX-OS Style CLI 282
Microsoft NLB 285
Configuring Microsoft NLB in Unicast Mode Using the NX-OS Style CLI 285
Configuring Microsoft NLB in Multicast Mode Using the NX-OS Style CLI 286
Configuring Microsoft NLB in IGMP Mode Using the NX-OS Style CLI 287
MLD Snooping 288
Configuring and Assigning an MLD Snooping Policy to a Bridge Domain using the NX-OS Style
CLI 288
Configuring HSRP 291
Configuring HSRP in Cisco APIC Using Inline Parameters in NX-OS Style CLI 291
Configuring HSRP in Cisco APIC Using Template and Policy in NX-OS Style CLI 292
Cisco ACI GOLF 294
Cisco ACI GOLF 294

Configuration Tasks to Configure Cisco ACI GOLF Services Using the NX-OS Style CLI 296
Configuring a Spine and the Infra Tenant for BGP EVPN, Using the NX-OS Style CLI 296
APIC GOLF Connections Shared by Multi-Site Sites 299
Recommended Shared GOLF Configuration Using the NX-OS Style CLI 300
Configuring BGP to Support BGP EVPN on a Spine, Using the NX-OS Style CLI 301
Configuring a Tenant for BGP EVPN Using the NX-OS Style CLI 302
Configuring a Route Map 304

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xii
 Contents

Enabling Distributing BGP EVPN Type-2 Host Routes to a DCIG Using the NX-OS Style CLI 306
Cisco ACI GOLF Configuration Example, Using the NX-OS Style CLI 307
Troubleshooting EVPN Type-2 Route Distribution to a DCIG 309
Multipod_Fabric 311
About Multipod Fabric 311
Assigning Switches in a Multipod Fabric 311
Configuring Fabric-External Connectivity for a Multipod Fabric 312
Configuring Spine Interfaces and OSPF for a Multipod Fabric 315
Remote Leaf Switches 318
About Remote Leaf Switches in the ACI Fabric 318
Remote Leaf Switch Hardware Requirements 319
Restrictions and Limitations 320
WAN Router and Remote Leaf Switch Configuration Guidelines 321
Configure Remote Leaf Switches Using the NX-OS Style CLI 322
Transit Routing 325
Transit Routing in the ACI Fabric 325
Transit Routing Related Topics 326
Transit Routing Overview 326
Guidelines for Transit Routing 328
Configure Transit Routing Using the NX-OS Style CLI 333
Example: Transit Routing 336

CHAPTER 8 Configuring Cisco ACI QoS 341

QoS for L3Outs 341
Configuring QoS for L3Outs Using the NX-OS Style CLI 341
Configuring QoS Directly on L3Out Using CLI 342
CoS Preservation 343
Preserving 802.1P Class of Service Settings 343
Enable Class Of Service (CoS) Preservation Using NX-OS Style CLI 344
Multipod QoS 345
Creating DSCP Translation Policy Using NX-OS Style CLI 345
Preserving QoS Priority Settings in a Multipod Fabric 346
Translating QoS Ingress Markings to Egress Markings 347
Translating QoS Ingress Markings to Egress Markings 347

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xiii
 Contents

Creating Custom QoS Policy Using NX-OS Style CLI 347

CHAPTER 9 Configuring Management Interfaces 349

Configuring Out-of-Band Management Access 349

Configuring Inband Management Access 351
Configuring Inband Management Access to a Switch from an Outside Network 351
Configuring Inband Management Access to a Controller from an Outside Network 353
Configuring Inband Management Connectivity to the Management Station 355
Configuring Inband Management Contract to Open HTTPS/SSH Ports 357

CHAPTER 10 Configuring Security 359

About Security Configuration 359
Configuring AAA 360
Configuring Security Servers 363
Configuring a RADIUS Server 363
Configuring a TACACS+ Server 366
Configuring an LDAP Server 367
Configuring the Password Policy 370
Configuring Users 373
Configuring a Locally Authenticated User 373
Configuring a Certificate and SSH-Key for a Local User 375
Configuring Public Key Infrastructure 377
Configuring a Certificate Authority and Chain of Trust 377
Configuring Keys and a Keyring 377
Generating a Certificate Signing Request 379
Configuring Webtokens 381
Configuring Communication Policies 382
Configuring the HTTP Policy 382
Configuring the HTTPS Policy 383
Configuring the SSH Policy 385
Configuring the Telnet Policy 386
Configuring AES Encryption 387
Configuring Fabric Secure Mode 388
Configuring COOP Authentication 389

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xiv
 Contents

About COOP Authentication 389

Configuring COOP Authentication 390
Configuring FIPS 390
About Federal Information Processing Standards (FIPS) 390
Guidelines and Limitations 391
Configuring FIPS for Cisco APIC Using NX-OS Style CLI 391
Configuring Control Plane Policing 392
Information About CoPP 392
Guidelines and Limitations for CoPP 394
Configuring CoPP Using the Cisco NX-OS CLI 394
Configuring Per Interface Per Protocol CoPP Policy Using the NX-OS Style CLI 395
Configuring First Hop Security 395
About First Hop Security 395
ACI FHS Deployment 396
Guidelines and Limitations 396
Configuring FHS Using the NX-OS CLI 397
Configuring 802.1x 403

802.1X Overview 403

Host Support 404
Authentication Modes 404
Guidelines and Limitations 404
Configuration Overview 405
Configuring 802.1X Node Authentication Using NX-OS Style CLI 406
Configuring 802.1X Port Authentication Using the NX-OS Style CLI 406

CHAPTER 11 Configuring Anycast Services 409

About Anycast Services 409

Configuring Anycast Services Using the NX-OS Style CLI 410

CHAPTER 12 Configuring VMM 415

Configuring VMM 415

CHAPTER 13 Configuring Layer 4 to Layer 7 Services 417

Configuring Layer 4 to Layer 7 Services 417

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xv
 Contents

CHAPTER 14 Configuring Global Policies 419

About Global Policies 419

Configuring Out-of-Band Management NTP 419
Configuring the System Clock 422
Configuring Error Disable Recovery 423
Configuring Link Level Discovery Protocol 424
Configuring Miscabling Protocol 424
Configuring the Endpoint Loop Protection Policy 426
Configuring the Rogue Endpoint Control Policy 427
About the Rogue Endpoint Control Policy 427
Configure Rogue Endpoint Control Using the NX-OS Style CLI 427
Configuring IP Aging 429
Overview 429
Configuring the IP Aging Policy Using the NX-OS-Style CLI 429
Configuring the Dynamic Load Balancer 429
Configuring Spanning Tree Protocol 431
Configuring IS-IS 432
Configuring BGP Route Reflectors 435
Decommissioning a Node 436
Configuring Power Management 436
Configuring a Scheduler 438
Configuring System MTU 440
About PTP 441
Guidelines and Limitations 442
Configuring PTP Using the NX-OS CLI 444

CHAPTER 15 Configuring Cisco Tetration Analytics 447

Overview 447
Configuring Cisco Tetration Analytics Using the NX-OS Style CLI 447

CHAPTER 16 Configuring NetFlow 451

About NetFlow 451

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xvi
 Contents

Configuring a NetFlow Exporter Policy for Virtual Machine Networking Using the NX-OS-Style CLI
452

Configuring NetFlow and Tetration Analytics Feature Priority Through Node Control Policy Using
NX-OS-Style CLI 452
Configuring NetFlow Node Policy Using the NX-OS-Style CLI 453
Configuring NetFlow Infra Selectors Using the NX-OS-Style CLI 453
Configuring NetFlow Overrides Using the NX-OS-Style CLI 456
Configuring NetFlow Tenant Hierarchy Using the NX-OS-Style CLI 456
Consuming a NetFlow Exporter Policy Under a VMM Domain Using the NX-OS-Style CLI for VMware
VDS 460
Enabling or Disabling NetFlow on an Endpoint Group Using the NX-OS-Style CLI for VMware
VDS 460

CHAPTER 17 Managing Firmware 463

Managing Firmware 463

Adding or Removing Repository Images 463
Changing Catalog Firmware 464
Upgrading Controller Firmware 465
Upgrading Switch Firmware 467

CHAPTER 18 Managing the Configuration with Snapshots 469

About Configuration Management and Snapshots 469

Exporting a Snapshot 469
Importing a Snapshot 471
Rollback Configuration Using Snapshots 472
Uploading or Downloading a Snapshot File to a Remote Path 473
Managing Snapshot Files and Jobs 475

CHAPTER 19 Configuring Monitoring 477

Configuring Syslog 477

Configuring a Logging Server Group 477
Configuring Syslog 479
Configuring Call Home 480
Configuring the Call Home Policy 480

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xvii
 Contents

Configuring a Call Home Destination Profile 482

Call Home Destination Profile Configuration Commands 484
Configuring a Call Home Query 485
Query Subtree Categories 486
Configuring TACACS External Logging 487
Creating a TACACS External Logging Destination Group Using the NX-OS-Style CLI 487
Creating a TACACS External Logging Source Using the NX-OS-Style CLI 488
Sending an On-Demand Tech Support File Using the NX-OS Style CLI 489
Configuring a Remote Path for File Export 490
Using Show Commands for Monitoring 491
About Using the Show Commands 491
Using the show faults Command 492
Using the show events Command 493
Using the show health Command 494
Using the show audits Command 495
Using the show stats Command 496
Entity Filters for Show Commands 497
Configuring SNMP 498
Configuring SNMP Policy Using CLI 499
Configuring Smart Callhome 501
About Smart Callhome 501
Creating a Smart Callhome Destination Group Using the NX-OS-Style CLI 501

CHAPTER 20 Configuring SPAN 505

Configuring SPAN and ERSPAN 505

SPAN Guidelines and Restrictions 505
Configuring Local SPAN in Access Mode 506
Configuring ERSPAN in Access Mode 508
Configuring ERSPAN in Fabric Mode 511
Configuring ERSPAN in Tenant Mode 514

CHAPTER 21 Applying the show running config Output to Another Cisco APIC 517
About Import and Export Configurations 517
Import and Export Configuration Guidelines and Limitations 517

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xviii
 Contents

Exporting a CLI Configuration 517

Importing a CLI Configuration 518

CHAPTER 22 Configuring a Forwarding Scale Profile Policy 521

Forwarding Scale Profile Policy Overview 521

Supported Platforms for Forwarding Scale Profile Policies 523
Configuring the Forwarding Scale Profile Policy Using the NX-OS-Style CLI 523

APPENDIX A Verified Scalability Using the CLI 527

CLI Scalability Limits 527

APPENDIX B Use Case: Three-Tier Application with Transit Topology 529

About Deploying a Three-Tier Application with Transit Topology 529
Deploying a Three-Tier Application 531
Transit Routing with OSPF and BGP 533

APPENDIX C Examples: Show Commands 535

Examples: Show Commands 535

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xix
Contents

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xx
 Preface
• Audience, on page xxi
• New and Changed Information, on page xxi
• Document Conventions, on page xxxii
• Related Documentation, on page xxxiv
• Documentation Feedback, on page xxxiv

Audience
This guide is intended for network and systems administrators who configure and maintain the Application
Centric Infrastructure fabric.

New and Changed Information

The following table provides an overview of the significant changes to this guide up to the current release.
The table does not provide an exhaustive list of all changes made to the guide or of the new features up to
this release.

Table 1: New and Changed Behavior in Cisco ACI, Release 3.2(1)

Feature Description Where Documented

Smart Licensing Smart Licensing is enabled in the Cisco ACI Smart Licensing, on
Cisco ACI fabric and by extension page 35
in the Cisco APIC as a Cisco Smart
Licensing-enabled product.
Layer 3 Routed and Sub-Interface Support for layer 3 port channels is #unique_7
Port Channels added.

Fibre Channel NPV Support for FC traffic over the Configuring Layer 2 External
Fabric. Connectivity, on page 75

802.1x enhancements Support for IP Phones Configuring Security, on page 359

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xxi
 Preface
Preface

Feature Description Where Documented

Anycast Services Anycast services are supported in Configuring Anycast Services, on

the Cisco ACI fabric. A typical use page 409
case is to support ASA firewalls in
the pods of a multipod fabric, but
Anycast could be used to enable
other services, such as DNS servers
or printing services.

Rogue Endpoint Control Support is added for global Rogue Configuring Global Policies, on
Endpoint Detection, to detect page 419
unauthorized EPs.

Enhanced Port Profile Support on Support is added on the Configuring Layer 2 External
N9K-C93180YC-FX Switches N9K-C93180YC-FX switch for Connectivity, on page 75
port profiles to change ports from
uplink to downlink or downlink to
uplink.

Enhanced Breakout Support on Support is added for 100 Gigabit Configuring Layer 2 External
Profiled QSFP Ports on (Gb) (4X25Gb) and 40Gb Connectivity, on page 75
N9K-C93180YC-FX Switches (4X10Gb) dynamic breakouts on
profiled QSFP ports on the
N9K-C93180YC-FX switch (in
ACI mode).

Contract and Subject Exceptions Contracts between EPGs are Configuring Tenants, on page 41
enhanced to include exceptions to
subjects or contracts. This enables
a subset of EPGs to be excluded in
contract filtering. For example, a
provider EPG can communicate
with all consumer EPGs except
those that match criteria configured
in a Subject Exception in the
contract governing their
communication.

Mixing the NX-OS style CLI and Cautions are added about mixing Using the APIC CLI, on page 1
the APIC GUI the two interfaces to configure the
fabric.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xxii
Preface
Preface

Feature Description Where Documented

Forwarding Scale Profile Policy The High LPM scale option is Configuring a Forwarding Scale
added to the forwarding scale Profile Policy, on page 521
profile policy. High longest prefix
match (LPM) provides scalability
similar to the dual-stack policy,
except that the LPM scale is
128,000 and the policy scale is
8,000.
Scale improvements in the other
forwarding scale options are also
added in this release.

Transit Routing Procedures to configure transit #unique_7

routing using the NX-OS-style CLI
are added to the guide.

Routed Connectivity to External New procedures to configure #unique_7

Networks L3Out connectivity to external
networks are added to the guide.

Table 2: New and Changed Behavior in Cisco ACI, Release 3.1(2m)

Feature Description Where Documented

Maximum MTU Increased Up to Cisco APIC Release 3.1(2), Global Policies

the range is 576 to 9000 bytes.
From release 3.1(2), and later, the
maximum MTU value is 9216. The
default has not changed from 9000.

QoS for L3Out QoS policy enforcement on L3Out Configuring Cisco ACI QoS
ingress traffic is enhanced. To
configure QoS policies in an
L3Out, the VRF must be set in
egress mode (Policy Control
Enforcement Direction = “egress”)
with policy control enabled (Policy
Control Enforcement Preference =
“Enforced”). You must configure
the QoS class priority or DSCP
setting in the contract that governs
the Layer 3 External network.

Neighbor Discovery Router RS/RA packets are used for auto Configuring Layer 3 External
Advertisement on Layer 3 Out configuration and are configurable Connectivity
on Layer 3 interfaces including
routed interface, Layer 3 sub
interface, and SVI.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xxiii
 Preface
Preface

Table 3: New and Changed Behavior in Cisco ACI, Release 3.1(1i)

Feature Description Where Documented

Configuring Flood in Encapsulation Beginning with Cisco ACI Release Configuring Flood in
3.1(1) on the Cisco ACI switches Encapsulation
with the Application Spine Engine
(ASE), all protocols are flooded in
encapsulation. Multiple EPGs are
now supported under one bridge
domain with an external switch.
When two EPGs share the same BD
and the Flood in Encapsulation
option is turned on, the EPG
flooding traffic does not reach the
other EPG. It overcomes the
challenges of using the Cisco ACI
switches with the Virtual Connect
(VC) tunnel network.

CoPP per interface per protocol Support for configuring CoPP on a Configuring Control Plane Policing
per interface per protocol basis.

Remote Leaf Switches With an ACI fabric deployed, you Remote Leaf Switches in
can extend ACI services and APIC Configuring Layer 3 External
management to remote datacenters Connectivity
with Cisco ACI leaf switches that
have no local spine switch or APIC
attached.

New Hardware Support for Multipod and GOLF are supported Cisco ACI GOLF and Multipod
Multipod and GOLF by all Cisco Nexus 9300 platform Fabric in Configuring Layer 3
ACI-mode switches and all of the External Connections
Cisco Nexus 9500 platform
ACI-mode switch line cards and
fabric modules. With Cisco APIC,
release 3.1(x) and higher, this
includes the N9K-C9364C switch.

MACsec MACsec provides MAC-layer Configuring MACsec

encryption over wired networks by
using out-of-band methods for
encryption keying. The MACsec
Key Agreement (MKA) Protocol
provides the required session keys
and manages the required
encryption keys.

Using Shared GOLF Connections Guidelines were added to avoid Cisco ACI GOLF in Configuring
Between Multi-Site Sites inter-VRF traffic issues for APIC Layer 3 External Connections
Sites in a Multi-Site topology, if
stretched VRFs share GOLF
connections.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xxiv
Preface
Preface

Feature Description Where Documented

SVI Auto State Allows for the SVI auto state in Configuring Layer 3 External
Switch Virtual Interface behavior Connectivity
to be enabled. This allows the SVI
state to be in the down state when
all the ports in the VLAN go down.
This feature is available in the
APIC Release 2.2(3x) release and
going forward with APIC Release
3.1(1). It is not supported in APIC
Release 3.0(x).
BFD support for spine switch Support for Bidirectional Configuring Bi-Directional Route
Forwarding Detection (BFD) spine Forwarding (BFD)
switch is added.

SNMP Trap Aggregation Enables SNMP traps from the Configuring SNMP
SNMP Trap Aggregation fabric
nodes to be delivered to one of the
APICs in the cluster.

Note The APIC Release 2.2(3x) feature is only available in this specific release. It is not supported in APIC Release
3.0(x) or Release 3.1(x).

Table 4: New and Changed Behavior in Cisco ACI, Release 2.3(3x)

Feature Description Where Documented

SVI Auto State Allows for the SVI auto state in Configuring Layer 3 External
Switch Virtual Interface behavior Connectivity
to be enabled. This allows the SVI
state to be in the down state when
all the ports in the VLAN go down.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xxv
 Preface
Preface

Table 5: New and Changed Behavior in Cisco ACI, Release 3.0(1k)

Feature Description Where Documented

Forwarding Scale Profile Policy The forwarding scale profile policy Configuring a Forwarding Scale
enables you to choose between Profile Policy
Dual Stack (the default profile) and
IPv4 Scale. A forwarding scale
profile policy that is set to Dual
Stack provides scalability of up to
6K endpoints for IPv6
configurations and up to 12K
endpoints for IPv4 configurations.
The IPv4 Scale option enables
systems with no IPv6
configurations to increase
scalability with up to 24K IPv4
endpoints.

Graceful Insertion and Removal The Graceful Insertion and Removing a Switch to Maintenance
(GIR) Mode Removal (GIR) mode or Mode Using the CLI
maintenance mode allows you to
isolate a switch from the network
with minimum service disruption.

Q-in-Q Encapsulation Mapping for Using Cisco APIC, you can map Configuring Q-in-Q Encapsulation
EPGs double-tagged VLAN traffic Mapping for EPGs in Configuring
ingressing on a regular interface, Layer 2 External Connectivity
PC, or VPC to an EPG. When this
feature is enabled, when
double-tagged traffic enters the
network for an EPG, both tags are
processed individually in the fabric
and restored to double-tags when
egressing the ACI switch.
Ingressing single-tagged and
untagged traffic is dropped.

802.1x Port Authentication With this release, you can configure Configuring 802.1x Port
an 802.1x Port Authentication Authentication Policy and
policy or 802.1x Node Configuring 802.1x Node
Authentication Policy. Authentication Policy in
Configuring Layer 2 Connectivity

First Hop Security Enables better IPv4 and IPv6 link Configuring First Hop Security in
security and management over the Configuring Security
layer 2 links.

Precision Time Protocol Time synchronization protocol Configuring PTP in Configuring

defined in IEEE 1588 for nodes Global Policies
distributed across the APIC.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xxvi
Preface
Preface

Feature Description Where Documented

Enforced Bridge Domain Enforced bridge domain is Enforced Bridge Domain in

supported, in which an endpoint in Configuring Tenants
a subject endpoint group (EPG) can
only ping subnet gateways within
the associated bridge domain.
With this configuration enabled,
you can create a global exception
list of IP addresses which can ping
any subnet gateway.

Table 6: New and Changed Behavior in Cisco ACI, Release 2.3(1e)

Feature Description Where Documented

Cisco APIC Quota Management Creates, deletes, and updates a Creating Quota Management
quota management configuration
which enables the admin to limit
what managed objects that can be
added under a given tenant or
globally across tenants.

Contract Inheritance To streamline associating contracts See Contract Inheritance in

to new EPGs, you can now enable Configuring Tenants
an EPG to inherit all the
(provided/consumed) contracts
associated directly to another EPG
in the same tenant. Contract
inheritance can be configured for
application, microsegmented,
L2Out, and L3Out EPGs. Any
changes you make to the EPG
contract master’s contracts, are
received by the inheriting EPG.

802.1Q Tunnel Enhancements Now you can configure ports on Configuring Layer 2 External
core-switches for use in Dot1q Connectivity
Tunnels for multiple customers.
You can also define access VLANs
to distinguish between customers
consuming the corePorts. You can
also disable MAC learning on
Dot1q Tunnels.

Control Plane Policing Protects the control plane and Configuring Security
separates it from the data plane,
which ensures network stability,
reachability, and packet delivery.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xxvii
 Preface
Preface

Feature Description Where Documented

Encapsulation scope for SVI across With this release you can configure See Configuring Layer 3 External
Layer 3 Outside networks the encapsulation scope for SVI Connectivity
across Layer 3 Outside networks.

Symmetric Hashing Symmetric hashing is now See Configuring Port Channels in

supported on port channels. Leaf Nodes Using the NX-OS CLI

Reflective relay (802.1Qbg) Reflective relay transfers switching See Configuring Fabric and
for virtual machines out of the host Interfaces
server to an external network
switch. It provides connectivity
between VMs on the same physical
server and the rest of the network.
It allows policies that you configure
on the Cisco APIC to apply to
traffic between the VMs on the
same server.

Microsegmentation for virtual Adds content for configuring See Configuring

switches microsegment EPGs on VMware Microsegmentation on Virtual
VDS, Cisco AVS, and Microsoft Switches
vSwitch.

Table 7: New Features and Changed Behavior in Cisco APIC 2.2(2e) Release

Feature or Change Description Where Documented

Per VRF per node BGP timer With this release, you can define Configuring Layer 3 External
and associate BGP timers on a per Connectivity
VRF per node basis.

Layer 3 Out to Layer 3 Out With this release, shared Layer 3 Configuring Layer 3 External
Inter-VRF Leaking Outs in different VRFs can Connectivity
communicate with each other using
a contract.

Multiple BGP communities With this release, multiple BGP Configuring Layer 3 External
assigned per route prefix communities can now be assigned Connectivity
per route prefix using the BGP
protocol.

Apply the show running config Two new CLI commands, export About Import and Export
command output to another Cisco config and import config, were Configurations in
APIC added to enable running the output
Applying the show running config
for the show running-config
Output to Another Cisco APIC
command on another Cisco APIC.

Name change Changed name of "Layer 3 EVPN Cisco ACI GOLF and Multipod in
Services for Fabric WAN" to Configuring Layer 3 External
"Cisco ACI GOLF Connectivity

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xxviii
Preface
Preface

Table 8: New Features and Changed Behavior in Cisco APIC 2.2(1n) Release

Feature Description Where Documented

802.1Q Tunnels You can configure 802.1Q tunnels Configuring 802.1Q Tunnels in
to enable point-to-multi-point Configuring Layer 2 External
tunneling of Ethernet frames in the Connectivity
fabric, with Quality of Service
(QoS) priority settings.

APIC Cluster High Availability Support is added to operate the APIC High Availability
APICs in a cluster in an
Active/Standby mode. In an APIC
cluster, the designated active APICs
share the load and the designated
standby APICs can act as an
replacement for any of the APICs
in an active cluster.

Contract Preferred Groups Support is added for contract Configuring Contract Preferred
preferred groups that enable greater Groups in Configuring Tenants
control of communication between
EPGs in a VRF. If most of the
EPGs in the VRF should have open
communication, but a few should
only have limited communication
with the other EPGs, you can
configure a combination of a
contract preferred group and
contracts with filters to control
communication precisely.

Dynamic Breakout Ports Support is added for connecting a Configuring Dynamic Breakout
40 Gigabit Ethernet (GE) leaf Ports in Configuring Layer 2
switch port to 4-10GE capable External Connectivity
(downlink) devices (with Cisco
40-Gigabit to 4X10-Gigabit
breakout cables).

FCoE over FEX You can now configure FCoE over Support Fibre Channel over
FEX ports. Ethernet Traffic on the ACI Fabric

CDP supported in policies on In this release, support is added for Configuring Fabric and Interfaces
interfaces to FEX devices CDP on interfaces to FEX devices.

HSRP Support is added for HSRP, a Configuring HSRP in Configuring

protocol that provides first-hop Layer 3 External Connectivity
routing redundancy for IP hosts on
Ethernet networks configured with
a default router IP address.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xxix
 Preface
Preface

Feature Description Where Documented

NetFlow Support is added for NetFlow Configuring NetFlow

technology, which provides the
metering base for a key set of
applications, including network
traffic accounting, usage-based
network billing, network planning,
as well as denial of services
monitoring, network monitoring,
outbound marketing, and data
mining for both service providers
and enterprise customers.

VLAN Domains Moved to Configuring Layer 2 Configuring VLAN Domains in

External Connectivity Configuring Layer 2 External
Connectivity

Table 9: New Features and Changed Behavior in Cisco APIC 2.1(1h) Release

Feature Description Where Documented

IP aging In this release, the IP aging, a Configuring IP Aging

policy for tracking and aging
unused IPs on an endpoint, is
supported.

Creating a route map/profile using In this release, the explicit prefix Creating a Route Map
explicit prefix list using a new list is supported through a new
match type. match type that is called match
route destination.
Configure FIPS In this release, support for FIPS. Configuring FIPS for Cisco APIC
FIPS specifies certain
cryptographic algorithms as secure,
and it also identifies which
algorithms should be used for a
module to be FIPS compliant.
Distribute EVPN Type-2 Host In this release, for optimal traffic Enabling Distributing EVPN
Routes forwarding in an EVPN topology, Type-2 Host Routes Using the
you can enable fabric spines to NX-OS in Configuring Layer 3
advertise host routes using EVPN EVPN Services over Fabric WAN
type-2 (MAC-IP) routes to the
DCIG along with public BD
subnets in the form of BGP EVPN
type-5 (IP Prefix) routes.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xxx
Preface
Preface

Feature Description Where Documented

Configure IGMP snoop layer 2 In this release, IGMP snoop support Enabling IGMP Snoop Static Port
multicast support is implemented which allows a Groups and Enabling IGMP Snoop
network switch to monitor IGMP Access Groups in Configuring
traffic and filter multicasts from Layer 2 IGMP Snoop Multicast
flooding layer 2 traffic. Among the
features implemented is static port
group configuration and access
group configuration.

Configuring network-based In this release you can configure Configuring Microsegmentation on

microsegmented EPGs in a microsegmented EPGs with IP Bare-Metall
bare-metal environment address attributes or MAC address
attributes for physical endpoint
devices.

Translating QoS CoS Settings In this release, you can enable the Translating QoS CoS Settings
ACI Fabric to classify the traffic Using the NX-OS CLI
for devices that classify the traffic
based only on the CoS value.

Table 10: New Features and Changed Behavior in Cisco APIC 2.0(2f) release

Feature Description Where Documented

Proxy ARP Proxy ARP in Cisco ACI is added About Proxy ARP, on page 144
to enable endpoints within a
network or subnet to communicate
with other endpoints without
knowing the real MAC address of
the endpoints.

Tetration Analytics Cisco Tetration Analytics agent Overview, on page 447

configuration is added.

Multipod QoS Support for Preserving CoS and Preserving QoS Priority Settings in
DSCP settings is added for a Multipod Fabric
Multipod topologies.

Layer 3 EVPN Services Over More detail was added on how to Configuration Tasks to Configure
Fabric WAN configure Layer 3 EVPN services. Cisco ACI GOLF Services Using
the NX-OS Style CLI, on page 296

Release Feature Where

2.0(1) Port Security About Port Security and

ACI, on page 136

2.0(1) COOP Authentication About COOP

Authentication, on page
389

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xxxi
 Preface
Document Conventions

Release Feature Where

2.0(1) Layer 3 Multicast Layer 3 Multicast, on

page 249

2.0(1) Layer 3 EVPN Services Over Fabric WAN Cisco ACI GOLF , on
page 294
2.0(1) Multipod Fabric About Multipod Fabric,
on page 311
2.0(1) Verified Scalability Using the CLI Verified Scalability Using
the CLI, on page 527
1.2(2) BFD About BFD, on page 234

Route Summarization Configuring an EIGRP

Interface, on page 221
Configuring OSPF, on
page 193

Route Dampening Configuring Layer 3

External Connectivity, on
page 161

Named Mode for configuring Layer 3 external Configuring Layer 3

connectivity External Connectivity, on
page 161

IPv6 support Configuring Layer 3

External Connectivity, on
page 161

1.2(1) Initial Release --

Document Conventions
Command descriptions use the following conventions:

Convention Description
bold Bold text indicates the commands and keywords that you enter literally
as shown.

Italic Italic text indicates arguments for which the user supplies the values.

[x] Square brackets enclose an optional element (keyword or argument).

[x | y] Square brackets enclosing keywords or arguments separated by a vertical

bar indicate an optional choice.

{x | y} Braces enclosing keywords or arguments separated by a vertical bar

indicate a required choice.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xxxii
Preface
Preface

Convention Description
[x {y | z}] Nested set of square brackets or braces indicate optional or required
choices within optional or required elements. Braces and a vertical bar
within square brackets indicate a required choice within an optional
element.

variable Indicates a variable for which you supply values, in context where italics
cannot be used.

string A nonquoted set of characters. Do not use quotation marks around the
string or the string will include the quotation marks.

Examples use the following conventions:

Convention Description
screen font Terminal sessions and information the switch displays are in screen font.

boldface screen font Information you must enter is in boldface screen font.

italic screen font Arguments for which you supply values are in italic screen font.

<> Nonprinting characters, such as passwords, are in angle brackets.

[] Default responses to system prompts are in square brackets.

!, # An exclamation point (!) or a pound sign (#) at the beginning of a line

of code indicates a comment line.

This document uses the following conventions:

Note Means reader take note. Notes contain helpful suggestions or references to material not covered in the manual.

Caution Means reader be careful. In this situation, you might do something that could result in equipment damage or
loss of data.

Warning IMPORTANT SAFETY INSTRUCTIONS

This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work
on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard
practices for preventing accidents. Use the statement number provided at the end of each warning to locate
its translation in the translated safety warnings that accompanied this device.
SAVE THESE INSTRUCTIONS

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xxxiii
 Preface
Related Documentation

Related Documentation
Cisco Application Centric Infrastructure (ACI) Documentation
The ACI documentation is available at the following URL: http://www.cisco.com/c/en/us/support/
cloud-systems-management/application-policy-infrastructure-controller-apic/
tsd-products-support-series-home.html.

Cisco Application Centric Infrastructure (ACI) Simulator Documentation

The Cisco ACI Simulator documentation is available at http://www.cisco.com/c/en/us/support/
cloud-systems-management/application-centric-infrastructure-simulator/tsd-products-support-series-home.html.

Cisco Nexus 9000 Series Switches Documentation

The Cisco Nexus 9000 Series Switches documentation is available at http://www.cisco.com/c/en/us/support/
switches/nexus-9000-series-switches/tsd-products-support-series-home.html.

Cisco Application Virtual Switch Documentation

The Cisco Application Virtual Switch (AVS) documentation is available at http://www.cisco.com/c/en/us/
support/switches/application-virtual-switch/tsd-products-support-series-home.html.

Cisco Application Centric Infrastructure (ACI) Integration with OpenStack Documentation

Cisco ACI integration with OpenStack documentation is available at http://www.cisco.com/c/en/us/support/
cloud-systems-management/application-policy-infrastructure-controller-apic/
tsd-products-support-series-home.html.

Documentation Feedback
To provide technical feedback on this document, or to report an error or omission, please send your comments
to [email protected]. We appreciate your feedback.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

xxxiv
 CHAPTER 1
Using the APIC CLI
• Accessing the NX-OS Style CLI, on page 1
• Using the NX-OS Style CLI for APIC, on page 2
• Differences in Usage from NX-OS, on page 5
• Mixing the NX-OS Style CLI and the APIC GUI, on page 5

Accessing the NX-OS Style CLI

Note From Cisco APIC Release 1.0 until Release 1.2, the default CLI was a Bash shell with commands to directly
operate on managed objects (MOs) and properties of the Management Information Model. Beginning with
Cisco APIC Release 1.2, the default CLI is a NX-OS style CLI. The object model CLI is available by typing
the bash command at the initial CLI prompt.

Procedure

Step 1 From a secure shell (SSH) client, open an SSH connection to APIC at username @ ip-address .
Use the administrator login name and the out-of-band management IP address that you configured during the
initial setup. For example, [email protected].

Step 2 When prompted, enter the administrator password.

What to do next
When you enter the NX-OS style CLI, the initial command level is the EXEC level. From this level, you can
reach these configuration modes:
• To continue in the NX-OS style CLI, you can stay in EXEC mode or you can type configure to enter
global configuration mode.
For information about NX-OS style CLI commands, see the Cisco APIC NX-OS Style CLI Command
Reference.
• To reach the object model CLI, type bash .

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

1
 Using the APIC CLI
Using the NX-OS Style CLI for APIC

For information about object mode CLI commands, see the Cisco APIC Command-Line Interface User
Guide, APIC Releases 1.0 and 1.1.

Using the NX-OS Style CLI for APIC

Using CLI Command Modes
The NX-OS style CLI is organized in a hierarchy of command modes with EXEC mode as the root, containing
a tree of configuration submodes beginning with global configuration mode. The commands available to you
depend on the mode you are in. To obtain a list of available commands in any mode, type a question mark
(?) at the system prompt.
This table lists and describes the two most commonly used modes (EXEC and global configuration) along
with an example submode (DNS). The table shows how to enter and exit the modes, and the resulting system
prompts. The system prompt helps to identify which mode you are in and the commands that are available to
you in that mode.

Mode Access Method Prompt Exit Method

EXEC From the APIC prompt, enter To exit to the login prompt, use
apic#
execsh. the exit command.

Global From EXEC mode, enter the To exit from a configuration

apic(config)#
configuration configure command. submode to its parent mode, use
the exit command.
DNS configuration From global configuration mode,
enter the dns command. apic(config-dns)# To exit from any configuration
mode or submode to EXEC
mode, use the end command.

CLI Command Hierarchy

Configuration mode has several submodes, with commands that perform similar functions grouped under the
same level. For example, all commands that display information about the system, configuration, or hardware
are grouped under the show command, and all commands that allow you to configure the switch are grouped
under the configure command.
To execute a command that is not available in EXEC mode, you navigate to its submode starting at the top
level of the hierarchy. For example, to configure DNS settings, use the configure command to enter the
global configuration mode, then enter the dns command. When you are in the DNS configuration submode,
you can query the available commands. as in this example:

apic1# configure
apic1(config)# dns
apic1(config-dns)# ?
address Configure the ip address for dns servers
domain Configure the domains for dns servers
exit Exit from current mode
fabric Show fabric related information
no Negate a command or set its defaults
show Show running system information
use-vrf Configure the management vrf for dns servers
where Show the current mode

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

2
Using the APIC CLI
Using the NX-OS Style CLI for APIC

apic1(config-dns)# end
apic1#

Each submode places you further down in the prompt hierarchy. To view the hierarchy for the current mode,
use the configure command, as shown in this example:

apic1# configure
apic1(config)# bgp-fabric
apic1(config-bgp-fabric)# where
configure t; bgp-fabric
apic1(config-bgp-fabric)#

To leave the current level and return to the previous level, type exit . To return directly to the EXEC level,
type end .

EXEC Mode Commands

When you start a CLI session, you begin in EXEC mode. From EXEC mode, you can enter configuration
mode. Most EXEC commands are one-time commands, such as show commands, which display the current
configuration status.

Configuration Mode Commands

Configuration mode allows you to make changes to the existing configuration. When you save the configuration,
these commands are saved across switch reboots. Once you are in configuration mode, you can enter a variety
of protocol-specific modes. Configuration mode is the starting point for all configuration commands.

Listing Commands and Syntax

In any command mode, you can obtain a list of available commands by entering a question mark (?).

apic1(config-dns)# ?
address Configure the ip address for dns servers
domain Configure the domains for dns servers
exit Exit from current mode
fabric Show fabric related information
no Negate a command or set its defaults
show Show running system information
use-vrf Configure the management vrf for dns servers
where Show the current mode

apic1(config-dns)# end
apic1#

To see a list of commands that begin with a particular character sequence, type those characters followed by
a question mark (?). Do not include a space before the question mark.
apic1(config)# sh ?
aaa Show AAA information
access-list Show Access-list Information
accounting Show accounting information
acllog Show acllog information
. . .

To complete a command after you begin typing, type a tab.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

3
 Using the APIC CLI
Using the NX-OS Style CLI for APIC

apic1# qu<TAB>
apic1# quota

To list keywords or arguments, enter a question mark in place of a keyword or argument. Include a space
before the question mark. This form of help is called command syntax help because it reminds you which
keywords or arguments are applicable based on the commands, keywords, and arguments you have already
entered.

apic1(config-dns)# use-vrf ?
inband-mgmt Configure dns on inband
oob-mgmt Configure dns on out-of-band

apic1(config-dns)#

You can also abbreviate a command if the abbreviation is unambiguous. In this example, the configure
command is abbreviated.

apic1# conf
apic1(config)#

Undoing or Reverting to Default Values or Conditions Using the 'no' Prefix

For many configuration commands, you can precede the command with the no keyword to remove a setting
or to restore a setting to the default value. This example shows how to remove a previously-configured DNS
address from the configuration.

apic1(config-dns)# address 192.0.20.123 preferred

apic1(config-dns)# show dns-address
Address Preferred
------------------- ---------
192.0.20.123 yes

apic1(config-dns)# no address 192.0.20.123

apic1(config-dns)# show dns-address
Address Preferred
------------------- ---------

Executing BASH Commands From the NX-OS Style CLI

To execute a single command in the bash shell, type bash -c ' path/command ' as shown in this example.

apic1# bash -c '/controller/sbin/acidiag avread'

You can execute a bash command from any mode or submode in the NX-OS style CLI.

Entering Configuration Text with Spaces or Special Characters

When a configuration field consists of user-defined text, special characters such as '$' should be escaped ('\$')
or the entire word or string should be wrapped in single quotes to avoid misinterpretation by Bash.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

4
 Using the APIC CLI
Differences in Usage from NX-OS

Differences in Usage from NX-OS

The usage of the NX-OS style CLI for APIC differs from the traditional NX-OS CLI in these ways:
• Global configuration mode is entered with the configure command instead of configure terminal .
• To perform node-level configuration on a particular leaf switch, you must first navigate to that switch
using the leaf command.
• The command syntax for specifying a physical port is slightly different. For example, an Ethernet port
is specified as eth x/y instead of ethx/y .
• When a configuration field consists of user-defined text, such as a password, special characters such as
'$' or '!' should be escaped with a backslash ('\$') or the entire word or string should be wrapped in single
quotes to avoid misinterpretation by Bash.
• Some command shortcuts are different due to Bash behavior:
• Ctrl-D exits a session.
• Ctrl-Z suspends a job.

• OSPF configuration adds area route-map and area connectivity commands.

Mixing the NX-OS Style CLI and the APIC GUI

Basic mode is deprecated since Cisco APIC Release 3.0(1). There is only one GUI as of that release.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

5
 Using the APIC CLI
About the Modes of Configuring Layer 3 External Connectivity

Caution Configurations done through the NX-OS style CLI are rendered in the APIC GUI. They can be seen, but
sometimes may not be editable in the GUI. Also changes made in the APIC GUI may be seen in the NX-OS
style CLI, but may only partially work. See the following examples:
• Do not mix the GUI and the CLI, when doing per-interface configuration on APIC. Configurations
performed in the GUI, may only partially work in the NX-OS CLI.
For example, if you configure a switch port in the GUI at Tenants > tenant-name > Application
Profiles > application-profile-name > Application EPGs > EPG-name > Static Ports > Deploy
Static EPG on PC, VPC, or Interface
Then you use the show running-config command in the NX-OS style CLI, you receive output such as:
leaf 102
interface ethernet 1/15
switchport trunk allowed vlan 201 tenant t1 application ap1 epg ep1
exit
exit

If you use these commands to configure a static port in the NX-OS style CLI, the following error occurs:
apic1(config)# leaf 102
apic1(config-leaf)# interface ethernet 1/15
apic1(config-leaf-if)# switchport trunk allowed vlan 201 tenant t1 application ap1 epg
ep1
No vlan-domain associated to node 102 interface ethernet1/15 encap vlan-201

This occurs because the CLI has validations that are not performed by the APIC GUI. For the commands
from the show running-config command to function in the NX-OS CLI, a vlan-domain must have been
previously configured. The order of configuration is not enforced in the GUI.

For the steps to remove such objects, see Troubleshooting Unwanted _ui_ Objects in the APIC Troubleshooting
Guide.

About the Modes of Configuring Layer 3 External Connectivity

Because APIC supports multiple user interfaces (UIs) for configuration, the potential exists for unintended
interactions when you create a configuration with one UI and later modify the configuration with another UI.
This section describes considerations for configuring Layer 3 external connectivity with the APIC NX-OS
style CLI, when you may also be using other APIC user interfaces.
When you configure Layer 3 external connectivity with the APIC NX-OS style CLI, you have the choice of
two modes:
• Implicit mode, a simpler mode, is not compatible with the APIC GUI or the REST API.
• Named (or Explicit) mode is compatible with the APIC GUI and the REST API.

In either case, the configuration should be considered read-only in the incompatible UI.

How the Modes Differ

In both modes, the configuration settings are defined within an internal container object, the "L3 Outside" (or
"L3Out"), which is an instance of the l3extOut class in the API. The main difference between the two modes
is in the naming of this container object instance:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

6
Using the APIC CLI
About the Modes of Configuring Layer 3 External Connectivity

• Implicit mode—the naming of the container is implicit and does not appear in the CLI commands. The
CLI creates and maintains these objects internally.
• Named mode—the naming is provided by the user. CLI commands in the Named Mode have an additional
l3Out field. To configure the named L3Out correctly and avoid faults, the user is expected to understand
the API object model for external Layer 3 configuration.

Note Except for the procedures in the Configuring Layer 3 External Connectivity Using the Named Mode section,
this guide describes Implicit mode procedures.

Guidelines and Restrictions

• In the same APIC instance, both modes can be used together for configuring Layer 3 external connectivity
with the following restriction: The Layer 3 external connectivity configuration for a given combination
of tenant, VRF, and leaf can be done only through one mode.
• For a given tenant VRF, the policy domain where the External-l3 EPG can be placed can be in either the
Named mode or in the Implicit mode. The recommended configuration method is to use only one mode
for a given tenant VRF combination across all the nodes where the given tenant VRF is deployed for
Layer 3 external connectivity. The modes can be different across different tenants or different VRFs and
no restrictions apply.
• In some cases, an incoming configuration to a Cisco APIC cluster will be validated against inconsistencies,
where the validations involve externally-visible configurations (northbound traffic through the L3Outs).
An Invalid Configuration error message will appear for those situations where the configuration is invalid.
• The external Layer 3 features are supported in both configuration modes, with the following exception:
• Route-peering and Route Health Injection (RHI) with a L4-L7 Service Appliance is supported only
in the Named mode. The Named mode should be used across all border leaf switches for the tenant
VRF where route-peering is involved.

• Layer 3 external network objects (l3extOut) created using the Implicit mode CLI procedures are identified
by names starting with “__ui_” and are marked as read-only in the GUI. The CLI partitions these
external-l3 networks by function, such as interfaces, protocols, route-map, and EPG. Configuration
modifications performed through the REST API can break this structure, preventing further modification
through the CLI.

For the steps to remove such objects, see Troubleshooting Unwanted _ui_ Objects in the APIC Troubleshooting
Guide.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

7
 Using the APIC CLI
About the Modes of Configuring Layer 3 External Connectivity

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

8
 CHAPTER 2
Configuring Fabric and Interfaces
• Fabric and Interface Configuration, on page 9
• Graceful Insertion and Removal (GIR) Mode, on page 10
• Configuring Physical Ports in Leaf Nodes and FEX Devices Using the NX-OS CLI, on page 11
• Configuring Port Channels in Leaf Nodes and FEX Devices Using the NX-OS CLI, on page 14
• Configuring Virtual Port Channels in Leaf Nodes and FEX Devices Using the NX-OS CLI, on page 20
• Configuring FEX Connections Using Profiles with the NX-OS Style CLI, on page 25
• Reflective Relay (802.1Qbg), on page 26
• Configuring Policy Groups for Interfaces, on page 28
• Configuring Overrides for Interfaces, on page 31
• About Forwarding Error Correction, on page 33

Fabric and Interface Configuration

To form the ACI fabric, Cisco Nexus 9000 Series ACI-mode switches are deployed in a leaf and spine topology
managed by the APIC controller. Each leaf node is connected to all spine nodes with no connectivity between
the leaf nodes. The interconnecting links between the leaf and spine nodes are called fabric links and the
respective ports are called fabric ports. The fabric ports do not require user configuration for normal operation
as these are auto discovered and factory default configuration is applied during fabric bring-up. All endpoint
devices are connected to the leaf nodes through access ports. The access ports must be configured similar to
those in NX-OS switches. Both fabric and access ports are represented as Interfaces as in NX-OS.
The leaf and spine nodes are considered different objects in the ACI model and support different sets of
policies. In the CLI, these nodes are represented as leaf and spine respectively while both are commonly
referred to as nodes. Leaf and spine node values are unique across all the pods in the fabric. FEX modules, if
attached to the leaf nodes, will have fex-id values unique only within each leaf. For example, two leaf nodes
can each have a FEX 101 attached.

Note Configuring FEX connections with FEX IDs 165 to 199 is not supported in the APIC GUI. To use one of
these FEX IDs, configure the profile using the NX-OS style CLI. For more information, see Configuring FEX
Connections Using Interface Profiles with the NX-OS Style CLI.

As of Cisco APIC, Release 3.0(1k), connections to FEX modules can be configured as profiles.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

9
 Configuring Fabric and Interfaces
Graceful Insertion and Removal (GIR) Mode

Interface Naming for Leaf and FEX Interfaces

In ACI fabric, most interface configuration is done for physical ports, port-channels, or vPCs (either directly
connected to leaf nodes or connected through FEX modules). The general command syntax for each interface
type is shown in the following table.

Interface Type Command Syntax Examples

Port interface ethernet slot/port interface eth 1/1

FEX Port interface ethernet fex-id/slot/port interface eth 101/1/1

Port-channel interface port-channel name interface port-channel foo

FEX Port-channel interface port-channel name fex interface port-channel foo fex 101
fex-id

Virtual Port-channel (VPC) interface vpc name interface vpc foo

vPC over FEX interface vpc name fex fex-a fex-b interface vpc foo fex 101 102

Graceful Insertion and Removal (GIR) Mode

The Graceful Insertion and Removal (GIR) mode, or maintenance mode, allows you to isolate a switch from
the network with minimum service disruption. In the GIR mode you can perform real-time debugging without
affecting traffic.
You can use graceful insertion and removal to gracefully remove a switch and isolate it from the network in
order to perform debugging operations. The switch is removed from the regular forwarding path with minimal
traffic disruption. When you are finished performing the debugging operations, you can use graceful insertion
to return the switch to its fully operational (normal) mode. In graceful removal, all external protocols are
gracefully brought down except the fabric protocol (IS-IS) and the switch is isolated from the network. During
maintenance mode, the maximum metric is advertised in IS-IS within the Cisco Application Centric
Infrastructure (Cisco ACI) fabric and therefore the maintenance mode TOR does not attract traffic from the
spine switches. In addition, all the front-panel interfaces are shutdown on the switch except the fabric interfaces.
In graceful insertion, the switch is automatically decommissioned, rebooted, and recommissioned. When
recommissioning is completed, all external protocols are restored and maximum metric in IS-IS is reset after
10 minutes.
The following protocols are supported:
• Border Gateway Protocol (BGP)
• Enhanced Interior Gateway Routing Protocol (EIGRP)
• Intermediate System-to-Intermediate System (IS-IS)
• Open Shortest Path First (OSPF)
• Link Aggregation Control Protocol (LACP)

Important Notes
• Upgrading or downgrading a switch in maintenance mode is not supported.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

10
 Configuring Fabric and Interfaces
Removing a Switch to Maintenance Mode Using the CLI

• While the switch is in maintenance mode, the Ethernet port module stops propagating the interface related
notifications. As a result, if the remote switch is rebooted or the fabric link is flapped during this time,
the fabric link will not come up afterward unless the switch is manually rebooted (using the acidiag
touch clean command), decommissioned, and recommissioned.
• For multi-pod, IS-IS metric for redistributed routes should be set to less than 63. To set the IS-IS
metric for redistributed routes, choose Fabric > Fabric Policies > Pod Policies > IS-IS Policy.
• Existing GIR supports all Layer 3 traffic diversion. With LACP, all the Layer 2 traffic is also diverted
to the redundant node. Once a node goes into maintenance mode, LACP running on the node immediately
informs neighbors that it can no longer be aggregated as part of port-channel. All traffic is then diverted
to the vPC peer node.
• For a GIR upgrade, Cisco Application Policy Infrastructure Controller (Cisco APIC)-connected leaf
switches must be put into different maintenance groups such that the Cisco APIC-connected leaf switches
get upgraded one at a time.

Removing a Switch to Maintenance Mode Using the CLI

Use this procedure to remove a switch to maintenance mode using the CLI.

Procedure

Command or Action Purpose

Step 1 [no]debug-switch node_id or node_name Removes the switch to maintenance mode.

Inserting a Switch to Operation Mode Using CLI

Use this procedure to insert a switch to operational mode using the CLI.

Procedure

Command or Action Purpose

Step 1 [no]no debug-switch node_id or node_name Inserts the switch to operational mode.

Configuring Physical Ports in Leaf Nodes and FEX Devices Using

the NX-OS CLI
The commands in the following examples create many managed objects (MOs) in the ACI policy model that
are fully compatible with the REST API/SDK and GUI. However, the CLI user can focus on the intended
network configuration instead of ACI model internals.
The following figure shows examples of Ethernet ports directly on leaf nodes or FEX modules attached to
leaf nodes and how each is represented in the CLI. For FEX ports, the fex-id is included in the naming of
the port itself as in ethernet 101/1/1 . While describing an interface range, the ethernet keyword need not
be repeated as in NX-OS. Example: interface ethernet 101/1/1-2, 102/1/1-2 .

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

11
 Configuring Fabric and Interfaces
Configuring Physical Ports in Leaf Nodes and FEX Devices Using the NX-OS CLI

• Leaf node ID numbers are global.

• The fex-id numbers are local to each leaf.
• Note the space after the keyword ethernet .

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf or leafs to be configured. The

node-id can be a single node ID or a range of
Example:
IDs, in the form node-id1 - node-id2 , to which
apic1(config)# leaf 102 the configuration will be applied.

Step 3 interface type Specifies the interface that you are configuring.
You can specify the interface type and identity.
Example:
For an Ethernet port, use “ethernet slot / port.”
apic1(config-leaf)# interface ethernet
1/2

Step 4 (Optional) fex associate node-id If the interface or interfaces to be configured

are FEX interfaces, you must use this command
Example:
to attach the FEX module to a leaf node before
apic1(config-leaf-if)# fex associate 101 configuration.
Note This step is required before creating
a port-channel using FEX ports.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

12
Configuring Fabric and Interfaces
Configuring Physical Ports in Leaf Nodes and FEX Devices Using the NX-OS CLI

Command or Action Purpose

Step 5 speed speed The speed setting is shown as an example. At
this point you can configure any of the interface
Example:
settings shown in the table below.
apic1(config-leaf-if)# speed 10G

The following table shows the interface settings that can be configured at this point.

Command Purpose

[no] shut Shut down physical interface

[no] speed speedValue Set the speed for physical interface

[no] link debounce time time Set link debounce

[no] negotiate auto Configure negotiate

[no] cdp enable Disable/enable Cisco Discovery Protocol (CDP)

[no] mcp enable Disable/enable Mis-cabling Protocol (MCP)

[no] lldp transmit Set the transmit for physical interface

[no] lldp receive Set the LLDP receive for physical interface

spanning-tree {bpduguard | bpdufilter} {enable | Configure spanning tree BPDU

disable}

[no] storm-control level percentage [ burst-rate Storm-control configuration (percentage)

percentage ]

[no] storm-control pps packets-per-second burst-rate Storm-control configuration (packets-per-second)

packets-per-second

Examples
Configure one port in a leaf node. The following example shows how to configure the interface
eth1/2 in leaf 101 for the following properties: speed, cdp, and admin state.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/2
apic1(config-leaf-if)# speed 10G
apic1(config-leaf-if)# cdp enable
apic1(config-leaf-if)# no shut

Configure multiple ports in multiple leaf nodes. The following example shows the configuration of
speed for interfaces eth1/1-10 for each of the leaf nodes 101-103.

apic1(config)# leaf 101-103

apic1(config-leaf)# interface eth 1/1-10

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

13
 Configuring Fabric and Interfaces
Configuring Port Channels in Leaf Nodes and FEX Devices Using the NX-OS CLI

apic1(config-leaf-if)# speed 10G

Attach a FEX to a leaf node. The following example shows how to attach a FEX module to a leaf
node. Unlike in NX-OS, the leaf port Eth1/5 is implicitly configured as fabric port and a FEX fabric
port-channel is created internally with the FEX uplink port(s). In ACI, the FEX fabric port-channels
use default configuration and no user configuration is allowed.

Note This step is required before creating a port-channel using FEX ports, as described in the next example.

apic1(config)# leaf 102

apic1(config-leaf)# interface eth 1/5
apic1(config-leaf-if)# fex associate 101

Configure FEX ports attached to leaf nodes. This example shows configuration of speed for interfaces
eth1/1-10 in FEX module 101 attached to each of the leaf nodes 102-103. The FEX ID 101 is included
in the port identifier. FEX IDs start with 101 and are local to a leaf.

apic1(config)# leaf 102-103

apic1(config-leaf)# interface eth 101/1/1-10
apic1(config-leaf-if)# speed 1G

Configuring Port Channels in Leaf Nodes and FEX Devices Using

the NX-OS CLI
Port-channels are logical interfaces in NX-OS used to aggregate bandwidth for multiple physical ports and
also for providing redundancy in case of link failures. In NX-OS, port-channel interfaces are identified by
user-specified numbers in the range 1 to 4096 unique within a node. Port-channel interfaces are either configured
explicitly (using the interface port-channel command) or created implicitly (using the channel-group
command). The configuration of the port-channel interface is applied to all the member ports of the port-channel.
There are certain compatibility parameters (speed, for example) that cannot be configured on the member
ports.
In the ACI model, port-channels are configured as logical entities identified by a name to represent a collection
of policies that can be assigned to set of ports in one or more leaf nodes. Such assignment creates one
port-channel interface in each of the leaf nodes identified by an auto-generated number in the range 1 to 4096
within the leaf node, which may be same or different among the nodes for the same port-channel name. The
membership of these port-channels may be same or different as well. When a port-channel is created on the
FEX ports, the same port-channel name can be used to create one port-channel interface in each of the FEX
devices attached to the leaf node. Thus, it is possible to create up to N+1 unique port-channel interfaces
(identified by the auto-generated port-channel numbers) for each leaf node attached to N FEX modules. This
is illustrated with the examples below. Port-channels on the FEX ports are identified by specifying the fex-id
along with the port-channel name ( interface port-channel foo fex 101 , for example).

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

14
Configuring Fabric and Interfaces
Configuring Port Channels in Leaf Nodes and FEX Devices Using the NX-OS CLI

• N+1 instances per leaf of port-channel foo are possible when each leaf is connected to N FEX nodes.
• Leaf ports and FEX ports cannot be part of the same port-channel instance.
• Each FEX node can have only one instance of port-channel foo.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 template port-channel channel-name Creates a new port-channel or configures an

existing port-channel (global configuration).
Example:
apic1(config)# template port-channel foo

Step 3 [no] switchport access vlan vlan-id tenant Deploys the EPG with the VLAN on all ports
tenant-name application application-name epg with which the port-channel is associated.
epg-name
Example:

apic1(config-po-ch-if)# switchport access

vlan 4 tenant ExampleCorp application
Web epg webEpg

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

15
 Configuring Fabric and Interfaces
Configuring Port Channels in Leaf Nodes and FEX Devices Using the NX-OS CLI

Command or Action Purpose

Step 4 channel-mode active Note The channel-mode command is
equivalent to the mode option in the
Example:
channel-group command in NX-OS.
apic1(config-po-ch-if)# channel-mode In ACI, however, this is supported
active
for the port-channel (not on a
Note To enable symmetric hashing, enter member port).
the lacp symmetric-hash
command: Symmetric hashing is not supported on the
apic1(config-po-ch-if)# lacp
following switches:
symmetric-hash • Cisco Nexus 93128TX
• Cisco Nexus 9372PX
• Cisco Nexus 9372PX-E
• Cisco Nexus 9372TX
• Cisco Nexus 9372TX-E
• Cisco Nexus 9396PX
• Cisco Nexus 9396TX

Step 5 exit Returns to configure mode.

Example:
apic1(config-po-ch-if)# exit

Step 6 leaf node-id Specifies the leaf switches to be configured.

The node-id can be a single node ID or a range
Example:
of IDs, in the form node-id1 - node-id2 , to
apic1(config)# leaf 101 which the configuration will be applied.

Step 7 interface type Specifies the interface or range of interfaces

that you are configuring to the port-channel.
Example:
apic1(config-leaf)# interface ethernet
1/1-2

Step 8 [no] channel-group channel-name Assigns the interface or range of interfaces to

the port-channel. Use the keyword no to
Example:
remove the interface from the port-channel. To
apic1(config-leaf-if)# channel-group foo change the port-channel assignment on an
interface, you can enter the channel-group
command without first removing the interface
from the previous port-channel.

Step 9 (Optional) lacp port-priority priority This setting and other per-port LACP properties
can be applied to member ports of a
Example:
port-channel at this point.
apic1(config-leaf-if)# lacp port-priority

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

16
Configuring Fabric and Interfaces
Configuring Port Channels in Leaf Nodes and FEX Devices Using the NX-OS CLI

Command or Action Purpose

1000 Note In the ACI model, these commands
apic1(config-leaf-if)# lacp rate fast are allowed only after the ports are
member of a port channel. If a port
is removed from a port channel,
configuration of these per-port
properties are removed as well.

The following table shows various commands for global configurations of port channel properties in the ACI
model. These commands can also be used for configuring overrides for port channels in a specific leaf in the
(config-leaf-if) CLI mode. The configuration made on the port-channel is applied to all member ports.

CLI Syntax Feature

[no] speed <speedValue> Set the speed for port-channel

[no] link debounce time <time> Set Link Debounce for port-channel

[no] negotiate auto Configure Negotiate for port-channel

[no] cdp enable Disable/Enable CDP for port-channel

[no] mcp enable Disable/Enable MCP for port-channel

[no] lldp transmit Set the transmit for port-channel

[no] lldp receive Set the lldp receive for port-channel

spanning-tree <bpduguard | bpdufilter> <enable | Configure spanning tree BPDU

disable>

[no] storm-control level <percentage> [ burst-rate Storm-control configuration (percentage)

<percentage> ]

[no] storm-control pps <packet-per-second> burst-rate Storm-control configuration (packets-per-second)

<packets-per-second>

[no] channel-mode { active | passive | on| mac-pinning LACP mode for the link in port-channel l
}

[no] lacp min-links <value> Set minimum number of links

[no] lacp max-links <value> Set maximum number of links

[no] lacp fast-select-hot-standby LACP fast select for hot standby ports

[no] lacp graceful-convergence LACP graceful convergence

[no] lacp load-defer LACP load defer member ports

[no] lacp suspend-individual LACP individual Port suspension

[no] lacp port-priority LACP port priority

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

17
 Configuring Fabric and Interfaces
Configuring Port Channels in Leaf Nodes and FEX Devices Using the NX-OS CLI

CLI Syntax Feature

[no] lacp rate LACP rate

Examples
Configure a port channel (global configuration). A logical entity foo is created that represents a
collection of policies with two configurations: speed and channel mode. More properties can be
configured as required.

Note The channel mode command is equivalent to the mode option in the channel group command in
NX-OS. In ACI, however, this supported for the port-channel (not on member port).

apic1(config)# template port-channel foo

apic1(config-po-ch-if)# switchport access vlan 4 tenant ExampleCorp application Web epg
webEpg
apic1(config-po-ch-if)# speed 10G
apic1(config-po-ch-if)# channel-mode active

Configure ports to a port-channel in a FEX. In this example, port channel foo is assigned to ports
Ethernet 1/1-2 in FEX 101 attached to leaf node 102 to create an instance of port channel foo. The
leaf node will auto-generate a number, say 1002 to identify the port channel in the switch. This port
channel number would be unique to the leaf node 102 regardless of how many instance of port
channel foo are created.

Note The configuration to attach the FEX module to the leaf node must be done before creating port
channels using FEX ports.

apic1(config)# leaf 102

apic1(config-leaf)# interface ethernet 101/1/1-2
apic1(config-leaf-if)# channel-group foo

In Leaf 102, this port channel interface can be referred to as interface port-channel foo FEX 101.
apic1(config)# leaf 102
apic1(config-leaf)# interface port-channel foo fex 101
apic1(config-leaf)# shut

Configure ports to a port channel in multiple leaf nodes. In this example, port channel foo is assigned
to ports Ethernet 1/1-2 in each of the leaf nodes 101-103. The leaf nodes will auto generate a number
unique in each node (which may be same or different among nodes) to represent the port-channel
interfaces.
apic1(config)# leaf 101-103
apic1(config-leaf)# interface ethernet 1/1-2
apic1(config-leaf-if)# channel-group foo

Add members to port channels. This example would add two members eth1/3-4 to the port-channel
in each leaf node, so that port-channel foo in each node would have members eth 1/1-4.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

18
Configuring Fabric and Interfaces
Configuring Port Channels in Leaf Nodes and FEX Devices Using the NX-OS CLI

apic1(config)# leaf 101-103

apic1(config-leaf)# interface ethernet 1/3-4
apic1(config-leaf-if)# channel-group foo

Remove members from port channels. This example would remove two members eth1/2, eth1/4 from
the port channel foo in each leaf node, so that port channel foo in each node would have members
eth 1/1, eth1/3.
apic1(config)# leaf 101-103
apic1(config-leaf)# interface eth 1/2,1/4
apic1(config-leaf-if)# no channel-group foo

Configure port-channel with different members in multiple leaf nodes. This example shows how to
use the same port-channel foo policies to create a port-channel interface in multiple leaf nodes with
different member ports in each leaf. The port-channel numbers in the leaf nodes may be same or
different for the same port-channel foo. In the CLI, however, the configuration will be referred as
interface port-channel foo. If the port-channel is configured for the FEX ports, it would be referred
to as interface port-channel foo fex <fex-id>.
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/1-2
apic1(config-leaf-if)# channel-group foo
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit
apic1(config)# leaf 102
apic1(config-leaf)# interface ethernet 1/3-4
apic1(config-leaf-if)# channel-group foo
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit
apic1(config)# leaf 103
apic1(config-leaf)# interface ethernet 1/5-8
apic1(config-leaf-if)# channel-group foo
apic1(config-leaf-if)# exit
apic1(config-leaf)# interface ethernet 101/1/1-2
apic1(config-leaf-if)# channel-group foo

Configure per port properties for LACP. This example shows how to configure member ports of a
port-channel for per-port properties for LACP.

Note In ACI model, these commands are allowed only after the ports are member of a port channel. If a
port is removed from a port channel, configuration of these per-port properties would be removed
as well.

apic1(config)# leaf 101

apic1(config-leaf)# interface ethernet 1/1-2
apic1(config-leaf-if)# channel-group foo
apic1(config-leaf-if)# lacp port-priority 1000
apic1(config-leaf-if)# lacp rate fast

Configure admin state for port channels. In this example, a port-channel foo is configured in each
of the leaf nodes 101-103 using the channel-group command. The admin state of port-channel(s) can
be configured in each leaf using the port-channel interface. In ACI model, the admin state of the
port-channel cannot be configured in the global scope.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

19
 Configuring Fabric and Interfaces
Configuring Virtual Port Channels in Leaf Nodes and FEX Devices Using the NX-OS CLI

// create port-channel foo in each leaf

apic1(config)# leaf 101-103
apic1(config-leaf)# interface ethernet 1/3-4
apic1(config-leaf-if)# channel-group foo

// configure admin state in specific leaf

apic1(config)# leaf 101
apic1(config-leaf)# interface port-channel foo
apic1(config-leaf-if)# shut

Override config is very helpful to assign specific vlan-domain, for example, to the port-channel
interfaces in each leaf while sharing other properties.
// configure a port channel global config
apic1(config)# interface port-channel foo
apic1(config-if)# speed 1G
apic1(config-if)# channel-mode active

// create port-channel foo in each leaf

apic1(config)# leaf 101-103
apic1(config-leaf)# interface ethernet 1/1-2
apic1(config-leaf-if)# channel-group foo

// override port-channel foo in leaf 102

apic1(config)# leaf 102
apic1(config-leaf)# interface port-channel foo
apic1(config-leaf-if)# speed 10G
apic1(config-leaf-if)# channel-mode on
apic1(config-leaf-if)# vlan-domain dom-foo

This example shows how to change port channel assignment for ports using the channel-group
command. There is no need to remove port channel membership before assigning to other port
channel.
apic1(config)# leaf 101-103
apic1(config-leaf)# interface ethernet 1/3-4
apic1(config-leaf-if)# channel-group foo
apic1(config-leaf-if)# channel-group bar

Configuring Virtual Port Channels in Leaf Nodes and FEX

Devices Using the NX-OS CLI
A virtual port channel (vPC) is an enhancement to port-channels that allows connection of a host or switch
to two upstream leaf nodes to improve bandwidth utilization and availability. In NX-OS, vPC configuration
is done in each of the two upstream switches and configuration is synchronized using peer link between the
switches.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

20
Configuring Fabric and Interfaces
Configuring Virtual Port Channels in Leaf Nodes and FEX Devices Using the NX-OS CLI

Note When creating a vPC domain between two leaf switches, both switches must be in the same switch generation,
one of the following:
• Generation 1 - Cisco Nexus N9K switches without “EX” on the end of the switch name; for example,
N9K-9312TX
• Generation 2 – Cisco Nexus N9K switches with “EX” on the end of the switch model name; for example,
N9K-93108TC-EX

Switches such as these two are not compatible vPC peers. Instead, use switches of the same generation.

The ACI model does not require a peer link and vPC configuration can be done globally for both the upstream
leaf nodes. A global configuration mode called vpc context is introduced in ACI and vPC interfaces are
represented using a type interface vpc that allows global configuration applicable to both leaf nodes.
Two different topologies are supported for vPC in the ACI model: vPC using leaf ports and vPC over FEX
ports. It is possible to create many vPC interfaces between a pair of leaf nodes and similarly, many vPC
interfaces can be created between a pair of FEX modules attached to the leaf node pairs in a straight-through
topology.
vPC considerations include:
• The vPC name used is unique between leaf node pairs. For example, only one vPC 'corp' can be created
per leaf pair (with or without FEX).
• Leaf ports and FEX ports cannot be part of the same vPC.
• Each FEX module can be part of only one instance of vPC corp.
• vPC context allows configuration
• The vPC context mode allows configuration of all vPCs for a given leaf pair. For vPC over FEX, the
fex-id pairs must be specified either for the vPC context or along with the vPC interface, as shown in
the following two alternative examples.

(config)# vpc context leaf 101 102

(config-vpc)# interface vpc Reg fex 101 101

or

(config)# vpc context leaf 101 102 fex 101 101

(config-vpc)# interface vpc Reg

In the ACI model, vPC configuration is done in the following steps (as shown in the examples below).

Note A VLAN domain is required with a VLAN range. It must be associated with the port-channel template.

1. VLAN domain configuration (global config) with VLAN range

2. vPC domain configuration (global config)

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

21
 Configuring Fabric and Interfaces
Configuring Virtual Port Channels in Leaf Nodes and FEX Devices Using the NX-OS CLI

3. Port-channel template configuration (global config)

4. Associate the port-channel template with the VLAN domain
5. Port-channel configuration for vPC (global config)
6. Configure ports to vPC in leaf nodes
7. Configure L2, L3 for vPC in the vpc context

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 vlan-domain name[dynamic] [type Configures a VLAN domain for the virtual
domain-type] port-channel (here with a port-channel
template).
Example:
apic1(config)# vlan-domain dom1 dynamic

Step 3 vlan range Configures a VLAN range for the VLAN

domain and exits the configuration mode. The
Example:
range can be a single VLAN or a range of
apic1(config-vlan)# vlan 1000-1999 VLANs.
apic1(config-vlan)# exit

Step 4 vpc domain explicit domain-id leaf node-id1 Configures a vPC domain between a pair of
node-id2 leaf nodes. You can specify the vPC domain
ID in the explicit mode along with the leaf
Example:
node pairs.
apic1(config)# vpc domain explicit 1
leaf 101 102 Alternative commands to configure a vPC
domain are as follows:
• vpc domain [consecutive | reciprocal]
The consecutive and reciprocal options
allow auto configuration of a vPC domain
across all leaf nodes in the ACI fabric.
• vpc domain consecutive domain-start
leaf start-node end-node
This command configures a vPC domain
consecutively for a selected set of leaf
node pairs.

Step 5 peer-dead-interval interval Configures the time delay the Leaf switch
waits to restore the vPC before receiving a
Example:
response from the peer. If it does not receive

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

22
Configuring Fabric and Interfaces
Configuring Virtual Port Channels in Leaf Nodes and FEX Devices Using the NX-OS CLI

Command or Action Purpose

apic1(config-vpc)# peer-dead-interval a response from the peer within this time, the
10 Leaf switch considers the peer dead and brings
up the vPC with the role as a master. If it does
receive a response from the peer it restores the
vPC at that point. The range is from 5 seconds
to 600 seconds. The default is 200 seconds.

Step 6 exit Returns to global configuration mode.

Example:
apic1(config-vpc)# exit

Step 7 template port-channel channel-name Creates a new port-channel or configures an

existing port-channel (global configuration).
Example:
apic1(config)# template port-channel All vPCs are configured as port-channels in
corp each leaf pair. The same port-channel name
must be used in a leaf pair for the same vPC.
This port-channel can be used to create a vPC
among one or more pairs of leaf nodes. Each
leaf node will have only one instance of this
vPC.

Step 8 vlan-domain member vlan-domain-name Associates the port channel template with the
previously configured VLAN domain.
Example:
vlan-domain member dom1

Step 9 switchport access vlan vlan-id tenant Deploys the EPG with the VLAN on all ports
tenant-name application application-name with which the port-channel is associated.
epg epg-name
Example:

apic1(config-po-ch-if)# switchport
access vlan 4 tenant ExampleCorp
application Web epg webEpg

Step 10 channel-mode active Note A port-channel must be in active

channel-mode for a vPC.
Example:
apic1(config-po-ch-if)# channel-mode
active

Step 11 exit Returns to configure mode.

Example:
apic1(config-po-ch-if)# exit

Step 12 leaf node-id1 node-id2 Specifies the pair of leaf switches to be

configured.
Example:
apic1(config)# leaf 101-102

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

23
 Configuring Fabric and Interfaces
Configuring Virtual Port Channels in Leaf Nodes and FEX Devices Using the NX-OS CLI

Command or Action Purpose

Step 13 interface type leaf/interface-range Specifies the interface or range of interfaces
that you are configuring to the port-channel.
Example:
apic1(config-leaf)# interface ethernet
1/3-4

Step 14 [no] channel-group channel-name vpc Assigns the interface or range of interfaces to
the port-channel. Use the keyword no to
Example:
remove the interface from the port-channel.
apic1(config-leaf-if)# channel-group To change the port-channel assignment on an
corp vpc
interface, you can enter the channel-group
command without first removing the interface
from the previous port-channel.
Note The vpc keyword in this command
makes the port-channel a vPC. If
the vPC does not already exist, a
vPC ID is automatically generated
and is applied to all member leaf
nodes.

Step 15 exit
Example:
apic1(config-leaf-if)# exit

Step 16 exit
Example:
apic1(config-leaf)# exit

Step 17 vpc context leaf node-id1 node-id2 The vPC context mode allows configuration
of vPC to be applied to both leaf node pairs.
Example:
apic1(config)# vpc context leaf 101
102

Step 18 interface vpc channel-name

Example:
apic1(config-vpc)# interface vpc blue
fex 102 102

Step 19 (Optional) [no] shutdown Administrative state configuration in the vPC

context allows changing the admin state of a
Example:
vPC with one command for both leaf nodes.
apic1(config-vpc-if)# no shut

Example
This example shows how to configure a basic vPC.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

24
 Configuring Fabric and Interfaces
Configuring FEX Connections Using Profiles with the NX-OS Style CLI

apic1# configure
apic1(config)# vlan-domain dom1 dynamic

apic1(config-vlan)# vlan 1000-1999

apic1(config-vlan)# exit
apic1(config)# vpc domain explicit 1 leaf 101 102
apic1(config-vpc)# peer-dead-interval 10

apic1(config-vpc)# exit
apic1(config)# template port-channel corp
apic1(config-po-ch-if)# vlan-domain member dom1

apic1(config-po-ch-if)# channel-mode active

apic1(config-po-ch-if)# exit
apic1(config)# leaf 101-102
apic1(config-leaf)# interface ethernet 1/3-4
apic1(config-leaf-if)# channel-group corp vpc
apic1(config-leaf-if)# exit
apic1(config)# vpc context leaf 101 102

This example shows how to configure vPCs with FEX ports.

apic1(config-leaf)# interface ethernet 101/1/1-2

apic1(config-leaf-if)# channel-group Reg vpc
apic1(config)# vpc context leaf 101 102
apic1(config-vpc)# interface vpc corp
apic1(config-vpc-if)# exit
apic1(config-vpc)# interface vpc red fex 101 101
apic1(config-vpc-if)# switchport
apic1(config-vpc-if)# exit
apic1(config-vpc)# interface vpc blue fex 102 102
apic1(config-vpc-if)# shut

Configuring FEX Connections Using Profiles with the NX-OS

Style CLI
Use this procedure to configure FEX connections to leaf nodes using the NX-OS style CLI.

Note Configuring FEX connections with FEX IDs 165 to 199 is not supported in the APIC GUI. To use one of
these FEX IDs, configure the profile using the following commands.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

25
 Configuring Fabric and Interfaces
Reflective Relay (802.1Qbg)

Command or Action Purpose

apic1# configure

Step 2 leaf-interface-profile name Specifies the leaf interface profile to be

configured.
Example:
apic1(config)# leaf-interface-profile
fexIntProf1

Step 3 leaf-interface-group name Specifies the interface group to be configured.

Example:
apic1(config-leaf-if-profile)#
leaf-interface-group leafIntGrp1

Step 4 fex associate fex-id [template template-type Attaches a FEX module to a leaf node. Use the
fex-template-name] optional template keyword to specify a template
to be used. If it does not exist, the system
Example:
creates a template with the name and type you
apic1(config-leaf-if-group)# fex specified.
associate 101

Example
This merged example configures a leaf interface profile for FEX connections with ID 101.
apic1# configure
apic1(config)# leaf-interface-profile fexIntProf1
apic1(config-leaf-if-profile)# leaf-interface-group leafIntGrp1
apic1(config-leaf-if-group)# fex associate 101

Reflective Relay (802.1Qbg)

Reflective relay is a switching option beginning with Cisco APIC Release 2.3(1). Reflective relay—the tagless
approach of IEEE standard 802.1Qbg—forwards all traffic to an external switch, which then applies policy
and sends the traffic back to the destination or target VM on the server as needed. There is no local switching.
For broadcast or multicast traffic, reflective relay provides packet replication to each VM locally on the server.
One benefit of reflective relay is that it leverages the external switch for switching features and management
capabilities, freeing server resources to support the VMs. Reflective relay also allows policies that you configure
on the Cisco APIC to apply to traffic between the VMs on the same server.
In the Cisco ACI, you can enable reflective relay, which allows traffic to turn back out of the same port it
came in on. You can enable reflective relay on individual ports, port channels, or virtual port channels as a
Layer 2 interface policy using the APIC GUI, NX-OS CLI, or REST API. It is disabled by default.
The term Virtual Ethernet Port Aggregator (VEPA) is also used to describe 802.1Qbg functionality.

Reflective Relay Support

Reflective relay supports the following:
• IEEE standard 802.1Qbg tagless approach, known as reflective relay.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

26
 Configuring Fabric and Interfaces
Enabling Reflective Relay Using the NX-OS CLI

Cisco APIC Release 2.3(1) release does not support the IEE standard 802.1Qbg S-tagged approach with
multichannel technology.
• Physical domains.
Virtual domains are not supported.
• Physical ports, port channels (PCs), and virtual port channels (vPCs).
Cisco Fabric Extender (FEX) and blade servers are not supported. If reflective relay is enabled on an
unsupported interface, a fault is raised, and the last valid configuration is retained. Disabling reflective
relay on the port clears the fault.
• Cisco Nexus 9000 series switches with EX or FX at the end of their model name.

Enabling Reflective Relay Using the NX-OS CLI

Reflective relay is disabled by default; however, you can enable it on a port, port channel, or virtual port
channel as a Layer 2 interface policy on the switch. In the NX-OS CLI, you can use a template to enable
reflective relay on multiple ports or you can enable it on individual ports.

Before you begin

This procedure assumes that you have set up the Cisco Application Centric Infrastructure (ACI) fabric and
installed the physical switches.

Procedure

Enable reflective relay on one or multiple ports:

Example:
This example enables reflective relay on a single port:
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/2
apic1(config-leaf-if)# switchport vepa enabled
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit

Example:
This example enables reflective relay on multiple ports using a template:
apic1(config)# template policy-group grp1
apic1(config-pol-grp-if)# switchport vepa enabled
apic1(config-pol-grp-if)# exit
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/2-4
apic1(config-leaf-if)# policy-group grp1

Example:
This example enables reflective relay on a port channel:
apic1(config)# leaf 101
apic1(config-leaf)# interface port-channel po2
apic1(config-leaf-if)# switchport vepa enabled
apic1(config-leaf-if)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

27
 Configuring Fabric and Interfaces
Configuring Policy Groups for Interfaces

apic1(config-leaf)# exit
apic1(config)#

Example:
This example enables reflective relay on multiple port channels:
apic1(config)# template port-channel po1
apic1(config-if)# switchport vepa enabled
apic1(config-if)# exit
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/3-4
apic1(config-leaf-if)# channel-group po1
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit

Example:
This example enables reflective relay on a virtual port channel:
apic1(config)# vpc domain explicit 1 leaf 101 102
apic1(config-vpc)# exit
apic1(config)# template port-channel po4
apic1(config-if)# exit
apic1(config)# leaf 101-102
apic1(config-leaf)# interface eth 1/11-12
apic1(config-leaf-if)# channel-group po4 vpc
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit
apic1(config)# vpc context leaf 101 102
apic1(config-vpc)# interface vpc po4
apic1(config-vpc-if)# switchport vepa enabled

Configuring Policy Groups for Interfaces

In data center networks, typically configuration of many interfaces is the same across multiple nodes. This
can be achieved in the ACI Policy Model by creating policy-groups to be shared by groups of interfaces across
multiple leaf nodes. The policy-group is identified by a name similar to the port-channel; however, in case of
port-channel the policies shared with the group of ports create one logical interface in each leaf while in case
of a policy-group, each of the ports sharing the policies are individual physical interfaces. The policy-group
concept is very similar to a port-profile in NX-OS.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 template policy-group policy-group-name Creates a new policy group or edits an existing
policy group.
Example:
apic1(config)# template policy-group pg1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

28
Configuring Fabric and Interfaces
Configuring Policy Groups for Interfaces

Command or Action Purpose

Step 3 [no] switchport access vlan vlan-id tenant
tenant-name application application-name epg
epg-name
Example:

apic1(config-pol-grp-if)# switchport
access vlan 4 tenant ExampleCorp
application Web epg webEpg

Step 4 (Apply configuration commands) The table at the end of these steps shows various
commands for configurations of policy-group
Example:
for interfaces.
apic1(config-pol-grp-if)# speed 10G
apic1(config-pol-grp-if)# cdp enable

Step 5 exit Returns to configure mode.

Example:
apic1(config-pol-grp-if)# exit

Step 6 leaf node-id Specifies the leaf or leafs to be configured. The

node-id can be a single node ID or a range of
Example:
IDs, in the form node-id1 - node-id2 , to which
apic1(config)# leaf 101-103 the configuration will be applied.

Step 7 interface type Specifies the interface or range of interfaces to

which you will apply the policy group.
Example:
apic1(config-leaf)# interface ethernet
1/1-24

Step 8 [no] policy-group policy-group-name [force] Applies the policy-group to the interface or
range of interfaces. Use the keyword no to
Example:
remove the policy-group from the interface. Use
apic1(config-leaf-if)# policy-group pg1 the keyword force to delete any override
configurations on the interfaces.
If the specified policy-group was not configured
prior to this command, this command would
not implicitly create the policy-group. However,
the policy-group would take effect on the
interface after the policy-group has been
configured in the global scope.
To change the policy-group assignment on an
interface, you can enter the policy-group
command without first removing the previous
policy-group from the interface.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

29
 Configuring Fabric and Interfaces
Configuring Policy Groups for Interfaces

Command or Action Purpose

Note If you apply a policy-group to an
interface and then assign the
interface to a port-channel, the
interface will lose the policy-group
configuration and the policies in the
port-channel will be applied.

The following table shows various commands for configurations of policy-group for interfaces.

CLI Syntax Feature

[no] speed <speedValue> Set the speed for Physical Interface

[no] link debounce time <time> Set link debounce for Physical Interface

[no] negotiate auto Configure Negotiate for Physical Interface

[no] cdp enable Disable/Enable CDP for Physical Interface

[no] mcp enable Disable/Enable MCP for Physical Interface

[no] lldp transmit Set the LLDP transmit for Physical Interface

[no] lldp receive Set the LLDP receive for Physical Interface

spanning-tree <bpduguard | bpdufilter> <enable | Configure spanning tree BPDU

disable>

[no] storm-control level <percentage> [ burst-rate Storm-control configuration (percentage)

<percentage> ]

[no] storm-control pps <packet-per-second> burst-rate Storm-control configuration (packets-per-second)

<packets-per-second>

Example
This example shows how to configure a policy-group and apply it to a range of ports in each of the
leaf nodes 101-103. Each of the ports sharing the policy-group in each leaf will have the same
configuration as defined in the policy-group pg1.

apic1# configure
apic1(config)# template policy-group pg1
apic1(config-pol-grp-if)# switchport access vlan 4 tenant ExampleCorp application Web epg
webEpg
apic1(config-pol-grp-if)# speed 10G
apic1(config-pol-grp-if)# cdp enable
apic1(config-pol-grp-if)# exit
apic1(config)# leaf 101-103
apic1(config-leaf)# interface ethernet 1/1-24
apic1(config-leaf-if)# policy-group pg1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

30
 Configuring Fabric and Interfaces
Configuring Overrides for Interfaces

Configuring Overrides for Interfaces

When policy-groups are used with large number of interfaces, it may be useful to have the option to configure
a set of ports for specific properties that will override the configuration in the assigned policy-group. Override
configuration is allowed only if the port is assigned to a policy-group. Override configuration is not allowed
for member ports of a port-channel. When a port is added to a port-channel, the override configuration is
automatically removed. However, during policy-group assignment to a port that has overrides configured, the
override configuration is not removed automatically and the user can decide to remove the override
configuration with the force option, if required, in the policy-group command.
When a policy-group assignment is removed from a port, the override config, if exists, does not change.
Similarly, the override config does not change if the port is assigned to a different policy-group (without the
force option). The override config takes effect once configured and it is not removed even if the user assigns
default values to all the properties in the override. To remove the override config, the user can reapply the
policy-group assignment with force option. The force option, however, is not displayed in the show
running-config as it is used to just remove the override config in the ACI model.
In the ACI model, overrides can be configured for a policy which may contain one or more properties. If a
policy has more than one property, it is not possible to override only one property within a policy. In the CLI
framework, when the user intends to override a property for which the corresponding policy has more than
one property, all other properties in the policy except the override property would be implicitly copied to the
override configuration to avoid ambiguity. Such implicit copy of configuration would be reflected in the
output of show running-config regardless of the value (including default values). Also, the copy is done only
once during the configuration of the override policy and any subsequent change to the policy-group for any
of the properties in that policy would have no effect on the port(s) on which the override is configured.
If the policy-group assigned to a port is not configured when the override is created, the implicit copy of
properties noted above is not possible; instead, default values are assigned to properties in the override config
for which the corresponding policy has more than one property. These properties shall not change for the
override config when the policy-group is configured afterwards. It is recommended that user create overrides
after configuring the policy-group itself or the user may need to configure the overrides in addition to the
config in policy-group to get desired configuration if the config for properties in override are set to default
implicitly before the configuration of the policy-group with non-default values for those properties.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf or leafs to be configured. The

node-id can be a single node ID or a range of
Example:
IDs, in the form node-id1 - node-id2 , to which
apic1(config)# leaf 102 the configuration will be applied.

Step 3 interface type Specifies the interface or range of interfaces

with an override configuration.
Example:
apic1(config-leaf)# interface ethernet
1/2

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

31
 Configuring Fabric and Interfaces
Configuring Overrides for Interfaces

Command or Action Purpose

Step 4 policy-group policy-group-name force Forces the policy-group to the interface or range
of interfaces, deleting any override
Example:
configurations on the interfaces.
apic1(config-leaf-if)# policy-group pg1
force

Examples
This example shows how to apply a policy-group and then override the speed configuration for port
eth1/1 in leaf node 101. In the ACI model, speed is part of a policy that also contains properties
autoneg and link debounce time. As a result, those properties are copied from the speed policy-group
when the override of pg1 is configured.

apic1# configure
apic1(config)# interface policy-group pg1
apic1(config-pol-grp-if)# speed 10G
apic1(config-pol-grp-if)# cdp enable
apic1(config-pol-grp-if)# exit
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/1-2
apic1(config-leaf-if)# policy-group pg1
apic1(config-pol-grp-if)# exit
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/1
apic1(config-leaf-if)# speed 1G
apic1(config-leaf-if)# show running-config

leaf 101
interface ethernet 1/1
policy-group pg1
speed 1G
autoneg on
link debounce time 100

interface ethernet 1/2

policy-group pg1

This example shows how to remove the override configuration from port eth1/1 in leaf node 101.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/1
apic1(config-leaf-if)# policy-group pg1 force
apic1(config-leaf-if)# show running-config

leaf 101
interface ethernet 1/1
policy-group pg1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

32
 Configuring Fabric and Interfaces
About Forwarding Error Correction

About Forwarding Error Correction

Forwarding Error Correction (FEC) is a method of obtaining error control in data transmission over an
unreliable or noisy channel in which the source (transmitter) encodes the data in a redundant way using Error
Correcting Code, and the destination (receiver) recognizes it and corrects the errors without requiring a
retransmission. The available options are as follows:
• CL74-FC-FEC—Supports 25 Gbps speed.
• CL91-RS-FEC—Supports 25 and 100 Gbps speeds.
• Disable-FEC—Disables FEC.
• Inherit—The switch uses FEC based on the port transceiver type. All copper (CR4) transceivers have
FC-FEC enabled on 25G. All interfaces with 100G transceivers have RS-FEC enabled.

The default is "Inherit".

Note FEC is only configurable on the front port and not on fabric ports.

Configuring FEC Using NX-OS Style CLI

Procedure

Command or Action Purpose

Step 1 Enter the configure mode. Enters the configuration mode.
Example:
apic1# configure

Step 2 Enter the switch mode. Enters the switch mode.

Example:
apic1(config)# leaf 104

Step 3 Specify the interface and port. Specifies the interface and port.
Example:
apic1(config-leaf)# int eth 1/4

Step 4 Configure FEC. Configures RS-FEC.

Example: Note The default forward-error-correction
apic1(config-leaf-if)# value is inherit.
forward-error-correction cl91-rs-fec

Step 5 Exit the interface mode. Exits the interface mode.

Example:
apic1(config-leaf-if)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

33
 Configuring Fabric and Interfaces
Configuring FEC Using NX-OS Style CLI

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

34
 CHAPTER 3
Cisco ACI Smart Licensing
This chapter contains the following sections:
• About Smart Licensing, on page 35

About Smart Licensing

Starting with Cisco Application Programming Infrastructure Controller (APIC) release 3.2(1), Smart Licensing
is enabled in the Cisco ACI fabric and by extension in the Cisco APIC as a Cisco Smart Licensing-enabled
product. Cisco Smart Licensing is a unified license management system that manages all the software licenses
across Cisco products.
Smart Licensing has the following advantages over a traditional license:
• For the purposes of Smart Licensing, APIC is occasionally referred to as the ACI controller product.
• CSSM (Cisco Smart Software Manager) provides a central portal view to customers. Customers can
view all the licenses they purchased and license usage and status. It helps prevent occurrences of license
violations, expiry of subscription-based licenses, and out-of-compliance licenses.
• To support Smart Licensing, the standard CLI commands and GUI view is implemented across different
Cisco products. This provides customers with a consistent user experience.
• Smart Licensing reduces the complexity of license management and makes it easier for customers to
troubleshoot license-related issues.

The following URLs provide you with additional information about Smart Licensing:
• The customer log in URL to access your CSSM account: https://software.cisco.com/
• Cisco Smart Accounts URL: https://www.cisco.com/c/en/us/products/software/smart-accounts.html

The following URLs are additional resources that you can refer:
• Training Materials and Resources
• Smart Accounts and Smart Licensing

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

35
 Cisco ACI Smart Licensing
Smart Licensing Usage Guidelines and Limitations

Smart Licensing Usage Guidelines and Limitations

Follow these Smart Licensing guidelines and limitations:
• The Evaluation Period countdown time is stored in the APIC. The countdown time remains intact during
a software downgrade. Therefore, if the customer upgrades their APIC software version 3.2 or later once
again after a downgrade, the countdown time will continue from the previous value before the downgrade.
The countdown time cannot be reset. After 90 days, if no action is taken to register, the license status
will display Evaluation Expired.
• If there is a license violation for a feature that is enabled on APIC, the feature functionality will not be
disabled, and there will be no impact on system functionality. The system will continue to operate, but
relevant faults will be raised to warn the user. The most severe fault that will be raised is major.
• If the registration fails, click the Faults tab in the APIC GUI System > Smart Licensing area. To see
details about a specific failure, double click the listed fault.
• The DLC tool is not supported when you use the Smart Software Manager Satellite transport setting.

Pre-Registration Verifications
Verification Checklist for CSSM Configurations
The following is a user checklist for readiness and configurations required with CSSM.
1. Verify that you have the appropriate Smart Account and Virtual Accounts created.
2. If you have purchased smart-enabled licenses from Cisco Commerce, then verify that your user-purchased
licenses are populated.
3. As you begin the APIC Smart Licensing registration, work with your Cisco TAC engineer to ensure that
you are ready with the appropriate CSSM items.

Verification Checklist for Smart Licensing and APIC Configurations

The following is a user checklist for readiness and configurations required with the APIC.
- Your DNS settings must be configured in APIC to resolve to https://software.cisco.com/.

Registering for Smart Licensing Using the CLI

Registering for Smart Licensing with Direct Connect to CSSM Using the CLI

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

36
 Cisco ACI Smart Licensing
Registering for Smart Licensing with Transport Gateway Using the CLI

Command or Action Purpose

Step 2 license smart transport-mode smart-licensing Configures the Smart Licensing mode.
Example:
apic1(config)# license smart
transport-mode smart-licensing

Step 3 license smart register idtoken id token from Registers with the CSSM account using the
cssm account token from the account.
Example:
apic1(config)# license smart register
idtoken <id token from cssm account>

Registering for Smart Licensing with Transport Gateway Using the CLI

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 license smart transport-mode satellite url Configures the Transport Gateway mode and
http(s)://10.0.0.0:8080/Transportgateway/services/DeviceRequestHandler URL.
Example:
apic1(config)# license smart
transport-mode satellite url
http(s)://<ip address|hostname of
transport gateway>:<http(s)
port>/Transportgateway/services/DeviceRequestHandler

Step 3 license smart register idtoken id token from Registers with the CSSM using the token from
cssm account the CSSM Smart account or the CSSM Virtual
account.
Example:
apic1(config)# license smart register
idtoken <id token from cssm account>

Registering for Smart Licensing with Smart Software Manager Satellite Using the CLI

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

37
 Cisco ACI Smart Licensing
Registering for Smart Licensing with HTTP or HTTPS Proxy Using the CLI

Command or Action Purpose

Step 2 license smart transport-mode satellite url Configures the Smart Software Manager
http(s)://10.0.10.1:8080/Transportgateway/services/DeviceRequestHandler Satellite mode and URL.
Example:
apic1(config)# license smart
transport-mode satellite url
http(s)://<ip address|hostname of
transport gateway>:<http(s)
port>/Transportgateway/services/DeviceRequestHandler

Step 3 license smart register idtoken id token from Registers with the Satellite using the token from
smart software manager satellite the Smart Software Manager Satellite account.
Example: Note Note : Do not use the token from the
apic1(config)# license smart register CSSM account.
idtoken <id token from smart software
manager satellite>

Registering for Smart Licensing with HTTP or HTTPS Proxy Using the CLI

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 license smart transport-mode proxy Configures the proxy mode, the IP address or
ip-address ip address port port number hostname and the http(s) port.
Example:
apic1(config)# license smart
transport-mode proxy ip-address
10.0.0.248 port 4440

Step 3 license smart register idtoken id token from Registers with the CSSM account using the
cssm account token from the CSSM smart account or the
CSSM virtual account.
Example:
apic1(config)# license smart register
idtoken <id token from cssm account>

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

38
 CHAPTER 4
Configuring APIC High Availability
• About Cold Standby for APIC Cluster, on page 39
• Switching Over Active APIC with Standby APIC Using CLI, on page 40

About Cold Standby for APIC Cluster

The Cold Standby functionality for an APIC cluster enables you to operate the APICs in a cluster in an
Active/Standby mode. In an APIC cluster, the designated active APICs share the load and the designated
standby APICs can act as a replacement for any of the APICs in an active cluster.
As an admin user, you can set up the Cold Standby functionality when the APIC is launched for the first time.
We recommend that you have at least three active APICs in a cluster, and one or more standby APICs. As an
admin user, you can initiate the switch over to replace an active APIC with a standby APIC.
Important Notes
• The standby APIC is automatically updated with firmware updates to keep the backup APIC at same
firmware version as the active cluster.
• During an upgrade process, once all the active APICs are upgraded, the standby APIC is also be upgraded
automatically.
• Temporary IDs are assigned to standby APICs. After a standby APIC is switched over to an active APIC,
a new ID is assigned.
• Admin login is not enabled on standby APIC. To troubleshoot Cold Standby, you must log in to the
standby using SSH as rescue-user.
• During switch over the replaced active APIC is powered down, to prevent connectivity to the replaced
APIC.
• Switch over fails under the following conditions:
• If there is no connectivity to the standby APIC.
• If the firmware version of the standby APIC is not the same as that of the active cluster.

• After switching over a standby APIC to active, if it was the only standby, you must configure a new
standby.
• The following limitations are observed for retaining out of band address for standby APIC after a fail
over.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

39
 Configuring APIC High Availability
Switching Over Active APIC with Standby APIC Using CLI

• Standby (new active) APIC may not retain its out of band address if more than 1 active APICs are
down or unavailable.
• Standby (new active) APIC may not retain its out of band address if it is in a different subnet than
active APIC. This limitation is only applicable for APIC release 2.x.
• Standby (new active) APIC may not retain its IPv6 out of band address. This limitation is not
applicable starting from APIC release 3.1x.
• Standby (new active) APIC may not retain its out of band address if you have configured non Static
OOB Management IP address policy for replacement (old active) APIC.

Note In case you observe any of the limitations, in order to retain standby APICs out
of band address, you must manually change the OOB policy for replaced APIC
after the replace operation is completed successfully.

• We recommend keeping standby APICs in same POD as the active APICs it may replace.
• There must be three active APICs in order to add a standby APIC.
• The standby APIC does not participate in policy configuration or management.
• No information is replicated to standby controllers, including admin credentials.

Switching Over Active APIC with Standby APIC Using CLI

Use this procedure to switch over an active APIC with a standby APIC.

Procedure

Command or Action Purpose

Step 1 replace-controller replace ID number Backup Replaces an active APIC with an standby APIC.
serial number
Example:
apic1#replace-controller replace 2
FCH1804V27L
Do you want to replace APIC 2 with a
backup? (Y/n): Y

Step 2 replace-controller reset ID number Resets fail over status of the active controller.
Example:
apic1# replace-controller reset 2
Do you want to reset failover status of
APIC 2? (Y/n): Y

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

40
 CHAPTER 5
Configuring Tenants
• Creating a Tenant, VRF, and Bridge Domain, on page 41
• Additional Bridge Domain Configuration, on page 44
• Configuring an Enforced Bridge Domain, on page 45
• Creating an Application Endpoint Group, on page 48
• Configuring Legacy Forwarding Mode in the Bridge Domain, on page 51
• Configuring Contracts, on page 52
• Contract Inheritance, on page 56
• Configuring Contract Preferred Groups, on page 65
• Exporting a Contract to Another Tenant, on page 68
• Configuring Contract or Subject Exceptions, on page 70
• Creating Quota Management, on page 72

Creating a Tenant, VRF, and Bridge Domain

This topic describes the following steps in the basic provisioning of a new tenant:
1. Create a tenant
2. Associate the tenant with a security domain
3. Create a VRF for the tenant
4. Create a bridge domain for endpoint groups within the tenant

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 tenant tenant-name Creates a tenant if it does not exist and enters
the tenant configuration mode.
Example:
apic1(config)# tenant exampleCorp

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

41
 Configuring Tenants
Creating a Tenant, VRF, and Bridge Domain

Command or Action Purpose

Step 3 security domain domain-name Associates the tenant with one or more security
domains.
Example:
apic1(config-tenant)# security domain
exampleCorp_dom1

Step 4 [no] vrf context vrf-name Creates a private network (VRF) for the tenant.
A tenant can have one or more VRFs
Example:
configured.
apic1(config-tenant)# vrf context
exampleCorp_v1

Step 5 [no] contract {provider | consumer} Provide or consume contracts for all the EPGs
contract-name under the VRF.
Example:
apic1(config-tenant-vrf)# contract
provider web

Step 6 exit Returns to the tenant configuration mode.

Example:
apic1(config-tenant-vrf)# exit

Step 7 [no] bridge-domain bd-name Creates or deletes a bridge domain under the
tenant. Enters bridge domain configuration
Example:
mode.
apic1(config-tenant)# bridge-domain
exampleCorp_b1

Step 8 [no] vrf member vrf-name Assigns the bridge-domain to a VRF.

Example:
apic1(config-tenant-bd)# vrf member
exampleCorp_v1

Step 9 exit Returns to the tenant configuration mode.

Example:
apic1(config-tenant-bd)# exit

Step 10 interface bridge-domain bd-name Enters tenant interface configuration mode to

enable routing and to apply interfaces to the
Example:
bridge domain.
apic1(config-tenant)# interface
bridge-domain exampleCorp_b1

Step 11 [no] {ip | ipv6} address address/mask-length Assigns or removes the gateway IP address of
[scope {private | public}] [secondary] the bridge domain and enters the IP address
mode to configure optional IP address
Example:
properties.
apic1(config-tenant-if)# ip address The scope of the gateway address can be one
172.1.1.1/24 of the following:
apic1(config-tenant-if)# ipv6 address
2001:1:1::1/64

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

42
Configuring Tenants
Creating a Tenant, VRF, and Bridge Domain

Command or Action Purpose

• Public —Can be advertised to external
Layer 3 networks through routing
protocols (BGP, OSPF, EIGRP).
• Private —Not advertised to external
Layer 3 networks.

The optional secondary keyword allows you

to configure a secondary gateway address.

Examples
This example shows the basic configuration of a tenant including assignment to a security domain,
creation of a VRF with contracts, and creation of a bridge domain.

apic1# configure
apic1(config)# tenant exampleCorp
apic1(config-tenant)# security domain exampleCorp_dom1
apic1(config-tenant)# vrf context exampleCorp_v1
apic1(config-tenant-vrf)# contract enforce
apic1(config-tenant-vrf)# contract provider web
apic1(config-tenant-vrf)# contract consumer db
apic1(config-tenant-vrf)# contract provider icmp
apic1(config-tenant-vrf)# contract consumer icmp
apic1(config-tenant-vrf)# exit
apic1(config-tenant)# bridge-domain exampleCorp_b1
apic1(config-tenant-bd)# vrf member exampleCorp_v1
apic1(config-tenant-bd)# exit
apic1(config-tenant)# interface bridge-domain exampleCorp_b1
apic1(config-tenant-interface)# ip address 172.1.1.1/24
apic1(config-tenant-interface)# ipv6 address 2001:1:1::1/64
apic1(config-tenant-interface)# exit

This example shows the VRF configuration specific to a leaf.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# vrf context exampleCorp_v1 tenant exampleCorp
apic1(config-leaf-vrf)# ip route 1.2.3.4 5.6.7.8

This example shows the VRF configuration specific to a leaf interface.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# int eth 1/1
apic1(config-leaf-if)# vrf member exampleCorp_v1 tenant exampleCorp

What to do next
Add an application profile, create an application endpoint group (EPG), and associate the EPG to the bridge
domain.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

43
 Configuring Tenants
Additional Bridge Domain Configuration

Additional Bridge Domain Configuration

This topic describes the following configurations for a bridge domain:
• Configuring a MAC address
• Configuring a DHCP relay address
• Configuring route leaking for shared services

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic# configure

Step 2 tenant tenant-name Enters the tenant configuration mode.

Example:
apic(config-tenant)# tenant exampleCorp

Step 3 interface bridge-domain bd-name Enters tenant interface configuration mode to

configure the bridge domain.
Example:
apic(config-tenant)# interface
bridge-domain exampleCorp_bd1

Step 4 (Optional) mac-address mac-address Configures the MAC address to be used in the
ARP reply for the pervasive gateway
Example:
functionality.
apic(config-tenant-interface)#
mac-address 1234.5678.abcd

Step 5 (Optional) no mac-address Changes the MAC address to its default.

Example:
apic(config-tenant-interface)# no
mac-address

Step 6 (Optional) [no] ip dhcp relay address tenant Sets or removes a DHCP relay address for the
tenant-name dhcp-address {application bridge-domain along with any supported
app-name epg epg-name | external-l2 options.
l2-epg-name | external-l3 l3-epg-name}
Example:
apic(config-tenant-interface)# ip dhcp
relay address 192.0.20.1 tenant
exampleCorp application app1 epg epg1

Step 7 (Optional) [no] {ip | ipv6} shared address Route leaking is allowed across VRFs to
address/mask-length provider application provide common services like DHCP, DNS for
app-name epg epg-name multiple tenant VRFs. Shared service is enabled

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

44
 Configuring Tenants
Configuring an Enforced Bridge Domain

Command or Action Purpose

Example: by marking subnets as provider or consumer
apic(config-tenant-interface)# ip shared subnets and specifying the EPGs providing the
address 7.8.9.1/24 provider application shared service.
app2 epg epg2

Step 8 (Optional) [no] {ip | ipv6} shared address See the previous step.
address/mask-length consumer application
any epg any
Example:
apic(config-tenant-interface)# ip shared
address 3.2.3.4/24 consumer application
any epg any

Examples

apic1# configure
apic1(config)# tenant exampleCorp
apic1(config-tenant)# interface bridge-domain exampleCorp_bd1
apic1(config-tenant-interface)# mac-address 1234.5678.abcd
apic(config-tenant-interface)# ip dhcp relay address 192.0.20.1 tenant exampleCorp application
app1 epg epg1
apic1(config-tenant-interface)# ip shared address 1.2.3.4/24 provider application any
apic1(config-tenant-interface)# ip shared address 3.2.3.4/24 consumer application any epg
any
apic1(config-tenant-interface)# exit
apic1(config-tenant)# exit
apic1(config)# tenant my_dhcp_provider
apic1(config-tenant)# interface bridge-domain bd_dhcp
apic1(config-tenant-interface)# ip shared address 7.8.9.1/24 provider application app2 epg
epg2

Configuring an Enforced Bridge Domain

An enforced bridge domain (BD) configuration entails creating an endpoint in a subject endpoint group (EPG)
which can only ping subnet gateways within the associated bridge domain.
With this configuration, you can then create a global exception list of IP addresses which can ping any subnet
gateway.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

45
 Configuring Tenants
Configuring an Enforced Bridge Domain

Figure 1: Enforced Bridge Domain

Note • The exception IP addresses can ping all of the BD gateways across all of your VRFs.
• A loopback interface configured for an L3 out does not enforce reachability to the IP address that is
configured for the subject loopback interface.
• When an eBGP peer IP address exists in a different subnet than the subnet of the L3out interface, the
peer subnet must be added to the allowed exception subnets.
Otherwise, eBGP traffic is blocked because the source IP address exists in a different subnet than the
L3out interface subnet.

Configuring an Enforced Bridge Domain

An enforced bridge domain (BD) configuration entails creating an endpoint in a subject endpoint group (EPG)
which can only ping subnet gateways within the associated bridge domain.
With this configuration, you can then create a global exception list of IP addresses which can ping any subnet
gateway.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

46
 Configuring Tenants
Configuring an Enforced Bridge Domain Using the NX-OS Style CLI

Figure 2: Enforced Bridge Domain

Note • The exception IP addresses can ping all of the BD gateways across all of your VRFs.
• A loopback interface configured for an L3 out does not enforce reachability to the IP address that is
configured for the subject loopback interface.
• When an eBGP peer IP address exists in a different subnet than the subnet of the L3out interface, the
peer subnet must be added to the allowed exception subnets.
Otherwise, eBGP traffic is blocked because the source IP address exists in a different subnet than the
L3out interface subnet.

Configuring an Enforced Bridge Domain Using the NX-OS Style CLI

This section provides information on how to configure your enforced bridge domain using the NX-OS style
command line interface (CLI).

Procedure

Step 1 Create and enable the tenant:

Example:
In the following example ("cokeVrf") is created and enabled.
apic1(config-tenant)# vrf context cokeVrf
apic1(config-tenant-vrf)# bd-enforce enable
apic1(config-tenant-vrf)# exit
apic1(config-tenant)#exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

47
 Configuring Tenants
Creating an Application Endpoint Group

Step 2 Add the subnet to the exception list.

Example:
apic1(config)#bd-enf-exp-ip add1.2.3.4/24
apic1(config)#exit

You can confirm if the enforced bridge domain is operational using the following type of command:
apic1# show running-config all | grep bd-enf
bd-enforce enable
bd-enf-exp-ip add 1.2.3.4/24

Example
The following command removes the subnet from the exception list:
apic1(config)# no bd-enf-exp-ip 1.2.3.4/24
apic1(config)#tenant coke
apic1(config-tenant)#vrf context cokeVrf

What to do next
To disable the enforced bridge domain run the following command:
apic1(config-tenant-vrf)# no bd-enforce enable

Creating an Application Endpoint Group

This topic describes the following steps in the basic provisioning of a static application EPG:
1. Create an application profile within the tenant
2. Create an EPG in the application profile
3. Assign a bridge domain to the EPG
4. Deploy the EPG to a Layer 2 interface

Before you begin

Before you can create an application profile and an application endpoint group (EPG), you must create a
VLAN domain, tenant, VRF, and bridge domain.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

48
Configuring Tenants
Creating an Application Endpoint Group

Command or Action Purpose

Step 2 tenant tenant-name Enters the tenant configuration mode.
Example:
apic1(config)# tenant exampleCorp

Step 3 [no] application app-name Creates an application profile and enters

application profile configuration mode.
Example:
apic1(config-tenant)# application
OnlineStore

Step 4 [no] epg epg-name Creates (or deletes) an EPG in the application
profile and enters EPG configuration mode.
Example:
apic1(config-tenant-app)# epg
exampleCorp_webepg1

Step 5 [no] bridge-domain member epg-name Associates the EPG to the bridge domain.
Every EPG must belong to a BD.
Example:
apic1(config-tenant-app-epg)#
bridge-domain member exampleCorp_b1

Step 6 exit Returns to the tenant application configuration

mode.
Example:
apic1(config-tenant-app-epg)# exit

Step 7 exit Returns to the tenant configuration mode.

Example:
apic1(config-tenant-app)# exit

Step 8 exit Returns to the global configuration mode.

Example:
apic1(config-tenant)# exit

Step 9 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 10 interface type Specifies the interface that you are configuring.
For an Ethernet port, use “ethernet slot / port.”
Example:
apic1(config-leaf)# interface eth 1/2

Step 11 (Optional) switchport Because layer 2 is the default state of a port,

this command is only needed when the port
Example:
must be converted from a layer 3
apic1(config-leaf-if)# switchport configuration.

Step 12 vlan-domain member domain-name Associates the interface with a VLAN domain.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

49
 Configuring Tenants
Creating an Application Endpoint Group

Command or Action Purpose

apic1(config-leaf-if)# vlan-domain
member dom1

Step 13 switchport trunk allowed vlan vlan-id tenant Deploys the EPG on the interface and identifies
tenant-name app app-name epg epg-name the EPG through EPG-to-VLAN mapping.
This configuration applies only to static EPG
Example:
deployment. If the VLAN is in use for another
apic1(config-leaf-if)# switchport trunk EPG or external SVI, you must delete the
allowed vlan 10 tenant exampleCorp
application OnlineStore epg
VLAN configuration before using it for this
exampleCorp_webepg1 EPG.
Note The interface must be associated
with a VLAN domain or this
command is rejected.

Examples
This example shows how to create an application EPG deployed to a layer 2 port.

apic1# configure
apic1(config)# tenant exampleCorp
apic1(config-tenant)# application OnlineStore
apic1(config-tenant-app)# epg exampleCorp_webepg1
apic1(config-tenant-app-epg)# bridge-domain member exampleCorp_b1
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)# exit
apic1(config-tenant)# exit

apic1(config)# leaf 101

apic1(config-leaf)# interface eth 1/2
apic1(config-leaf-if)# switchport
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# switchport trunk allowed vlan 10 tenant exampleCorp application
OnlineStore epg exampleCorp_webepg1

This example shows how to deploy the EPG to a port channel.

apic1(config)# leaf 101

apic1(config-leaf)# interface port-channel po1
apic1(config-leaf-if)# switchport
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# switchport trunk allowed vlan 10 tenant exampleCorp application
OnlineStore epg exampleCorp_webepg1

What to do next
Map a VLAN on a port to the EPG.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

50
 Configuring Tenants
Configuring Legacy Forwarding Mode in the Bridge Domain

Configuring Legacy Forwarding Mode in the Bridge Domain

Legacy forwarding mode allows switching and routing without the use of contracts or EPGs. In this mode,
the VLAN on a port directly maps to a bridge domain. The legacy forwarding vlan command automatically
creates all necessary objects so that no EPG-related configuration is required.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:

configure

Step 2 tenant tenant-name Enters the tenant configuration mode.

Example:
apic1(config)# tenant exampleCorp

Step 3 bridge-domain bd-name Enters tenant interface configuration mode to

configure the bridge domain.
Example:
apic1(config-tenant)# bridge-domain
exampleCorp_b1

Step 4 [no] legacy forwarding vlan vlan-id Maps the VLAN to the bridge domain.
vlan-domain vlan-domain-name
Example:
apic1(config-tenant-bd)#
legacy-forwarding vlan 50 vlan-domain
dom1

Step 5 exit Returns to the tenant configuration mode.

Example:
apic1(config-tenant-bd)# exit

Step 6 exit Returns to the global configuration mode.

Example:
apic1(config-tenant)# exit

Step 7 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 8 interface type Specifies the interface that you are configuring.
For an Ethernet port, use ethernet slot/port .
Example:
apic1(config-leaf)# interface eth 1/1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

51
 Configuring Tenants
Configuring Contracts

Command or Action Purpose

Step 9 [no] switchport trunk allowed vlan vlan-id Enables the VLAN on the interface and
tenant tenant-name legacy-forwarding associates it to the tenant bridge domain that
uses the VLAN in the legacy forwarding mode.
Example:
apic1(config-leaf-if)# switchport trunk
allowed vlan 50 tenant exampleCorp
legacy-forwarding

Examples
This example shows how to configure legacy forwarding mode for forwarding between bridge
domains.

apic1# configure
apic1(config)# tenant exampleCorp
apic1(config-tenant)# bridge-domain exampleCorp_b1
apic1(config-tenant-bd)# legacy-forwarding vlan 50 vlan-domain dom1
apic1(config-tenant-bd)# exit
apic1(config-tenant)# bridge-domain exampleCorp_b2
apic1(config-tenant-bd)# legacy-forwarding vlan 60 vlan-domain dom1
apic1(config-tenant-bd)# exit
apic1(config-tenant)# exit

apic1(config)# leaf 101

apic1(config-leaf)# interface eth 1/1
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# switchport trunk allowed vlan 50 tenant exampleCorp legacy-forwarding
apic1(config-leaf-if)# exit
apic1(config-leaf)# interface eth 1/2
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# switchport trunk allowed vlan 60 tenant exampleCorp legacy-forwarding

Configuring Contracts
Contracts are configured under a tenant with the following tasks:
• Define filters as access lists
• Define the contract and subjects
• Link the contract to an EPG

The tasks need not follow this order. For example, you can link a contract name to an EPG before you have
defined the contract.

Note Filters (ACLs) in APIC use match instead of permit | deny as in the traditional NX-OS ACL. The purpose
of a filter entry is only to match a given traffic flow. The traffic will be permitted or denied when the ACL is
applied on a contract or on a taboo contract.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

52
Configuring Tenants
Configuring Contracts

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 tenant tenant-name Creates a tenant if it does not exist and enters
the tenant configuration mode.
Example:

tenant exampleCorp

Step 3 access-list acl-name Creates an access list (filter) that can be used
in a contract.
Example:
apic1(config-tenant)# access-list
http_acl

Step 4 (Optional) match {arp | icmp | ip} Creates a rule to match traffic of the selected
protocol.
Example:
apic1(config-tenant-acl)# match arp

Step 5 (Optional) match {tcp | udp} [src from[-to]] Creates a rule to match TCP or UDP traffic.
[dest from[-to]]
Example:

apic1(config-tenant-acl)# match tcp dest

80
apic1(config-tenant-acl)# match tcp dest
443

Step 6 (Optional) match raw options Creates a rule to match a raw vzEntry.
Example:
apic1(config-tenant-acl)#

Step 7 exit Returns to the tenant configuration mode.

Example:
apic1(config-tenant-acl)# exit

Step 8 contract contract-name Creates a contract and enters the contract

configuration mode.
Example:
apic1(config-tenant)# contract web80

Step 9 subject subject-name Creates a contract subject and enters the

subject configuration mode.
Example:
apic1(config-tenant-contract)# subject
web80

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

53
 Configuring Tenants
Configuring Contracts

Command or Action Purpose

Step 10 (Optional) [no] access-group acl-name [in | Adds (removes) an access list from the
out | both] contract, specifying the direction of the traffic
to be matched.
Example:
apic1(config-tenant-contract-subj)#
access-group http_acl both

Step 11 (Optional) [no] label name label-name Adds (removes) a provider or consumer label
{provider | consumer} to the subject.
Example:
apic1(config-tenant-contract-subj)#

Step 12 (Optional) [no] label match {provider | Specifies the match type for the provider or
consumer} [any | one | all | none] consumer label:
Example: • any —Match if any label is found in the
apic1(config-tenant-contract-subj)# contract relation.
• one —Match if exactly one label is found
in the contract relation.
• all —Match if all labels are found in the
contract relation.
• none —Match if no labels are found in
the contract relation.

Step 13 exit Returns to the contract configuration mode.

Example:
apic1(config-tenant-contract-subj)# exit

Step 14 exit Returns to the tenant configuration mode.

Example:
apic1(config-tenant-contract)# exit

Step 15 application app-name Enters application configuration mode.

Example:
apic1(config-tenant)# application
OnlineStore

Step 16 epg epg-name Enters configuration mode for the EPG to be

linked to the contract.
Example:
apic1(config-tenant-app)# epg
exampleCorp_webepg1

Step 17 bridge-domain member bd-name Specifies the bridge domain for this EPG.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

54
Configuring Tenants
Configuring Contracts

Command or Action Purpose

apic1(config-tenant-app-epg)#
bridge-domain member exampleCorp_bd1

Step 18 contract provider provider-contract-name Specifies the provider contract for this EPG.
Communication with this EPG can be initiated
Example:
from other EPGs as long as the communication
apic1(config-tenant-app-epg)# contract complies with this provider contract.
provider web80

Step 19 contract consumer consumer-contract-name Specifies the consumer contract for this EPG.
The endpoints in this EPG may initiate
Example:
communication with any endpoint in an EPG
apic1(config-tenant-app-epg)# contract that is providing this contract.
consumer rmi99

Examples
This example shows how to create and apply contracts to an EPG.

apic1# configure
apic1(config)# tenant exampleCorp

# CREATE FILTERS
apic1(config-tenant)# access-list http_acl
apic1(config-tenant-acl)# match tcp dest 80
apic1(config-tenant-acl)# match tcp dest 443
apic1(config-tenant-acl)# exit

# CREATE CONTRACT WITH FILTERS

apic1(config-tenant)# contract web80
apic1(config-tenant-contract)# subject web80
apic1(config-tenant-contract-subj)# access-group http_acl both
apic1(config-tenant-contract-subj)# exit
apic1(config-tenant-contract)# exit

# ASSOCIATE CONTRACTS TO EPG

apic1(config-tenant)# application OnlineStore
apic1(config-tenant-app)# epg exampleCorp_webepg1
apic1(config-tenant-app-epg)# bridge-domain member exampleCorp_bd1
apic1(config-tenant-app-epg)# contract consumer rmi99
apic1(config-tenant-app-epg)# contract provider web80
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)#exit
apic1(config-tenant)#exit

# ASSOCIATE PORT AND VLAN TO EPG

apic1(config)#leaf 101
apic1(config-leaf)# interface ethernet 1/4
apic1(config-leaf-if)# switchport trunk allowed vlan 102 tenant exampleCorp application
OnlineStore epg exampleCorp_webepg1

This example shows a simpler method for defining a contract by declaring the filters inline in the
contract itself.

apic1# configure

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

55
 Configuring Tenants
Contract Inheritance

apic1(config)# tenant exampleCorp

apic1(config-tenant)# contract web80
apic1(config-tenant-contract)# match tcp 80
apic1(config-tenant-contract)# match tcp 443

Contract Inheritance
About Contract Inheritance
To streamline associating contracts to new EPGs, you can now enable an EPG to inherit all the (provided and
consumed) contracts associated directly to another EPG in the same tenant. Contract inheritance can be
configured for application, microsegmented, L2Out, and L3Out EPGs.
With Release 3.x, you can also configure contract inheritance for Inter-EPG contracts, both provided and
consumed. Inter-EPG contracts are supported on Cisco Nexus 9000 Series switches with EX or FX at the end
of their model name or later models.
You can enable an EPG to inherit all the contracts associated directly to another EPG, using the APIC GUI,
NX-OS style CLI, and the REST API.
Figure 3: Contract Inheritance

In the diagram above, EPG A is configured to inherit Provided-Contract 1 and 2 and Consumed-Contract 3
from EPG B (contract master for EPG A).

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

56
 Configuring Tenants
Configuring Application or uSeg EPG Contract Inheritance Using the NX-OS Style CLI

Use the following guidelines when configuring contract inheritance:

• Contract inheritance can be configured for application, microsegmented (uSeg), external L2Out EPGs,
and external L3Out EPGs. The relationships must be between EPGs of the same type.
• Both provided and consumed contracts are inherited from the contract master when the relationship is
established.
• Contract masters and the EPGs inheriting contracts must be within the same tenant.
• Changes to the masters’ contracts are propagated to all the inheritors. If a new contract is added to the
master, it is also added to the inheritors.
• An EPG can inherit contracts from multiple contract masters.
• Contract inheritance is only supported to a single level (cannot be chained) and a contract master cannot
inherit contracts.
• Contract subject label and EPG label inheritance is supported. When EPG A inherits a contract from
EPG B, if different subject labels are configured under EPG A and EPG B, APIC only uses the subject
label configured under EPG B and not a collection of labels from both EPGs.
• Whether an EPG is directly associated to a contract or inherits a contract, it consumes entries in TCAM.
So contract scale guidelines still apply. For more information, see the Verified Scalability Guide for your
release.
• vzAny security contracts and taboo contracts are not supported.

For information about configuring Contract Inheritance and viewing inherited and standalone contracts, see
Cisco APIC Basic Configuration Guide.

Configuring Application or uSeg EPG Contract Inheritance Using the NX-OS

Style CLI
To configure contract inheritance for application or uSeg EPGs, use the following commands:

Before you begin

Configure the tenant, application profile, and bridge-domain to be used by the EPGs.
Configure the contracts to be shared by the EPGs at the VRF level.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 tenant tenant-name Creates or specifies the tenant to be configured;

and enters into tenant configuration mode.
Example:
apic1# (config) tenant Tn1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

57
 Configuring Tenants
Configuring Application or uSeg EPG Contract Inheritance Using the NX-OS Style CLI

Command or Action Purpose

Step 3 application application-name Creates or specifies an application and enters
into application mode.
Example:
apic1(config-tenant)# application AP1

Step 4 epg epg-name [type micro-segmented] Creates or specifies the application or uSeg
EPG to be configured and enters into EPG
Example:
configuration mode. For uSeg EPGs add the
apic1(config-tenant-app)# epg AEPg403 type.
In this example, this is the application EPG
contract master.

Step 5 bridge-domain member bd-name Associates the EPG with the bridge domain.
Example:
apic1(config-tenant-app-epg)#
bridge-domain member T1BD1

Step 6 contract consumer contract-name Adds a contract to be consumed by this EPG.

Example:
apic1(config-tenant-app-epg)# contract
consumer cctr5

Step 7 contract provider [label label] Adds a contract to be provided by this EPG,
including an optional list of subject or EPG
Example:
labels (must be previously configured).
apic1(config-tenant-app-epg)# contract
provider T1ctrl_cif

Step 8 exit Exits the configuration mode

Example:
apic1(config-tenant-app-epg)# exit

Step 9 epg epg-name [type micro-segmented] Creates or specifies the application or uSeg
EPG to be configured and enters into EPG
Example:
configuration mode. For uSeg EPGs add the
apic1(config-tenant-app)# epg AEPg404 type.
In this example, this is the EPG inheriting
contracts.

Step 10 bridge-domain member bd-name Associates the EPG with the bridge domain.
Example:
apic1(config-tenant-app-epg)#
bridge-domain member T1BD1

Step 11 inherit-from-epg application Configures this EPG to inherit contracts from

application-name epg the EPG contract master.
EPG-contract-master-name]
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

58
Configuring Tenants
Configuring Application or uSeg EPG Contract Inheritance Using the NX-OS Style CLI

Command or Action Purpose

apic1(config-tenant-app-epg)#
inherit-from-epg application AP1 epg
AEPg403

Step 12 exit Exits the configuration mode

Example:
apic1(config-tenant-app-epg)# exit

Step 13 epg epg-name [type micro-segmented] Creates or specifies the application or uSeg
EPG to be configured and enters into EPG
Example:
configuration mode.
apic1(config-tenant-app)# epg
uSeg1_403_10 type micro-segmented In this example, this is the uSeg EPG contract
master.

Step 14 bridge-domain member bd-name Associates the EPG with the bridge domain.
Example:
apic1(config-tenant-app-epg)#
bridge-domain member T1BD1

Step 15 contract provider [label label] Adds a contract to be provided by this EPG,
including an optional list of subject or EPG
Example:
labels (must be previously configured).
apic1(config-tenant-app-epg)# contract
provider T1ctrl_uSeg_l3out

Step 16 attribute-logical-expression Adds a logical expression to the uSeg EPG as

logical-expression matching criteria.
Example:
apic1(config-tenant-app-epg)#
attribute-logical-expression 'ip equals
192.168.103.10 force'

Step 17 exit Exits the configuration mode

Example:
apic1(config-tenant-app-epg)# exit

Step 18 epg epg-name [type micro-segmented] Creates or specifies the application or uSeg
EPG to be configured and enters into EPG
Example:
configuration mode.
apic1(config-tenant-app)# epg
uSeg1_403_30 type micro-segmented In this example, this is the uSeg EPG that
inherits contracts from the EPG contract
master.

Step 19 bridge-domain member bd-name Associates the EPG with the bridge domain.
Example:
apic1(config-tenant-app-epg)#
bridge-domain member T1BD1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

59
 Configuring Tenants
Configuring Application or uSeg EPG Contract Inheritance Using the NX-OS Style CLI

Command or Action Purpose

Step 20 attribute-logical-expression Adds a logical expression to the uSeg EPG as
logical-expression criteria.
Example:
apic1(config-tenant-app-epg)#
attribute-logical-expression 'ip equals
192.168.103.30 force'

Step 21 inherit-from-epg application Configures this EPG to inherit contracts from

application-name epg the EPG contract master.
EPG-contract-master-name
Example:
apic1(config-tenant-app-epg)#
inherit-from-epg application AP1 epg
uSeg1_403_10

Step 22 exit Exits the configuration mode

Example:
apic1(config-tenant-app-epg)# exit

Step 23 exit Exits the configuration mode

Example:
apic1(config-tenant-app)# exit

Step 24 exit Exits the configuration mode

Example:
apic1(config-tenant)# exit

Step 25 exit Exits the configuration mode

Example:
apic1(config)# exit

Example
ifav90-ifc1# show running-config tenant Tn1 application AP1
# Command: show running-config tenant Tn1 application AP1
# Time: Fri Apr 28 17:28:32 2017
tenant Tn1
application AP1
epg AEPg403
bridge-domain member T1BD1
contract consumer cctr5 imported
contract provider T1ctr1_cif
exit
epg AEPg404
bridge-domain member T1BD1
inherit-from-epg application AP1 epg AEPg403
exit
epg uSeg1_403_10 type micro-segmented
bridge-domain member T1BD1
contract provider T1Ctr1_uSeg_l3out

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

60
 Configuring Tenants
Configuring L2Out EPG Contract Inheritance Using the NX-OS Style CLI

attribute-logical-expression 'ip equals 192.168.103.10 force'

exit
epg uSeg1_403_30 type micro-segmented
bridge-domain member T1BD1
attribute-logical-expression 'ip equals 192.168.103.30 force'
inherit-from-epg application AP1 epg uSeg1_403_10
exit
exit
exit

Configuring L2Out EPG Contract Inheritance Using the NX-OS Style CLI
To configure contract inheritance for an external L2Out EPG, use the following commands:

Before you begin

Configure the tenant, VRF, and bridge-domain to be used by the EPGs.
Configure the Layer 2 outside network (L2Out) that the EPGs will use.
Configure the contracts to be shared by the EPGs, at the VRF level.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 tenant tenant-name Creates or specifies the tenant to be configured;

and enters into tenant configuration mode.
Example:
apic1(config)# tenant Tn1

Step 3 external-l2 epg external-l2-epg-name Configures or specifies an external L2Out

EPG. In this example, this is the L2out contract
Example:
master.
apic1(config-tenant)# external-l2 epg
l2out1:l2Ext1

Step 4 bridge-domain member bd-name Associates the L2Out EPG with a bridge
domain.
Example:

apic1(config-tenant-l2ext-epg)#
bridge-domain member T1BD1

Step 5 contract provider contract-name [label label] Adds a contract to be provided by this EPG.
Example:
apic1(config-tenant-l2ext-epg)# contract
provider T1ctr_tcp

Step 6 exit Exits the configuration mode

Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

61
 Configuring Tenants
Configuring L2Out EPG Contract Inheritance Using the NX-OS Style CLI

Command or Action Purpose

apic1(config-tenant-l2ext-epg)# exit

Step 7 external-l2 epg external-l2-epg-name Configures an external L2Out EPG. In this

example, this is the EPG that inherits contracts
Example:
from the L2out contract master.
apic1(config-tenant)# external-l2 epg
L2out12:l2Ext12

Step 8 bridge-domain member bd-name Associates the L2out EPG with the bridge
domain.
Example:
apic1(config-tenant-l2ext-epg)#
bridge-domain member T1BD1

Step 9 inherit-from-epg Configures this EPG to inherit contracts from

L2Out-contract-master-name the L2Out contract master.
Example:
apic1(config-tenant-l2ext-epg)#
inherit-from-epg epg l2out1:l2Ext1

Step 10 exit Exits the configuration mode

Example:
apic1(config-tenant-l2ext-epg)# exit

Example
The steps above are taken from the following example:
apic1# show running-config tenant Tn1 external-l2
# Command: show running-config tenant Tn1 external-l2
# Time: Thu May 11 13:10:14 2017
tenant Tn1
external-l2 epg l2out1:l2Ext1
bridge-domain member T1BD1
contract provider T1ctr_tcp
exit
external-l2 epg l2out10:l2Ext10
bridge-domain member T1BD10
contract provider T1ctr_tcp
exit
external-l2 epg l2out11:l2Ext11
bridge-domain member T1BD11
contract provider T1ctr_udp
exit
external-l2 epg l2out12:l2Ext12
bridge-domain member T1BD12
inherit-from-epg epg l2out1:l2Ext1
inherit-from-epg epg l2out10:l2Ext10
inherit-from-epg epg l2out11:l2Ext11
inherit-from-epg epg l2out2:l2Ext2
inherit-from-epg epg l2out3:l2Ext3
inherit-from-epg epg l2out4:l2Ext4
inherit-from-epg epg l2out5:l2Ext5
inherit-from-epg epg l2out6:l2Ext6
inherit-from-epg epg l2out7:l2Ext7
inherit-from-epg epg l2out8:l2Ext8

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

62
 Configuring Tenants
Configuring External L3Out EPG Contract Inheritance Using the NX-OS Style CLI

inherit-from-epg epg l2out9:l2Ext9

exit
external-l2 epg l2out2:l2Ext2
bridge-domain member T1BD2
contract provider T1ctr_tcp
exit
external-l2 epg l2out3:l2Ext3
bridge-domain member T1BD3
contract provider T1ctr_tcp
exit
external-l2 epg l2out4:l2Ext4
bridge-domain member T1BD4
contract provider T1ctr_tcp
exit
external-l2 epg l2out5:l2Ext5
bridge-domain member T1BD5
contract provider T1ctr_tcp
exit
external-l2 epg l2out6:l2Ext6
bridge-domain member T1BD6
contract provider T1ctr_tcp
exit
external-l2 epg l2out7:l2Ext7
bridge-domain member T1BD7
contract provider T1ctr_tcp
exit
external-l2 epg l2out8:l2Ext8
bridge-domain member T1BD8
contract provider T1ctr_tcp
exit
external-l2 epg l2out9:l2Ext9
bridge-domain member T1BD9
contract provider T1ctr_tcp
exit
exit

Configuring External L3Out EPG Contract Inheritance Using the NX-OS Style
CLI
To configure contract inheritance for an external L3Out EPG, use the following commands:

Before you begin

Configure the tenant, VRF, and bridge-domain to be used by the EPGs.
Configure the Layer 3 outside network (L3Out) that the EPGs will use.
Configure the contracts to be shared by the EPGs, at the VRF level.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

63
 Configuring Tenants
Configuring External L3Out EPG Contract Inheritance Using the NX-OS Style CLI

Command or Action Purpose

Step 2 tenant tenant-name Creates or specifies the tenant to be configured;
and enters into tenant configuration mode.
Example:
apic1(config)# tenant Tn1

Step 3 external-l3 epg external-l3-epg-name l3out Configures an external L3Out EPG. In this
l3out-name example, this is the L3out contract master.
Example:
apic1(config-tenant-app)# external-l3
epg l3Ext108 l3out T1L3out1

Step 4 vrf member vrf-name Associates the L3out with the VRF.
Example:
apic1(tenant-l3out)# vrf member T1ctx1

Step 5 match ip ip-address-and-mask Adds a subnet that identifies hosts as part of

the EPG and adds the optional shared scope
Example:
for the subnet.
apic1(config-tenant-l3ext-epg)# match
ip 192.168.110.0/24 shared

Step 6 contract provider contract-name [label label] Adds a contract to be provided by this EPG.
Example:
apic1(config-tenant-l3ext-epg)# contract
provider T1ctrl-L3out

Step 7 exit Exits the configuration mode

Example:
apic1(config-tenant-l3ext-epg)# exit

Step 8 external-l3 epg external-l3-epg-name l3out Configures an external L3Out EPG. In this
l3out-name example, this is the EPG that inherits contracts
from the L3out contract master.
Example:
apic1(config-tenant-app)# external-l3
epg l3Ext110 l3out T1L3out1

Step 9 vrf member vrf-name Associates the L3out with the VRF.
Example:
apic1(tenant-l3out)# vrf member T1ctx1

Step 10 match ip ip-address-and-mask Adds a subnet that identifies hosts as part of

the EPG and adds the optional shared scope
Example:
for the subnet.
apic1(config-tenant-l3ext-epg)# match
ip 192.168.112.0/24 shared

Step 11 inherit-from-epg Configures this EPG to inherit contracts from

L3Out-contract-master-name the L3Out contract master.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

64
 Configuring Tenants
Configuring Contract Preferred Groups

Command or Action Purpose

apic1(config-tenant-l3ext-epg)#
inherit-from-epg l3Ext108

Step 12 exit Exits the configuration mode

Example:
apic1(config-tenant-l3ext-epg)# exit

Example
ifav90-ifc1# show running-config tenant Tn1 external-l3 epg l3Ext110
# Command: show running-config tenant Tn1 external-l3 epg l3Ext110
# Time: Fri Apr 28 17:36:15 2017
tenant Tn1
external-l3 epg l3Ext108 l3out T1L3out1
vrf member T1ctx1
match ip 192.168.110.0/24 shared
contract provider T1ctrl-L3out
exit
external-l3 epg l3Ext110 l3out T1L3out1
vrf member T1ctx1
match ip 192.168.112.0/24 shared
inherit-from-epg epg l3Ext108
exit
exit

Configuring Contract Preferred Groups

About Contract Preferred Groups
There are two types of policy enforcements available for EPGs in a VRF with a contract preferred group
configured:
• Included EPGs: EPGs can freely communicate with each other without contracts, if they have membership
in a contract preferred group. This is based on the source-any-destination-any-permit default rule.
• Excluded EPGs: EPGs that are not members of preferred groups require contracts to communicate with
each other. Otherwise, the default source-any-destination-any-deny rule applies.

The contract preferred group feature enables greater control of communication between EPGs in a VRF. If
most of the EPGs in the VRF should have open communication, but a few should only have limited
communication with the other EPGs, you can configure a combination of a contract preferred group and
contracts with filters to control inter-EPG communication precisely.
EPGs that are excluded from the preferred group can only communicate with other EPGs if there is a contract
in place to override the source-any-destination-any-deny default rule.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

65
 Configuring Tenants
About Contract Preferred Groups

Figure 4: Contract Preferred Group Overview

Limitations
The following limitations apply to contract preferred groups:
• In topologies where an L3Out and application EPG are configured in a Contract Preferred Group, and
the EPG is deployed only on a VPC, you may find that only one leaf switch in the VPC has the prefix
entry for the L3Out. In this situation, the other leaf switch in the VPC does not have the entry, and
therefore drops the traffic.
To workaround this issue, you can do one of the following:
• Disable and reenable the contract group in the VRF
• Delete and recreate the prefix entries for the L3Out EPG

• Also, where the provider or consumer EPG in a service graph contract is included in a contract group,
the shadow EPG can not be excluded from the contract group. The shadow EPG will be permitted in the
contract group, but it does not trigger contract group policy deployment on the node where the shadow
EPG is deployed. To download the contract group policy to the node, you deploy a dummy EPG within
the contract group .

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

66
 Configuring Tenants
Configuring Contract Preferred Groups Using the NX-OS Style CLI

Configuring Contract Preferred Groups Using the NX-OS Style CLI

You can use the APIC NX-OS style CLI to configure a contract preferred group. In this example, a contract
preferred group is configured for a VRF. One of the EPGs using the VRF is included in the preferred group.

Before you begin

Create the tenants, VRFs, and EPGs that will consume the contract preferred group.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode
Example:
apic1# configure
apic1(config)#

Step 2 tenant tenant-name Creates a tenant or enters into tenant

configuration mode
Example:
apic1(config)# tenant tenant64

Step 3 vrf context vrf-name Creates a VRF or enters into VRF

configuration mode.
Example:
apic1(config-tenant)# vrf context vrf64

Step 4 whitelist-blacklist-mix Enables a contract preferred group for the VRF

and then returns to tenant configuration mode.
Example:
apic1(config-tenant-vrf)#
whitelist-blacklist-mix
apic1(config-tenant-vrf)# exit

Step 5 bridge-domain bd-name Creates a bridge-domain for the VRF or enters

into BD configuration mode.
Example:
apic1(config-tenant)# bridge-domain
bd64

Step 6 vrf member vrf-name Associates the VRF with the bridge-domain
and returns to teanant configuration mode.
Example:
apic1(config-tenant-bd)# vrf member
vrf64
apic1(config-tenant-bd)# exit

Step 7 application app-name Creates an application or enters into application

configuration mode.
Example:
apic1(config-tenant)# application
app-ldap

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

67
 Configuring Tenants
Exporting a Contract to Another Tenant

Command or Action Purpose

Step 8 epg epg-name Creates an EPG or enters into EPG tenant-app
EPG configuration mode.
Example:
apic1(config-tenant-app)# epg epg-ldap

Step 9 bridge-domain member bd-name Associates the EPG with the bridge-domain .
Example:
apic1(config-tenant-app-epg)#
bridge-domain member bd64

Step 10 vrf-blacklist-mode Configures this EPG to be included in the

contract preferred group.
Example:
apic1(config-tenant-app-epg)#
vrf-blacklist-mode

Example
The following example creates a contract preferred group for vrf64 and includes epg-ldap in it.
apic1# configure
apic1(config)# tenant tenant64
apic1(config-tenant)# vrf context vrf64
apic1(config-tenant-vrf)# whitelist-blacklist-mix
apic1(config-tenant-vrf)# exit

apic1(config-tenant)# bridge-domain bd64

apic1(config-tenant-bd)# vrf member vrf64
apic1(config-tenant-bd)# exit

apic1(config-tenant)# application app-ldap

apic1(config-tenant-app)# epg epg-ldap
apic1(config-tenant-app-epg)# bridge-domain member bd64
apic1(config-tenant-app-epg)# vrf-blacklist-mode

Exporting a Contract to Another Tenant

You can export a contract from one tenant and import it to another. In the tenant that imports the contract, the
contract can be applied only as a consumer contract. The contract can be renamed during the export.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

68
Configuring Tenants
Exporting a Contract to Another Tenant

Command or Action Purpose

Step 2 tenant tenant-name Enters the tenant configuration mode for the
exporting tenant.
Example:
apic1(config)# tenant RedCorp

Step 3 contract contract-name Enters the contract configuration mode for the
contract to be exported.
Example:
apic1(config-tenant)# contract web80

Step 4 scope {application | exportable | tenant | vrf} Configures how the contract can be shared.
The scope can be:
Example:
apic1(config-tenant-contract)# scope • application —Can be shared among the
exportable EPGs of the same application.
• exportable —Can be shared across
tenants.
• tenant —Can be shared among the EPGs
of the same tenant.
• vrf —Can be shared among the EPGs of
the same VRF.

Step 5 export to tenant other-tenant-name as Exports the contract to the other tenant. You
new-contract-name can use the same contract name or you can
rename it.
Example:
apic1(config-tenant-contract)# export
to tenant BlueCorp as webContract1

Step 6 exit Returns to the tenant configuration mode.

Example:
apic1(config-tenant-contract)# exit

Step 7 exit Returns to the global configuration mode.

Example:
apic1(config-tenant)# exit

Step 8 tenant tenant-name Enters the tenant configuration mode for the
importing tenant.
Example:

tenant BlueCorp

Step 9 application app-name Enters application configuration mode.

Example:
apic1(config-tenant)# application
BlueStore

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

69
 Configuring Tenants
Configuring Contract or Subject Exceptions

Command or Action Purpose

Step 10 epg epg-name Enters configuration mode for the EPG to be
linked to the contract.
Example:
apic1(config-tenant-app)# epg BlueWeb

Step 11 contract consumer consumer-contract-name Specifies the imported consumer contract for
imported this EPG. The endpoints in this EPG may
initiate communication with any endpoint in
Example:
an EPG that is providing this contract.
apic1(config-tenant-app-epg)# contract
consumer webContract1 imported

Examples
This example shows how to export a contract from the tenant RedCorp to the tenant BlueCorp, where
it will be a consumer contract.

apic# configure
apic1(config)# tenant RedCorp
apic1(config-tenant)# contract web80
apic1(config-tenant-contract)# scope exportable
apic1(config-tenant-contract)# export to tenant BlueCorp as webContract1
apic1(config-tenant-contract)# exit
apic1(config-tenant)# exit
apic1(config)# tenant BlueCorp
apic1(config-tenant)# application BlueStore
apic1(config-tenant-application)# epg BlueWeb
apic1(config-tenant-application-epg)# contract consumer webContract1 imported

Configuring Contract or Subject Exceptions

Configuring Contract or Subject Exceptions for Contracts
In Cisco APIC Release 3.2(1), contracts between EPGs are enhanced to enable denying a subset of contract
providers or consumers from participating in the contract. Inter-EPG contracts and Intra-EPG contracts are
supported with this feature.
You can enable a provider EPG to communicate with all consumer EPGs except those that match criteria
configured in a subject or contract exception. For example, if you want to enable an EPG to provide services
to all EPGs for a tenant, except a subset, you can enable those EPGs to be excluded. To configure this, you
create an exception in the contract or one of the subjects in the contract. The subset is then denied access to
providing or consuming the contract.
Labels, counters, and permit and deny logs are supported with contracts and subject exceptions.
To apply an exception to all subjects in a contract, add the exception to the contract. To apply an exception
only to a single subject in the contract, add the exception to the subject.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

70
 Configuring Tenants
Configure a Contract or Subject Exception Using the NX-OS Style CLI

When adding filters to subjects, you can set the action of the filter (to permit or deny objects that match the
filter criteria). Also for Deny filters, you can set the priority of the filter. Permit filters always have the default
priority. Marking the subject-to-filter relation to deny automatically applies to each pair of EPGs where there
is a match for the subject. Contracts and subjects can include multiple subject-to-filter relationships that can
be independently set to permit or deny the objects that match the filters.

Exception Types
Contract and subject exceptions can be based on the following types and include regular expressions, such as
the * wildcard:

Exception criteria exclude these Example Description

objects as defined in the Consumer
Regex and Provider Regex fields

Tenant <vzException consRegex= This example, excludes EPGs using

“common” field= “Tenant” name= the common tenant from consuming
“excep03” provRegex= “t1” /> contracts provided by the t1 tenant.

VRF <vzException consRegex= “ctx1” This example excludes members of

field= “Ctx” name= “excep05” ctx1 from consuming the services
provRegex= “ctx1” /> provided by the same VRF.

EPG <vzException consRegex= The example assumes that multiple

“EPgPa*” field= “EPg” name= EPGs exist, with names starting
“excep03” provRegex= “EPg03” with EPGPa, and they should all be
/> denied as consumers for the
contract provided by EPg03

Dn <vzException consRegex= This example excludes epg193 from

“uni/tn-t36/ap-customer/epg-epg193” consuming the contract provided
field= “Dn” name=“excep04” by epg200.
provRegex=
“uni/tn-t36/ap-customer/epg-epg200”
/>

Tag <vzException consRegex= “red” The example excludes objects

field= “Tag” name= “excep01” marked with the red tag from
provRegex= “green” /> consuming and those marked with
the green tag from participating in
the contract.

Configure a Contract or Subject Exception Using the NX-OS Style CLI

In this task, you configure a contract that will allow most of the EPGs to communicate, but deny access to a
subset of them. Multiple exceptions can be added to a contract or a subject.

Before you begin

Configure the tenant, VRF, application profile, and EPGs to provide and consume the contract.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

71
 Configuring Tenants
Creating Quota Management

Procedure

Step 1 Configure filters for HTTP and HTTPS, using commands as in the following example:
Example:
apic1(config)# tenant t2
apic1(config-tenant)# access-list ac1
apic1(config-tenant-acl)# match ip
apic1(config-tenant-acl)# match tcp dest 80
apic1(config-tenant-acl)# exit
apic1(config-tenant)# access-list ac2
apic1(config-tenant-acl)# match ip
apic1(config-tenant-acl)# match tcp dest 443

Step 2 Configure a contract that excludes EPg01 from consuming it and EPg03 from providing it.
Example:

apic1(config-tenant)# contract webCtrct

apic1(config-tenant-contract)# subject https-subject
apic1(config-tenant-contract-subj)# exception name EPG consumer-regexp EPg01 field EPg
provider-regexp EPg03
apic1(config-tenant-contract-subj)# access-group ac1 in blacklist
apic1(config-tenant-contract-subj)# access-group ac2 in whitelist

Creating Quota Management

About APIC Quota Management Configuration
Starting in the Cisco Application Policy Infrastructure Controller (APIC) Release 2.3(1), there are limits on
number of objects a tenant admin can configure. This enables the admin to limit what managed objects that
can be added under a given tenant or globally across tenants.
This feature is useful when you want to limit any tenant or group of tenants from exceeding ACI maximums
per leaf or per fabric or unfairly consuming a majority of available resources, potentially affecting other tenants
on the same fabric.

Creating a Quota Management Configuration Using the NX-OS Style CLI

This procedure explains how to create a quota management configuration using the NX-OS Style CLI.

Procedure

Create a quota management configuration using the NX-OS CLI:

Example:
apic1# conf t
apic1(config)# quota fvBD max 100 scope uni/tn-green exceed-action fault
apic1(config)# quota fvBD max 1000 scope uni exceed-action fail

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

72
Configuring Tenants
Creating a Quota Management Configuration Using the NX-OS Style CLI

apic1(config)# quota fvBD max 34 tenant red

Syntax:

[no] quota <className> max <maxValue> [exceed-action {fail|fault}] \

[{scope <containerDn>| tenant <tenantName> \
[{bridge-domain <bd>|application <ap> [epg <epgName>]}]}]

where <className> is the managed object className such as fvBD or fvCtx etc. All the eligible classes
accordingly to the presence of the quota flag in the model are accepted.
where <maxValue> is the value after which the <exceed-action> is applied.
where <exceed-action> is the action to be taken after the <maxValue> is exceeded, can either be:
• fail: when you want to fail the transaction exceeding the limit.
• fault: raise a fault.

where <containerDn> is the tree under which the limit will be enforced. "uni" will be across the whole ACI
policy model, "tenant green" will be for the tenant green.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

73
 Configuring Tenants
Creating a Quota Management Configuration Using the NX-OS Style CLI

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

74
 CHAPTER 6
Configuring Layer 2 External Connectivity
• Configuring Layer 2 External Connectivity, on page 75
• Configuring VLAN Domains, on page 79
• Configuring Q-in-Q Encapsulation Mapping for EPGs, on page 86
• Support Fibre Channel over Ethernet Traffic on the ACI Fabric, on page 88
• Fibre Channel NPV, on page 102
• Configuring 802.1Q Tunnels, on page 108
• Configuring Dynamic Breakout Ports, on page 113
• Configuring Port Profiles, on page 118
• Microsegmentation on Virtual Switches, on page 124
• Configuring Microsegmentation on Bare-Metal , on page 127
• Configuring Layer 2 IGMP Snoop Multicast, on page 129
• Configuring Port Security, on page 136
• Configuring Proxy ARP, on page 144
• Configuring Traffic Storm Control, on page 152
• Configuring MACsec, on page 155

Configuring Layer 2 External Connectivity

Layer 2 External Connectivity represents the switching network between the ACI leaf switches (aka border
leaf) and an External Router. The VLAN representing the external L2 network is mapped to one of the
bridge-domains within the fabric, which provides the Layer 2 extension for the bridge-domain and lets the
EPGs using the bridge-domain talk to the outside network. The outside network is mapped to an EPG, which
helps in realizing contracts between different internal applications and different L2 outside VLANs across
nodes.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

75
 Configuring Layer 2 External Connectivity
Configuring Layer 2 External Connectivity

Caution Do not mix the GUI and the CLI, when doing per-interface configuration on APIC. Configurations performed
in the GUI, may only partially work in the NX-OS CLI.
For example, if you configure a switch port in the GUI at Tenants > tenant-name > Application Profiles >
application-profile-name > Application EPGs > EPG-name > Static Ports > Deploy Static EPG on PC,
VPC, or Interface
Then you use the show running-config command in the NX-OS style CLI, you receive output such as:
leaf 102
interface ethernet 1/15
switchport trunk allowed vlan 201 tenant t1 application ap1 epg ep1
exit
exit

If you use these commands to configure a static port in the NX-OS style CLI, the following error occurs:
apic1(config)# leaf 102
apic1(config-leaf)# interface ethernet 1/15
apic1(config-leaf-if)# switchport trunk allowed vlan 201 tenant t1 application ap1 epg ep1

No vlan-domain associated to node 102 interface ethernet1/15 encap vlan-201

This occurs because the CLI has validations that are not performed by the APIC GUI. For the commands from
the show running-config command to function in the NX-OS CLI, a vlan-domain must have been previously
configured. The order of configuration is not enforced in the GUI.

The configuration for Layer2 external connectivity is similar to a static application EPG, where you map a
VLAN on a node port to an EPG and map the EPG to a bridge-domain to provide/consume contracts.

Procedure

Command or Action Purpose

Step 1 Access configuration mode.
Example:
apic1# configure

Step 2 Enter tenant configuration mode.

Example:
apic1(config)# tenant exampleCorp

Step 3 [no] external-l2 epg epg-name Create (or delete ) an external layer 2 EPG.
Example:
apic1(config-tenant)# external-l2 epg
extendBD1

Step 4 Assign a bridge domain to the EPG.

Example:
apic1(config-tenant-extl2epg)#
bridge-domain member bd1

Step 5 Return to tenant configuration mode.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

76
Configuring Layer 2 External Connectivity
Configuring Layer 2 External Connectivity

Command or Action Purpose

Example:
apic1(config-tenant-extl2epg)# exit

Step 6 Return to global configuration mode.

Example:
apic1(config-tenant)# exit

Step 7 Specify the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 8 Specify a port for the external EPG.

Example:
apic1(config-leaf)# interface eth 1/2

Step 9 By default, a port is in Layer 2 trunk mode. If

the port is in Layer 3 mode, convert it to Layer
2 trunk mode using this command.
Example:
apic1(config-leaf-if)# switchport

Step 10 Associate the interface with a VLAN domain.

Example:
apic1(config-leaf-if)# vlan-domain
member dom1

Step 11 Assigns a VLAN on the leaf port and maps the Note The interface must be associated
VLAN to a layer 2 external EPG, with the with a VLAN domain or this
switchport trunk allowed vlan vlan-id tenant command is rejected.
tenant-name external-l2 epg epg-name
command.
Example:
apic1(config-leaf-if)# switchport trunk
allowed vlan 10 tenant exampleCorp
external-l2 epg extendBD1

Step 12 Assign a VLAN on the leaf port and map the Note The interface must be associated
VLAN to an external SVI with the switchport with a VLAN domain or this
{trunk allowed | trunk native | access} vlan command is rejected.
vlan-id tenant tenant-name external-svi
command.
Example:
apic1(config-leaf-if)# switchport trunk
allowed vlan 10 tenant exampleCorp
external-svi

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

77
 Configuring Layer 2 External Connectivity
Configuring Layer 2 External Connectivity

Examples
This example shows how to deploy a layer 2 port for external connectivity.

apic1# configure
apic1(config)# tenant exampleCorp
apic1(config-tenant)# external-l2 epg extendBD1
apic1(config-tenant-extl2epg)# bridge-domain member bd1
apic1(config-tenant-extl2epg)# exit
apic1(config-tenant)# exit

apic1(config)# leaf 101

apic1(config-leaf)# interface eth 1/2
apic1(config-leaf-if)# switchport
apic1(config-leaf-if)# switchport mode trunk
apic1(config-leaf-if)# switchport trunk allowed vlan 10 tenant exampleCorp external-l2 epg
extendBD1

This example shows how to deploy a layer 2 port channel or vPC for external connectivity.

...

apic1(config)# leaf 101

apic1(config-leaf)# interface port-channel po1
apic1(config-leaf-if)# switchport trunk allowed vlan 10 tenant exampleCorp external-l2 epg
extendBD1

These examples show how to configure SVI on a layer 2 interface for external connectivity.

apic1(config)# leaf 101

pic1(config-leaf)# interface ethernet 1/5

apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# switchport trunk allowed vlan 10 tenant exampleCorp external-svi
apic1(config-leaf-if)# no switchport trunk allowed vlan 10 tenant exampleCorp external-svi

apic1(config-leaf)# interface ethernet 1/37

apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# switchport access vlan 11 tenant exampleCorp external-svi
apic1(config-leaf-if)# no switchport access vlan 11 tenant exampleCorp external-svi

apic1(config-leaf)# interface port-channel po34

apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# switchport trunk native vlan 12 tenant exampleCorp external-svi
apic1(config-leaf-if)# no switchport trunk native vlan 12 tenant exampleCorp external-svi

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

78
 Configuring Layer 2 External Connectivity
Configuring VLAN Domains

Configuring VLAN Domains

About VLAN Domains
ACI fabric can be partitioned into groups of 4K VLANs to allow a large number of layer 2 domains across
the fabric, which can be used by multiple tenants. A VLAN domain represents a set of VLANs that can be
configured on group of nodes and ports. VLAN domains let multiple tenants share and independently manage
common fabric resources such as nodes, ports, and VLANs. A tenant can be provided access to one or more
VLAN domains. For more information about VLAN pools, see Endpoint Groups in the ACI Policy Model
chapter in Cisco Application Centric Infrastructure Fundamentals.
VLAN domains can be static or dynamic. Static VLAN domains support static VLAN pools, while dynamic
VLAN domains can support both static and dynamic VLAN pools. VLANs in static pools are managed by
the user and are used for applications such as connectivity to bare metal hosts. VLANs in the dynamic pool
are allocated and managed by the APIC without user intervention and are used for applications such as VMM.
The default type for VLAN domains and VLAN pools within the domain is static.
The fabric administrator performs the following steps before tenants can start using the fabric resources for
their L2/L3 configurations:
1. Create VLAN domains and assign VLANs in each VLAN domain.
2. Assign the external facing ports on the leaf switches to one or more VLAN domains.
3. Convert a port to L2/L3 by using the [no] switchport command. The default state of a port is
L2(switchport) in trunk mode.
4. For an L2 port, set the scope of a VLAN on a port to be global or local. The default is global.

The fabric administrator can update any configuration in these steps even after VLAN domains are assigned
to tenants and are in use by tenant applications.

A Note About Spanning Tree and VLAN Domains

Although the ACI fabric does not participate in spanning tree, it can partition a spanning tree domain based
on access policy configuration. ACI does not rely on a bridge domain or its settings to determine spanning
tree domains. Instead, leaf switches flood BPDUs within the same VLAN encapsulation, if a VLAN Pool is
assigned to EPG domains. The VLAN pool assigned to EPG domains ultimately serves as the spanning tree
domain.
Using multiple EPG domains tied to different VLAN Pools does not allow BPDUs to flood across endpoints
properly, even if they are all using the same VLAN ID. The type of EPG domain, (physical or Layer 2 external),
does not change this behavior.
Because the ACI Fabric floods all BPDUs from all devices within a spanning-tree domain, this may trigger
behaviors on external devices that are verifying BPDU info, such as the MAC address per interface. An
example of a feature that activates is "spanning-tree EtherChannel misconfig guard" found on IOS devices.
These features should be taken into account when utilizing ACI as a Layer 2 Tunnel.

Note Multiple Spanning Tree (MST) is not supported on interfaces configured with the Per Port VLAN feature
(configuring multiple EPGs on a leaf switch using the same VLAN ID with localPort scope).

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

79
 Configuring Layer 2 External Connectivity
Basic VLAN Domain Configuration

Basic VLAN Domain Configuration

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] vlan-domain domain-name [dynamic] Creates a VLAN domain or edits an existing
domain. Include the dynamic keyword to
Example:
create a dynamic VLAN pool. The default is
static.
apic1(config)# vlan-domain dom2 dynamic

Step 3 [no] vlan range [dynamic] Assigns a range or a comma-separated list of

VLANs to the VLAN domain.
Example:
apic1(config-vlan)# vlan 1000-1999,4001 A VLAN can be either static or dynamic. A
static VLAN is configured by the user, such as
for providing connectivity from a host to an
external switched network, while VLANs in the
dynamic range are configured internally by an
APIC application, such as a VMM or L4-L7
services. The default type is static.
Note A static domain cannot contain
dynamic VLANs.
A VLAN on a given port can map to
only one vlan-domain. This is
enforced during configuration.

Examples
This example shows how to configure basic VLAN domains.

apic1# configure
apic1(config)# vlan-domain dom1
apic1(config-vlan)# vlan 1000-1999,4001
apic1(config-vlan)# exit
apic1(config)# vlan-domain dom2 dynamic
apic1(config-vlan)# vlan 101-200
apic1(config-vlan)# vlan 301-400 dynamic

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

80
 Configuring Layer 2 External Connectivity
Advanced VLAN Domain Configuration

Advanced VLAN Domain Configuration

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] vlan-domain domain-name [dynamic] Creates a VLAN domain or edits an existing
[type {phys | l2ext | l3ext}] domain. Include the dynamic keyword to
create a dynamic VLAN pool. The default is
Example:
static.
apic1(config)# vlan-domain dom1 type phys The type option is visible and mandatory if
one or more of the following conditions exist:
• If all three vlan-domain types are not
present for this domain name
• If the three vlan-domain types have
different VLAN pools
• If the three vlan-domain types share the
same VLAN pool but if the pool name
differs from the vlan-domain name

Step 3 [no] vlan-pool vlan-pool-name Creates a VLAN pool. This command is

available only when the type option is present
Example:
in the vlan-domain command. You must
apic1(config-leaf)# vlan-pool myVlanPool3 declare the VLAN pool before adding VLANs
with the vlan command.

Step 4 [no] vlan range [dynamic] Assigns a range or a comma-separated list of

VLANs to the VLAN domain.
Example:
apic1(config-vlan-domain)# vlan A VLAN can be either static or dynamic. A
1000-1999,4001 static VLAN is configured by the user, such as
for providing connectivity from a host to an
external switched network, while VLANs in the
dynamic range are configured internally by an
APIC application, such as a VMM or L4-L7
services. The default type is static.
Note A static domain cannot contain
dynamic VLANs.
A VLAN on a given port can map to
only one vlan-domain. This is
enforced during configuration.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

81
 Configuring Layer 2 External Connectivity
Associating a VLAN Domain to a Port

Command or Action Purpose

Step 5 show vlan-domain [name domain-name] [vlan Displays vlan-domain usage for applications
vlan-id] [leaf leaf-id] such as App-EPG, sub-interface, external SVI,
and external-L2.
Example:
apic1(config-vlan-domain)# show
vlan-domain name dom1 vlan 1002 leaf 102

Examples
This example shows how to configure a VLAN domain with a VLAN pool.

apic1# configure
(config)# vlan-domain dom1 type phys
(config-vlan-domain)# vlan-pool myVlanPool3
(config-vlan-domain)# vlan 1000-1999, 4001

Associating a VLAN Domain to a Port

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 leaf node-id1-node-id2 Specifies the pair of leafs to be configured.

Example:
apic1(config)# leaf 101-102

Step 3 interface type Specifies a port or range of ports to be

associated with the VLAN domain.
Example:
apic1(config-leaf)# int eth 1/1-24

Step 4 [no] vlan-domain member domain-name Assigns the specified ports to the VLAN
domain.
Example:
apic1(config-leaf-if)# vlan-domain member
dom1

Step 5 [no] switchport By default, a port is in Layer 2 trunk mode. If

the port is in Layer 3 mode, it must be converted
Example:
to Layer 2 trunk mode using this command.
apic1(config-leaf-if)# switchport

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

82
 Configuring Layer 2 External Connectivity
Associating a VLAN Domain to a Port-Channel

Command or Action Purpose

Step 6 (Optional) [no] switchport vlan scope local By default, the scope of a VLAN is global to
the node. One VLAN can be mapped to only
Example:
one EPG in the node. When the VLAN scope
apic1(config-leaf-if)# switchport vlan is local to the port, the mapping from VLAN to
scope local
EPG can be different for different ports on the
same node.
To return the scope to global, use the no
command prefix.

Step 7 show vlan-domain [name domain-name] [vlan Displays vlan-domain usage for applications
vlan-id] [leaf leaf-id] such as App-EPG, external SVI, and
external-L2.
Example:
apic1(config-leaf-if)# show vlan-domain
name dom1 vlan 1002 leaf 102

Examples
This example shows how to associate a VLAN domain to ports.

apic1# configure
(config) # leaf 101-102
(config-leaf) # int eth 1/1-24
(config-leaf-if) # vlan-domain member dom1

(config-leaf) # int eth 1/1-12

(config-leaf-if) # no switchport
(config-leaf) # int eth 1/13-24
(config-leaf-if) # switchport

(config) # leaf 101-102

(config-leaf) # int eth 1/1-12
(config-leaf-if) # switchport vlan scope local

(config-leaf) # int eth 1/13

(config-leaf-if) # no switchport vlan scope local

Associating a VLAN Domain to a Port-Channel

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

83
 Configuring Layer 2 External Connectivity
Associating a VLAN Domain to a Template Policy-Group

Command or Action Purpose

Step 2 leaf node-id1-node-id2 Specifies the pair of leafs to be configured.
Example:
apic1(config)# leaf 101-102

Step 3 interface port-channel port-channel-name Specifies a port-channel to be associated with

the VLAN domain.
Example:
apic1(config-leaf)# int port-channel pc1

Step 4 [no] vlan-domain member domain-name Assigns the specified port-channel to the VLAN
domain.
Example:
apic1(config-leaf-if)# vlan-domain member
dom1

Examples

apic1# configure
apic1(config)# leaf 101-102
apic1(config-leaf)# int port-channel pc1
apic1(config-leaf-if)# vlan-domain member dom1

Associating a VLAN Domain to a Template Policy-Group

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 template policy-group policy-group-name Specifies the template policy-group to be

configured.
Example:
apic1(config)# template policy-group
myPolGp5

Step 3 [no] vlan-domain member domain-name Assigns the specified template policy-group to
the VLAN domain.
Example:
apic1(config-pol-grp-if)# vlan-domain
member dom1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

84
 Configuring Layer 2 External Connectivity
Associating a VLAN Domain to a Template Port-Channel

Examples

apic1# configure
apic1(config)# template policy-group myPolGp5
apic1(config-pol-grp-if)# vlan-domain member dom1

Associating a VLAN Domain to a Template Port-Channel

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 template port-channel policy-group-name Specifies the template port-channel to be

configured.
Example:
apic1(config)# template port-channel
myPC7

Step 3 [no] vlan-domain member domain-name Assigns the specified template port-channel to
the VLAN domain.
Example:
apic1(config-if)# vlan-domain member dom1

Examples

apic1# configure
apic1(config)# template port-channel myPC7
apic1(config-po-ch-if)# vlan-domain member dom1

Associating a VLAN Domain to a Virtual Port-Channel

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

85
 Configuring Layer 2 External Connectivity
Configuring Q-in-Q Encapsulation Mapping for EPGs

Command or Action Purpose

Step 2 vpc context leaf node-id1 node-id2 [fex fex-id1 Specifies the VPC and leafs to be configured.
fex-id2]
Example:
apic1(config)# vpc context leaf 101 102

Step 3 interface vpc vpc-name [fex fex-id1 fex-id2] Specifies a port-channel to be associated with
the VLAN domain.
Example:
apic1(config-vpc)# int vpc vpc1

Step 4 [no] vlan-domain member domain-name Assigns the specified VPC to the VLAN
domain.
Example:
apic1(config-vpc-if)# vlan-domain member
dom1

Examples

apic1# configure
apic1(config)# vpc context leaf 101 102
apic1(config-vpc)# int vpc vpc1
apic1(config-vpc-if)# vlan-domain member dom1

Configuring Q-in-Q Encapsulation Mapping for EPGs

Q-in-Q Encapsulation Mapping for EPGs
Using Cisco APIC, you can map double-tagged VLAN traffic ingressing on a regular interface, PC, or vPC
to an EPG. When this feature is enabled, when double-tagged traffic enters the network for an EPG, both tags
are processed individually in the fabric and restored to double-tags when egressing the ACI switch. Ingressing
single-tagged and untagged traffic is dropped.
This feature is only supported on Nexus 9300-FX platform switches.
Both the outer and inner tag must be of EtherType 0x8100.
MAC learning and routing are based on the EPG port, sclass, and VRF, not on the access encapsulations.
QoS priority settings are supported, derived from the outer tag on ingress, and rewritten to both tags on egress.
EPGs can simultaneously be associated with other interfaces on a leaf switch, that are configured for
single-tagged VLANs.
Service graphs are supported for provider and consumer EPGs that are mapped to Q-in-Q encapsulated
interfaces. You can insert service graphs, as long as the ingress and egress traffic on the service nodes is in
single-tagged encapsulated frames.
The following features and options are not supported with this feature:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

86
 Configuring Layer 2 External Connectivity
Mapping EPGs to Q-in-Q Encapsulated Leaf Interfaces Using the NX-OS Style CLI

• Per-Port VLAN feature

• FEX connections
• Mixed Mode is not supported. For example, an interface in Q-in-Q encapsulation mode can have a static
path binding to an EPG with double-tagged encapsulation only, not with regular VLAN encapsulation.
• STP and the “Flood in Encapsulation” option
• Untagged and 802.1p mode
• Multi-pod and Multi-Site
• Legacy bridge domain
• L2Out and L3Out connections
• VMM integration
• Changing a port mode from routed to Q-in-Q encapsulation mode is not supported
• Per-vlan MCP is not supported between ports in Q-in-Q encapsulation mode and ports in regular trunk
mode.
• When vPC ports are enabled for Q-in-Q encapsulation mode, VLAN consistency checks are not performed.

Mapping EPGs to Q-in-Q Encapsulated Leaf Interfaces Using the NX-OS Style
CLI
Enable an interface for Q-in-Q encapsulation and associate the interface with an EPG.

Before you begin

Create the tenant, application profile, and application EPG that will be mapped with an interface configured
for Q-in-Q mode.

Procedure

Command or Action Purpose

Step 1 Configure Enters global configuration mode.
Example:
apic1# configure

Step 2 leaf number Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101
Step 3 interface ethernet slot/port Specifies the interface to be configured.
Example:
apic1 (config-leaf)# interface ethernet
1/25

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

87
 Configuring Layer 2 External Connectivity
Support Fibre Channel over Ethernet Traffic on the ACI Fabric

Command or Action Purpose

Step 4 switchport mode dot1q-tunnel doubleQtagPort Enables an interface for Q-in-Q encapsulation.
Example:
apic1(config-leaf-if)# switchport mode
dot1q-tunnel doubleQtagPort

Step 5 switchport trunk qinq outer-vlan Associates the interface with an EPG.
vlan-number inner-vlan vlan-number tenant
tenant-name application application-name epg
epg-name
Example:
apic1(config-leaf-if)# switchport trunk
qinq outer-vlan 202 inner-vlan 203
tenant tenant64 application AP64 epg
EPG64

Example
The following example enables Q-in-Q encapsulation (with outer-VLAN ID 202 and inner-VLAN
ID 203) on the leaf interface 101/1/25, and associates the interface with EPG64.
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/25
apic1(config-leaf-if)#switchport mode dot1q-tunnel doubleQtagPort
apic1(config-leaf-if)# switchport trunk qinq outer-vlan 202 inner-vlan 203 tenant tenant64
application AP64 epg EPG64

Support Fibre Channel over Ethernet Traffic on the ACI Fabric

Supporting Fibre Channel over Ethernet Traffic on the ACI Fabric
Cisco ACI enables you to configure and manage support for Fibre Channel over Ethernet (FCoE) traffic on
the ACI fabric.
FCoE is a protocol that encapsulates Fibre Channel (FC) packets within Ethernet packets, thus enabling storage
traffic to move seamlessly between a Fibre Channel SAN and an Ethernet network.
A typical implementation of FCoE protocol support on the ACI fabric enables hosts located on the
Ethernet-based ACI fabric to communicate with SAN storage devices located on an FC network. The hosts
are connecting through virtual F ports deployed on an ACI leaf switch. The SAN storage devices and FC
network are connected through a Fibre Channel Forwarding (FCF) bridge to the ACI fabric through a virtual
NP port, deployed on the same ACI leaf switch as is the virtual F port. Virtual NP ports and virtual F ports
are also referred to generically as virtual Fibre Channel (vFC) ports.

Note In the FCoE topology, the role of the ACI leaf switch is to provide a path for FCoE traffic between the locally
connected SAN hosts and a locally connected FCF device. The leaf switch does not perform local switching
between SAN hosts, and the FCoE traffic is not forwarded to a spine switch.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

88
Configuring Layer 2 External Connectivity
Supporting Fibre Channel over Ethernet Traffic on the ACI Fabric

Topology Supporting FCoE Traffic Through ACI

The topology of a typical configuration supporting FCoE traffic over the ACI fabric consists of the following
components:
Figure 5: ACI Topology Supporting FCoE Traffic

• One or more ACI leaf switches configured through FC SAN policies to function as an NPV backbone.
• Selected interfaces on the NPV-configured leaf switches configured to function as virtual F ports, which
accommodate FCoE traffic to and from hosts running SAN management or SAN-consuming applications.
• Selected interfaces on the NPV-configured leaf switches configured to function as virtual NP ports, which
accommodate FCoE traffic to and from a Fibre Channel Forwarding (FCF) bridge.

The FCF bridge receives FC traffic from fibre channel links typically connecting SAN storage devices and
encapsulates the FC packets into FCoE frames for transmission over the ACI fabric to the SAN management
or SAN Data-consuming hosts. It receives FCoE traffic and repackages it back to FC for transmission over
the fibre channel network.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

89
 Configuring Layer 2 External Connectivity
Supporting Fibre Channel over Ethernet Traffic on the ACI Fabric

Note In the above ACI topology, FCoE traffic support requires direct connections between the hosts and virtual F
ports and direct connections between the FCF device and the virtual NP port.

APIC servers enable an operator to configure and monitor the FCoE traffic through the APIC GUI, the APIC
NX-OS style CLI, or through application calls to the APIC REST API.

Topology Supporting FCoE Initialization

In order for FCoE traffic flow to take place as described, you must also set up separate VLAN connectivity
over which SAN Hosts broadcast FCoE Initialization protocol (FIP) packets to discover the interfaces enabled
as F ports.

vFC Interface Configuration Rules

Whether you set up the vFC network and EPG deployment through the APIC GUI, NX-OS style CLI, or the
REST API, the following general rules apply across platforms:
• F port mode is the default mode for vFC ports. NP port mode must be specifically configured in the
Interface policies.
• The load balancing default mode is for leaf-switch or interface level vFC configuration is src-dst-ox-id.
• One VSAN assignment per bridge domain is supported.
• The allocation mode for VSAN pools and VLAN pools must always be static.
• vFC ports require association with a VSAN domain (also called Fibre Channel domain) that contains
VSANs mapped to VLANs.

FCoE Guidelines and Limitations

FCoE is supported on the following switches:
• N9K-C93180LC-EX (When 40 Gigabit Ethernet (GE) ports are enabled as FCoE F or NP ports, they
cannot be enabled for 40GE port breakout. FCoE is not supported on breakout ports.)
• N9K-C93108TC-EX
• N9K-C93180YC-EX
• N9K-C93180LC-EX (FCoE support on FEX ports)
• N9K-C93180YC-FX (FCoE support on FEX ports, 40G ports (1/49-54), and 4x10G breakout ports)

FCoE is supported on the following Nexus FEX devices:

• N2K-C2348UPQ-10GE
• N2K-C2348TQ-10GE
• N2K-C2232PP-10GE
• N2K-B22DELL-P
• N2K-B22HP-P

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

90
 Configuring Layer 2 External Connectivity
FCoE NX-OS Style CLI Configuration

• N2K-B22IBM-P
• N2K-B22DELL-P-FI

The vlan used for FCoE should have vlanScope set to Global. vlanScope set to portLocal is not supported for
FCoE. The value is set via the L2 Interface Policy l2IfPol.

FCoE NX-OS Style CLI Configuration

Configuring FCoE Connectivity Without Policies or Profiles Using the NX-OS

Style CLI
The following sample NX-OS style CLI sequences configure FCoE connectivity for EPG e1 under tenant t1
without configuring or applying switch-level and interface-level policies and profiles.

Procedure

Command or Action Purpose

Step 1 Under the target tenant configure a bridge The sample command sequence creates bridge
domain to support FCoE traffic. domain b1 under tenant t1 configured to support
FCoE connectivity.
Example:
apic1(config)# tenant t1
apic1(config-tenant)# vrf context v1
apic1(config-tenant-vrf)# exit
apic1(config-tenant)# bridge-domain b1
apic1(config-tenant-bd)# fc
apic1(config-tenant-bd)# vrf member v1
apic1(config-tenant-bd)# exit
apic1(config-tenant)# exit

Step 2 Under the same tenant, associate the target EPG The sample command sequence creates EPG
with the FCoE-configured bridge domain. e1 and associates that EPG with the
FCoE-configured bridge domain b1.
Example:
apic1(config)# tenant t1
apic1(config-tenant)# application a1
apic1(config-tenant-app)# epg e1
apic1(config-tenant-app-epg)#
bridge-domain member b1
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)# exit
apic1(config-tenant)# exit

Step 3 Create a VSAN domain, VSAN pools, VLAN In Example A, the sample command sequence
pools and VSAN to VLAN mapping. creates VSAN domain, dom1 with VSAN pools
and VLAN pools, maps VSAN 1 to VLAN 1
Example:
and maps VSAN 2 to VLAN 2
A
In Example B, an alternate sample command
apic1(config)# vsan-domain dom1
sequence creates a reusable VSAN attribute
apic1(config-vsan)# vsan 1-10
apic1(config-vsan)# vlan 1-10 template pol1 and then creates VSAN domain

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

91
 Configuring Layer 2 External Connectivity
Configuring FCoE Connectivity Without Policies or Profiles Using the NX-OS Style CLI

Command or Action Purpose

apic1(config-vsan)# fcoe vsan 1 vlan 1 dom1, which inherits the attributes and
loadbalancing src-dst-ox-id
mappings from that template.
apic1(config-vsan)# fcoe vsan 2 vlan 2

Example:
B
apic1(config)# template vsan-attribute
pol1
apic1(config-vsan-attr)# fcoe vsan 2
vlan 12 loadbalancing src-dst-ox-id
apic1(config-vsan-attr)# fcoe vsan 3
vlan 13 loadbalancing src-dst-ox-id
apic1(config-vsan-attr)# exit
apic1(config)# vsan-domain dom1
apic1(config-vsan)# vsan 1-10
apic1(config-vsan)# vlan 1-10
apic1(config-vsan)# inherit
vsan-attribute pol1
apic1(config-vsan)# exit

Step 4 Create the physical domain to support the FCoE In the example, the command sequence creates
Initialization (FIP) process. a regular VLAN domain, fipVlanDom, which
includes VLAN 120 to support the FIP process.
Example:

apic1(config)# vlan-domain fipVlanDom

apic1(config-vlan)# vlan 120
apic1(config-vlan)# exit

Step 5 Under the target tenant configure a regular In the example, the command sequence creates
bridge domain. bridge domain fip-bd.
Example:
apic1(config)# tenant t1
apic1(config-tenant)# vrf context v2
apic1(config-tenant-vrf)# exit
apic1(config-tenant)# bridge-domain
fip-bd
apic1(config-tenant-bd)# vrf member v2
apic1(config-tenant-bd)# exit
apic1(config-tenant)# exit

Step 6 Under the same tenant, associate this EPG with In the example, the command sequence
the configured regular bridge domain. associates EPG epg-fip with bridge domain
fip-bd.
Example:
apic1(config)# tenant t1
apic1(config-tenant)# application a1
apic1(config-tenant-app)# epg epg-fip
apic1(config-tenant-app-epg)#
bridge-domain member fip-bd
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)# exit
apic1(config-tenant)# exit

Step 7 Configure a VFC interface with F mode. In example A the command sequence enables
interface 1/2 on leaf switch 101 to function as
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

92
Configuring Layer 2 External Connectivity
Configuring FCoE Connectivity Without Policies or Profiles Using the NX-OS Style CLI

Command or Action Purpose

A an F port and associates that interface with
apic1(config)# leaf 101
VSAN domain dom1.
apic1(config-leaf)# interface ethernet
Each of the targeted interfaces must be assigned
1/2
apic1(config-leaf-if)# vlan-domain member one (and only one) VSAN in native mode. Each
fipVlanDom interface may be assigned one or more
apic1(config-leaf-if)# switchport trunk additional VSANs in regular mode.
native vlan 120 tenant t1 application
a1 epg epg-fip The sample command sequence associates the
apic1(config-leaf-if)# exit target interface 1/2 with:
apic1(config-leaf)# exit • VLAN 120 for FIP discovery and
apic1(config-leaf)# interface vfc 1/2
associates it with EPG epg-fip and
apic1(config-leaf-if)# switchport mode
f application a1 under tenant t1.
apic1(config-leaf-if)# vsan-domain member
dom1 • VSAN 2 as a native VSAN and associates
apic1(config-leaf-if)# switchport vsan it with EPG e1 and application a1 under
2 tenant t1 application a1 epg e1 tenant t1.
apic1(config-leaf-if)# switchport trunk
allowed vsan 3 tenant t1 application a1 • VSAN 3 as a regular VSAN.
epg e2
apic1(config-leaf-if)# exit
In example B, the command sequence
Example: configures a vFC over a vPC with the same
B VSAN on both the legs. From the CLI you
cannot specify different VSANs on each log.
apic1(config)# vpc context leaf 101 102
apic1(config-vpc)# interface vpc vpc1 The alternate configuration can be carried out
apic1(config-vpc-if)# vlan-domain member in the APIC advanced GUI.
vfdom100
apic1(config-vpc-if)# vsan-domain member
dom1
apic1(config-vpc-if)# #For FIP discovery
apic1(config-vpc-if)# switchport trunk
native vlan 120 tenant t1 application a1
epg epg-fip
apic1(config-vpc-if)# switchport vsan 2
tenant t1 application a1 epg e1
apic1(config-vpc-if)# exit
apic1(config-vpc)# exit
apic1(config)# leaf 101-102
apic1(config-leaf)# interface ethernet
1/3
apic1(config-leaf-if)# channel-group vpc1
vpc
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit

Example:
C
apic1(config)# leaf 101
apic1(config-leaf)# interface vfc-po pc1
apic1(config-leaf-if)# vsan-domain member
dom1
apic1(config-leaf-if)# switchport vsan
2 tenant t1 application a1 epg e1
apic1(config-leaf-if)# exit
apic1(config-leaf)# interface ethernet
1/2

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

93
 Configuring Layer 2 External Connectivity
Configuring FCoE Connectivity With Policies and Profiles Using the NX-OS Style CLI

Command or Action Purpose

apic1(config-leaf-if)# channel-group pc1
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit

Step 8 Configure a VFC interface with NP mode. The sample command sequence enables
interface 1/4 on leaf switch 101 to function as
Example:
an NP port and associates that interface with
apic1(config)# leaf 101
VSAN domain dom1.
apic1(config-leaf)# interface vfc 1/4
apic1(config-leaf-if)# switchport mode
np
apic1(config-leaf-if)# vsan-domain member
dom1

Step 9 Assign the targeted FCoE-enabled interfaces a Each of the targeted interfaces must be assigned
VSAN. one (and only one) VSAN in native mode. Each
interface may be assigned one or more
Example:
additional VSANs in regular mode.
apic1(config-leaf-if)# switchport trunk
allowed vsan 1 tenant t1 application a1 The sample command sequence assigns the
epg e1 target interface to VSAN 1 and associates it
apic1(config-leaf-if)# switchport vsan
2 tenant t4 application a4 epg e4
with EPG e1 and application a1 under tenant
t1. "trunk allowed" assigns vsan 1 regular mode
status. The command sequence also assigns the
interface a required native mode VSAN 2. As
this example shows, it is permissible for
different VSANs to provide different EPGs
running under different tenants access to the
same interfaces.

Configuring FCoE Connectivity With Policies and Profiles Using the NX-OS
Style CLI
The following sample NX-OS style CLI sequences create and use policies to configure FCoE connectivity
for EPG e1 under tenant t1.

Procedure

Command or Action Purpose

Step 1 Under the target tenant configure a bridge The sample command sequence creates bridge
domain to support FCoE traffic. domain b1 under tenant t1 configured to
support FCoE connectivity.
Example:
apic1# configure
apic1(config)# tenant t1
apic1(config-tenant)# vrf context v1
apic1(config-tenant-vrf)# exit
apic1(config-tenant)# bridge-domain b1

apic1(config-tenant-bd)# fc
apic1(config-tenant-bd)# vrf member v1
apic1(config-tenant-bd)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

94
Configuring Layer 2 External Connectivity
Configuring FCoE Connectivity With Policies and Profiles Using the NX-OS Style CLI

Command or Action Purpose

apic1(config-tenant)# exit
apic1(config)#

Step 2 Under the same tenant, associate your target The sample command sequence creates EPG
EPG with the FCoE configured bridge domain. e1 associates that EPG with FCoE-configured
bridge domain b1.
Example:
apic1(config)# tenant t1
apic1(config-tenant)# application a1
apic1(config-tenant-app)# epg e1
apic1(config-tenant-app-epg)#
bridge-domain member b1
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)# exit
apic1(config-tenant)# exit
apic1(config)#

Step 3 Create a VSAN domain, VSAN pools, VLAN In Example A, the sample command sequence
pools and VSAN to VLAN mapping. creates VSAN domain, dom1 with VSAN
pools and VLAN pools, maps VSAN 1 VLAN
Example:
1 and maps VSAN 2 to VLAN 2
A
In Example B, an alternate sample command
apic1(config)# vsan-domain dom1
sequence creates a reusable vsan attribute
apic1(config-vsan)# vsan 1-10
apic1(config-vsan)# vlan 1-10 template pol1 and then creates VSAN domain
apic1(config-vsan)# fcoe vsan 1 vlan 1 dom1, which inherits the attributes and
loadbalancing mappings from that template.
src-dst-ox-id
apic1(config-vsan)# fcoe vsan 2 vlan 2

Example:
B
apic1(config)# template vsan-attribute
pol1
apic1(config-vsan-attr)# fcoe vsan 2
vlan 12
loadbalancing
src-dst-ox-id
apic1(config-vsan-attr)# fcoe vsan 3
vlan 13
loadbalancing
src-dst-ox-id
apic1(config-vsan-attr)# exit
apic1(config)# vsan-domain dom1
apic1(config-vsan)# inherit
vsan-attribute pol1
apic1(config-vsan)# exit

Step 4 Create the physical domain to support the

FCoE Initialization (FIP) process.
Example:
apic1(config)# vlan-domain fipVlanDom
apic1(config)# vlan-pool fipVlanPool

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

95
 Configuring Layer 2 External Connectivity
Configuring FCoE Connectivity With Policies and Profiles Using the NX-OS Style CLI

Command or Action Purpose

Step 5 Configure a Fibre Channel SAN policy. The sample command sequence creates Fibre
Channel SAN policy ffp1 to specify a
Example:
combination of error-detect timeout values
apic1# (EDTOV), resource allocation timeout values
apic1# configure
apic1(config)# template fc-fabric-policy (RATOV), and the default FC map values for
ffp1 FCoE-enabled interfaces on a target leaf
apic1(config-fc-fabric-policy)# fctimer switch.
e-d-tov 1111
apic1(config-fc-fabric-policy)# fctimer
r-a-tov 2222
apic1(config-fc-fabric-policy)# fcoe
fcmap 0E:FC:01
apic1(config-fc-fabric-policy)# exit

Step 6 Create a Fibre Channel node policy. The sample command sequence creates Fibre
Channel node policy flp1 to specify a
Example:
combination of disruptive load-balancing
apic1(config)# template fc-leaf-policy enablement and FIP keep-alive values. These
flp1
apic1(config-fc-leaf-policy)# fcoe values also apply to all the FCoE-enabled
fka-adv-period 44 interfaces on a target leaf switch.
apic1(config-fc-leaf-policy)# exit

Step 7 Create Node Policy Group. The sample command sequence creates a Node
Policy group, lpg1, which combines the values
Example:
of the Fibre Channel SAN policy ffp1 and
apic1(config)# template
Fibre Channel node policy, flp1. The combined
leaf-policy-group lpg1
apic1(config-leaf-policy-group)# inherit values of this node policy group can be applied
fc-fabric-policy ffp1 to Node profiles configured later.
apic1(config-leaf-policy-group)# inherit
fc-leaf-policy flp1
apic1(config-leaf-policy-group)# exit
apic1(config)# exit
apic1#

Step 8 Create a Node Profile. The sample command sequence creates node
profile lp1 associates it with node policy group
Example:
lpg1, node group lg1, and leaf switch 101.
apic1(config)# leaf-profile lp1
apic1(config-leaf-profile)# leaf-group
lg1
apic1(config-leaf-group)# leaf 101
apic1(config-leaf-group)#
leaf-policy-group lpg1

Step 9 Create an interface policy group for F port The sample command sequence creates
interfaces. interface policy group ipg1 and assigns a
combination of values that determine priority
Example:
flow control enablement, F port enablement,
apic1(config)# template policy-group and slow-drain policy values for any interface
ipg1
apic1(config-pol-grp-if)# that this policy group is applied to.
priority-flow-control mode auto
apic1(config-pol-grp-if)# switchport
mode f
apic1(config-pol-grp-if)# slow-drain
pause timeout 111

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

96
Configuring Layer 2 External Connectivity
Configuring FCoE Connectivity With Policies and Profiles Using the NX-OS Style CLI

Command or Action Purpose

apic1(config-pol-grp-if)# slow-drain
congestion-timeout count 55
apic1(config-pol-grp-if)# slow-drain
congestion-timeout action log

Step 10 Create an interface policy group for NP port The sample command sequence creates
interfaces. interface policy group ipg2 and assigns a
combination of values that determine priority
Example:
flow control enablement, NP port enablement,
apic1(config)# template policy-group and slow-drain policy values for any interface
ipg2
apic1(config-pol-grp-if)# that this policy group is applied to.
priority-flow-control mode auto
apic1(config-pol-grp-if)# switchport
mode np
apic1(config-pol-grp-if)# slow-drain
pause timeout 111
apic1(config-pol-grp-if)# slow-drain
congestion-timeout count 55
apic1(config-pol-grp-if)# slow-drain
congestion-timeout action log

Step 11 Create an interface profile for F port interfaces. The sample command sequence creates an
interface profile lip1 for F port interfaces,
Example:
associates the profile with F port specific
apic1# configure interface policy group ipg1, and specifies the
apic1(config)# leaf-interface-profile
lip1 interfaces to which this profile and its
apic1(config-leaf-if-profile)# associated policies applies.
description 'test description lip1'
apic1(config-leaf-if-profile)#
leaf-interface-group lig1
apic1(config-leaf-if-group)# description
'test description lig1'
apic1(config-leaf-if-group)#
policy-group ipg1
apic1(config-leaf-if-group)# interface
ethernet 1/2-6, 1/9-13

Step 12 Create an interface profile for NP port The sample command sequence creates an
interfaces. interface profile lip2 for NP port interfaces,
associates the profile with NP port specific
Example:
interface policy group ipg2, and specifies the
apic1# configure
interface to which this profile and its associated
apic1(config)#
leaf-interface-profile lip2 policies applies.
apic1(config-leaf-if-profile)#
description 'test description lip2'
apic1(config-leaf-if-profile)#
leaf-interface-group lig2
apic1(config-leaf-if-group)#
description 'test description lig2'
apic1(config-leaf-if-group)#
policy-group ipg2
apic1(config-leaf-if-group)# interface
ethernet 1/14

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

97
 Configuring Layer 2 External Connectivity
Configuring FCoE Over FEX Using NX-OS Style CLI

Command or Action Purpose

Step 13 Configure QoS Class Policy for Level 1. The sample command sequence specifies the
QoS level of FCoE traffic to which priority
Example:
flow control policy might be applied and
apic1(config)# qos parameters level1 pauses no-drop packet handling for Class of
apic1(config-qos)# pause no-drop cos 3
Service level 3.

Configuring FCoE Over FEX Using NX-OS Style CLI

FEX ports are configured as port VSANs.

Procedure

Step 1 Configure Tenant and VSAN domain:

Example:
apic1# configure
apic1(config)# tenant t1
apic1(config-tenant)# vrf context v1
apic1(config-tenant-vrf)# exit
apic1(config-tenant)# bridge-domain b1
apic1(config-tenant-bd)# fc
apic1(config-tenant-bd)# vrf member v1
apic1(config-tenant-bd)# exit
apic1(config-tenant)# application a1
apic1(config-tenant-app)# epg e1
apic1(config-tenant-app-epg)# bridge-domain member b1
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)# exit
apic1(config-tenant)# exit

apic1(config)# vsan-domain dom1

apic1(config-vsan)# vlan 1-100
apic1(config-vsan)# vsan 1-100
apic1(config-vsan)# fcoe vsan 2 vlan 2 loadbalancing src-dst-ox-id
apic1(config-vsan)# fcoe vsan 3 vlan 3 loadbalancing src-dst-ox-id
apic1(config-vsan)# fcoe vsan 5 vlan 5
apic1(config-vsan)# exit

Step 2 Associate FEX to an interface:

Example:
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/12
apic1(config-leaf-if)# fex associate 111
apic1(config-leaf-if)# exit

Step 3 Configure FCoE over FEX per port, port-channel, and VPC:
Example:
apic1(config-leaf)# interface vfc 111/1/2
apic1(config-leaf-if)# vsan-domain member dom1
apic1(config-leaf-if)# switchport vsan 2 tenant t1 application a1 epg e1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

98
Configuring Layer 2 External Connectivity
Configuring FCoE Over FEX Using NX-OS Style CLI

apic1(config-leaf-if)# exit
apic1(config-leaf)# interface vfc-po pc1 fex 111
apic1(config-leaf-if)# vsan-domain member dom1
apic1(config-leaf-if)# switchport vsan 2 tenant t1 application a1 epg e1
apic1(config-leaf-if)# exit
apic1(config-leaf)# interface ethernet 111/1/3
apic1(config-leaf-if)# channel-group pc1
apic1(config-leaf-if# exit
apic1(config-leaf)# exit
apic1(config)# vpc domain explicit 12 leaf 101 102
apic1(config-vpc)# exit
apic1(config)# vpc context leaf 101 102
apic1(config-vpc)# interface vpc vpc1 fex 111 111
apic1(config-vpc-if)# vsan-domain member dom1
apic1(config-vpc-if)# switchport vsan 2 tenant t1 application a1 epg e1
apic1(config-vpc-if)# exit
apic1(config-vpc)# exit
apic1(config)# leaf 101-102
apic1(config-leaf)# interface ethernet 1/2
apic1(config-leaf-if)# fex associate 111
apic1(config-leaf-if)# exit
apic1(config-leaf)# interface ethernet 111/1/2
apic1(config-leaf-if)# channel-group vpc1 vpc
apic1(config-leaf-if)# exit

Step 4 Verify the configuration with the following command:

Example:
apic1(config-vpc)# show vsan-domain detail
vsan-domain : dom1

vsan : 1-100

vlan : 1-100

Leaf Interface Vsan Vlan Vsan-Mode Port-Mode Usage

Operational State
------------ ---------------- ---- ---- ----------- ---------
--------------------------------------
101 vfc111/1/2 2 2 Native Tenant: t1
Deployed
App: a1
Epg: e1

101 PC:pc1 5 5 Native Tenant: t1

Deployed
App: a1
Epg: e1

101 vfc111/1/3 3 3 Native F Tenant: t1

Deployed
App: a1
Epg: e1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

99
 Configuring Layer 2 External Connectivity
Verifying FCoE Configuration Using the NX-OS Style CLI

Verifying FCoE Configuration Using the NX-OS Style CLI

The following show command verifies the FCoE configuration on your leaf switch ports.

Procedure

Use the show vsan-domain command to verify FCoE is enabled on the target switch.
The command example confirms FCoE enabled on the listed leaf switches and its FCF connection details.
Example:

ifav-isim8-ifc1# show vsan-domain detail

vsan-domain : iPostfcoeDomP1

vsan : 1-20 51-52 100-102 104-110 200 1999 3100-3101 3133

2000

vlan : 1-20 51-52 100-102 104-110 200 1999 3100-3101 3133

2000

Vsan Port Operational

Leaf Interface Vsan Vlan Mode Mode Usage State
---- --------- ---- ---- ------- ---- ---------------- ------------
101 vfc1/11 1 1 Regular F Tenant: iPost101 Deployed

App: iPost1

Epg: iPost1

101 vfc1/12 1 1 Regular NP Tenant: iPost101 Deployed

App: iPost1

Epg: iPost1

101 PC:infraAccBndl 4 4 Regular NP Tenant: iPost101 Deployed

Grp_pc01 App: iPost4

Epg: iPost4

101 vfc1/30 2000 Native Tenant: t1 Not deployed

App: a1 (invalid-path)

Epg: e1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

100
 Configuring Layer 2 External Connectivity
Undeploying FCoE Elements Using the NX-OS Style CLI

Undeploying FCoE Elements Using the NX-OS Style CLI

Any move to undeploy FCoE connectivity from the ACI fabric requires that you remove the FCoE components
on several levels.

Procedure

Step 1 List the attributes of the leaf port interface, set its mode setting to default, and then remove its EPG deployment
and domain association.
The example sets the port mode setting of interface vfc 1/2 to default and then removes the deployment of
EPG e1 and the association with VSAN Domain dom1 from that interface.
Example:

apic1(config)# leaf 101

apic1(config-leaf)# interface vfc 1/2
apic1(config-leaf-if)# show run
# Command: show running-config leaf 101 interface vfc 1 / 2
# Time: Tue Jul 26 09:41:11 2016
leaf 101
interface vfc 1/2
vsan-domain member dom1
switchport vsan 2 tenant t1 application a1 epg e1
exit
exit
apic1(config-leaf-if)# no switchport mode
apic1(config-leaf-if)# no switchport vsan 2 tenant t1 application a1 epg e1
apic1(config-leaf-if)# no vsan-domain member dom1
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit

Step 2 List and remove the VSAN/VLAN mapping and the VLAN and VSAN pools.
The example removes the VSAN/VLAN mapping for vsan 2, VLAN pool 1-10, and VSAN pool 1-10 from
VSAN domain dom1.
Example:
apic1(config)# vsan-domain dom1
apic1(config-vsan)# show run
# Command: show running-config vsan-domain dom1
# Time: Tue Jul 26 09:43:47 2016
vsan-domain dom1
vsan 1-10
vlan 1-10
fcoe vsan 2 vlan 2
exit
apic1(config-vsan)# no fcoe vsan 2
apic1(config-vsan)# no vlan 1-10
apic1(config-vsan)# no vsan 1-10
apic1(config-vsan)# exit

#################################################################################
NOTE: To remove a template-based VSAN to VLAN mapping use an alternate sequence:
#################################################################################

apic1(config)# template vsan-attribute <template_name>

apic1(config-vsan-attr)# no fcoe vsan 2

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

101
 Configuring Layer 2 External Connectivity
Fibre Channel NPV

Step 3 Delete the VSAN Domain.

The example deletes VSAN domain dom1.
Example:

apic1(config)# no vsan-domain dom1

Step 4 You can delete the associated tenant, EPG, and selectors if you do not need them.

Fibre Channel NPV

Fibre Channel Connectivity Overview
A switch is in NPV mode after enabling NPV. NPV mode applies to an entire switch. All end devices connected
to a switch that are in NPV mode must log in as an N port to use this feature (loop-attached devices are not
supported). All links from the edge switches (in NPV mode) to the NPV core switches are established as NP
ports (not E ports), which are used for typical inter-switch links.

FC NPV Benefits
FC NPV provides the following:
• Increased number of hosts that connect to the fabric without adding domain IDs in the fabric
• Connection of FC and FCoE hosts and targets to SAN fabrics using FC interfaces
• Automatic traffic mapping
• Static traffic mapping
• Disruptive automatic load balancing

FC NPV Mode
Feature-set fcoe-npv in ACI will be enabled automatically by default when first FCoE/FC configuration is
pushed.

FC Topology
The topology of a typical configuration supporting FC traffic over the ACI fabric consists of the following
components:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

102
 Configuring Layer 2 External Connectivity
Fibre Channel N-Port Virtualization Guidelines and Limitations

• A Leaf can be connected to a FC switch by using FCoE NP port or native FC NP port.

• An ACI Leaf can be directly connected with a server/Storage using FCoE links.
• FC/FCoE traffic is not sent to fabric/spine. A Leaf switch does not do local switching for FCoE traffic.
The switching is done by a core switch which is connected with a leaf switch via FC/FCoE NPV link.
• Multiple FDISC followed by Flogi is supported with FCoE host and FC/FCoE NP links.

Fibre Channel N-Port Virtualization Guidelines and Limitations

When configuring Fibre Channel N-Port Virtualization (NPV), note the following guidelines and limitations:
• Fibre Channel NP ports support trunk mode, but Fibre Channel F ports do not.
• On a trunk Fibre Channel port, internal login happens on the highest VSAN.
• On the core switch, the following features must be enabled:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

103
 Configuring Layer 2 External Connectivity
Configuring FC Connectivity Without Policies or Profiles Using the NX-OS CLI

feature npiv
feature fport-channel-trunk

• To use an 8G uplink speed, you must configure the IDLE fill pattern on the core switch.

Note Following is an example of configuring IDLE fill pattern on a Cisco MDS switch:

Switch(config)# int fc2/3

Switch(config)# switchport fill-pattern IDLE speed 8000
Switch(config)# show run int fc2/3

interface fc2/3
switchport speed 8000
switchport mode NP
switchport fill-pattern IDLE speed 8000
no shutdown

• In the Cisco APIC 4.1(1) release and later, Fibre Channel NPV support is limited to the Cisco
N9K-C93180YC-FX switch.
• You can use ports 1 through 48 for Fibre Channel configuration. Ports 49 through 54 cannot be Fibre
Channel ports.
• If you convert a port from Ethernet to Fibre Channel or the other way around, you must reload the switch.
Currently, you can convert only one contiguous range of ports to Fibre Channel ports, and this range
must be a multiple of 4, ending with a port number that is a multiple of 4. For example, 1-4, 1-8, or 21-24.
• Fibre Channel Uplink (NP) connectivity to Brocade Port Blade Fibre Channel 16-32 is not supported
when a Cisco N9K-93180YC-FX leaf switch port is configured in 8G speed.
• The selected port speed must be supported by the SFP. For example, because a 32G SFP supports
8/16/32G, a 4G port speed requires an 8G or 16G SFP. Because a 16G SFP supports 4/8/16G, a 32G
port speed requires a 32G SFP.
• Speed autonegotiation is supported. The default speed is 'auto'.
• You cannot use Fibre Channel on 40G and breakout ports.
• FEX cannot be directly connected to FC ports.
• FEX HIF ports cannot be converted to FC.

Configuring FC Connectivity Without Policies or Profiles Using the NX-OS CLI

The sample command sequence below creates bridge domain b1 under tenant t1 configured to support FCoE
connectivity.

Before you begin

• Under the target tenant configure a bridge domain to support FCoE traffic.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

104
Configuring Layer 2 External Connectivity
Configuring FC Connectivity Without Policies or Profiles Using the NX-OS CLI

Procedure

Step 1 To create a bridge domain for FCoE connectivity:

Example:
apic1(config)# tenant t1
apic1(config-tenant)# vrf context v1
apic1(config-tenant-vrf)# exit
apic1(config-tenant)# bridge-domain b1
apic1(config-tenant-bd)# fc
apic1(config-tenant-bd)# vrf member v1
apic1(config-tenant-bd)# exit
apic1(config-tenant)# exit

Step 2 Under the same tenant, associate the target EPG with the FCoE-configured bridge domain. The sample
command sequence below creates EPG e1 and associates that EPG with the FCoE-configured bridge domain
b1:
Example:
apic1(config)# tenant t1
apic1(config-tenant)# application a1
apic1(config-tenant-app)# epg e1
apic1(config-tenant-app-epg)# bridge-domain member b1
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)# exit
apic1(config-tenant)# exit

Step 3 The following example creates vsan domain dom1 with vsans 1-10:
Example:
apic1(config)# vsan-domain dom1
apic1(config-vsan)# vsan 1-10

Step 4 Convert range of ports from Ethernet to FC mode. The following example converts port 1/1-4 on switch 101
to FC:
Example:
apic1# config
apic1(config)# leaf 101
apic1(config-leaf)# slot 1
apic1(config-leaf-slot)# port 1 4 type fc
apic1(config-leaf-slot)# exit
apic1(config-leaf)# exit

Note Port conversion from Ethernet to FC and vice versa requires reload of switch.

Step 5 Configure FC interface in NP mode. The following example sets various interface properties on interface fc
1/10 and associates that interface with VSAN domain dom1. Each of the targeted interfaces must be assigned
one (and only one) VSAN in native mode. The sample command sequence associates the target interface 1/10
with VSAN 10 as a native VSAN and associates it with EPG e1 and application a1 under tenant t1.
Example:
apic1(config-leaf)# interface fc 1/10
apic1(config-leaf-fc-if)# switchport mode [f | np]
apic1(config-leaf-fc-if)# switchport rxbbcredit <16-64>
apic1(config-leaf-fc-if)# switchport speed [16G | 32G | 4G | 8G | auto | unknown]
apic1(config-leaf-fc-if)# vsan-domain member dom1
apic1(config-leaf-fc-if)# switchport vsan 10 tenant t1 application a1 epg e1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

105
 Configuring Layer 2 External Connectivity
Configuring FC Connectivity With Policies or Profiles Using the NX-OS CLI

Step 6 Create traffic map to pin server ports to uplink ports. The following example creates Traffic map for vFC 1/47
server interface pinned to FC 1/7 uplink interface:
Example:
apic1# config
apic1(config)# leaf 101
apic1(config-leaf)# npv traffic-map server-interface vfc 1/47 label label1 tenant tenant1
application app1 epg epg1
apic1(config-leaf)# npv traffic-map external-interface fc 1/7 tenant tenant1 label label1

Configuring FC Connectivity With Policies or Profiles Using the NX-OS CLI

The sample command sequence below creates bridge domain b1 under tenant t1 configured to support FCoE
connectivity.

Before you begin

• Under the target tenant configure a bridge domain to support FCoE traffic.

Procedure

Step 1 To create a bridge domain for FCoE connectivity:

Example:
apic1(config)# tenant t1
apic1(config-tenant)# vrf context v1
apic1(config-tenant-vrf)# exit
apic1(config-tenant)# bridge-domain b1
apic1(config-tenant-bd)# fc
apic1(config-tenant-bd)# vrf member v1
apic1(config-tenant-bd)# exit
apic1(config-tenant)# exit

Step 2 Under the same tenant, associate the target EPG with the FCoE-configured bridge domain. The sample
command sequence below creates EPG e1 and associates that EPG with the FCoE-configured bridge domain
b1:
Example:
apic1(config)# tenant t1
apic1(config-tenant)# application a1
apic1(config-tenant-app)# epg e1
apic1(config-tenant-app-epg)# bridge-domain member b1
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)# exit
apic1(config-tenant)# exit

Step 3 Create a VSAN domain. The following example creates vsan domain dom1 with vsans 1-10:
Example:
apic1(config)# vsan-domain dom1
apic1(config-vsan)# vsan 1-10

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

106
Configuring Layer 2 External Connectivity
Configuring FC Connectivity With Policies or Profiles Using the NX-OS CLI

Step 4 Create an interface policy group for NP port interfaces. The sample command sequence creates FC interface
policy group ipg2 and assigns a combination of values that determine values for any interface that this policy
group is applied to:
Example:
apic1(config)# template fc-policy-group ipg1
apic1(config-fc-pol-grp-if)# switchport ?
fill-pattern Configure fill pattern for fc interface
mode Configure port mode for fc interface
rxbbcredit Configure rxBBCredit for fc interface
speed Configure speed for fc interface
trunk-mode Configure trunk-mode for fc interface
apic1(config-fc-pol-grp-if)# switchport fill-pattern [ARBFF | IDLE]
apic1(config-fc-pol-grp-if)# switchport mode [f | np]
apic1(config-fc-pol-grp-if)# switchport rxbbcredit <16-64>
apic1(config-fc-pol-grp-if)# switchport speed [16G | 32G | 4G | 8G | auto | unknown]
apic1(config-fc-pol-grp-if)# vsan-domain member dom1

Step 5 Create an interface profile for FC port interfaces. The sample command sequence creates an interface profile
lip1 for FC port interfaces, associates the profile with FC interface policy group ipg1, and specifies the
interfaces to which this profile and its associated policies applies:
Example:
apic1# configure
apic1(config)# leaf-interface-profile lip1
apic1(config-leaf-if-profile)# description 'test description lip1'
apic1(config-leaf-if-profile)# leaf-interface-group lig1
apic1(config-leaf-if-group)# description 'test description lig1'
apic1(config-leaf-if-group)# fc-policy-group ipg1
apic1(config-leaf-if-group)# interface fc 1/1-4

Step 6 Create a leaf profile, assign the leaf interface profile to the leaf profile, and assign the leaf IDs on which the
profile will be applied:
Example:
apic1(config)# leaf-profile lp103
apic1(config-leaf-profile)# leaf-interface-profile lip1
apic1(config-leaf-profile)# leaf-group range
apic1(config-leaf-group)# leaf 103
apic1(config-leaf-group)#

Note After associating leaf interface profile to leaf, reload of leaf is required to bring up these ports as
FC ports.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

107
 Configuring Layer 2 External Connectivity
Configuring 802.1Q Tunnels

Configuring 802.1Q Tunnels

About ACI 802.1Q Tunnels
Figure 6: ACI 802.1Q Tunnels

With Cisco ACI and Cisco APIC Release 2.2(1x) and higher, you can configure 802.1Q tunnels on edge
(tunnel) ports to enable point-to-multi-point tunneling of Ethernet frames in the fabric, with Quality of Service
(QoS) priority settings. A Dot1q Tunnel transports untagged, 802.1Q tagged, and 802.1ad double-tagged
frames as-is across the fabric. Each tunnel carries the traffic from a single customer and is associated with a
single bridge domain. ACI front panel ports can be part of a Dot1q Tunnel. Layer 2 switching is done based
on Destination MAC (DMAC) and regular MAC learning is done in the tunnel. Edge-port Dot1q Tunnels
are supported on second-generation (and later) Cisco Nexus 9000 series switches with "EX" on the end of the
switch model name.
With Cisco ACI and Cisco APIC Release 2.3(x) and higher, you can also configure multiple 802.1Q tunnels
on the same core port to carry double-tagged traffic from multiple customers, each distinguished with an
access encapsulation configured for each 802.1Q tunnel. You can also disable MAC Address Learning on
802.1Q tunnels. Both edge ports and core ports can belong to an 802.1Q tunnel with access encapsulation and
disabled MAC Address Learning. Both edge ports and core ports in Dot1q Tunnels are supported on
third-generation Cisco Nexus 9000 series switches with "FX" on the end of the switch model name.
Terms used in this document may be different in the Cisco Nexus 9000 Series documents.

Table 11: 802.1Q Tunnel Terminology

ACI Documents Cisco Nexus 9000 Series Documents

Edge Port Tunnel Port

Core Port Trunk Port

The following guidelines and restrictions apply:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

108
Configuring Layer 2 External Connectivity
About ACI 802.1Q Tunnels

• Layer 2 tunneling of VTP, CDP, LACP, LLDP, and STP protocols is supported with the following
restrictions:
• Link Aggregation Control Protocol (LACP) tunneling functions as expected only with point-to-point
tunnels using individual leaf interfaces. It is not supported on port-channels (PCs) or virtual
port-channels (vPCs).
• CDP and LLDP tunneling with PCs or vPCs is not deterministic; it depends on the link it chooses
as the traffic destination.
• To use VTP for Layer 2 protocol tunneling, CDP must be enabled on the tunnel.
• STP is not supported in an 802.1Q tunnel bridge domain when Layer 2 protocol tunneling is enabled
and the bridge domain is deployed on Dot1q Tunnel core ports.
• ACI leaf switches react to STP TCN packets by flushing the end points in the tunnel bridge domain
and flooding them in the bridge domain.
• CDP and LLDP tunneling with more than two interfaces flood packets on all interfaces.
• With Cisco APIC Release 2.3(x) or higher, the destination MAC address of Layer 2 protocol packets
tunneled from edge to core ports is rewritten as 01-00-0c-cd-cd-d0 and the destination MAC address
of Layer 2 protocol packets tunneled from core to edge ports is rewritten with the standard default
MAC address for the protocol.

• If a PC or vPC is the only interface in a Dot1q Tunnel and it is deleted and reconfigured, remove the
association of the PC/VPC to the Dot1q Tunnel and reconfigure it.
• With Cisco APIC Release 2.2(x) the Ethertypes for double-tagged frames must be 0x9100 followed by
0x8100.
However, with Cisco APIC Release 2.3(x) and higher, this limitation no longer applies for edge ports,
on third-generation Cisco Nexus switches with "FX" on the end of the switch model name.
• For core ports, the Ethertypes for double-tagged frames must be 0x8100 followed by 0x8100.
• You can include multiple edge ports and core ports (even across leaf switches) in a Dot1q Tunnel.
• An edge port may only be part of one tunnel, but a core port can belong to multiple Dot1q tunnels.
• With Cisco APIC Release 2.3(x) and higher, regular EPGs can be deployed on core ports that are used
in 802.1Q tunnels.
• L3Outs are not supported on interfaces enabled for Dot1q Tunnels.
• FEX interfaces are not supported as members of a Dot1q Tunnel.
• Interfaces configured as breakout ports do not support 802.1Q tunnels.
• Interface-level statistics are supported for interfaces in Dot1q Tunnels, but statistics at the tunnel level
are not supported.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

109
 Configuring Layer 2 External Connectivity
Configuring 802.1Q Tunnels Using the NX-OS Style CLI

Configuring 802.1Q Tunnels Using the NX-OS Style CLI

Note You can use ports, port-channels, or virtual port channels for interfaces included in a Dot1q Tunnel. Detailed
steps are included for configuring ports. See the examples below for the commands to configure edge and
core port-channels and virtual port channels.

Create a Dot1q Tunnel and configure the interfaces for use in the tunnel using the NX-OS Style CLI, with
the following steps:

Note Dot1q Tunnels must include 2 or more interfaces. Repeat the steps (or configure two interfaces together), to
mark each interface for use in a Dot1q Tunnel. In this example, two interfaces are configured as edge-switch
ports, used by a single customer.

Use the following steps to configure a Dot1q Tunnel using the NX-OS style CLI:
1. Configure at least two interfaces for use in the tunnel.
2. Create a Dot1q Tunnel.
3. Associate all the interfaces with the tunnel.

Before you begin

Configure the tenant that will use the Dot1q Tunnel.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 Configure two interfaces for use in an 802.1Q

tunnel, with the following steps:
Step 3 leaf ID Identifies the leaf where the interfaces of the
Dot1q Tunnel will be located.
Example:
apic1(config)# leaf 101

Step 4 interface ethernet slot/port Identifies the interface or interfaces to be

marked as ports in a tunnel.
Example:
apic1(config-leaf)# interface ethernet
1/13-14

Step 5 switchport mode dot1q-tunnel {edgePort | Marks the interfaces for use in an 802.1Q
corePort} tunnel, and then leaves the configuration mode.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

110
 Configuring Layer 2 External Connectivity
Example: Configuring an 802.1Q Tunnel Using Ports with the NX-OS Style CLI

Command or Action Purpose

Example: The example shows configuring some
apic1(config-leaf-if)# switchport mode interfaces for edge port use. Repeat steps 3 to
dot1q-tunnel edgePort 5 to configure more interfaces for the tunnel.
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit
apic1(config)# exit

Step 6 Create an 802.1Q tunnel with the following

steps:
Step 7 leaf ID Returns to the leaf where the interfaces are
located.
Example:

apic1(config)# leaf 101

Step 8 interface ethernet slot/port Returns to the interfaces included in the tunnel.
Example:

apic1(config-leaf)# interface ethernet

1/13-14

Step 9 switchport tenant tenant-name dot1q-tunnel Associates the interfaces to the tunnel and exits
tunnel-name the configuration mode.
Example:

apic1(config-leaf-if)# switchport
tenant tenant64 dot1q-tunnel
vrf64_edgetunnel
apic1(config-leaf-if)# exit

Step 10 Repeat steps 7 to 10 to associate other

interfaces with the tunnel.

Example: Configuring an 802.1Q Tunnel Using Ports with the NX-OS Style CLI
The example marks two ports as edge port interfaces to be used in a Dot1q Tunnel, marks two more ports to
be used as core port interfaces, creates the tunnel, and associates the ports with the tunnel.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/13-14
apic1(config-leaf-if)# switchport mode dot1q-tunnel edgePort
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit
apic1(config)leaf 102
apic1(config-leaf)# interface ethernet 1/10, 1/21
apic1(config-leaf-if)# switchport mode dot1q-tunnel corePort
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit

apic1(config)# tenant tenant64

apic1(config-tenant)# dot1q-tunnel vrf64_tunnel
apic1(config-tenant-tunnel)# l2protocol-tunnel cdp
apic1(config-tenant-tunnel)# l2protocol-tunnel lldp

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

111
 Configuring Layer 2 External Connectivity
Example: Configuring an 802.1Q Tunnel Using Port-Channels with the NX-OS Style CLI

apic1(config-tenant-tunnel)# access-encap 200

apic1(config-tenant-tunnel)# mac-learning disable

apic1(config-tenant-tunnel)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/13-14
apic1(config-leaf-if)# switchport tenant tenant64 dot1q-tunnel vrf64_tunnel
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit
apic1(config)# leaf 102
apic1(config-leaf)# interface ethernet 1/10, 1/21
apic1(config-leaf-if)# switchport tenant tenant64 dot1q-tunnel vrf64_tunnel
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit

Example: Configuring an 802.1Q Tunnel Using Port-Channels with the NX-OS Style CLI
The example marks two port-channels as edge-port 802.1Q interfaces, marks two more port-channels as
core-port 802.1Q interfaces, creates a Dot1q Tunnel, and associates the port-channels with the tunnel.

apic1# configure
apic1(config)# tenant tenant64
apic1(config-tenant)# dot1q-tunnel vrf64_tunnel
apic1(config-tenant-tunnel)# l2protocol-tunnel cdp
apic1(config-tenant-tunnel)# l2protocol-tunnel lldp

apic1(config-tenant-tunnel)# access-encap 200

apic1(config-tenant-tunnel)# mac-learning disable

apic1(config-tenant-tunnel)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 101
apic1(config-leaf)# interface port-channel pc1
apic1(config-leaf-if)# exit
apic1(config-leaf)# interface ethernet 1/2-3
apic1(config-leaf-if)# channel-group pc1
apic1(config-leaf-if)# exit
apic1(config-leaf)# interface port-channel pc1
apic1(config-leaf-if)# switchport mode dot1q-tunnel edgePort
apic1(config-leaf-if)# switchport tenant tenant64 dot1q-tunnel vrf64_tunnel
apic1(config-tenant-tunnel)# exit
apic1(config-tenant)# exit

apic1(config)# leaf 102

apic1(config-leaf)# interface port-channel pc2
apic1(config-leaf-if)# exit
apic1(config-leaf)# interface ethernet 1/4-5
apic1(config-leaf-if)# channel-group pc2
apic1(config-leaf-if)# exit
apic1(config-leaf)# interface port-channel pc2
apic1(config-leaf-if)# switchport mode dot1q-tunnel corePort
apic1(config-leaf-if)# switchport tenant tenant64 dot1q-tunnel vrf64_tunnel

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

112
 Configuring Layer 2 External Connectivity
Example: Configuring an 802.1Q Tunnel Using Virtual Port-Channels with the NX-OS Style CLI

Example: Configuring an 802.1Q Tunnel Using Virtual Port-Channels with the NX-OS Style CLI
The example marks two virtual port-channels (vPCs) as edge-port 802.1Q interfaces for theDot1q Tunnel,
marks two more vPCs as core-port interfaces for the tunnel, creates the tunnel, and associates the virtual
port-channels with the tunnel.

apic1# configure
apic1(config)# vpc domain explicit 1 leaf 101 102
apic1(config)# vpc context leaf 101 102
apic1(config-vpc)# interface vpc vpc1
apic1(config-vpc-if)# switchport mode dot1q-tunnel edgePort
apic1(config-vpc-if)# exit
apic1(config-vpc)# exit
apic1(config)# vpc domain explicit 1 leaf 103 104
apic1(config)# vpc context leaf 103 104
apic1(config-vpc)# interface vpc vpc2
apic1(config-vpc-if)# switchport mode dot1q-tunnel corePort
apic1(config-vpc-if)# exit
apic1(config-vpc)# exit
apic1(config)# tenant tenant64
apic1(config-tenant)# dot1q-tunnel vrf64_tunnel
apic1(config-tenant-tunnel)# l2protocol-tunnel cdp
apic1(config-tenant-tunnel)# l2protocol-tunnel lldp

apic1(config-tenant-tunnel)# access-encap 200

apic1(config-tenant-tunnel)# mac-learning disable

apic1(config-tenant-tunnel)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 103
apic1(config-leaf)# interface ethernet 1/6
apic1(config-leaf-if)# channel-group vpc1 vpc
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit
apic1(config)# leaf 104
apic1(config-leaf)# interface ethernet 1/6
apic1(config-leaf-if)# channel-group vpc1 vpc
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit
apic1(config-vpc)# interface vpc vpc1
apic1(config-vpc-if)# switchport tenant tenant64 dot1q-tunnel vrf64_tunnel
apic1(config-vpc-if)# exit

Configuring Dynamic Breakout Ports

Configuration of Dynamic Breakout Ports
Breakout cables are suitable for very short links and offer a cost effective way to connect within racks and
across adjacent racks.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

113
 Configuring Layer 2 External Connectivity
Configuring Dynamic Breakout Ports Using the NX-OS Style CLI

Breakout enables a 40 Gigabit (Gb) port to be split into four independent and logical 10Gb ports or a 100Gb
port to be split into four independent and logical 25Gb ports.
Before you configure breakout ports, connect a 40Gb port to four 10Gb ports or a 100Gb port to four 25Gb
ports with one of the following cables:
• Cisco QSFP-4SFP10G
• Cisco QSFP-4SFP25G

The 40Gb to 10Gb dynamic breakout feature is supported on the access facing ports of the following switches:
• N9K-C9332PQ
• N9K-C93180LC-EX
• N9K-C9336C-FX

The 100Gb to 25Gb breakout feature is supported on the access facing ports of the following switches:
• N9K-C93180LC-EX
• N9K-C9336C-FX2

Observe the following guidelines and restrictions:

• In general, breakouts and port profiles (ports changed from uplink to downlink) are not supported on the
same port.
• Fast Link Failover policies are not supported on the same port with the dynamic breakout feature.
• Breakout subports can be used in the same way other port types in the policy model are used.
• When a port is enabled for dynamic breakout, other policies (expect monitoring policies) on the parent
port are no longer valid.
• When a port is enabled for dynamic breakout, other EPG deployments on the parent port are no longer
valid.
• A breakout sub-port can not be further broken out using a breakout policy group.

Configuring Dynamic Breakout Ports Using the NX-OS Style CLI

Use the following steps to configure a breakout port, verify the configuration, and configure an EPG on a sub
port, using the NX-OS style CLI.

Before you begin

• The ACI fabric is installed, APIC controllers are online, and the APIC cluster is formed and healthy.
• An APIC fabric administrator account is available that will enable creating the necessary fabric
infrastructure configurations.
• The target leaf switches are registered in the ACI fabric and available.
• The 40GE or 100GE leaf switch ports are connected with Cisco breakout cables to the downlink ports.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

114
Configuring Layer 2 External Connectivity
Configuring Dynamic Breakout Ports Using the NX-OS Style CLI

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf ID Selects the leaf switch where the breakout port
will be located and enters leaf configuration
Example:
mode.
apic1(config)# leaf 101

Step 3 interface ethernet slot/port Identifies the interface to be enabled as a 40

Gigabit Ethernet (GE) breakout port.
Example:
apic1(config-leaf)# interface ethernet
1/16

Step 4 breakout 10g-4x | 25g-4x Enables the selected interface for breakout.
Example: Note For switch support for the Dynamic
apic1(config-leaf-if)# breakout 10g-4x Breakout Port feature, see
Configuration of Dynamic Breakout
Ports, on page 113.

Step 5 show run Verifies the configuration by showing the

running configuration of the interface and
Example:
returns to global configuration mode.
apic1(config-leaf-if)# show run
# Command: show running-config leaf 101
interface ethernet 1 / 16
# Time: Fri Dec 2 18:13:39 2016
leaf 101
interface ethernet 1/16
breakout 10g-4x
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit

Step 6 tenant tenant-name Selects or creates the tenant that will consume
the breakout ports and enters tenant
Example:
configuration mode.
apic1(config)# tenant tenant64

Step 7 vrf context vrf-name Creates or identifies the Virtual Routing and
Forwarding (VRF) instance associated with
Example:
the tenant and exits the configuration mode.
apic1(config-tenant)# vrf context vrf64
apic1(config-tenant-vrf)# exit

Step 8 bridge-domain bridge-domain-name Creates or identifies the bridge-domain

associated with the tenant and enters BD
Example:
configuration mode.
apic1(config-tenant)# bridge-domain bd64

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

115
 Configuring Layer 2 External Connectivity
Configuring Dynamic Breakout Ports Using the NX-OS Style CLI

Command or Action Purpose

Step 9 vrf member vrf-name Associates the VRF with the bridge-domain
and exits the configuration mode.
Example:
apic1(config-tenant-bd)# vrf member
vrf64
apic1(config-tenant-bd)# exit

Step 10 application application-profile-name Creates or identifies the application profile

associated with the tenant and the EPG.
Example:
apic1(config-tenant)# application app64

Step 11 epg epg-name Creates or identifies the EPG and enters into
EPG configuration mode.
Example:
apic1(config-tenant)# epg epg64

Step 12 bridge-domain member bridge-domain-name Associates the EPG with the bridge domain
and returns to global configuration mode.
Example:
apic1(config-tenant-app-epg)# Configure the sub ports as desired, for
bridge-domain member bd64 example, use the speed command in leaf
apic1(config-tenant-app-epg)# exit interface mode to configure a sub port.
apic1(config-tenant-app)# exit
apic1(config-tenant)# exit

Step 13 leaf leaf-name Associates the EPG with a break-out port.

Example:

apic1(config)# leaf 1017

apic1(config-leaf)# interface ethernet
1/13
apic1(config-leaf-if)# vlan-domain
member dom1
apic1(config-leaf-if)# switchport trunk
allowed vlan 20 tenant t1 application
AP1 epg EPG1

Note The vlan-domain and vlan-domain

member commands mentioned in
the above example are a
pre-requisite for deploying an EPG
on a port.

Step 14 speed interface-speed Enters leaf interface mode, sets the speed of
an interface, and exits the configuration mode.
Example:

apic1(config)# leaf 101

apic1(config-leaf)# interface ethernet
1/16/1
apic1(config-leaf-if)# speed 10G
apic1(config-leaf-if)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

116
Configuring Layer 2 External Connectivity
Configuring Dynamic Breakout Ports Using the NX-OS Style CLI

Command or Action Purpose

Step 15 show run After you have configured the sub ports,
entering this command in leaf configuration
Example:
mode displays the sub port details.
apic1(config-leaf)# show run

The port on leaf 101 at interface 1/16 is confirmed enabled for breakout with sub ports 1/16/1, 1/16/2, 1/16/3,
and 1/16/4.

Example
This example configures the port for breakout:
apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/16
apic1(config-leaf-if)# breakout 10g-4x

This example configures the EPG for the sub ports.

apic1(config)# tenant tenant64
apic1(config-tenant)# vrf context vrf64
apic1(config-tenant-vrf)# exit
apic1(config-tenant)# bridge-domain bd64
apic1(config-tenant-bd)# vrf member vrf64
apic1(config-tenant-bd)# exit
apic1(config-tenant)# application app64
apic1(config-tenant-app)# epg epg64
apic1(config-tenant-app-epg)# bridge-domain member bd64
apic1(config-tenant-app-epg)# end

This example sets the speed for the breakout sub ports to 10G.
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/16/1
apic1(config-leaf-if)# speed 10G
apic1(config-leaf-if)# exit

apic1(config-leaf)# interface ethernet 1/16/2

apic1(config-leaf-if)# speed 10G
apic1(config-leaf-if)# exit
apic1(config-leaf)# interface ethernet 1/16/3
apic1(config-leaf-if)# speed 10G
apic1(config-leaf-if)# exit
apic1(config-leaf)# interface ethernet 1/16/4
apic1(config-leaf-if)# speed 10G
apic1(config-leaf-if)# exit

This example shows the four sub ports connected to leaf 101, interface 1/16.
apic1#(config-leaf)# show run
# Command: show running-config leaf 101
# Time: Fri Dec 2 00:51:08 2016
leaf 101
interface ethernet 1/16/1
speed 10G
negotiate auto
link debounce time 100
exit
interface ethernet 1/16/2
speed 10G
negotiate auto

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

117
 Configuring Layer 2 External Connectivity
Configuring Port Profiles

link debounce time 100

exit
interface ethernet 1/16/3
speed 10G
negotiate auto
link debounce time 100
exit
interface ethernet 1/16/4
speed 10G
negotiate auto
link debounce time 100
exit
interface ethernet 1/16
breakout 10g-4x
exit
interface vfc 1/16

Configuring Port Profiles

Configuring Port Profiles
Prior to Cisco APIC, Release 3.1(1), conversion from uplink port to downlink port or downlink port to uplink
port (in a port profile) was not supported on Cisco ACI leaf switches. Starting with Cisco APIC Release 3.1(1),
uplink and downlink conversion is supported on Cisco Nexus 9000 series switches with names that end in
EX or FX, and later (for example, N9K-C9348GC-FXP). A FEX connected to converted downlinks is also
supported.
This functionality is supported on the following Cisco switches:
• N9K-C9348GC-FXP
• N9K-C93180LC-EX and N9K-C93180YC-FX
• N9K-93180YC-EX, N9K-C93180YC-EX, and N9K-C93180YC-EXU
• N9K-C93108TC-EX and N9K-C93108TC-FX
• N9K-C9336C-FX2 (only downlink to uplink conversion supported)

Restrictions
Fast Link Failover policies and port profiles are not supported on the same port. If port profile is enabled,
Fast Link Failover cannot be enabled or vice versa.
The last 2 uplink ports of supported TOR switches cannot be converted to downlink ports (they are reserved
for uplink connections.)
Up to Cisco APIC Release 3.2, port profiles and breakout ports are not supported on the same ports.

Guidelines
In converting uplinks to downlinks and downlinks to uplinks, consider the following guidelines.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

118
Configuring Layer 2 External Connectivity
Configuring Port Profiles

Subject Guideline

Decommissioning nodes If a decommissioned node has the Port Profile feature deployed on it, the port
with port profiles conversions are not removed even after decommissioning the node. It is
necessary to manually delete the configurations after decommission, for the
ports to return to the default state. To do this, log onto the switch, run the
setup-clean-config.sh script, and wait for it to run. Then, enter the reload
command.

FIPS When you enable or disable Federal Information Processing Standards (FIPS)
on a Cisco ACI fabric, you must reload each of the switches in the fabric for
the change to take effect. The configured scale profile setting is lost when you
issue the first reload after changing the FIPS configuration. The switch remains
operational, but it uses the default scale profile. This issue does not happen on
subsequent reloads if the FIPS configuration has not changed.
FIPS is supported on Cisco NX-OS release 13.1(1) or later.
If you must downgrade the firmware from a release that supports FIPS to a
release that does not support FIPS, you must first disable FIPS on the Cisco
ACI fabric and reload all the switches in the fabric for the FIPS configuration
change.

Maximum uplink port limit When the maximum uplink port limit is reached and ports 25 and 27 are
converted from uplink to downlink and back to uplink on Cisco 93180LC-EX
switches:
On Cisco 93180LC-EX Switches, ports 25 and 27 are the native uplink ports.
Using the port profile, if you convert port 25 and 27 to downlink ports, ports
29, 30, 31, and 32 are still available as four native uplink ports. Because of the
threshold on the number of ports (which is maximum of 12 ports) that can be
converted, you can convert 8 more downlink ports to uplink ports. For example,
ports 1, 3, 5, 7, 9, 13, 15, 17 are converted to uplink ports and ports 29, 30, 31
and 32 are the 4 native uplink ports (the maximum uplink port limit on Cisco
93180LC-EX switches).
When the switch is in this state and if the port profile configuration is deleted
on ports 25 and 27, ports 25 and 27 are converted back to uplink ports, but
there are already 12 uplink ports on the switch (as mentioned earlier). To
accommodate ports 25 and 27 as uplink ports, 2 random ports from the port
range 1, 3, 5, 7, 9, 13, 15, 17 are denied the uplink conversion and this situation
cannot be controlled by the user.
Therefore, it is mandatory to clear all the faults before reloading the leaf node
to avoid any unexpected behavior regarding the port type. It should be noted
that if a node is reloaded without clearing the port profile faults, especially
when there is a fault related to limit-exceed, the port might not be in an expected
operational state.

Breakout Limitations

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

119
 Configuring Layer 2 External Connectivity
Port Profile Configuration Summary

Switch Releases Limitations

N9K-C9332PQ Cisco APIC 2.2 (1n) and • 40Gb dynamic breakouts into 4X10Gb ports
higher are supported.
• Ports 13 and 14 do not support breakouts.
• Port profiles and breakouts are not supported
on the same port.

N9K-C93180LC-EX Cisco APIC 3.1(1i) and • 40Gb and 100Gb dynamic breakouts are
higher supported on ports 1 through 24 on odd
numbered ports.
• When the top ports (odd ports) are broken out,
then the bottom ports (even ports) are error
disabled.
• Port profiles and breakouts are not supported
on the same port.

N9K-C9336C-FX2 Cisco APIC 3.1(2m) and • 40Gb and 100Gb dynamic breakouts are
higher supported on ports 1 through 30.
• Port profiles and breakouts are not supported
on the same port.

Port Profile Configuration Summary

The following table summarizes supported uplinks and downlinks for the switches that support port profile
conversions from Uplink to Downlink and Downlink to Uplink.

Switch Model Default Links Max Uplinks (Fabric Max Downlinks Release
Ports) (Server Ports) Supported

N9K-C9348GC-FXP 48 x 100M/1G 48 x 100M/1G Same as default port 3.1(1i)

BASE-T downlinks BASE-T downlinks configuration
4 x 10/25-Gbps SFP28 4 x 10/25-Gbps SFP28
downlinks uplinks
2 x 40/100-Gbps 2 x 40/100-Gbps
QSFP28 uplinks QSFP28 uplinks

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

120
Configuring Layer 2 External Connectivity
Port Profile Configuration Summary

Switch Model Default Links Max Uplinks (Fabric Max Downlinks Release
Ports) (Server Ports) Supported

N9K-C93180LC-EX 24 x 40-Gbps QSFP28 18 x 40-Gbps QSFP28 24 x 40-Gbps 3.1(1i)

downlinks(1-24) downlinks (from 1-24) QSFP28
downlinks(1-24)
2 x 40/100-Gbps 6 x 40-Gbps QSFP28
QSFP28 uplinks(25, uplinks(from 1-24) 2 x 40/100-Gbps
27) QSFP28
2 x 40/100-Gbps
downlinks(25, 27)
4 x 40/100-Gbps QSFP28 uplinks(25,
QSFP28 27) 4 x 40/100-Gbps
uplinks(29-32) QSFP28
4 x 40/100-Gbps
uplinks(29-32)
Or QSFP28
uplinks(29-32) Or
12 x 100-Gbps
QSFP28 Or 12 x 100-Gbps
downlinks(odd number QSFP28
6 x 100-Gbps QSFP28
from 1-24) downlinks(odd
downlinks(odd number
number from 1-24)
2 x 40/100-Gbps from 1-24)
QSFP28 uplinks(25, 2 x 40/100-Gbps
6 x 100-Gbps QSFP28
27) QSFP28 downlinks
uplinks(odd number
(25, 27)
4 x 40/100-Gbps from 1-24)
QSFP28 4 x 40/100-Gbps
2 x 40/100-Gbps
uplinks(29-32) QSFP28
QSFP28 uplinks(25,
uplinks(29-32)
27)
4 x 40/100-Gbps
QSFP28
uplinks(29-32)

N9K-C93180YC-EX 48 x 10/25-Gbps fiber Same as default port 48 x 10/25-Gbps fiber 3.1(1i)

downlinks configuration downlinks
N9K-C93180YC-FX
6 x 40/100-Gbps 48 x 10/25-Gbps fiber 4 x 40/100-Gbps 4.0(1)
QSFP28 uplinks uplinks QSFP28 downlinks

6 x 40/100-Gbps 2 x 40/100-Gbps
QSFP28 uplinks QSFP28 uplinks

N9K-C93108TC-EX 48 x 10GBASE-T Same as default port 48 x 10/25-Gbps fiber 3.1

downlinks configuration downlinks
N9K-C93108TC-FX
6 x 40/100-Gbps 4 x 40/100-Gbps
QSFP28 uplinks QSFP28 downlinks
2 x 40/100-Gbps
QSFP28 uplinks

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

121
 Configuring Layer 2 External Connectivity
Configuring a Port Profile Using the NX-OS Style CLI

Switch Model Default Links Max Uplinks (Fabric Max Downlinks Release
Ports) (Server Ports) Supported

N9K-C9336C-FX2 30 x 40/100-Gbps 18 x 40/100-Gbps Same as default port 3.2(1i)

QSFP28 downlinks QSFP28 downlinks configuration
6 x 40/100-Gbps 18 x 40/100-Gbps
QSFP28 uplinks QSFP28 uplinks

18 x 40/100-Gbps 34 x 40/100-Gbps 3.2(3i)

QSFP28 downlinks QSFP28 downlinks
18 x 40/100-Gbps 2 x 40/100-Gbps
QSFP28 uplinks QSFP28 uplinks

36 x 40/100-Gbps 34 x 40/100-Gbps 4.1

QSFP28 uplinks QSFP28 downlinks
2 x 40/100-Gbps
QSFP28 uplinks

N9K-93240YC-FX2 48 x 10/25-Gbps fiber Same as default port 48 x 10/25-Gbps fiber 4.0(1)

downlinks configuration downlinks
12 x 40/100-Gbps 48 x 10/25-Gbps fiber 10 x 40/100-Gbps 4.1
QSFP28 uplinks uplinks QSFP28 downlinks

12 x 40/100-Gbps 2 x 40/100-Gbps
QSFP28 uplinks QSFP28 uplinks

N9K-C93216TC-FX2 96 x 10G BASE-T Same as default port 96 x 10G BASE-T 4.1.2

downlinks configuration downlinks
12 x 40/100-Gbps 10 x 40/100-Gbps
QSFP28 uplinks QSFP28 downlinks
2 x 40/100-Gbps
QSFP28 uplinks

N9K-C93360YC-FX2 96 x 10/25-Gbps 44 x 10/25Gbps SFP28 96 x 10/25-Gbps 4.1.2

SFP28 downlinks downlinks SFP28 downlinks
12 x 40/100-Gbps 52 x 10/25Gbps SFP28 10 x 40/100-Gbps
QSFP28 uplinks uplinks QSFP28 downlinks
12 x 40/100Gbps 2 x 40/100-Gbps
QSFP28 uplinks QSFP28 uplinks

Configuring a Port Profile Using the NX-OS Style CLI

To configure a port profile using the NX-OS style CLI, perform the following steps:

Before you begin

• The ACI fabric is installed, APIC controllers are online, and the APIC cluster is formed and healthy.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

122
 Configuring Layer 2 External Connectivity
Verifying Port Profile Configuration and Conversion Using the NX-OS Style CLI

• An APIC fabric administrator account is available that will enable creating or modifying the necessary
fabric infrastructure configurations.
• The target leaf switches are registered in the ACI fabric and available.

Procedure

Step 1 configure
Enters global configuration mode.
Example:
apic1# configure

Step 2 leaf node-id

Specifies the leaf or leaf switches to be configured.
Example:
apic1(config)# leaf 102

Step 3 interface type

Specifies the interface that you are configuring. You can specify the interface type and identity. For an Ethernet
port, use ethernet slot / port.
Example:
apic1(config-leaf)# interface ethernet 1/2

Step 4 port-direction {uplink | downlink}

Determines the port direction or changes it. This example configures the port to be a downlink.
Note On the N9K-C9336C-FX switch, changing a port from uplink to downlink is not supported.

Example:
apic1(config-leaf-if)# port-direction downlink

Step 5 Log on to the leaf switch where the port is located and enter the setup-clean-config.sh -k command, then the
reload command.

Verifying Port Profile Configuration and Conversion Using the NX-OS Style CLI
You can verify the configuration and the conversion of the ports using the show interface brief CLI command.

Note Port profile can be deployed only on the top ports of a Cisco N9K-C93180LC-EX switch, for example, 1, 3,
5, 7, 9, 11, 13, 15, 17, 19, 21, and 23. When the top port is converted using the port profile, the bottom ports
are hardware disabled. For example, if Eth 1/1 is converted using the port profile, Eth 1/2 is hardware disabled.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

123
 Configuring Layer 2 External Connectivity
Microsegmentation on Virtual Switches

Procedure

Step 1 This example displays the output for converting an uplink port to downlink port. Before converting an uplink
port to downlink port, the output is displayed in the example. The keyword routed denotes the port as uplink
port.
Example:

switch# show interface brief

<snip>
Eth1/49 -- eth routed down sfp-missing 100G(D) --
Eth1/50 -- eth routed down sfp-missing 100G(D) --
<snip>

Step 2 After configuring the port profile and reloading the switch, the output is displayed in the example. The keyword
trunk denotes the port as downlink port.
Example:

switch# show interface brief

<snip>
Eth1/49 0 eth trunk down sfp-missing 100G(D) --
Eth1/50 0 eth trunk down sfp-missing 100G(D) --
<snip>

Microsegmentation on Virtual Switches

Configuring Microsegmentation on Virtual Switches
Microsegmentation with the Cisco Application Centric Infrastructure (ACI) provides the ability to automatically
assign endpoints to logical security zones called endpoint groups (EPGs) based on various network-based or
virtual machine (VM)-based attributes. This section contains instructions for configuring microsegment (uSeg)
EPGs on virtual switches.
Microsegmentation with Cisco ACI provides support for virtual endpoints attached to the following:
• VMware vSphere Distributed Switch (VDS)
• Cisco Application Virtual Switch (AVS)
• Microsoft vSwitch

See the Cisco ACI Virtualization Guide for information about how Microsegmentation with Cisco ACI
works, prerequisites, guidelines, and scenarios.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

124
 Configuring Layer 2 External Connectivity
Configuring Microsegmentation with Cisco ACI Using the NX-OS-Style CLI

Configuring Microsegmentation with Cisco ACI Using the NX-OS-Style CLI

This section describes how to configure Microsegmentation with Cisco ACI for Cisco ACI Virtual Edge,
Cisco AVS, VMware VDS or Microsoft Hyper-V Virtual Switch using VM-based attributes within an
application EPG.

Procedure

Step 1 In the CLI, enter configuration mode:

Example:
apic1# configure
apic1(config)#

Step 2 Create the uSeg EPG:

Example:
This example is for an application EPG.
Note The command to allow microsegmentation in the following example is required for VMware VDS
only.
apic1(config)# tenant cli-ten1
apic1(config-tenant)# application cli-a1
apic1(config-tenant-app)# epg cli-baseEPG1
apic1(config-tenant-app-epg)# bridge-domain member cli-bd1
apic1(config-tenant-app-epg)# vmware-domain member cli-vmm1 allow-micro-segmentation

Example:
(Optional) This example sets match EPG precedence for the uSeg EPG:
apic1(config)# tenant Coke
apic1(config-tenant)# application cli-a1
apic1(config-tenant-app)# epg cli-uepg1 type micro-segmented
apic1(config-tenant-app-uepg)# bridge-domain member cli-bd1
apic1(config-tenant-app-uepg)# match-precedence 10

Example:
This example uses a filter based on the attribute VM Name.

apic1(config)# tenant cli-ten1

apic1(config-tenant)# application cli-a1
apic1(config-tenant-app)# epg cli-uepg1 type micro-segmented
apic1(config-tenant-app-uepg)# bridge-domain member cli-bd1
apic1(config-tenant-app-uepg)# attribute-logical-expression ‘vm-name contains <cos1>’

Example:
This example uses a filter based on an IP address.

apic1(config)# tenant cli-ten1

apic1(config-tenant)# application cli-a1
apic1(config-tenant-app)# epg cli-uepg1 type micro-segmented
apic1(config-tenant-app-uepg)# bridge-domain member cli-bd1
apic1(config-tenant-app-uepg)# attribute-logical-expression ‘ip equals <FF:FF:FF:FF:FF:FF>’

Example:
This example uses a filter based on a MAC address.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

125
 Configuring Layer 2 External Connectivity
Configuring Microsegmentation with Cisco ACI Using the NX-OS-Style CLI

apic1(config)# tenant cli-ten1

apic1(config-tenant)# application cli-a1
apic1(config-tenant-app)# epg cli-uepg1 type micro-segmented
apic1(config-tenant-app-uepg)# bridge-domain member cli-bd1
apic1(config-tenant-app-uepg)# attribute-logical-expression ‘mac equals <FF-FF-FF-FF-FF-FF>’

Example:
This example uses the operator AND to match all attributes and the operator OR to match any attribute.
apic1(config)# tenant cli-ten1
apic1(config-tenant)# application cli-a1
apic1(config-tenant-app)# epg cli-uepg1 type micro-segmented
apic1(config-tenant-app-uepg)# attribute-logical-expression 'hv equals host-123 OR (guest-os
equals "Ubuntu Linux (64-bit)" AND domain contains fex)'

Example:
This example uses a filter based on the attribute VM-Custom Attribute.
apic1(config)# tenant cli-ten1
apic1(config-tenant)# application cli-a1
apic1(config-tenant-app)# epg cli-uepg1 type micro-segmented
apic1(config-tenant-app-uepg)# bridge-domain member cli-bd1
apic1(config-tenant-app-uepg)# attribute-logical-expression 'custom <Custom Attribute Name>
equals <Custom Attribute value>'

Step 3 (Cisco ACI Virtual Edge only): Attach the uSeg EPG to a Cisco ACI Virtual Edge VMM domain, specifying
the switching and encapsulation modes:
Example:
vmware-domain member AVE-CISCO
switching-mode AVE
encap-mode vxlan
exit

Step 4 Verify the uSeg EPG creation:

Example:
The following example is for a uSeg EPG with a VM name attribute filter

apic1(config-tenant-app-uepg)# show running-config

# Command: show running-config tenant cli-ten1 application cli-a1 epg cli-uepg1 type
micro-segmented # Time: Thu Oct 8 11:54:32 2015
tenant cli-ten1
application cli-a1
epg cli-uepg1 type micro-segmented
bridge-domain cli-bd1
attribute-logical-expression ‘vm-name contains cos1 force’
{vmware-domain | microsoft-domain} member cli-vmm1
exit
exit
exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

126
 Configuring Layer 2 External Connectivity
Configuring Microsegmentation on Bare-Metal

Configuring Microsegmentation on Bare-Metal

Using Microsegmentation with Network-based Attributes on Bare Metal
You can use Cisco APIC to configure Microsegmentation with Cisco ACI to create a new, attribute-based
EPG using a network-based attribute, a MAC address or one or more IP addresses. You can configure
Microsegmentation with Cisco ACI using network-based attributes to isolate VMs or physical endpoints
within a single base EPG or VMs or physical endpoints in different EPGs.

Using an IP-based Attribute

You can use an IP-based filter to isolate a single IP address, a subnet, or multiple of noncontiguous IP addresses
in a single microsegment. You might want to isolate physical endpoints based on IP addresses as a quick and
simply way to create a security zone, similar to using a firewall.

Using a MAC-based Attribute

You can use a MAC-based filter to isolate a single MAC address or multiple MAC addresses. You might
want to do this if you have a server sending bad traffic int he network. By creating a microsegment with a
MAC-based filter, you can isolate the server.

Configuring a Network-Based Microsegmented EPG in a Bare-Metal

Environment Using the NX-OS Style CLI
This section describes how to configure microsegmentation with Cisco ACI using network-based attributes
(IP address or MAC address) within a base EPG in a bare-metal environment.

Procedure

Command or Action Purpose

Step 1 In the CLI, enter configuration mode:
Example:
apic1# configure
apic1(config)#

Step 2 Create the microsegment:

Example:
This example uses a filter based on an IP
address.
apic1(config)# tenant cli-ten1
apic1(config-tenant)# application cli-a1
apic1(config-tenant-app)# epg cli-uepg1
type micro-segmented
apic1(config-tenant-app-uepg)#
bridge-domain member cli-bd1
apic1(config-tenant-app-uepg)# attribute
cli-upg-att match ip <X.X.X.X>

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

127
 Configuring Layer 2 External Connectivity
Configuring a Network-Based Microsegmented EPG in a Bare-Metal Environment Using the NX-OS Style CLI

Command or Action Purpose

#Schemes to express the ip
A.B.C.D IP Address
A.B.C.D/LEN IP Address and mask

Example:
This example uses a filter based on a MAC
address.
apic1(config)# tenant cli-ten1
apic1(config-tenant)# application cli-a1
apic1(config-tenant-app)# epg cli-uepg1
type micro-segmented
apic1(config-tenant-app-uepg)#
bridge-domain member cli-bd1
apic1(config-tenant-app-uepg)# attribute
cli-upg-att match mac
<FF-FF-FF-FF-FF-FF>
#Schemes to express the mac
E.E.E MAC address (Option 1)
EE-EE-EE-EE-EE-EE MAC address (Option 2)
EE:EE:EE:EE:EE:EE MAC address (Option 3)
EEEE.EEEE.EEEE MAC address (Option 4)

Example:
This example uses a filter based on a MAC
address and enforces intra-EPG isolation
between all members of this uSeg EPG:
apic1(config)# tenant cli-ten1
apic1(config-tenant)# application cli-a1
apic1(config-tenant-app)# epg cli-uepg1
type micro-segmented
apic1(config-tenant-app-uepg)# isolation
enforced
apic1(config-tenant-app-uepg)#
bridge-domain member cli-bd1
apic1(config-tenant-app-uepg)# attribute
cli-upg-att match mac
<FF-FF-FF-FF-FF-FF>
#Schemes to express the mac
E.E.E MAC address (Option 1)
EE-EE-EE-EE-EE-EE MAC address (Option 2)
EE:EE:EE:EE:EE:EE MAC address (Option 3)
EEEE.EEEE.EEEE MAC address (Option 4)

Step 3 Deploy the EPG.

Example:
This example deploys the EPG and bids to the
leaf.
apic1(config)# leaf 101
apic1(config-leaf)# deploy-epg tenant
cli-ten1 application cli-a1 epg cli-uepg1
type micro-segmented

Step 4 Verify the microsegment creation:

Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

128
 Configuring Layer 2 External Connectivity
Configuring Layer 2 IGMP Snoop Multicast

Command or Action Purpose

apic1(config-tenant-app-uepg)# show
running-config
# Command: show running-config tenant
cli-ten1 application cli-app1 epg
cli-uepg1 type micro-segmented
# Time: Thu Oct 8 11:54:32 2015
tenant cli-ten1
application cli-app1
epg cli-esx1bu type micro-segmented

bridge-domain cli-bd1
attribute cli-uepg-att match mac
00:11:22:33:44:55
exit
exit
exit

Configuring Layer 2 IGMP Snoop Multicast

About Cisco APIC and IGMP Snooping
IGMP snooping is the process of listening to Internet Group Management Protocol (IGMP) network traffic.
The feature allows a network switch to listen in on the IGMP conversation between hosts and routers and
filter multicasts links that do not need them, thus controlling which ports receive specific multicast traffic.
Cisco APIC provides support for the full IGMP snooping feature included on a traditional switch such as the
N9000 standalone.
• Policy-based IGMP snooping configuration per bridge domain
APIC enables you to configure a policy in which you enable, disable, or customize the properties of
IGMP Snooping on a per bridge-domain basis. You can then apply that policy to one or multiple bridge
domains.
• Static port group implementation
IGMP static port grouping enables you to pre-provision ports, already statically-assigned to an application
EPG, as the switch ports to receive and process IGMP multicast traffic. This pre-provisioning prevents
the join latency which normally occurs when the IGMP snooping stack learns ports dynamically.
Static group membership can be pre-provisioned only on static ports (also called, static-binding ports)
assigned to an application EPG.
• Access group configuration for application EPGs
An “access-group” is used to control what streams can be joined behind a given port.
An access-group configuration can be applied on interfaces that are statically assigned to an application
EPG in order to ensure that the configuration can be applied on ports that will actually belong to the that
EPG.
Only Route-map-based access groups are allowed.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

129
 Configuring Layer 2 External Connectivity
Enabling IGMP Snooping Static Port Groups

Note You can use vzAny to enable protocols such as IGMP Snooping for all the EPGs in a VRF. For more
information about vzAny, see Use vzAny to Automatically Apply Communication Rules to all EPGs in a
VRF.
To use vzAny, navigate to Tenants > tenant-name > Networking > VRFs > vrf-name > EPG Collection
for VRF.

Enabling IGMP Snooping Static Port Groups

IGMP static port grouping enables you to pre-provision ports, that were previously statically-assigned to an
application EPG, to enable the switch ports to receive and process IGMP multicast traffic. This pre-provisioning
prevents the join latency which normally occurs when the IGMP snooping stack learns ports dynamically.
Static group membership can be pre-provisioned only on static ports assigned to an application EPG.
Static group membership can be configured through the APIC GUI, CLI, and REST API interfaces.

Configuring and Assigning an IGMP Snooping Policy to a Bridge Domain using

the NX-OS Style CLI
Before you begin
• Create the tenant that will consume the IGMP Snooping policy.
• Create the bridge domain for the tenant, where you will attach he IGMP Snooping policy.

Procedure

Command or Action Purpose

Step 1 Create a snooping policy based on default The example NX-OS style CLI sequence:
values.
• Creates an IGMP Snooping policy named
Example: cookieCut1 with default values.

apic1(config-tenant)# template ip igmp

• Displays the default IGMP Snooping
snooping policy cookieCut1 values for the policy cookieCut1.
apic1(config-tenant-template-ip-igmp-snooping)#
show run all

# Command: show running -config all

tenant foo template ip igmp snooping
policy cookieCut1
# Time: Thu Oct 13 18:26:03 2016
tenant t_10
template ip igmp snooping policy
cookieCut1
ip igmp snooping
no ip igmp snooping fast-leave
ip igmp snooping
last-member-query-interval 1
no ip igmp snooping querier

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

130
Configuring Layer 2 External Connectivity
Configuring and Assigning an IGMP Snooping Policy to a Bridge Domain using the NX-OS Style CLI

Command or Action Purpose

ip igmp snooping query-interval
125
ip igmp snooping
query-max-response-time 10
ip igmp snooping
stqrtup-query-count 2
ip igmp snooping
startup-query-interval 31
no description
exit
exit
apic1(config-tenant-template-ip-igmp-snooping)#

Step 2 Modify the snooping policy as necessary. The example NX-OS style CLI sequence:
Example: • Specifies a custom value for the
query-interval value in the IGMP Snooping
apic1(config-tenant-template-ip-igmp-snooping)# policy named cookieCut1.
ip igmp snooping query-interval 300
apic1(config-tenant-template-ip-igmp-snooping)# • Confirms the modified IGMP Snooping
show run all value for the policy cookieCut1.
# Command: show running -config all
tenant foo template ip igmp snooping
policy cookieCut1
#Time: Thu Oct 13 18:26:03 2016
tenant foo
template ip igmp snooping policy
cookieCut1
ip igmp snooping
no ip igmp snooping fast-leave
ip igmp snooping
last-member-query-interval 1
no ip igmp snooping querier
ip igmp snooping query-interval
300
ip igmp snooping
query-max-response-time 10
ip igmp snooping
stqrtup-query-count 2
ip igmp snooping
startup-query-interval 31
no description
exit
exit
apic1(config-tenant-template-ip-igmp-snooping)#
exit
apic1(config--tenant)#

Step 3 Assign the policy to a bridge domain. The example NX-OS style CLI sequence:
Example: • Navigates to bridge domain, BD3. for the
query-interval value in the IGMP Snooping
apic1(config-tenant)# int bridge-domain policy named cookieCut1.
bd3
apic1(config-tenant-interface)# ip igmp • Assigns the IGMP Snooping policy with
snooping policy cookieCut1 a modified IGMP Snooping value for the
policy cookieCut1.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

131
 Configuring Layer 2 External Connectivity
Enabling IGMP Snooping and Multicast on Static Ports in the NX-OS Style CLI

What to do next
You can assign the IGMP Snooping policy to multiple bridge domains.

Enabling IGMP Snooping and Multicast on Static Ports in the NX-OS Style CLI
You can enable IGMP snooping and multicast on ports that have been statically assigned to an EPG. Then
you can create and assign access groups of users that are permitted or denied access to the IGMP snooping
and multicast traffic enabled on those ports.
The steps described in this task assume the pre-configuration of the following entities:
• Tenant: tenant_A
• Application: application_A
• EPG: epg_A
• Bridge Domain: bridge_domain_A
• vrf: vrf_A -- a member of bridge_domain_A
• VLAN Domain: vd_A (configured with a range of 300-310)
• Leaf switch: 101 and interface 1/10
The target interface 1/10 on switch 101 is associated with VLAN 305 and statically linked with tenant_A,
application_A, epg_A
• Leaf switch: 101 and interface 1/11
The target interface 1/11 on switch 101 is associated with VLAN 309 and statically linked with tenant_A,
application_A, epg_A

Before you begin

Before you begin to enable IGMP snooping and multicasting for an EPG, complete the following tasks.
• Identify the interfaces to enable this function and statically assign them to that EPG

Note For details on static port assignment, see Deploying an EPG on a Specific Port
with APIC Using the NX-OS Style CLI in the Cisco APIC Layer 2 Networking
Configuration Guide.

• Identify the IP addresses that you want to be recipients of IGMP snooping multicast traffic.

Procedure

Command or Action Purpose

Step 1 On the target interfaces enable IGMP snooping The example sequences enable:
and layer 2 multicasting

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

132
 Configuring Layer 2 External Connectivity
Enabling IGMP Snoop Access Groups

Command or Action Purpose

Example: • IGMP snooping on the statically-linked
apic1# conf t target interface 1/10 and associates it with
apic1(config)# tenant tenant_A a multicast IP address, 225.1.1.1
apic1(config-tenant)# application
application_A • IGMP snooping on the statically-linked
apic1(config-tenant-app)# epg epg_A target interface 1/11 and associates it with
apic1(config-tenant-app-epg)# ip igmp
snooping static-group 225.1.1.1 leaf 101
a multicast IP address, 227.1.1.1
interface ethernet 1/10 vlan 305
apic1(config-tenant-app-epg)# end

apic1# conf t
apic1(config)# tenant tenant_A;
application application_A; epg epg_A
apic1(config-tenant-app-epg)# ip igmp
snooping static-group 227.1.1.1 leaf 101
interface ethernet 1/11 vlan 309
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)# exit

Enabling IGMP Snoop Access Groups

An “access-group” is used to control what streams can be joined behind a given port.
An access-group configuration can be applied on interfaces that are statically assigned to an application EPG
in order to ensure that the configuration can be applied on ports that will actually belong to the that EPG.
Only Route-map-based access groups are allowed.
IGMP snoop access groups can be configured through the APIC GUI, CLI, and REST API interfaces.

Enabling Group Access to IGMP Snooping and Multicast using the NX-OS
Style CLI
After you have enabled IGMP snooping and multicast on ports that have been statically assigned to an EPG,
you can then create and assign access groups of users that are permitted or denied access to the IGMP snooping
and multicast traffic enabled on those ports.
The steps described in this task assume the pre-configuration of the following entities:
• Tenant: tenant_A
• Application: application_A
• EPG: epg_A
• Bridge Domain: bridge_domain_A
• vrf: vrf_A -- a member of bridge_domain_A
• VLAN Domain: vd_A (configured with a range of 300-310)
• Leaf switch: 101 and interface 1/10
The target interface 1/10 on switch 101 is associated with VLAN 305 and statically linked with tenant_A,
application_A, epg_A

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

133
 Configuring Layer 2 External Connectivity
Enabling Group Access to IGMP Snooping and Multicast using the NX-OS Style CLI

• Leaf switch: 101 and interface 1/11

The target interface 1/11 on switch 101 is associated with VLAN 309 and statically linked with tenant_A,
application_A, epg_A

Note For details on static port assignment, see Deploying an EPG on a Specific Port with APIC Using the NX-OS
Style CLI in the Cisco APIC Layer 2 Networking Configuration Guide.

Procedure

Command or Action Purpose

Step 1 Define the route-map "access groups." The example sequences configure:
Example: • Route-map-access group "foobroker"
apic1# conf t linked to multicast group 225.1.1.1/24,
apic1(config)# tenant tenant_A; access permited
application application_A; epg epg_A
apic1(config-tenant)# route-map fooBroker • Route-map-access group "foobroker"
permit linked to multicast group 227.1.1.1/24,
apic1(config-tenant-rtmap)# match ip
multicast group 225.1.1.1/24
access denied
apic1(config-tenant-rtmap)# exit

apic1(config-tenant)# route-map fooBroker

deny
apic1(config-tenant-rtmap)# match ip
multicast group 227.1.1.1/24
apic1(config-tenant-rtmap)# exit

Step 2 Verify route map configurations.

Example:
apic1(config-tenant)# show running-config
tenant test route-map fooBroker
# Command: show running-config tenant
test route-map fooBroker
# Time: Mon Aug 29 14:34:30 2016
tenant test
route-map fooBroker permit 10
match ip multicast group
225.1.1.1/24
exit
route-map fooBroker deny 20
match ip multicast group
227.1.1.1/24
exit
exit

Step 3 Specify the access group connection path. The example sequences configure:
Example: • Route-map-access group "foobroker"
apic1(config-tenant)# application connected through leaf switch 101,
application_A interface 1/10, and VLAN 305.
apic1(config-tenant-app)# epg epg_A

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

134
 Configuring Layer 2 External Connectivity
Deploying an EPG on a Specific Port with APIC Using the NX-OS Style CLI

Command or Action Purpose

apic1(config-tenant-app-epg)# ip igmp • Route-map-access group "newbroker"
snooping access-group route-map fooBroker
connected through leaf switch 101,
leaf 101 interface ethernet 1/10 vlan
305 interface 1/10, and VLAN 305.
apic1(config-tenant-app-epg)# ip igmp
snooping access-group route-map newBroker
leaf 101 interface ethernet 1/10 vlan
305

Step 4 Verify the access group connections.

Example:
apic1(config-tenant-app-epg)# show run
# Command: show running-config tenant
tenant_A application application_A epg
epg_A
# Time: Mon Aug 29 14:43:02 2016
tenant tenent_A
application application_A
epg epg_A
bridge-domain member
bridge_domain_A

ip igmp snooping access-group

route-map fooBroker leaf 101 interface
ethernet 1/10 vlan 305
ip igmp snooping access-group
route-map fooBroker leaf 101 interface
ethernet 1/11 vlan 309
ip igmp snooping access-group
route-map newBroker leaf 101 interface
ethernet 1/10 vlan 305
ip igmp snooping static-group
225.1.1.1 leaf 101 interface ethernet
1/10 vlan 305
ip igmp snooping static-group
225.1.1.1 leaf 101 interface ethernet
1/11 vlan 309
exit
exit
exit

Deploying an EPG on a Specific Port with APIC Using the NX-OS Style CLI
Procedure

Step 1 Configure a VLAN domain:

Example:

apic1(config)# vlan-domain dom1

apic1(config-vlan)# vlan 10-100

Step 2 Create a tenant:

Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

135
 Configuring Layer 2 External Connectivity
Configuring Port Security

apic1# configure
apic1(config)# tenant t1

Step 3 Create a private network/VRF:

Example:

apic1(config-tenant)# vrf context ctx1

apic1(config-tenant-vrf)# exit

Step 4 Create a bridge domain:

Example:

apic1(config-tenant)# bridge-domain bd1

apic1(config-tenant-bd)# vrf member ctx1
apic1(config-tenant-bd)# exit

Step 5 Create an application profile and an application EPG:

Example:

apic1(config-tenant)# application AP1

apic1(config-tenant-app)# epg EPG1
apic1(config-tenant-app-epg)# bridge-domain member bd1
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)# exit
apic1(config-tenant)# exit

Step 6 Associate the EPG with a specific port:

Example:

apic1(config)# leaf 1017

apic1(config-leaf)# interface ethernet 1/13
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# switchport trunk allowed vlan 20 tenant t1 application AP1 epg EPG1

Note The vlan-domain and vlan-domain member commands mentioned in the above example are a
pre-requisite for deploying an EPG on a port.

Configuring Port Security

About Port Security and ACI
The port security feature protects the ACI fabric from being flooded with unknown MAC addresses by limiting
the number of MAC addresses learned per port. The port security feature support is available for physical
ports, port channels, and virtual port channels.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

136
 Configuring Layer 2 External Connectivity
Port Security Guidelines and Restrictions

Port Security Guidelines and Restrictions

The guidelines and restrictions are as follows:
• Port security is available per port.
• Port security is supported for physical ports, port channels, and virtual port channels (vPCs).
• Static and dynamic MAC addresses are supported.
• MAC address moves are supported from secured to unsecured ports and from unsecured ports to secured
ports.
• The MAC address limit is enforced only on the MAC address and is not enforced on a MAC and IP
address.
• Port security is not supported with the Fabric Extender (FEX).

Port Security at Port Level

In the APIC, the user can configure the port security on switch ports. Once the MAC limit has exceeded the
maximum configured value on a port, all traffic from the exceeded MAC addresses is forwarded. The following
attributes are supported:
• Port Security Timeout—The current supported range for the timeout value is from 60 to 3600 seconds.
• Violation Action—The violation action is available in protect mode. In the protect mode, MAC learning
is disabled and MAC addresses are not added to the CAM table. Mac learning is re-enabled after the
configured timeout value.
• Maximum Endpoints—The current supported range for the maximum endpoints configured value is
from 0 to 12000. If the maximum endpoints value is 0, the port security policy is disabled on that port.

Configuring a Port Security Policy Group Template

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1#
configure

Step 2 [no] template policy-group policy-group-name Creates (or deletes) a policy group template.
Example:
apic1(config)#
template policy-group
PortSecGrp1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

137
 Configuring Layer 2 External Connectivity
Configuring a Port Security Policy Group Template

Command or Action Purpose

Step 3 [no] switchport access vlan vlan-id tenant
tenant-name application application-name epg
epg-name
Example:

apic1(config-pol-grp-if)# switchport
access vlan 4 tenant ExampleCorp
application Web epg webEpg

Step 4 [no] switchport port-security maximum Sets the maximum number of secure MAC
number-of-addresses addresses for the port. The range is 0 to 12000
addresses. The default is 1 address.
Example:
apic1(config-pol-grp-if)#
switchport port-security maximum
1

Step 5 [no] switchport port-security violation Sets the action to be taken when a security
protect violation is detected. The protect action drops
packets with unknown source addresses until
Example:
you remove a sufficient number of secure MAC
apic1(config-pol-grp-if)# addresses to drop below the maximum value.
switchport port-security
violation protect

Step 6 exit Returns to global configuration mode.

Example:
apic1(config-pol-grp-if)#
exit

Example
This example shows how to create a port security policy group template.

apic1# configure
apic1(config)# template policy-group PortSecGrp1
apic1(config-pol-grp-if)# switchport port-security maximum 20
apic1(config-pol-grp-if)# switchport port-security violation protect
apic1(config-pol-grp-if)# exit

What to do next
Apply the port security template to an interface.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

138
 Configuring Layer 2 External Connectivity
Configuring Port Security on an Interface Using a Template

Configuring Port Security on an Interface Using a Template

Before you begin
Create a port security policy group template.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 interface type-or-range Specifies a port or a range of ports to be

configure.
Example:
apic1(config-leaf)# interface eth 1/2-4

Step 4 [no] policy-group policy-group-name Applies the policy group template to the port
or range of ports.
Example:
apic1(config-leaf-if)# policy-group
PortSecGrp1

Example
This example shows how to apply a port security policy group template to a range of Ethernet ports.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface eth 1/2-4
apic1(config-leaf-if)# policy-group PortSecGrp1

This example shows how to configure port security on a port channel using a template.

apic1# configure
apic1(config)# template port-channel po1
apic1(config-if)# switchport port-security maximum 10
apic1(config-if)# switchport port-security violation protect
apic1(config-if)# exit
apic1(config)# leaf 101
apic1(config-leaf)# interface eth 1/3-4
apic1(config-leaf-if)# channel-group po1
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

139
 Configuring Layer 2 External Connectivity
Configuring Port Security on an Interface Using Overrides

Configuring Port Security on an Interface Using Overrides

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 interface type-or-range Specifies an interface or a range of interfaces

to be configured.
Example:
apic1(config-leaf)# interface eth 1/2-4

Step 4 [no] switchport port-security maximum Sets the maximum number of secure MAC
number-of-addresses addresses for the interface. The range is 0 to
12000 addresses. The default is 1 address.
Example:
apic1(config-leaf-if)# switchport
port-security maximum 1

Step 5 [no] switchport port-security violation Sets the action to be taken when a security
protect violation is detected. The protect action drops
packets with unknown source addresses until
Example:
you remove a sufficient number of secure MAC
apic1(config-leaf-if)# switchport addresses to drop below the maximum value.
port-security violation protect

Example
This example shows how to configure port security on an Ethernet interface.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface eth 1/2
apic1(config-leaf-if)# switchport port-security maximum 10
apic1(config-leaf-if)# switchport port-security violation protect

This example shows how to configure port security on a port channel.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface port-channel po2
apic1(config-leaf-if)# switchport port-security maximum 10
apic1(config-leaf-if)# switchport port-security violation protect

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

140
 Configuring Layer 2 External Connectivity
802.1x Port and Node Authentication

This example shows how to configure port security on a virtual port channel (VPC).

apic1# configure
apic1(config)# vpc domain explicit 1 leaf 101 102
apic1(config-vpc)# exit
apic1(config)# template port-channel po4
apic1(config-if)# exit
apic1(config)# leaf 101-102
apic1(config-leaf)# interface eth 1/11-12
apic1(config-leaf-if)# channel-group po4 vpc
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit
apic1(config)# vpc context leaf 101 102
apic1(config-vpc)# interface vpc po4
apic1(config-vpc-if)# switchport port-security maximum 10
apic1(config-vpc-if)# switchport port-security violation protect

802.1x Port and Node Authentication

802.1x Port and Node Authentication
IEEE 802.1x is a port-based authentication mechanism to prevent unauthorized devices from gaining access
to the network. You can configure 802.1x port and node authentication using the NX-OS style CLI.

Configuring a Port Authentication Policy

Procedure

Step 1 In the CLI, enter configuration mode:

Example:
apic1# configure
apic1(config)#

Step 2 Create a policy group:

Example:
apic1(config)# template policy-group mypol

Step 3 Configure port-level authentication policy in the policy group:

Example:
apic1(config-pol-grp-if)# switchport port-authentication mydot1x

Step 4 Configure host mode (two modes are supported: multi-host and single-host - single being the default setting):
Example:
apic1(config-port-authentication)# host-mode multi-host

Step 5 Enable this policy (policy is disabled by default):

Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

141
 Configuring Layer 2 External Connectivity
Configuring a Node Authentication Policy

apic1(config-port-authentication)# no shutdown
apic1(config-port-authentication)# exit
apic1(config-pol-grp-if)# exit
apic1(config)#

Step 6 Configure the leaf interface profile:

Example:
apic1(config)#leaf-interface-profile myprofile

Step 7 Configure a policy group for the leaf switch interface profile:
Example:
apic1(config-leaf-if-profile)#leaf-interface-group mygroup

Step 8 Specify ports and/or interfaces for your interface group:

Example:
apic1(config-leaf-if-group)# interface ethernet 1/10-12

Step 9 Apply the policy on your interface group:

Example:
apic1(config-leaf-if-group)# policy-group mypol
apic1(config-leaf-if-group)# exit
apic1(config-leaf-if-profile)# exit

Step 10 Configure the leaf profile :

Example:
apic1(config)#
apic1(config)# leaf-profile myleafprofile

Step 11 Configure the leaf policy group and specify leaf switch nodes for the group:
Example:
apic1(config-leaf-profile)# leaf-group myleafgrp
apic1(config-leaf-group)# leaf 101
apic1(config-leaf-group)# exit

Step 12 Apply an interface policy on the leaf switch profile:

Example:
apic1(config-leaf-profile)# leaf-interface-profile myprofile
apic1(config-leaf-group)# exit
apic1(config)#

Configuring a Node Authentication Policy

Procedure

Step 1 In the CLI, enter configuration mode:

Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

142
Configuring Layer 2 External Connectivity
Configuring a Node Authentication Policy

apic1# configure
apic1(config)#

Step 2 Configure the radius authentication group:

Example:
apic1(config)# aaa group server radius myradiusgrp
apic1(config-radius)#server 192.168.0.100 priority 1
apic1(config-radius)#exit

Step 3 Configure node level port authentication policy:

Example:
apic1(config)# policy-map type port-authentication mydot1x
apic1(config-pmap-port-authentication)#radius-provider-group myradiusgrp

Step 4 [Optional] Override the defaul VLAN ID if authentication fails. :

Example:
apic1(config-pmap-port-authentication)#fail-auth-vlan 2001

Step 5 [Optional] Override defaul EPG if authentication fails:

Example:
apic1(config-pmap-port-authentication)#fail-auth-epg tenant tn1 application ap1 epg epg256
apic1(config)# exit

Step 6 Configure policy group and specify port authentication policy in the group:
Example:
apic1(config)#template leaf-policy-group lpg2
apic1(config-leaf-policy-group)# port-authentication mydot1x
apic1(config-leaf-policy-group)#exit

Step 7 Configure the leaf switch profile:

Example:
apic1(config)# leaf-profile mylp2

Step 8 Configure a group for the leaf switch profile and specify the policy group:
Example:
apic1(config-leaf-profile)#leaf-group mylg2
apic1(config-leaf-group)# leaf-policy-group lpg2
apic1(config-leaf-group)#exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

143
 Configuring Layer 2 External Connectivity
Configuring Proxy ARP

Configuring Proxy ARP

About Proxy ARP
Proxy ARP in Cisco ACI enables endpoints within a network or subnet to communicate with other endpoints
without knowing the real MAC address of the endpoints. Proxy ARP is aware of the location of the traffic
destination, and offers its own MAC address as the final destination instead.
To enable Proxy ARP, intra-EPG endpoint isolation must be enabled on the EPG see the following figure for
details. For more information about intra-EPG isolation and Cisco ACI, see the Cisco ACI Virtualization
Guide.
Figure 7: Proxy ARP and Cisco APIC

Proxy ARP within the Cisco ACI fabric is different from the traditional proxy ARP. As an example of the
communication process, when proxy ARP is enabled on an EPG, if an endpoint A sends an ARP request for
endpoint B and if endpoint B is learned within the fabric, then endpoint A will receive a proxy ARP response
from the bridge domain (BD) MAC. If endpoint A sends an ARP request for endpoint B, and if endpoint B
is not learned within the ACI fabric already, then the fabric will send a proxy ARP request within the BD.
Endpoint B will respond to this proxy ARP request back to the fabric. At this point, the fabric does not send
a proxy ARP response to endpoint A, but endpoint B is learned within the fabric. If endpoint A sends another
ARP request to endpoint B, then the fabric will send a proxy ARP response from the BD MAC.
The following example describes the proxy ARP resolution steps for communication between clients VM1
and VM2:
1. VM1 to VM2 communication is desired.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

144
Configuring Layer 2 External Connectivity
About Proxy ARP

Figure 8: VM1 to VM2 Communication is Desired.

Table 12: ARP Table State

Device State

VM1 IP = * MAC = *

ACI fabric IP = * MAC = *

VM2 IP = * MAC = *

2. VM1 sends an ARP request with a broadcast MAC address to VM2.

Figure 9: VM1 sends an ARP Request with a Broadcast MAC address to VM2

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

145
 Configuring Layer 2 External Connectivity
About Proxy ARP

Table 13: ARP Table State

Device State

VM1 IP = VM2 IP; MAC = ?

ACI fabric IP = VM1 IP; MAC = VM1 MAC

VM2 IP = * MAC = *

3. The ACI fabric floods the proxy ARP request within the bridge domain (BD).
Figure 10: ACI Fabric Floods the Proxy ARP Request within the BD

Table 14: ARP Table State

Device State

VM1 IP = VM2 IP; MAC = ?

ACI fabric IP = VM1 IP; MAC = VM1 MAC

VM2 IP = VM1 IP; MAC = BD MAC

4. VM2 sends an ARP response to the ACI fabric.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

146
Configuring Layer 2 External Connectivity
About Proxy ARP

Figure 11: VM2 Sends an ARP Response to the ACI Fabric

Table 15: ARP Table State

Device State

VM1 IP = VM2 IP; MAC = ?

ACI fabric IP = VM1 IP; MAC = VM1 MAC

VM2 IP = VM1 IP; MAC = BD MAC

5. VM2 is learned.
Figure 12: VM2 is Learned

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

147
 Configuring Layer 2 External Connectivity
About Proxy ARP

Table 16: ARP Table State

Device State

VM1 IP = VM2 IP; MAC = ?

ACI fabric IP = VM1 IP; MAC = VM1 MAC

IP = VM2 IP; MAC = VM2 MAC

VM2 IP = VM1 IP; MAC = BD MAC

6. VM1 sends an ARP request with a broadcast MAC address to VM2.

Figure 13: VM1 Sends an ARP Request with a Broadcast MAC Address to VM2

Table 17: ARP Table State

Device State

VM1 IP = VM2 IP MAC = ?

ACI fabric IP = VM1 IP; MAC = VM1 MAC

IP = VM2 IP; MAC = VM2 MAC

VM2 IP = VM1 IP; MAC = BD MAC

7. The ACI fabric sends a proxy ARP response to VM1.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

148
 Configuring Layer 2 External Connectivity
Guidelines and Limitations

Figure 14: ACI Fabric Sends a Proxy ARP Response to VM1

Table 18: ARP Table State

Device State

VM1 IP = VM2 IP; MAC = BD MAC

ACI fabric IP = VM1 IP; MAC = VM1 MAC

IP = VM2 IP; MAC = VM2 MAC

VM2 IP = VM1 IP; MAC = BD MAC

Guidelines and Limitations

Consider these guidelines and limitations when using Proxy ARP:
• Proxy ARP is supported only on isolated EPGs. If an EPG is not isolated, a fault will be raised. For
communication to happen within isolated EPGs with proxy ARP enabled, you must configure uSeg
EPGs. For example, within the isolated EPG, there could be multiple VMs with different IP addresses,
and you can configure a uSeg EPG with IP attributes matching the IP address range of these VMs.
• ARP requests from isolated endpoints to regular endpoints and from regular endpoints to isolated endpoints
do not use proxy ARP. In such cases, endpoints communicate using the real MAC addresses of destination
VMs.

Configuring Proxy ARP Using the Cisco NX-OS Style CLI

Before you begin
• The appropriate tenant, VRF, bridge domain, application profile and EPG must be created.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

149
 Configuring Layer 2 External Connectivity
Configuring Proxy ARP Using the Cisco NX-OS Style CLI

• Intra-EPG isolation must be enabled on the EPG where proxy ARP has to be enabled.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 tenant tenant-name Enters the tenant configuration mode.

Example:
apic1(config)# tenant Tenant1

Step 3 application application-profile-name Creates an application profile and enters the

application mode.
Example:

apic1(config-tenant)# application
Tenant1-App

Step 4 epg application-profile-EPG-name Creates an EPG and enter the EPG mode.
Example:

apic1(config-tenant-app)# epg
Tenant1-epg1

Step 5 proxy-arp enable Enables proxy ARP.

Example: Note You can disable proxy-arp with the
apic1(config-tenant-app-epg)# proxy-arp no proxy-arp command.
enable

Step 6 exit Returns to application profile mode.

Example:
apic1(config-tenant-app-epg)# exit

Step 7 exit Returns to tenant configuration mode.

Example:
apic1(config-tenant-app)# exit

Step 8 exit Returns to global configuration mode.

Example:
apic1(config-tenant)# exit

Examples
This example shows how to configure proxy ARP.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

150
 Configuring Layer 2 External Connectivity
Configuring Flood in Encapsulation

apic1# conf t
apic1(config)# tenant Tenant1
apic1(config-tenant)# application Tenant1-App
apic1(config-tenant-app)# epg Tenant1-epg1
apic1(config-tenant-app-epg)# proxy-arp enable
apic1(config-tenant-app-epg)#
apic1(config-tenant)#

Configuring Flood in Encapsulation

The configuration for Layer 2 external connectivity is similar to a static application EPG, where you map a
VLAN on a node port to an EPG and map the EPG to a bridge-domain to provide/consume contracts.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 tenant tenant-name Enters the tenant configuration mode.

Example:
apic1(config)# tenant Tenant1

Step 3 application application-profile-name Creates an application profile and enters the

application mode.
Example:
apic1(config)# application Tenant1-App

Step 4 epg application-profile-EPG-name Creates an EPG and enter the EPG mode.
Example:
apic1(config)# epg Tenant1-epg1

Step 5 flood-on-encapsulation enable Enables flood-on-encapsulation.

Example:
apic1(config-tenant-app-epg)#
flood-on-encapsulation enable

Step 6 exit Returns to application profile mode.

Example:
apic1(config-tenant-app-epg)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

151
 Configuring Layer 2 External Connectivity
Configuring Traffic Storm Control

Configuring Traffic Storm Control

About Traffic Storm Control
A traffic storm occurs when packets flood the LAN, creating excessive traffic and degrading network
performance. You can use traffic storm control policies to prevent disruptions on Layer 2 ports by broadcast,
unknown multicast, or unknown unicast traffic storms on physical interfaces.
By default, storm control is not enabled in the ACI fabric. ACI bridge domain (BD) Layer 2 unknown unicast
flooding is enabled by default within the BD but can be disabled by an administrator. In that case, a storm
control policy only applies to broadcast and unknown multicast traffic. If Layer 2 unknown unicast flooding
is enabled in a BD, then a storm control policy applies to Layer 2 unknown unicast flooding in addition to
broadcast and unknown multicast traffic.
Traffic storm control (also called traffic suppression) allows you to monitor the levels of incoming broadcast,
multicast, and unknown unicast traffic over a one second interval. During this interval, the traffic level, which
is expressed either as percentage of the total available bandwidth of the port or as the maximum packets per
second allowed on the given port, is compared with the traffic storm control level that you configured. When
the ingress traffic reaches the traffic storm control level that is configured on the port, traffic storm control
drops the traffic until the interval ends. An administrator can configure a monitoring policy to raise a fault
when a storm control threshold is exceeded.

Storm Control Guidelines

Configure traffic storm control levels according to the following guidelines and limitations:
• Typically, a fabric administrator configures storm control in fabric access policies on the following
interfaces:
• A regular trunk interface.
• A direct port channel on a single leaf switch.
• A virtual port channel (a port channel on two leaf switches).

• Beginning with the APIC Release 4.2(1), support is now available for triggering SNMP traps from Cisco
ACI when storm control thresholds are met, with the following restrictions:
• There are two actions associated with storm control: drop and shutdown. With the shutdown action,
interface traps will be raised, but the storm control traps to indicate that the storm is active or clear
is not determined by the shutdown action. Storm control traps with the shutdown action on the
policy should therefore be ignored.
• If the ports flap with the storm control policy on, clear and active traps are seen together when the
stats are collected. Clear and active traps are typically not seen together, but this is expected behavior
in this case.

• For port channels and virtual port channels, the storm control values (packets per second or percentage)
apply to all individual members of the port channel. Do not configure storm control on interfaces that
are members of a port channel.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

152
Configuring Layer 2 External Connectivity
Storm Control Guidelines

Note On switch hardware starting with the APIC 1.3(x) and switch 11.3(x) release, for
port channel configurations, the traffic suppression on the aggregated port may
be up to two times the configured value. The new hardware ports are internally
subdivided into these two groups: slice-0 and slice-1. To check the slicing map,
use the vsh_lc command show platform internal hal l2 port gpd and look
for slice 0 or slice 1 under the Sl column. If port-channel members fall on
both slice-0 and slice-1, allowed storm control traffic may become twice the
configured value because the formula is calculated based on each slice.

• When configuring by percentage of available bandwidth, a value of 100 means no traffic storm control
and a value of 0.01 suppresses all traffic.
• Due to hardware limitations and the method by which packets of different sizes are counted, the level
percentage is an approximation. Depending on the sizes of the frames that make up the incoming traffic,
the actual enforced level might differ from the configured level by several percentage points.
Packets-per-second (PPS) values are converted to percentage based on 256 bytes.
• Maximum burst is the maximum accumulation of rate that is allowed when no traffic passes. When traffic
starts, all the traffic up to the accumulated rate is allowed in the first interval. In subsequent intervals,
traffic is allowed only up to the configured rate. The maximum supported is 65535 KB. If the configured
rate exceeds this value, it is capped at this value for both PPS and percentage.
• The maximum burst that can be accumulated is 512 MB.
• On an egress leaf switch in optimized multicast flooding (OMF) mode, traffic storm control will not be
applied.
• On an egress leaf switch in non-OMF mode, traffic storm control will be applied.
• On a leaf switch for FEX, traffic storm control is not available on host-facing interfaces.
• Traffic storm control unicast/multicast differentiation is not supported on Cisco Nexus C93128TX,
C9396PX, C9396TX, C93120TX, C9332PQ, C9372PX, C9372TX, C9372PX-E, or C9372TX-E switches.
• SNMP traps for traffic storm control are not supported on Cisco Nexus C93128TX, C9396PX, C9396TX,
C93120TX, C9332PQ, C9372PX, C9372TX, C9372PX-E, C9372TX-E switches.
• Traffic storm control traps is not supported on Cisco Nexus C93128TX, C9396PX, C9396TX, C93120TX,
C9332PQ, C9372PX, C9372TX, C9372PX-E, or C9372TX-E switches.
• Storm Control Action is supported only on physical Ethernet interfaces and port-channel interfaces.
Starting with release 4.1(1), Storm Control Shutdown option is supported. When the shutdown action
is selected for an interface with the default Soak Instance Count, the packets exceeding the threshold are
dropped for 3 seconds and the port is shutdown on the 3rd second. The default action is Drop. When
Shutdown action is selected, the user has the option to specify the soaking interval. The default soaking
interval is 3 seconds. The configurable range is from 3 to 10 seconds.
• Starting with release 4.1(1), Error Disable Recovery option for storm-control is supported for the ports
which are in-error-disabled state due to storm shutdown action.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

153
 Configuring Layer 2 External Connectivity
Configuring a Traffic Storm Control Policy Using the NX-OS Style CLI

Configuring a Traffic Storm Control Policy Using the NX-OS Style CLI
Procedure

Command or Action Purpose

Step 1 Enter the following commands to create a PPS
policy:
Example:
(config)# template policy-group pg1
(config-pol-grp-if)# storm-control pps
10000 burst-rate 10000

Step 2 Enter the following commands to create a

percent policy:
Example:
(config)# template policy-group pg2
(config-pol-grp-if)# storm-control level
50 burst-rate 60

Step 3 Configure storm control on physical ports, port

channels, or virtual port channels:
Example:
[no] storm-control
[unicast|multicast|broadcast] level
<percentage> [burst-rate <percentage>]
[no] storm-control
[unicast|multicast|broadcast] pps
<packet-per-second> [burst-rate
<packet-per-second>]

sd-tb2-ifc1# configure terminal

sd-tb2-ifc1(config)# leaf 102

sd-tb2-ifc1(config-leaf)# interface
ethernet 1/19
sd-tb2-ifc1(config-leaf-if)#
storm-control unicast level 35 burst-rate
45
sd-tb2-ifc1(config-leaf-if)#
storm-control broadcast level 36
burst-rate 36
sd-tb2-ifc1(config-leaf-if)#
storm-control broadcast level 37
burst-rate 38
sd-tb2-ifc1(config-leaf-if)#

sd-tb2-ifc1# configure terminal

sd-tb2-ifc1(config)# leaf 102

sd-tb2-ifc1(config-leaf)# interface
ethernet 1/19
sd-tb2-ifc1(config-leaf-if)#
storm-control broadcast pps 5000
burst-rate 6000

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

154
 Configuring Layer 2 External Connectivity
Configuring MACsec

Command or Action Purpose

sd-tb2-ifc1(config-leaf-if)#
storm-control unicast pps 7000 burst-rate
7000
sd-tb2-ifc1(config-leaf-if)#
storm-control unicast pps 8000 burst-rate
10000
sd-tb2-ifc1(config-leaf-if)#

Configuring MACsec
About MACsec
MACsec is an IEEE 802.1AE standards based Layer 2 hop-by-hop encryption that provides data confidentiality
and integrity for media access independent protocols.
MACsec, provides MAC-layer encryption over wired networks by using out-of-band methods for encryption
keying. The MACsec Key Agreement (MKA) Protocol provides the required session keys and manages the
required encryption keys.
The 802.1AE encryption with MKA is supported on all types of links, that is, host facing links (links between
network access devices and endpoint devices such as a PC or IP phone), or links connected to other switches
or routers.
MACsec encrypts the entire data except for the Source and Destination MAC addresses of an Ethernet packet.
The user also has the option to skip encryption up to 50 bytes after the source and destination MAC address.
To provide MACsec services over the WAN or Metro Ethernet, service providers offer Layer 2 transparent
services such as E-Line or E-LAN using various transport layer protocols such as Ethernet over Multiprotocol
Label Switching (EoMPLS) and L2TPv3.
The packet body in an EAP-over-LAN (EAPOL) Protocol Data Unit (PDU) is referred to as a MACsec Key
Agreement PDU (MKPDU). When no MKPDU is received from a participants after 3 hearbeats (each hearbeat
is of 2 seconds), peers are deleted from the live peer list. For example, if a client disconnects, the participant
on the switch continues to operate MKA until 3 heartbeats have elapsed after the last MKPDU is received
from the client.

APIC Fabric MACsec

The APIC will be responsible for the MACsec keychain distribution to all the nodes in a Pod or to particular
ports on a node. Below are the supported MACsec keychain and MACsec policy distribution supported by
the APIC.
• A single user provided keychain and policy per Pod
• User provided keychain and user provided policy per fabric interface
• Auto generated keychain and user provided policy per Pod

A node can have multiple policies deployed for more than one fabric link. When this happens, the per fabric
interface keychain and policy are given preference on the affected interface. The auto generated keychain and
associated MACsec policy are then given the least preference.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

155
 Configuring Layer 2 External Connectivity
Guidelines and Limitations for MACsec

APIC MACsec supports two security modes. The MACsec must secure only allows encrypted traffic on the
link while the should secure allows both clear and encrypted traffic on the link. Before deploying MACsec
in must secure mode, the keychain must be deployed on the affected links or the links will go down. For
example, a port can turn on MACsec in must secure mode before its peer has received its keychain resulting
in the link going down. To address this issue the recommendation is to deploy MACsec in should secure
mode and once all the links are up then change the security mode to must secure.

Note Any MACsec interface configuration change will result in packet drops.

MACsec policy definition consists of configuration specific to keychain definition and configuration related
to feature functionality. The keychain definition and feature functionality definitions are placed in separate
policies. Enabling MACsec per Pod or per interface involves deploying a combination of a keychain policy
and MACsec functionality policy.

Note Using internal generated keychains do not require the user to specify a keychain.

APIC Access MACsec

MACsec is used to secure links between leaf switch L3out interfaces and external devices. APIC provides
GUI and CLI to allow users to program the MACsec keys and MacSec configuration for the L3Out interfaces
on the fabric on a per physical/pc/vpc interface basis. It is the responsibility of the user to make sure that the
external peer devices are programmed with the correct MacSec information.

Guidelines and Limitations for MACsec

Configure MACsec according to the following guidelines and limitations:
• Beginning with Cisco ACI Release 4.0, MACsec is supported on remote leaf switches.
• Fex ports are not supported for MACsec.
• Must-secure mode is not supported at Pod level.
• A MACsec policy with name ‘default’ is not supported.
• Auto key generation is only supported at the Pod level for fabric ports.
• Do not clean reboot a node if the fabric ports of that node is running MACsec in must-secure mode.
• Adding a new node to a Pod or stateless reboot of a node in a Pod which is running MACsec, must-secure
mode requires changing the mode to should-secure in order for the node to join the Pod.
• Upgrade/Downgrade should only be initiated if the fabric links are in should-secure mode. Once
upgrade/downgrade has completed, then the mode can be changed to must-secure.
Upgrading/Downgrading in must-secure mode will result in nodes losing connectivity to the fabric.
Recovering from connectivity loss requires that the fabric links of the nodes visible to the APIC be
configured to should-secure mode. If the fabric was downgraded to a version which does not support
MACsec, then nodes which are out of fabric will need to be clean rebooted.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

156
Configuring Layer 2 External Connectivity
Guidelines and Limitations for MACsec

• For PC/vPC interface, MACsec can be deployed via policy groups per PC/vPC interface. Port selectors
are used to deploy the policies to a particular set of ports. Therefore, it is the user’s responsibility to
create the right port selector corresponding to the L3Out interfaces.
• It is recommended that MACsec polices be configured to should-secure mode before a configuration is
exported.
• All the links on a spine are considered fabric links. However, if a spine link is used for IPN connectivity,
then this link will be treated as an access link. This means that MACsec access policy needs to be used
to deploy MACsec on these links.
• If a remote leaf fabric link is used for IPN connectivity, then this link will be treated as an access link.
A MACsec access policy needs to be used to deploy MACsec on these links.
• Improper deployment of must-secure mode on remote leaf fabric links can result in loss of connectivity
to the fabric. Follow the instructions provided in Deploying must-secure mode, on page 157 to prevent
such issues.
• MACSEC Sessions may take up to a minute to form or tear down when a new key is added to an empty
keychain or an active key is deleted from keychain.

Deploying must-secure mode

Incorrect deployment procedure of a policy that is configured for must-secure mode can result in a loss of
connectivity. The procedure below should be followed in order to prevent such issues:
• It is necessary to ensure that each link pair has their keychains before enabling MACsec must-secure
mode. To ensure this, the recommendation is to deploy the policy in should-secure mode, and once
MACsec sessions are active on the expected links, change the mode to must-secure.
• Attempting to replace the keychain on a MACsec policy that is configured to must-secure can cause
links to go down. The recommended procedure outlined below should be followed in this case:
• Change MACsec policy that is using the new keychain to should-secure mode.
• Verify that the affected interfaces are using should-secure mode.
• Update MACsec policy to use new keychain.
• Verify that relevant interfaces with active MACsec sessions are using the new keychain.
• Change MACsec policy to must-secure mode.

• The following procedure should be followed to disable/remove a MACsec policy deployed in must-secure
mode:
• Change the MACsec policy to should-secure.
• Verify that the affected interfaces are using should-secure mode.
• Disable/remove the MACsec policy.

Keychain Definition
• There should be one key in the keychain with a start time of now. If must-secure is deployed with a
keychain that doesn’t have a key that is immediately active then traffic will be blocked on that link until

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

157
 Configuring Layer 2 External Connectivity
Configuring MACsec Using the NX-OS Style CLI

the key becomes current and a MACsec session is started. If should-secure mode is being used then
traffic will be unencrypted until the key becomes current and a MACsec session has started.
• There should be one key in the keychain with an end time of infinite. When a keychain expires, then
traffic is blocked on affected interfaces which are configured for must-secure mode. Interfaces configured
for should-secure mode transmit unencrypted traffic.
• There should be overlaps in the end time and start time of keys that are used sequentially to ensure the
MACsec session stays up when there is a transition between keys.

Configuring MACsec Using the NX-OS Style CLI

Procedure

Step 1 Configure MACsec Security Policy for access interfaces

Example:
apic1# configure
apic1(config)# template macsec access security-policy accmacsecpol1
apic1(config-macsec-param)# cipher-suite gcm-aes-128
apic1(config-macsec-param)# conf-offset offset-30
apic1(config-macsec-param)# description 'description for mac sec parameters'
apic1(config-macsec-param)# key-server-priority 1
apic1(config-macsec-param)# sak-expiry-time 110
apic1(config-macsec-param)# security-mode must-secure
aapic1(config-macsec-param)# window-size 1
apic1(config-macsec-param)# exit
apic1(config)#

Step 2 Configure MACsec key chain for access interface:

PSK can be configured in 2 ways:
Note • Inline with the psk-string command as illustrated in key 12ab below. The PSK is not secure
because it is logged and exposed.
• Entered separately in a new command Enter PSK string after the psk-string command as
illustrated in key ab12. The PSK is secured because it is only echoed locally and is not logged.

Example:
apic1# configure
apic1(config)# template macsec access keychain acckeychainpol1
apic1(config-macsec-keychain)# description 'macsec key chain kc1'
apic1(config-macsec-keychain)# key 12ab
apic1(config-macsec-keychain-key)# life-time start 2017-09-19T12:03:15 end
2017-12-19T12:03:15
apic1(config-macsec-keychain-key)# psk-string 123456789a223456789a323456789abc
apic1(config-macsec-keychain-key)# exit
apic1(config-macsec-keychain)# key ab12
apic1(config-macsec-keychain-key)# life-time start now end infinite
apic1(config-macsec-keychain-key)# life-time start now end infinite
apic1(config-macsec-keychain-key)# psk-string
Enter PSK string: 123456789a223456789a323456789abc
apic1(config-macsec-keychain-key)# exit
apic1(config-macsec-keychain)# exit
apic1(config)#

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

158
Configuring Layer 2 External Connectivity
Configuring MACsec Using the NX-OS Style CLI

Step 3 Configure MACsec interface policy for access interface:

Example:
apic1# configure
apic1(config)# template macsec access interface-policy accmacsecifpol1
apic1(config-macsec-if-policy)# inherit macsec security-policy accmacsecpol1 keychain
acckeychainpol1
apic1(config-macsec-if-policy)# exit
apic1(config)#

Step 4 Associate MACsec interface policy to access interfaces on leaf (or spine):
Example:
apic1# configure
apic1(config)# template macsec access interface-policy accmacsecifpol1
apic1(config-macsec-if-policy)# inherit macsec security-policy accmacsecpol1 keychain
acckeychainpol1
apic1(config-macsec-if-policy)# exit
apic1(config)

Step 5 Configure MACsec Security Policy for fabric interfaces:

Example:
apic1# configure
apic1(config)# template macsec fabric security-policy fabmacsecpol1
apic1(config-macsec-param)# cipher-suite gcm-aes-xpn-128
apic1(config-macsec-param)# description 'description for mac sec parameters'
apic1(config-macsec-param)# window-size 1
apic1(config-macsec-param)# sak-expiry-time 100
apic1(config-macsec-param)# security-mode must-secure
apic1(config-macsec-param)# exit
apic1(config)#

Step 6 Configure MACsec key chain for fabric interface:

PSK can be configured in 2 ways:
Note • Inline with the psk-string command as illustrated in key 12ab below. The PSK is not secure
because it is logged and exposed.
• Entered separately in a new command Enter PSK string after the psk-string command as
illustrated in key ab12. The PSK is secured because it is only echoed locally and is not logged.

Example:
apic1# configure
apic1(config)# template macsec fabric security-policy fabmacsecpol1
apic1(config-macsec-param)# cipher-suite gcm-aes-xpn-128
apic1(config-macsec-param)# description 'description for mac sec parameters'
apic1(config-macsec-param)# window-size 1
apic1(config-macsec-param)# sak-expiry-time 100
apic1(config-macsec-param)# security-mode must-secure
apic1(config-macsec-param)# exit
apic1(config)# template macsec fabric keychain fabkeychainpol1
apic1(config-macsec-keychain)# description 'macsec key chain kc1'
apic1(config-macsec-keychain)# key 12ab
apic1(config-macsec-keychain-key)# psk-string 123456789a223456789a323456789abc
apic1(config-macsec-keychain-key)# life-time start 2016-09-19T12:03:15 end
2017-09-19T12:03:15
apic1(config-macsec-keychain-key)# exit
apic1(config-macsec-keychain)# key cd78
apic1(config-macsec-keychain-key)# psk-string

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

159
 Configuring Layer 2 External Connectivity
Configuring MACsec Using the NX-OS Style CLI

Enter PSK string: 123456789a223456789a323456789abc

apic1(config-macsec-keychain-key)# life-time start now end infinite
apic1(config-macsec-keychain-key)# exit
apic1(config-macsec-keychain)# exit
apic1(config)#

Step 7 Associate MACsec interface policy to fabric interfaces on leaf (or spine):
Example:
apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# fabric-interface ethernet 1/52-53
apic1(config-leaf-if)# inherit macsec interface-policy fabmacsecifpol2
apic1(config-leaf-if)# exit
apic1(config-leaf)#

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

160
 CHAPTER 7
Configuring Layer 3 External Connectivity
• About the Modes of Configuring Layer 3 External Connectivity, on page 161
• Configuring Layer 3 External Connectivity, on page 163
• Routed Connectivity to External Networks, on page 163
• Layer 3 Routed and Sub-Interface Port Channels, on page 175
• Layer 3 Out to Layer 3 Out Inter-VRF Leaking, on page 181
• About SVI External Encapsulation Scope, on page 185
• About SVI Auto State , on page 188
• Configuring an Interface and Static Route , on page 190
• OSPF Configuration, on page 193
• BGP Configuration, on page 200
• EIGRP Configuration, on page 217
• Configuring Route-Maps, on page 224
• Configuring Bi-Directional Route Forwarding (BFD), on page 234
• Configuring Layer 3 Multicast, on page 249
• Configuring External-L3 EPGs, on page 264
• Configuring Layer 3 External Connectivity Using the Named Mode, on page 266
• IPv6 Neighbor Discovery, on page 280
• Microsoft NLB, on page 285
• MLD Snooping, on page 288
• Configuring HSRP, on page 291
• Cisco ACI GOLF, on page 294
• Multipod_Fabric, on page 311
• Remote Leaf Switches, on page 318
• Transit Routing, on page 325

About the Modes of Configuring Layer 3 External Connectivity

Because APIC supports multiple user interfaces (UIs) for configuration, the potential exists for unintended
interactions when you create a configuration with one UI and later modify the configuration with another UI.
This section describes considerations for configuring Layer 3 external connectivity with the APIC NX-OS
style CLI, when you may also be using other APIC user interfaces.
When you configure Layer 3 external connectivity with the APIC NX-OS style CLI, you have the choice of
two modes:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

161
 Configuring Layer 3 External Connectivity
About the Modes of Configuring Layer 3 External Connectivity

• Implicit mode, a simpler mode, is not compatible with the APIC GUI or the REST API.
• Named (or Explicit) mode is compatible with the APIC GUI and the REST API.

In either case, the configuration should be considered read-only in the incompatible UI.

How the Modes Differ

In both modes, the configuration settings are defined within an internal container object, the "L3 Outside" (or
"L3Out"), which is an instance of the l3extOut class in the API. The main difference between the two modes
is in the naming of this container object instance:
• Implicit mode—the naming of the container is implicit and does not appear in the CLI commands. The
CLI creates and maintains these objects internally.
• Named mode—the naming is provided by the user. CLI commands in the Named Mode have an additional
l3Out field. To configure the named L3Out correctly and avoid faults, the user is expected to understand
the API object model for external Layer 3 configuration.

Note Except for the procedures in the Configuring Layer 3 External Connectivity Using the Named Mode section,
this guide describes Implicit mode procedures.

Guidelines and Restrictions

• In the same APIC instance, both modes can be used together for configuring Layer 3 external connectivity
with the following restriction: The Layer 3 external connectivity configuration for a given combination
of tenant, VRF, and leaf can be done only through one mode.
• For a given tenant VRF, the policy domain where the External-l3 EPG can be placed can be in either the
Named mode or in the Implicit mode. The recommended configuration method is to use only one mode
for a given tenant VRF combination across all the nodes where the given tenant VRF is deployed for
Layer 3 external connectivity. The modes can be different across different tenants or different VRFs and
no restrictions apply.
• In some cases, an incoming configuration to a Cisco APIC cluster will be validated against inconsistencies,
where the validations involve externally-visible configurations (northbound traffic through the L3Outs).
An Invalid Configuration error message will appear for those situations where the configuration is invalid.
• The external Layer 3 features are supported in both configuration modes, with the following exception:
• Route-peering and Route Health Injection (RHI) with a L4-L7 Service Appliance is supported only
in the Named mode. The Named mode should be used across all border leaf switches for the tenant
VRF where route-peering is involved.

• Layer 3 external network objects (l3extOut) created using the Implicit mode CLI procedures are identified
by names starting with “__ui_” and are marked as read-only in the GUI. The CLI partitions these
external-l3 networks by function, such as interfaces, protocols, route-map, and EPG. Configuration
modifications performed through the REST API can break this structure, preventing further modification
through the CLI.

For the steps to remove such objects, see Troubleshooting Unwanted _ui_ Objects in the APIC Troubleshooting
Guide.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

162
 Configuring Layer 3 External Connectivity
Configuring Layer 3 External Connectivity

Configuring Layer 3 External Connectivity

Configuration of layer 3 (L3) routing connectivity to an external network consists of the following components:
• Interface—Interface configuration for layer 3 ports, sub-interfaces, external SVI that are used to connect
to external routers.
• Routing Protocol Configuration—CLI supports static route, BGP, OSPF, EIGRP protocol configuration.
• Route-map control—A route map is used to match prefixes/BD public subnets and apply route-control
policies. Once created, it can be associated with routing protocols in a direction, such as “in” (BGP or
OSPF), “out”(BGP, OSPF, EIGRP).
Configurations pertaining to interface, routing protocols, and route-maps are maintained per leaf switch
under the config-leaf configuration mode.
• External-L3 EPG—A list of external subnets on a tenant VRF that are classified as one endpoint group
for applying contract and QoS policies. External-L3 EPGs (also called prefix EPGs) can have contracts
with other external-L3 EPGs and application EPGs. External-L3 EPG configuration is maintained under
tenant configuration. The external-L3 EPGs can be deployed on a subset of nodes where the VRF is
configured.

The steps for configuring layer 3 external connectivity can be summarized as follows:
1. Create a VRF under a tenant.
2. Configure and deploy the VRF on the border leaf switch.
3. Configure layer 3 interfaces on the border leaf Interfaces.
4. Configure route-maps on the leaf switch.
5. Configure routing protocols (BGP, OSPF, EIGRP) under leaf and leaf-interface.
6. Create and configure an external-L3 EPG under a tenant.
7. Deploy the external-L3 EPG on the border leaf switch.

Routed Connectivity to External Networks

About Routed Connectivity to Outside Networks
A Layer 3 outside network configuration (L3Out) defines how traffic is forwarded outside of the fabric. Layer
3 is used to discover the addresses of other nodes, select routes, select quality of service, and forward the
traffic that is entering, exiting, and transiting the fabric.

Note For guidelines and cautions for configuring and maintaining Layer 3 outside connections, see Guidelines for
Routed Connectivity to Outside Networks, on page 165.

For information about the types of L3Outs, see External Layer 3 Outside Connection Types, on page 167.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

163
 Configuring Layer 3 External Connectivity
Layer 3 Out for Routed Connectivity to External Networks

Layer 3 Out for Routed Connectivity to External Networks

Routed connectivity to external networks is enabled by associating a fabric access (infraInfra) external
routed domain (l3extDomP) with a tenant Layer 3 external instance profile (l3extInstP or external EPG) of
a Layer 3 external outside network (l3extOut), in the hierarchy in the following diagram:
Figure 15: Policy Model for Layer 3 External Connections

A Layer 3 external outside network (l3extOut object) includes the routing protocol options (BGP, OSPF, or
EIGRP or supported combinations) and the switch-specific and interface-specific configurations. While the
l3extOut contains the routing protocol (for example, OSPF with its related Virtual Routing and Forwarding
(VRF) and area ID), the Layer 3 external interface profile contains the necessary OSPF interface details. Both
are needed to enable OSPF.
The l3extInstP EPG exposes the external network to tenant EPGs through a contract. For example, a tenant
EPG that contains a group of web servers could communicate through a contract with the l3extInstP EPG
according to the network configuration contained in the l3extOut. The outside network configuration can
easily be reused for multiple nodes by associating the nodes with the L3 external node profile. Multiple nodes
that use the same profile can be configured for fail-over or load balancing. Also, a node can be added to

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

164
 Configuring Layer 3 External Connectivity
Guidelines for Routed Connectivity to Outside Networks

multiple l3extOuts resulting in VRFs that are associated with the l3extOuts also being deployed on that node.
For scalability information, refer to the current Verified Scalability Guide for Cisco ACI.

Guidelines for Routed Connectivity to Outside Networks

Use the following guidelines when creating and maintaining Layer 3 outside connections.

Topic Caution or Guideline

Updates through CLI For Layer 3 external networks created through the API or GUI and
updated through the CLI, protocols need to be enabled globally on
the external network through the API or GUI, and the node profile
for all the participating nodes needs to be added through the API or
GUI before doing any further updates through the CLI.

Loopbacks for Layer 3 networks on When configuring two Layer 3 external networks on the same node,
same node the loopbacks need to be configured separately for both Layer 3
networks.

Ingress-based policy enforcement Starting with Cisco APIC release 1.2(1), ingress-based policy
enforcement enables defining policy enforcement for Layer 3 Outside
(L3Out) traffic for both egress and ingress directions. The default is
ingress. During an upgrade to release 1.2(1) or higher, existing L3Out
configurations are set to egress so that the behavior is consistent with
the existing configuration. You do not need any special upgrade
sequence. After the upgrade, you change the global property value
to ingress. When it has been changed, the system reprograms the
rules and prefix entries. Rules are removed from the egress leaf and
installed on the ingress leaf, if not already present. If not already
configured, an Actrl prefix entry is installed on the ingress leaf.
Direct server return (DSR), and attribute EPGs require ingress based
policy enforcement. vzAny and taboo contracts ignore ingress based
policy enforcement. Transit rules are applied at ingress.

Bridge Domains with L3Outs A bridge domain in a tenant can contain a public subnet that is
advertised through an l3extOut provisioned in the common tenant.

Bridge domain route advertisement For When both OSPF and EIGRP are enabled on the same VRF on a
OSPF and EIGRP node and if the bridge domain subnets are advertised out of one of
the L3Outs, it will also get advertised out of the protocol enabled on
the other L3Out.
For OSPF and EIGRP, the bridge domain route advertisement is per
VRF and not per L3Out. The same behavior is expected when
multiple OSPF L3Outs (for multiple areas) are enabled on the same
VRF and node. In this case, the bridge domain route will be
advertised out of all the areas, if it is enabled on one of them.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

165
 Configuring Layer 3 External Connectivity
Guidelines for Routed Connectivity to Outside Networks

Topic Caution or Guideline

BGP Maximum Prefix Limit Starting with Cisco APIC release 1.2(1x), tenant policies for BGP
l3extOut connections can be configured with a maximum prefix
limit, that enables monitoring and restricting the number of route
prefixes received from a peer. Once the maximum prefix limit has
been exceeded, a log entry is recorded, and further prefixes are
rejected. The connection can be restarted if the count drops below
the threshold in a fixed interval, or the connection is shut down. Only
one option can be used at a time. The default setting is a limit of
20,000 prefixes, after which new prefixes are rejected. When the
reject option is deployed, BGP accepts one more prefix beyond the
configured limit, before the APIC raises a fault.

MTU Cisco ACI does not support IP fragmentation. Therefore, when you
configure Layer 3 Outside (L3Out) connections to external routers,
or multipod connections through an Inter-Pod Network (IPN), it is
critical that the interface MTU is set appropriately on both ends of
a link. On some platforms, such as Cisco ACI, Cisco NX-OS, and
Cisco IOS, the configurable MTU value does not take into account
the ethernet headers (matching IP MTU, and excluding the 14-18
ethernet header size), while other platforms, such as IOS-XR, include
the ethernet header in the configured MTU value. A configured value
of 9000 results in a max IP packet size of 9000 bytes in Cisco ACI,
Cisco NX-OS, and Cisco IOS, but results in a max IP packet size of
8986 bytes for an IOS-XR untagged interface.
For the appropriate MTU values for each platform, see the relevant
configuration guides.
Cisco highly recommends that you test the MTU with CLI-based
commands. For example, on the Cisco NX-OS CLI, use a command
such as ping 1.1.1.1 df-bit packet-size 9000
source-interface ethernet 1/1.

Layer 4 to Layer 7 When you are using a multinode service graph, you must have the
two EPGs in separate VRF instances. For these functions, the system
must do a Layer 3 lookup, so the EPGs must be in separate VRFs.
This limitation follows legacy service insertion, based on Layer 2
and Layer 3 lookups.

QoS for L3Outs To configure QoS policies for an L3Out and enable the policies to
be enforced on the BL switch where the L3Out is located, use the
following guidelines:
• The VRF Policy Control Enforcement Direction must be set
toEgress.
• The VRF Policy Control Enforcement Preference must be set
to Enabled.
• When configuring the contract that controls communication
between the EPGs using the L3Out, include the QoS class or
Target DSCP in the contract or subject of the contract.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

166
 Configuring Layer 3 External Connectivity
External Layer 3 Outside Connection Types

External Layer 3 Outside Connection Types

ACI supports the following External Layer 3 Outside connection options:
• Static Routing (supported for IPv4 and IPv6)
• OSPFv2 for normal and NSSA areas (IPv4)
• OSPFv3 for normal and NSSA areas (IPv6)
• iBGP (IPv4 and IPv6)
• eBGP (IPv4 and IPv6)
• EIGRP (IPv4 and IPv6)

The External Layer 3 Outside connections are supported on the following interfaces:
• Layer 3 Routed Interface
• Subinterface with 802.1Q tagging - With subinterface, you can use the same physical interface to provide
a Layer 2 outside connection for multiple private networks.
• Switched Virtual Interface (SVI) - With an SVI interface, the same physical interface that supports Layer
2 and Layer 3 and the same physical interface can be used for a Layer 2 outside connection and a Layer
3 outside connection.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

167
 Configuring Layer 3 External Connectivity
External Layer 3 Outside Connection Types

Figure 16: ACI Layer 3 Managed Objects

The managed objects that are used for the L3Outside connections are:
• External Layer 3 Outside (L3ext): Routing protocol options (OSPF area type, area, EIGRP autonomous
system, BGP), private network, External Physical domain.
• Logical Node Profile: Profile where one or more nodes are defined for the External Layer 3 Outside
connections. The configurations of the router-IDs and the loopback interface are defined in the profile.

Note Use the same router-ID for the same node across multiple External Layer 3 Outside
connections.

Note Within a single L3Out, a node can only be part of one Logical Node Profile.
Configuring the node to be a part of multiple Logical Node Profiles in a single
L3Out might result in unpredictable behavior, such as having a loopback address
pushed from one Logical Node Profile but not from the other. Use more path
bindings under the existing Logical Interface Profiles or create a new Logical
Interface Profile under the existing Logical Node Profile instead.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

168
 Configuring Layer 3 External Connectivity
Configuring a Layer 3 Outside for Tenant Networks Using the NX-OS Style CLI

• Logical Interface Profile: IP interface configuration for IPv4 and IPv6 interfaces. It is supported on the
Route Interfaces, Routed subinterfaces, and SVIs. The SVIs can be configured on physical ports,
port-channels, or vPCs.
• OSPF Interface Policy: Includes details such as OSPF Network Type and priority.
• EIGRP Interface Policy: Includes details such as Timers and split horizon.
• BGP Peer Connectivity Profile: The profile where most BGP peer settings, remote-as, local-as, and BGP
peer connection options are configured. You can associate the BGP peer connectivity profile with the
logical interface profile or the loopback interface under the node profile. This determines the update-source
configuration for the BGP peering session.
• External Network Instance Profile (EPG) (l3extInstP): The external EPG is also referred to as the
prefix-based EPG or InstP. The import and export route control policies, security import policies, and
contract associations are defined in this profile. You can configure multiple external EPGs under a single
L3Out. You may use multiple external EPGs when a different route or a security policy is defined on a
single External Layer 3 Outside connections. An external EPG or multiple external EPGs combine into
a route-map. The import/export subnets defined under the external EPG associate to the IP prefix-list
match clauses in the route-map. The external EPG is also where the import security subnets and contracts
are associated. This is used to permit or drop traffic for this L3out.
• Action Rules Profile: The action rules profile is used to define the route-map set clauses for the L3Out.
The supported set clauses are the BGP communities (standard and extended), Tags, Preference, Metric,
and Metric type.
• Route Control Profile: The route-control profile is used to reference the action rules profiles. This can
be an ordered list of action rules profiles. The Route Control Profile can be referenced by a tenant BD,
BD subnet, external EPG, or external EPG subnet.

There are more protocol settings for BGP, OSPF, and EIGRP L3Outs. These settings are configured per tenant
in the ACI Protocol Policies section in the GUI.

Note When configuring policy enforcement between external EPGs (transit routing case), you must configure the
second external EPG (InstP) with the default prefix 0/0 for export route control, aggregate export, and external
security. In addition, you must exclude the preferred group, and you must use an any contract (or desired
contract) between the transit InstPs.

Configuring a Layer 3 Outside for Tenant Networks Using the NX-OS Style CLI
These steps describe how to configure a Layer 3 outside network for tenant networks. This example shows
how to deploy a node and L3 port for tenant VRF external L3 connectivity using the NX-OS CLI.
This example is broken into steps for clarity. For a merged example, see NX-OS Style CLI Example: L3Out,
on page 173.

Before you begin

• Configure the node, port, functional profile, AEP, and Layer 3 domain.
• Configure a VLAN domain using the vlan-domain domain and vlan vlan-range commands.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

169
 Configuring Layer 3 External Connectivity
Configuring a Layer 3 Outside for Tenant Networks Using the NX-OS Style CLI

• Configure a BGP route reflector policy to propagate the routed within the fabric.

For an example using the commands for these prerequisites, see NX-OS Style CLI Example: L3Out
Prerequisites, on page 173.

Procedure

Step 1 Configure the tenant and VRF.

This example configures tenant t1 with VRF v1. They are not yet deployed.
Example:
apic1# configure
apic1(config)# tenant t1
apic1(config-tenant)# vrf context v1
apic1(config-tenant-vrf)# exit
apic1(congig-tenant)# exit
apic1(config)#

Step 2 Configure the node and interface for the L3Out.

This example configures VRF v1 on node 103 (the border leaf switch), which is named nodep1, with router
ID 11.11.11.103. It also configures interface eth1/3 as a routed interface (Layer 3 port), with IP address
12.12.12.3/24 and Layer 3 domain dom1.

Example:
apic1(config)# leaf 103
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# router-id 11.11.11.103
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# interface ethernet 1/3
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# vrf member tenant t1 vrf v1
apic1(config-leaf-if)# ip address 12.12.12.3/24
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit

Step 3 Configure the routing protocol.

This example configures BGP as the primary routing protocol, with a BGP peer address, 15.15.15.2 and
ASN 100.
Example:

apic1(config)# leaf 103

apic1(config-leaf)# router bgp 100
apic1(config-leaf-bgp)# vrf member tenant t1 vrf v1
apic1(config-leaf-bgp-vrf)# neighbor 15.15.15.2
apic1(config-leaf-bgp-vrf-neighbor)# exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf-bgp)# exit
apic1(config-leaf)# exit

Step 4 Optional. Configure a connectivity routing protocol.

This example configures OSPF as the communication protocol, with regular area ID 0.0.0.0, with loopback
address 30.30.30.0.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

170
Configuring Layer 3 External Connectivity
Configuring a Layer 3 Outside for Tenant Networks Using the NX-OS Style CLI

Example:

apic1(config)# leaf 103

apic1(config-leaf)# router ospf default
apic1(config-leaf-ospf)# vrf member tenant t1 vrf v1
apic1(config-leaf-ospf-vrf)# area 0.0.0.0 loopback 30.30.30.0
apic1(config-leaf-ospf-vrf)# exit
apic1(config-leaf-ospf)# exit
apic1(config-leaf)# exit

Step 5 Configure the external EPG on node 103.

In this example, the network 20.20.20.0/24 is configured as the external network extnw1.
Example:
apic1(config)# tenant t1
apic1(config-tenant)# external-l3 epg extnw1
apic1(config-tenant-l3ext-epg)# vrf member v1
apic1(config-tenant-l3ext-epg)# match ip 20.20.20.0/24
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 103
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# external-l3 epg extnw1
apic1(config-leaf-vrf)# exit

Step 6 Optional. Configure a route map.

This example configures a route map rp1 for the BGP peer in the outbound direction. The route map is applied
for routes that match a destination of 200.3.2.0/24. Also, on a successful match (if the route matches this
range) the route AS PATH attribute is updated to 200 and 100.
Example:
apic1(config-leaf)# template route group match-rule1 tenant t1
apic1(config-route-group)# ip prefix permit 200.3.2.0/24
apic1(config-route-group)# exit
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# route-map rp1
apic1(config-leaf-vrf-route-map)# match route group match-rule1 order 0
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# exit
apic1(config)# leaf 103
apic1(config-leaf)# router bgp 100
apic1(config-leaf-bgp)# vrf member tenant t1 vrf v1
apic1(config-leaf-bgp-vrf)# neighbor 15.15.15.2
apic1(config-leaf-bgp-vrf-neighbor)# route-map rp1 in
apic1(config-leaf-bgp-vrf-neighbor)# exit
apic1(config-leaf-bgp-vrf)#exit
apic1(config-leaf-bgp)# exit
apic1(config-leaf)# exit

Step 7 Add a bridge domain.

Example:
apic1(config)# tenant t1
apic1(config-tenant)# bridge-domain bd1
apic1(config-tenant-bd)# vrf member v1
apic1(config-tenant-bd)# exit
apic1(config-tenant)# interface bridge-domain bd1
apic1(config-tenant-interface)# ip address 44.44.44.1/24 scope public
apic1(config-tenant-interface)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

171
 Configuring Layer 3 External Connectivity
Configuring a Layer 3 Outside for Tenant Networks Using the NX-OS Style CLI

apic1(config-tenant)# exit
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# route-map rp1
apic1(config-leaf-vrf-route-map)# match bridge-domain bd1 tenant t1
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# exit

Step 8 Create an application EPG on node 101.

Example:
apic1(config)# tenant t1
apic1(config-tenant)# application app1
apic1(config-tenant-app)# epg epg1
apic1(config-tenant-app-epg)# bridge-domain member bd1
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/3
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# switchport trunk allowed vlan 2011 tenant t1 application app1 epg
epg1
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit
apic1(config)#

Step 9 Create filters (access-lists) and contracts.

Example:
apic1(config)# tenant t1
apic1(config-tenant)# access-list http-filter
apic1(config-tenant-acl)# match ip
apic1(config-tenant-acl)# match tcp dest 80
apic1(config-tenant-acl)# exit
apic1(config-tenant)# contract httpCtrct
apic1(config-tenant-contract)# scope vrf
apic1(config-tenant-contract)# subject subj1
apic1(config-tenant-contract-subj)# access-group http-filter both
apic1(config-tenant-contract-subj)# exit
apic1(config-tenant-contract)# exit

Step 10 Configure contracts and associate them with EPGs.

Example:
apic1(config-tenant)# external-l3 epg extnw1
apic1(config-tenant-l3ext-epg)# vrf member v1
apic1(config-tenant-l3ext-epg)# contract provider httpCtrct
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# application app1
apic1(config-tenant-app)# epg epg1
apic1(config-tenant-app-epg)# contract consumer httpCtrct
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)# exit
apic1(config-tenant)# exit
apic1(config)#

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

172
 Configuring Layer 3 External Connectivity
NX-OS Style CLI Example: L3Out Prerequisites

NX-OS Style CLI Example: L3Out Prerequisites

Before you can configure an L3Out, perform the following steps:
1. Configure a VLAN domain:
apic1# configure
apic1(config)# vlan-domain dom1
apic1(config-vlan)# vlan 1024-2048
apic1(config-vlan)# exit

2. Configure BGP route reflectors:

apic1(config)# bgp-fabric
apic1(config-bgp-fabric)# asn 100
apic1(config-bgp-fabric)# route-reflector spine 104,105

NX-OS Style CLI Example: L3Out

The following example provides a merged version of the steps to configure an L3Out using the NX-OS style
CLI. Configure the following prerequisites before configuring the L3Out.
apic1# configure
apic1(config)# tenant t1
apic1(config-tenant)# vrf context v1
apic1(config-tenant-vrf)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 103
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# router-id 11.11.11.103
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# interface ethernet 1/3
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# vrf member tenant t1 vrf v1
apic1(config-leaf-if)# ip address 12.12.12.3/24
apic1(config-leaf-if)# exit
apic1(config-leaf)# router bgp 100
apic1(config-leaf-bgp)# vrf member tenant t1 vrf v1
apic1(config-leaf-bgp-vrf)# neighbor 15.15.15.2
apic1(config-leaf-bgp-vrf-neighbor)# exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf-bgp)# exit
apic1(config-leaf)# router ospf default
apic1(config-leaf-ospf)# vrf member tenant t1 vrf v1
apic1(config-leaf-ospf-vrf)# area 0.0.0.0 loopback 30.30.30.0
apic1(config-leaf-ospf-vrf)# exit
apic1(config-leaf-ospf)# exit
apic1(config-leaf)# exit
apic1(config)# tenant t1
apic1(config-tenant)# external-l3 epg extnw1
apic1(config-tenant-l3ext-epg)# vrf member v1
apic1(config-tenant-l3ext-epg)# match ip 20.20.20.0/24
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 103
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# external-l3 epg extnw1
apic(config-leaf-vrf)# exit
apic1(config-leaf)# template route group match-rule1 tenant t1
apic1(config-route-group)# ip prefix permit 200.3.2.0/24

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

173
 Configuring Layer 3 External Connectivity
NX-OS Style CLI Example: L3Out

apic1(config-route-group)# exit
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# route-map rp1
apic1(config-leaf-vrf-route-map)# match route group match-rule1 order 0
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# router bgp 100
apic1(config-leaf-bgp)# vrf member tenant t1 vrf v1
apic1(config-leaf-bgp-vrf)# neighbor 15.15.15.2
apic1(config-leaf-bgp-vrf-neighbor)# route-map rp1 in
apic1(config-leaf-bgp-vrf-neighbor)#exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf-bgp)# exit
apic1(config-leaf)# exit
apic1(config)# tenant t1
apic1(config-tenant)# bridge-domain bd1
apic1(config-tenant-bd)# vrf member v1
apic1(config-tenant-bd)# exit
apic1(config-tenant)# interface bridge-domain bd1
apic1(config-tenant-interface)# ip address 44.44.44.1/24 scope public
apic1(config-tenant-interface)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# route-map map1
apic1(config-leaf-vrf-route-map)# match bridge-domain bd1 tenant t1
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# exit
apic1(config)# tenant t1
apic1(config-tenant)# application app1
apic1(config-tenant-app)# epg epg1
apic1(config-tenant-app-epg)# bridge-domain member bd1
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/3
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# switchport trunk allowed vlan 2011 tenant t1 application app1 epg
epg1
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit
apic1(config)# tenant t1
apic1(config-tenant)# access-list http-filter
apic1(config-tenant-acl)# match ip
apic1(config-tenant-acl)# match tcp dest 80
apic1(config-tenant-acl)# exit
apic1(config-tenant)# contract httpCtrct
apic1(config-tenant-contract)# scope vrf
apic1(config-tenant-contract)# subject subj1
apic1(config-tenant-contract-subj)# access-group http-filter both
apic1(config-tenant-contract-subj)# exit
apic1(config-tenant-contract)# exit
apic1(config-tenant)# external-l3 epg extnw1
apic1(config-tenant-l3ext-epg)# vrf member v1
apic1(config-tenant-l3ext-epg)# contract provider httpCtrct
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# application app1
apic1(config-tenant-app)# epg epg1
apic1(config-tenant-app-epg)# contract consumer httpCtrct
apic1(config-tenant-app-epg)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

174
 Configuring Layer 3 External Connectivity
Layer 3 Routed and Sub-Interface Port Channels

apic1(config-tenant-app)# exit
apic1(config-tenant)# exit
apic1(config)#

Layer 3 Routed and Sub-Interface Port Channels

About Layer 3 Port Channels
Previously, Cisco APIC supported only Layer 2 port channels. Starting with release 3.2(1), Cisco APIC now
supports Layer 3 port channels.
Figure 17: Switch Port Channel Configuration

Note Layer 3 routed and sub-interface port channels on border leaf switches are supported only on new generation
switches, which are switch models with "EX", "FX" or "FX2" at the end of the switch name.

Configuring a Layer 3 Routed Port-Channel Using the NX-OS CLI

This procedure configures a Layer 3 routed port channel.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf switch or leaf switches to be

configured. The node-id can be a single node
Example:
ID or a range of IDs, in the form
apic1(config)# leaf 101 node-id1-node-id2, to which the configuration
will be applied.

Step 3 interface port-channel channel-name Enters the interface configuration mode for the
specified port channel.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

175
 Configuring Layer 3 External Connectivity
Configuring a Layer 3 Routed Port-Channel Using the NX-OS CLI

Command or Action Purpose

apic1(config-leaf)# interface
port-channel po1

Step 4 no switchport Makes the interface Layer 3 capable.

Example:
apic1(config-leaf-if)# no switchport

Step 5 vrf member vrf-name tenant tenant-name Associates this port channel to this virtual
routing and forwarding (VRF) instance and L3
Example:
outside policy, where:
apic1(config-leaf-if)# vrf member v1
tenant t1 • vrf-name is the VRF name. The name can
be any case-sensitive, alphanumeric string
up to 32 characters.
• tenant-name is the tenant name. The name
can be any case-sensitive, alphanumeric
string up to 32 characters.

Step 6 vlan-domain member vlan-domain-name Associates the port channel template with the
previously configured VLAN domain.
Example:
apic1(config-leaf-if)# vlan-domain
member dom1

Step 7 ip address ip-address/subnet-mask Sets the IP address and subnet mask for the
specified interface.
Example:
apic1(config-leaf-if)# ip address
10.1.1.1/24

Step 8 ipv6 address sub-bits/prefix-length preferred Configures an IPv6 address based on an IPv6
general prefix and enables IPv6 processing on
Example:
an interface, where:
apic1(config-leaf-if)# ipv6 address
2001::1/64 preferred • sub-bits is the subprefix bits and host bits
of the address to be concatenated with the
prefixes provided by the general prefix
specified with the prefix-name argument.
The sub-bits argument must be in the
form documented in RFC 2373 where the
address is specified in hexadecimal using
16-bit values between colons.
• prefix-length is the length of the IPv6
prefix. A decimal value that indicates how
many of the high-order contiguous bits
of the address comprise the prefix (the
network portion of the address). A slash
mark must precede the decimal value.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

176
 Configuring Layer 3 External Connectivity
Configuring a Layer 3 Sub-Interface Port-Channel Using the NX-OS CLI

Command or Action Purpose

Step 9 ipv6 link-local ipv6-link-local-address Configures an IPv6 link-local address for an
interface.
Example:
apic1(config-leaf-if)# ipv6 link-local
fe80::1

Step 10 mac-address mac-address Manually sets the interface MAC address.

Example:
apic1(config-leaf-if)# mac-address
00:44:55:66:55::01

Step 11 mtu mtu-value Sets the MTU for this class of service.
Example:
apic1(config-leaf-if)# mtu 1500

Example
This example shows how to configure a basic Layer 3 port channel.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface port-channel po1
apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# vrf member v1 tenant t1
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# ip address 10.1.1.1/24
apic1(config-leaf-if)# ipv6 address 2001::1/64 preferred
apic1(config-leaf-if)# ipv6 link-local fe80::1
apic1(config-leaf-if)# mac-address 00:44:55:66:55::01
apic1(config-leaf-if)# mtu 1500

Configuring a Layer 3 Sub-Interface Port-Channel Using the NX-OS CLI

This procedure configures a Layer 3 sub-interface port channel.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf switch or leaf switches to be

configured. The node-id can be a single node
Example:
ID or a range of IDs, in the form

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

177
 Configuring Layer 3 External Connectivity
Configuring a Layer 3 Sub-Interface Port-Channel Using the NX-OS CLI

Command or Action Purpose

apic1(config)# leaf 101 node-id1-node-id2, to which the configuration
will be applied.

Step 3 vrf member vrf-name tenant tenant-name Associates this port channel to this virtual
routing and forwarding (VRF) instance and L3
Example:
outside policy, where:, where:
apic1(config-leaf-if)# vrf member v1
tenant t1 • vrf-name is the VRF name. The name can
be any case-sensitive, alphanumeric string
up to 32 characters.
• tenant-name is the tenant name. The name
can be any case-sensitive, alphanumeric
string up to 32 characters.

Step 4 vlan-domain member vlan-domain-name Associates the port channel template with the
previously configured VLAN domain.
Example:
apic1(config-leaf-if)# vlan-domain
member dom1

Step 5 ip address ip-address / subnet-mask Sets the IP address and subnet mask for the
specified interface.
Example:
apic1(config-leaf-if)# ip address
10.1.1.1/24

Step 6 ipv6 address sub-bits / prefix-length Configures an IPv6 address based on an IPv6
preferred general prefix and enables IPv6 processing on
an interface, where:
Example:
apic1(config-leaf-if)# ipv6 address • sub-bits is the subprefix bits and host bits
2001::1/64 preferred of the address to be concatenated with the
prefixes provided by the general prefix
specified with the prefix-name argument.
The sub-bits argument must be in the
form documented in RFC 2373 where the
address is specified in hexadecimal using
16-bit values between colons.
• prefix-length is the length of the IPv6
prefix. A decimal value that indicates how
many of the high-order contiguous bits
of the address comprise the prefix (the
network portion of the address). A slash
mark must precede the decimal value.

Step 7 ipv6 link-local ipv6-link-local-address Configures an IPv6 link-local address for an

interface.
Example:
apic1(config-leaf-if)# ipv6 link-local
fe80::1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

178
Configuring Layer 3 External Connectivity
Configuring a Layer 3 Sub-Interface Port-Channel Using the NX-OS CLI

Command or Action Purpose

Step 8 mac-address mac-address Manually sets the interface MAC address.
Example:
apic1(config-leaf-if)# mac-address
00:44:55:66:55::01

Step 9 mtu mtu-value Sets the MTU for this class of service.
Example:
apic1(config-leaf-if)# mtu 1500

Step 10 exit Returns to configure mode.

Example:
apic1(config-leaf-if)# exit

Step 11 interface port-channel channel-name Enters the interface configuration mode for the
specified port channel.
Example:
apic1(config-leaf)# interface
port-channel po1

Step 12 vlan-domain member vlan-domain-name Associates the port channel template with the
previously configured VLAN domain.
Example:
apic1(config-leaf-if)# vlan-domain
member dom1

Step 13 exit Returns to configure mode.

Example:
apic1(config-leaf-if)# exit

Step 14 interface port-channel channel-name.number Enters the interface configuration mode for the
specified sub-interface port channel.
Example:
apic1(config-leaf)# interface
port-channel po1.2001

Step 15 vrf member vrf-name tenant tenant-name Associates this port channel to this virtual
routing and forwarding (VRF) instance and L3
Example:
outside policy, where:, where:
apic1(config-leaf-if)# vrf member v1
tenant t1 • vrf-name is the VRF name. The name can
be any case-sensitive, alphanumeric string
up to 32 characters.
• tenant-name is the tenant name. The name
can be any case-sensitive, alphanumeric
string up to 32 characters.

Step 16 exit Returns to configure mode.

Example:
apic1(config-leaf-if)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

179
 Configuring Layer 3 External Connectivity
Adding Ports to the Layer 3 Port-Channel Using the NX-OS CLI

Example
This example shows how to configure a basic Layer 3 sub-interface port-channel.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface vlan 2001
apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# vrf member v1 tenant t1
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# ip address 10.1.1.1/24
apic1(config-leaf-if)# ipv6 address 2001::1/64 preferred
apic1(config-leaf-if)# ipv6 link-local fe80::1
apic1(config-leaf-if)# mac-address 00:44:55:66:55::01
apic1(config-leaf-if)# mtu 1500
apic1(config-leaf-if)# exit
apic1(config-leaf)# interface port-channel po1
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# exit
apic1(config-leaf)# interface port-channel po1.2001
apic1(config-leaf-if)# vrf member v1 tenant t1
apic1(config-leaf-if)# exit

Adding Ports to the Layer 3 Port-Channel Using the NX-OS CLI

This procedure adds ports to a Layer 3 port channel that you configured previously.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf switch or leaf switches to be

configured. The node-id can be a single node
Example:
ID or a range of IDs, in the form
apic1(config)# leaf 101 node-id1-node-id2, to which the configuration
will be applied.

Step 3 interface Ethernet slot/port Enters interface configuration mode for the
interface you want to configure.
Example:
apic1(config-leaf)# interface Ethernet
1/1-2

Step 4 channel-group channel-name Configures the port in a channel group.

Example:
apic1(config-leaf-if)# channel-group p01

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

180
 Configuring Layer 3 External Connectivity
Layer 3 Out to Layer 3 Out Inter-VRF Leaking

Example
This example shows how to add ports to a Layer 3 port-channel.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface Ethernet 1/1-2
apic1(config-leaf-if)# channel-group p01

Layer 3 Out to Layer 3 Out Inter-VRF Leaking

Starting with Cisco APIC release 2.2(2e) , when there are two Layer 3 Outs in two different VRFs, inter-VRF
leaking is supported.
For this feature to work, the following conditions must be satisfied:
• A contract between the two Layer 3 Outs is required.
• Routes of connected and transit subnets for a Layer 3 Out are leaked by enforcing contracts (L3Out-L3Out
as well as L3Out-EPG) and without leaking the dynamic or static routes between VRFs.
• Dynamic or static routes are leaked for a Layer 3 Out by enforcing contracts (L3Out-L3Out as well as
L3Out-EPG) and without advertising directly connected or transit routes between VRFs.
• Shared Layer 3 Outs in different VRFs can communicate with each other.
• There is no associated L3Out required for the bridge domain. When an Inter-VRF shared L3Out is used,
it is not necessary to associate the user tenant bridge domains with the L3Out in tenant common. If you
had a tenant-specific L3Out, it would still be associated to your bridge domains in your respective tenants.
• Two Layer 3 Outs can be in two different VRFs, and they can successfully exchange routes.
• This enhancement is similar to the Application EPG to Layer 3 Out inter-VRF communications. The
only difference is that instead of an Application EPG there is another Layer 3 Out. Therefore, in this
case, the contract is between two Layer 3 Outs.

In the following figure, there are two Layer 3 Outs with a shared subnet. There is a contract between the Layer
3 external instance profile (l3extInstP) in both the VRFs. In this case, the Shared Layer 3 Out for VRF1 can
communicate with the Shared Layer 3 Out for VRF2.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

181
 Configuring Layer 3 External Connectivity
Configuring Shared Layer 3 Out Inter-VRF Leaking Using the NX-OS Style CLI - Named Example

Figure 18: Shared Layer 3 Outs Communicating Between Two VRFs

Configuring Shared Layer 3 Out Inter-VRF Leaking Using the NX-OS Style CLI
- Named Example
Procedure

Command or Action Purpose

Step 1 Enter the configure mode.
Example:
apic1# configure

Step 2 Configure the provider Layer 3 Out.

Example:
apic1(config)# tenant t1_provider
apic1(config-tenant)# external-l3 epg
l3extInstP-1 l3out T0-o1-L3OUT-1
apic1(config-tenant-l3ext-epg)# vrf
member VRF1
apic1(config-tenant-l3ext-epg)# match ip
192.168.2.0/24 shared
apic1(config-tenant-l3ext-epg)# contract
provider vzBrCP-1
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant
t1_provider vrf VRF1 l3out T0-o1-L3OUT-1
apic1(config-leaf-vrf)# route-map
T0-o1-L3OUT-1_shared
apic1(config-leaf-vrf-route-map)# ip
prefix-list l3extInstP-1 permit
192.168.2.0/24
apic1(config-leaf-vrf-route-map)# match

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

182
 Configuring Layer 3 External Connectivity
Configuring Shared Layer 3 Out Inter-VRF Leaking Using the NX-OS Style CLI - Implicit Example

Command or Action Purpose

prefix-list l3extInstP-1
apic1(config-leaf-vrf-route-map-match)#
exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# exit

Step 3 Configure the consumer Layer 3 Out.

Example:
apic1(config)# tenant t1_consumer
apic1(config-tenant)# external-l3 epg
l3extInstP-2 l3out T0-o1-L3OUT-1
apic1(config-tenant-l3ext-epg)# vrf
member VRF2
apic1(config-tenant-l3ext-epg)# match ip
199.16.2.0/24 shared
apic1(config-tenant-l3ext-epg)# contract
consumer vzBrCP-1 imported
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant
t1_consumer vrf VRF2 l3out T0-o1-L3OUT-1
apic1(config-leaf-vrf)# route-map
T0-o1-L3OUT-1_shared
apic1(config-leaf-vrf-route-map)# ip
prefix-list l3extInstP-2 permit
199.16.2.0/24
apic1(config-leaf-vrf-route-map)# match
prefix-list l3extInstP-2
apic1(config-leaf-vrf-route-map-match)#
exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# exit
apic1(config)#

Configuring Shared Layer 3 Out Inter-VRF Leaking Using the NX-OS Style CLI
- Implicit Example
Procedure

Command or Action Purpose

Step 1 Enter the configure mode.
Example:
apic1# configure

Step 2 Configure the provider tenant and VRF.

Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

183
 Configuring Layer 3 External Connectivity
Configuring Shared Layer 3 Out Inter-VRF Leaking Using the NX-OS Style CLI - Implicit Example

Command or Action Purpose

apic1(config)# tenant t1_provider
apic1(config-tenant)# vrf context VRF1
apic1(config-tenant-vrf)# exit
apic1(config-tenant)# exit

Step 3 Configure the consumer tenant and VRF.

Example:
apic1(config)# tenant t1_consumer
apic1(config-tenant)# vrf context VRF2
apic1(config-tenant-vrf)# exit
apic1(config-tenant)# exit

Step 4 Configure the contract.

Example:
apic1(config)# tenant t1_provider
apic1(config-tenant)# contract vzBrCP-1
type permit
apic1(config-tenant-contract)# scope
exportable
apic1(config-tenant-contract)# export to
tenant t1_consumer
apic1(config-tenant-contract)# exit

Step 5 Configure the provider External Layer 3 EPG.

Example:
apic1(config-tenant)# external-l3 epg
l3extInstP-1
apic1(config-tenant-l3ext-epg)# vrf
member VRF1
apic1(config-tenant-l3ext-epg)# match ip
192.168.2.0/24 shared
apic1(config-tenant-l3ext-epg)# contract
provider vzBrCP-1
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# exit

Step 6 Configure the provider export map.

Example:
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant
t1_provider vrf VRF1
apic1(config-leaf-vrf)# route-map map1
apic1(config-leaf-vrf-route-map)# ip
prefix-list p1 permit 192.168.2.0/24
apic1(config-leaf-vrf-route-map)# match
prefix-list p1
apic1(config-leaf-vrf-route-map-match)#
exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# export map map1
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# exit

Step 7 Configure the consumer external Layer 3 EPG.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

184
 Configuring Layer 3 External Connectivity
About SVI External Encapsulation Scope

Command or Action Purpose

Example:
apic1(config)# tenant t1_consumer
apic1(config-tenant)# external-l3 epg
l3extInstP-2
apic1(config-tenant-l3ext-epg)# vrf
member VRF2
apic1(config-tenant-l3ext-epg)# match ip
199.16.2.0/24 shared
apic1(config-tenant-l3ext-epg)# contract
consumer vzBrCP-1 imported
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# exit

Step 8 Configure the consumer export map.

Example:
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant
t1_consumer vrf VRF2
apic1(config-leaf-vrf)# route-map map2
apic1(config-leaf-vrf-route-map)# ip
prefix-list p2 permit 199.16.2.0/24
apic1(config-leaf-vrf-route-map)# match
prefix-list p2
apic1(config-leaf-vrf-route-map-match)#
exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# export map map2
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# exit
apic1(config)#

About SVI External Encapsulation Scope

In the context of a Layer 3 Out configuration, a switch virtual interfaces (SVI), is configured to provide
connectivity between the ACI leaf switch and a router.
By default, when a single Layer 3 Out is configured with SVI interfaces, the VLAN encapsulation spans
multiple nodes within the fabric. This happens because the ACI fabric configures the same bridge domain
(VXLAN VNI) across all the nodes in the fabric where the Layer 3 Out SVI is deployed as long as all SVI
interfaces use the same external encapsulation (SVI) as shown in the figure.
However, when different Layer 3 Outs are deployed, the ACI fabric uses different bridge domains even if
they use the same external encapsulation (SVI) as shown in the figure:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

185
 Configuring Layer 3 External Connectivity
About SVI External Encapsulation Scope

Figure 19: Local Scope Encapsulation and One Layer 3 Out

Figure 20: Local Scope Encapsulation and Two Layer 3 Outs

Starting with Cisco APIC release 2.3, it is now possible to choose the behavior when deploying two (or more)
Layer 3 Outs using the same external encapsulation (SVI).
The encapsulation scope can now be configured as Local or VRF:
• Local scope (default): The example behavior is displayed in the figure titled Local Scope Encapsulation
and Two Layer 3 Outs.
• VRF scope: The ACI fabric configures the same bridge domain (VXLAN VNI) across all the nodes and
Layer 3 Out where the same external encapsulation (SVI) is deployed. See the example in the figure
titled VRF Scope Encapsulation and Two Layer 3 Outs.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

186
 Configuring Layer 3 External Connectivity
Encapsulation Scope Syntax

Figure 21: VRF Scope Encapsulation and Two Layer 3 Outs

Encapsulation Scope Syntax

The options for configuring the scope of the encapsulation used for the Layer 3 Out profile are as follows:
• Ctx—The same external SVI in all Layer 3 Outs in the same VRF for a given VLAN encapsulation. This
is a global value.
• Local —A unique external SVI per Layer 3 Out. This is the default value.

The mapping among the CLI, API, and GUI syntax is as follows:

Table 19: Encapsulation Scope Syntax

CLI API GUI

l3out local Local

vrf ctx VRF

Note The CLI commands to configure encapsulation scope are only supported when the VRF is configured through
a named Layer 3 Out configuration.

Configuring SVI Interface Encapsulation Scope Using NX-OS Style CLI

The following example displaying steps for an SVI interface encapsulation scope setting is through a named
Layer 3 Out configuration.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

187
 Configuring Layer 3 External Connectivity
About SVI Auto State

Procedure

Command or Action Purpose

Step 1 Enter the configure mode. Enters the configuration mode.
Example:
apic1# configure

Step 2 Enter the switch mode. Enters the switch mode.

Example:
apic1(config)# leaf 104

Step 3 Create the VLAN interface. Creates the VLAN interface. The VLAN range
is 1-4094.
Example:
apic1(config-leaf)# interface vlan 2001

Step 4 Specify the encapsulation scope. Specifies the encapsulation scope.

Example:
apic1(config-leaf-if)# encap scope vrf
context

Step 5 Exit the interface mode. Exits the interface mode.

Example:
apic1(config-leaf-if)# exit

About SVI Auto State

Note This feature is available in the APIC Release 2.2(3x) release and going forward with APIC Release 3.1(1). It
is not supported in APIC Release 3.0(x).

The Switch Virtual Interface (SVI) represents a logical interface between the bridging function and the routing
function of a VLAN in the device. SVI can have members that are physical ports, direct port channels, or
virtual port channels. The SVI logical interface is associated with VLANs, and the VLANs have port
membership.
The SVI state does not depend on the members. The default auto state behavior for SVI in Cisco APIC is that
it remains in the up state when the auto state value is disabled. This means that the SVI remains active even
if no interfaces are operational in the corresponding VLAN/s.
If the SVI auto state value is changed to enabled, then it depends on the port members in the associated VLANs.
When a VLAN interface has multiple ports in the VLAN, the SVI goes to the down state when all the ports
in the VLAN go down.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

188
 Configuring Layer 3 External Connectivity
Guidelines and Limitations for SVI Auto State Behavior

Table 20: SVI Auto State

SVI Auto State Description of SVI State

Disabled SVI remains in the up state even if no interfaces are operational

in the corresponding VLAN/s.
Disabled is the default SVI auto state value.

Enabled SVI depends on the port members in the associated VLANs.

When a VLAN interface contains multiple ports, the SVI goes
into the down state when all the ports in the VLAN go down.

Guidelines and Limitations for SVI Auto State Behavior

Read the following guidelines:
• When you enable or disable the auto state behavior for SVI, you configure the auto state behavior per
SVI. There is no global command.

Configuring SVI Auto State Using NX-OS Style CLI

Before you begin
• The tenant and VRF configured.
• A Layer 3 Out is configured and a logical node profile and a logical interface profile under the Layer 3
Out is configured.

Procedure

Command or Action Purpose

Step 1 Enter the configure mode. Enters the configuration mode.
Example:
apic1# configure

Step 2 Enter the switch mode. Enters the switch mode.

Example:
apic1(config)# leaf 104

Step 3 Create the VLAN interface. Creates the VLAN interface. The VLAN range
is 1-4094.
Example:
apic1(config-leaf)# interface vlan 2001

Step 4 Enable SVI auto state. Enables SVI auto state.

Example: By default, the SVI auto state value is not
apic1(config-leaf-if)# autostate enabled.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

189
 Configuring Layer 3 External Connectivity
Configuring an Interface and Static Route

Command or Action Purpose

Step 5 Exit the interface mode. Exits the interface mode.
Example:
apic1(config-leaf-if)# exit

Configuring an Interface and Static Route

Before you begin
Configure a tenant and VRF.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 [no] vrf context tenant tenant-name vrf Configures a tenant VRF on the node.
vrf-name
Example:
apic1(config-leaf)# vrf context tenant
exampleCorp vrf v1

Step 4 (Optional) [no] router-id ipv4-address Assigns a router ID for routing protocols
running on the VRF. If you do not assign a
Example:
router ID, an ID is generated internally that is
apic1(config-leaf-vrf)# router-id unique to each leaf switch.
1.2.3.4

Step 5 [no] {ip | ipv6} route ip-prefix/masklen Configures static route information for the
next-hop-address [preferred] VRF.
Example:

apic1(config-leaf-vrf)# ip route
21.1.1.1/32 32.1.1.1
apic1(config-leaf-vrf)# ipv6 route
5001::1/128 6002::1

Step 6 exit Returns to leaf configuration mode.

Example:
apic1(config-leaf-vrf)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

190
Configuring Layer 3 External Connectivity
Configuring an Interface and Static Route

Command or Action Purpose

Step 7 interface type Specifies a port for the external interface.
Example:
apic1(config-leaf)# interface eth 1/1

Step 8 vlan-domain member domain-name Assign a VLAN domain to the interface. The
VLAN domain must have already been created
Example:
using the vlan-domain command in the
apic1(config-leaf-if)# vlan-domain global configuration mode.
member dom1

Step 9 no switchport Configures the interface as a layer 3 interface,

exposing the layer 3 commands in the
Example:
configuration options.
apic1(config-leaf-if)# no switchport

Step 10 vrf member tenant tenant-name vrf vrf-name Attaches the interface to the tenant VRF.
Example:
apic1(config-leaf-if)# vrf member tenant
exampleCorp vrf v1

Step 11 [no] {ip | ipv6} address ip-prefix/masklen Configures IP addresses on the interface. The
[eui64] [secondary] [preferred] specified address can be declared as either:
Example: • preferred —The default source address
for traffic from the interface.
apic1(config-leaf-if)# ip address
10.1.1.1/24 • secondary —The secondary address of
apic1(config-leaf-if)# ipv6 address the interface.
2001::1/64 preferred
With the optional eui64 keyword, the host
can assign itself a 64-bit Extended Unique
Identifier (EUI).
In this mode, you can also configure ipv6
link-local , mac address , mtu , and other
layer 3 properties on the interface.

Step 12 [[no]] ip dhcp relay address tenant Sets or removes a DHCP relay address for the
tenant-name dhcp-address{application external interface along with any supported
app-name epg epg-name|external-12 options.
12-epg-name|external-13 13-epg-name}
Example:

apic(config-leaf-if)# ip dhcp relay

address 192.0.20.1 tenant exampleCorp
application app1 epg epg1

Examples
This example shows how to deploy a layer 3 port for external connectivity.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

191
 Configuring Layer 3 External Connectivity
Configuring an Interface and Static Route

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant exampleCorp vrf v1
apic1(config-leaf-vrf)# router-id 1.2.3.4
apic1(config-leaf-vrf)# ip route 21.1.1.1/32 32.1.1.1
apic1(config-leaf-vrf)# ipv6 route 5001::1/128 6002::1 preferred
apic1(config-leaf-vrf)# exit

apic1(config-leaf)# interface eth 1/1

apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1
apic1(config-leaf-if)# ip address 10.1.1.1/24
apic1(config-leaf-if)# ip address 11.1.1.1/24 secondary
apic1(config-leaf-if)# ipv6 address 2001::1/64 preferred
apic1(config-leaf-if)# ipv6 link-local fe80::1
apic1(config-leaf-if)# mac-address 00:44:55:66:55::01
apic1(config-leaf-if)# mtu 4470

This example shows how to configure a layer 3 subinterface port for external connectivity. In this
example, the subinterface ID (the "100" in 1/2.100) is actually the VLAN encapsulation instead of
an ID. All properties supported on a layer 3 port are available on the subinterface as well.

apic1# configure
apic1(config)# leaf 101
# SAME VRF CONTEXT CONFIGURATION AS PREVIOUS EXAMPLE

apic1(config-leaf)# interface eth 1/2.100

apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1
# SAME L3 PROPERTIES CONFIGURATION AS PREVIOUS EXAMPLE

This example shows the methods to configure a switched virtual interface (SVI) for external
connectivity. Each external SVI is uniquely identified by its encap VLAN denoted in the SVI ID.

apic1# configure
apic1(config)# leaf 101
# SAME VRF CONTEXT CONFIGURATION AS PREVIOUS EXAMPLE

apic1(config-leaf)# interface vlan 200

apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1
apic1(config-leaf-if)# ip address 13.1.1.1/24

# HOW TO ATTACH A PORT TO THE EXTERNAL SVI:

apic1(config)# leaf 101
apic1(config-leaf)# interface eth 1/4
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf)# switchport trunk allowed vlan 10 tenant exampleCorp external-svi

# HOW TO ATTACH A PORT CHANNEL TO THE EXTERNAL SVI:

apic1(config)# leaf 102
apic1(config-leaf)# interface port-channel po1
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf)# switchport trunk allowed vlan 10 tenant exampleCorp external-svi

# HOW TO ATTACH A VIRTUAL PORT CHANNEL (vPC) TO THE EXTERNAL SVI:

apic1(config)# vpc context leaf 101 102
apic1(config-leaf)# interface vpc vpc103
apic1(config-leaf-if)# vlan-domain member dom1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

192
 Configuring Layer 3 External Connectivity
OSPF Configuration

apic1(config-leaf)# switchport trunk allowed vlan 10 tenant exampleCorp external-svi

Note An external SVI must be configured on each of the participating nodes. This allows you to configure
different IP addresses on each of the nodes for the same SVI. If the vPC is part of an external SVI,
you must individually create an SVI on each of the participating vPC peers and you can provide
different IP addresses on each SVI.

OSPF Configuration
Configuring OSPF
OSPF can operate in one of the following modes in an area:
• When OSPF is used as the main routing protocol for the tenant VRF in the node, OSPF will import and
export routes defined in the route-map configured in the OSPF area. The route-map contains the export
routes.
• When OSPF is used as a connectivity protocol for BGP, OSPF advertises the loopback address which is
used as the source of the BGP session. Note that the loopback IP address and not the loopback ID is used.
In this case, a BGP session relying on OSPF will use the same loopback IP address in its update-source
command.

There is no need for separate configuration of OSPF and OSPFv3. The router OSPF mode handles both
OSPFv2 and OSPFv3 implicitly for the areas running under OSPF.
OSPF sessions are supported on all types of layer 3 Interfaces in the leaf, including:
• Layer 3 ports
• Subinterfaces
• External SVI

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

193
 Configuring Layer 3 External Connectivity
Configuring OSPF

Command or Action Purpose

Step 3 router ospf default Creates an OSPF routing process and enters
OSPF policy configuration.
Example:
apic1(config-leaf)# router ospf default

Step 4 vrf member tenant tenant-name vrf vrf-name Enables a VRF in the OSPF session.
Example:
apic1(config-leaf-ospf)# vrf member
tenant exampleCorp vrf v100

Step 5 (Optional) default-information originate Causes the switch to generate a default route.
[always]
Example:
apic1(config-leaf-ospf-vrf)#
default-information originate

Step 6 area area-id nssa [no-redistribution] Defines a not-so-stubby area (NSSA).

[default-information-originate]
Example:
apic1(config-leaf-ospf-vrf)# area 0 nssa

Step 7 area area-id stub Defines an area to be a stub area.

Example:
apic1(config-leaf-ospf-vrf)# area 17
stub

Step 8 area area-id default-cost cost Sets OSPF default area cost to a value between
0 and 16777215.
Example:
apic1(config-leaf-ospf-vrf)# area 17
default-cost 20

Step 9 area area-id route-map map-name out Specifies a route-map for outbound filtering.
Example:
apic1(config-leaf-ospf-vrf)# area 17
route-map ospf-to-eigrp out

Step 10 area area-id loopback loopback-address When OSPF is used as a connectivity protocol
for BGP, OSPF advertises the loopback
Example:
address which is used as the source of the BGP
apic1(config-leaf-ospf-vrf)# area 17 session. Note that the loopback IP address and
loopback 192.0.20.11/32
not the loopback ID is used. In this case, a
BGP session relying on OSPF will use the
same loopback IP address in its update-source
command.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

194
Configuring Layer 3 External Connectivity
Configuring OSPF

Command or Action Purpose

Step 11 inherit {ipv4 | ipv6} ospf vrf-policy Inherits the OSPF Template Policy under this
policy-name VRF.
Example:
apic1(config-leaf-ospf-vrf)# inherit
ipv4 ospf vrf-policy vrfTemplate2

Step 12 summary-address ip-address Configures external route summarization. Enter

the summary address for external routes
Example:
learned from other protocols.
apic1(config-leaf-ospf-vrf)#
summary-address 182.1.20.0/24

Step 13 area area-id range address-range cost cost Configures inter-area route summarization,
which summarizes the networks between areas.
Example:
apic1(config-leaf-ospf-vrf)# area 17 The range is the summary route to be
range 192.0.20.0/24 cost 20 advertised in areas. The cost is a value
between 0 and 16777215.

Step 14 exit Returns to OSPF configuration mode.

Example:
apic1(config-leaf-ospf-vrf)# exit

Step 15 exit Returns to leaf configuration mode.

Example:
apic1(config-leaf-ospf)# exit

Step 16 interface slot/port Specifies a port for the OSPF interface. The
interface could also be specified as interface
Example:
slot/port.vlan-id or interface vlan vlan-id .
apic1(config-leaf)# interface eth 1/2

Step 17 {ip | ipv6} router ospf default area area-id Creates an OSPF routing process and enters
OSPF policy configuration.
Example:
apic1(config-leaf-if)# ip router ospf
default area 17

Step 18 {ip | ipv6} ospf inherit interface-policy Inherits the OSPF interface template policy
if-policy-name tenant tenant-name under this tenant.
Example:
apic1(config-leaf-if)# ip ospf inherit
interface-policy ifPolicy3 tenant
exampleCorp

Step 19 [no] {ip | ipv6} ospf prefix-suppression Prevents OSPF from advertising all IP prefixes
{enable | disable | inherit} that belong to a specific interface, except for
prefixes that are associated with secondary IP
Example:
addresses.
apic1(config-leaf-if)# ip ospf
prefix-suppression enable

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

195
 Configuring Layer 3 External Connectivity
Creating OSPF VRF and Interface Templates

Command or Action Purpose

Step 20 [no] {ip | ipv6} ospf passive-interface Suppresses routing updates on the interface.
Example:
apic1(config-leaf-if)# ip ospf
passive-interface

Step 21 [no] ip ospf authentication {md5 | none | Specifies the authentication type.
simple}
Example:
apic1(config-leaf-if)# ip ospf
authentication md5

Step 22 ip ospf authentication-key key Specifies the authentication key.

Example:
apic1(config-leaf-if)# ip ospf
authentication-key c1$c0123

Examples

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# router ospf default
apic1(config-leaf-ospf)# vrf member tenant exampleCorp vrf v100
apic1(config-leaf-ospf-vrf)# area 0 nssa
apic1(config-leaf-ospf-vrf)# area 17 stub
apic1(config-leaf-ospf-vrf)# area 17 default-cost 20
apic1(config-leaf-ospf-vrf)# area 17 route-map ospf-to-eigrp out
apic1(config-leaf-ospf-vrf)# area 17 loopback 192.0.20.11/32
apic1(config-leaf-ospf-vrf)# inherit ipv4 ospf vrf-policy vrfTemplate2
apic1(config-leaf-ospf-vrf)# summary-address 182.1.20.0/24
apic1(config-leaf-ospf-vrf)# area 17 range 192.0.20.0/24 cost 20
apic1(config-leaf-ospf-vrf)# exit
apic1(config-leaf-ospf)# exit
apic1(config-leaf)# interface eth 1/3
apic1(config-leaf-if)# ip router ospf default area 17
apic1(config-leaf-if)# ip ospf inherit interface-policy ifPolicy3 tenant exampleCorp
apic1(config-leaf-if)# ip ospf prefix-suppression enable
apic1(config-leaf-if)# ip ospf passive-interface
apic1(config-leaf-if)# ip ospf authentication md5
apic1(config-leaf-if)# ip ospf authentication-key c1$c0123

Creating OSPF VRF and Interface Templates

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

196
Configuring Layer 3 External Connectivity
Creating OSPF VRF and Interface Templates

Command or Action Purpose

apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 template ospf vrf-policy vrf-policy-name Creates the OSPF VRF policy template under
tenant tenant-name the specified tenant.
Example:
apic1(config-leaf)# template ospf
vrf-policy vrfTemplate3 tenant
exampleCorp

Step 4 timers throttle lsa start-time hold-interval Sets the start-interval, hold-interval, and
max-time max-interval for link-state advertisements
(LSA).
Example:
apic1(config-vrf-policy)# timers
throttle lsa 200 10000 45000

Step 5 timers lsa-group-pacing seconds Sets the interval in which LSAs are grouped
and refreshed, checksummed, or aged.
Example:
apic1(config-vrf-policy)# timers
lsa-group-pacing 240

Step 6 timers lsa-arrival milliseconds Sets the minimum interval between the arrival
of each LSA.
Example:
apic1(config-vrf-policy)# timers
lsa-arrival 1000

Step 7 timers throttle spf spf-start spf-hold Sets the SPF init-interval, hold-interval, and
spf-max-wait max-interval for LSA.
Example:
apic1(config-vrf-policy)# timers
throttle spf 5 1000 90000

Step 8 auto-cost reference-bandwidth bandwidth Sets OSPF Policy Bandwidth Reference in

Mbps.
Example:
apic1(config-vrf-policy)# auto-cost
reference-bandwidth 1000

Step 9 distance distance Sets OSPF Policy Preferred Administrative

Distance.
Example:
apic1(config-vrf-policy)# distance 200

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

197
 Configuring Layer 3 External Connectivity
Creating OSPF VRF and Interface Templates

Command or Action Purpose

Step 10 maximum-paths max-paths Sets the maximum number of parallel routes
that OSPF can install in a routing table. The
Example:
range is from 1 to 16 routes.
apic1(config-vrf-policy)# maximum-paths
8

Step 11 graceful-restart helper-disable Disables the graceful restart helper mode.

Example:
apic1(config-vrf-policy)#
graceful-restart helper-disable

Step 12 prefix-suppression Prevents OSPF from advertising all IP prefixes

except prefixes that are associated with
Example:
loopbacks, secondary IP addresses, and passive
apic1(config-vrf-policy)# interfaces.
prefix-suppression

Step 13 name-lookup Configures OSPF to look up DNS names.

Example:
apic1(config-vrf-policy)# name-lookup

Step 14 exit Returns to leaf configuration mode.

Example:
apic1(config-vrf-policy)# exit

Step 15 template ospf interface-policy if-policy-name Creates the OSPF interface policy template
tenant tenant-name under the specified tenant.
Example:
apic1(config-leaf)# template ospf
interface-policy ifTemplate5 tenant
exampleCorp

Step 16 [no] advertise-subnet Advertises the primary IP address subnet mask

instead of /32.
Example:
apic1(config-interface-policy)#
advertise-subnet

Step 17 [no] cost if-cost Sets the OSPF cost for the interface. The range
is 0 to 65535.
Example:
apic1(config-interface-policy)# cost
300

Step 18 [no] dead-interval seconds Sets the interval in seconds at which hello
packets must not be seen before neighbors
Example:
declare the router down. The range is 1 to
apic1(config-interface-policy)# 65535 seconds.
dead-interval 60

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

198
Configuring Layer 3 External Connectivity
Creating OSPF VRF and Interface Templates

Command or Action Purpose

Step 19 [no] hello-interval seconds Specifies the interval between hello packets in
seconds. The range is 1 to 65535 seconds.
Example:
apic1(config-interface-policy)#
hello-interval 10

Step 20 [no] mtu-ignore Disables MTU mismatch detection on the

interface.
Example:
apic1(config-interface-policy)#
mtu-ignore

Step 21 [no] network {bcast | p2p | unspecified} Sets the OSPF interface policy network type,
which can be broadcast or point-to-point.
Example:
apic1(config-interface-policy)# network
p2p

Step 22 [no] passive-interface Suppresses OSPF routing updates on the

interface.
Example:
apic1(config-interface-policy)#
passive-interface

Step 23 [no] priority priority Sets OSPF interface priority, which is used to
determine the designated router (DR) on a
Example:
specific network. The range is 0 to 255.
apic1(config-interface-policy)# priority
4

Step 24 [no] retransmit-interval seconds Specifies the time between link-state

advertisement (LSA) retransmissions for
Example:
adjacencies belonging to the interface. The
apic1(config-interface-policy)# range is 1 to 65535 seconds.
retransmit-interval 5

Step 25 [no] transmit-delay seconds Sets the estimated time required to send a
link-state update packet on the interface. The
Example:
range is from 1 to 450 seconds.
apic1(config-interface-policy)#
transmit-delay 2

Examples
This example shows how to configure a VRF template and an interface template.

apic1# configure
apic1(config)# leaf 101

# CONFIGURING THE VRF TEMPLATE:

apic1(config-leaf)# template ospf vrf-policy vrfTemplate3 tenant exampleCorp
apic1(config-vrf-policy)# timers throttle lsa 200 10000 45000
apic1(config-vrf-policy)# timers lsa-group-pacing 240
apic1(config-vrf-policy)# timers lsa-arrival 1000
apic1(config-vrf-policy)# timers throttle spf 5 1000 90000

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

199
 Configuring Layer 3 External Connectivity
BGP Configuration

apic1(config-vrf-policy)# auto-cost reference-bandwidth 1000

apic1(config-vrf-policy)# distance 200
apic1(config-vrf-policy)# maximum-paths 8
apic1(config-vrf-policy)# graceful-restart helper-disable
apic1(config-vrf-policy)# prefix-suppression
apic1(config-vrf-policy)# name-lookup
apic1(config-vrf-policy)# exit

# CONFIGURING THE INTERFACE TEMPLATE:

apic1(config-leaf)# template ospf interface-policy ifTemplate5 tenant exampleCorp
apic1(config-ospf-if-policy)# advertise-subnet
apic1(config-ospf-if-policy)# cost 300
apic1(config-ospf-if-policy)# dead-interval 60
apic1(config-ospf-if-policy)# hello-interval 10
apic1(config-ospf-if-policy)# mtu-ignore
apic1(config-ospf-if-policy)# network p2p
apic1(config-ospf-if-policy)# passive-interface
apic1(config-ospf-if-policy)# priority 4
apic1(config-ospf-if-policy)# retransmit-interval 5
apic1(config-ospf-if-policy)# transmit-delay 2

BGP Configuration
Configuring BGP
Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 bgp-fabric Enters BGP configuration mode for the fabric.

Example:
apic1(config)# bgp-fabric

Step 3 asn asn-number Specifies the BGP autonomous system number

(ASN).
Example:
apic1(config-bgp-fabric)# asn 100

Step 4 route-reflector spine spine-id Configures the specified spine switch to be a

BGP route reflector.
Example:
apic1(config-bgp-fabric)# route-reflector
spine 105

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

200
 Configuring Layer 3 External Connectivity
Creating BGP Address Family and Timer Templates

Examples

apic1# configure
apic1(config)# bgp-fabric
apic1(config-bgp-fabric)# asn 100
apic1(config-bgp-fabric)# route-reflector spine 105

What to do next
Configure BGP address family and counters.

Creating BGP Address Family and Timer Templates

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 template bgp timers timer-policy-name Creates the BGP timers policy template under
tenant tenant-name the specified tenant.
Example:
apic1(config-leaf)# template bgp timers
bgpTimers tenant exampleCorp
This template will be available on all
leaves
where tenant exampleCorp has a VRF
deployment

Step 4 graceful-restart-helper Configure BGP Policy Graceful Restart Helper

apic1(config-bgp-timers)#
graceful-restart-helper

Step 5 graceful-restart stalepath-time seconds Sets the maximum time that BGP keeps stale
routes from the restarting BGP peer. The range
is 1 to 3600 seconds.
apic1(config-bgp-timers)#
graceful-restart stalepath-time 3600

Step 6 timers bgp keep-alive-seconds hold-seconds Sets the keep-alive timer and hold timer values.
The range for both is 1 to 3600 seconds.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

201
 Configuring Layer 3 External Connectivity
Configuring BGP Address Family and Timers

Command or Action Purpose

apic1(config-bgp-timers)# timers bgp 10
20

Step 7 exit apic1(config-bgp-timers)# exit

Step 8 template bgp address-family family-name Creates the BGP address family template under
tenant tenant-name the specified tenant.
Example:
apic1(config-leaf)# template bgp
address-family bgpAf1 tenant exampleCorp
This template will be available on all
leaves
where tenant exampleCorp has a VRF
deployment

Step 9 distance ebgp-distance ibgp-distance Sets the administrative distance for eBGP
local-distance routes, iBGP routes, and local routes. The
range is 1 to 255.
apic1(config-bgp-af)# distance 250 240
230

Step 10 exit Returns to leaf configuration mode.

apic1(config-bgp-af)# exit

Examples
This example shows how to create a BGP timer template and an address family template.

apic1# configure
apic1(config)# leaf 101

# CREATE A TIMER TEMPLATE

apic1(config-leaf)# template bgp timers bgpTimers tenant exampleCorp
This template will be available on all leaves where tenant exampleCorp has a VRF deployment
apic1(config-bgp-timers)# timers bgp 10 20
apic1(config-bgp-timers)# graceful-restart stalepath-time 3600
apic1(config-bgp-timers)# exit

# CREATE AN ADDRESS FAMILY TEMPLATE

apic1(config-leaf)# template bgp address-family bgpAf1 tenant bgp_t1
This template will be available on all leaves where tenant exampleCorp has a VRF deployment
apic1(config-bgp-af)# distance 250 240 230
apic1(config-bgp-af)# exit
apic1(config-leaf)# exit

Configuring BGP Address Family and Timers

Before you begin
Create a BGP address family template and timer template.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

202
Configuring Layer 3 External Connectivity
Configuring BGP Address Family and Timers

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 router bgp asn-number Enters BGP policy configuration.

Example:
apic1(config-leaf)# router bgp 100

Step 4 vrf member tenant tenant-name vrf vrf-name Specifies the VRF instance to associate with
subsequent address family configuration mode
Example:
commands.
apic1(config-bgp)# vrf member tenant
exampleCorp vrf v100

Step 5 inherit bgp timer timer-name Applies an existing timer configuration.

Example:
apic1(config-leaf-bgp-vrf)# inherit bgp
timer bgpTimers
This template will be inherited on all
leaves where VRF v100 has been deployed

Step 6 address-family {ipv4 | ipv6} unicast Declares neighbors with whom we want to
exchange normal IPv4 unicast routes.
Example:
apic1(config-leaf-bgp-vrf)#
address-family ipv4 unicast

Step 7 inherit bgp address-family family-name Adds the specified address family to this
address family.
Example:
apic1(config-leaf-bgp-vrf-af)# inherit
bgp address-family ipv4-af-pol
This template will be inherited on all
leaves where VRF v100 has been deployed

Step 8 exit
Example:
apic1(config-leaf-bgp-vrf-af)# exit

Examples
This example shows how to inherit a BGP timer configuration and IPv4 and IPv6 address families.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

203
 Configuring Layer 3 External Connectivity
Configuring a BGP Neighbor

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# router bgp 100
apic1(config-bgp)# vrf member tenant exampleCorp vrf v100
apic1(config-leaf-bgp-vrf)# inherit bgp timer bgpTimers
This template will be inherited on all leaves where VRF v100 has been deployed
apic1(config-leaf-bgp-vrf)# address-family ipv4 unicast
apic1(config-leaf-bgp-vrf-af)# inherit bgp address-family ipv4-af-pol
This template will be inherited on all leaves where VRF v100 has been deployed
apic1(config-leaf-bgp-vrf-af)# exit
apic1(config-leaf-bgp-vrf)# address-family ipv6 unicast
apic1(config-leaf-bgp-vrf-af)# inherit bgp address-family ipv6-af-pol
This template will be inherited on all leaves where VRF v100 has been deployed
apic1(config-leaf-bgp-vrf-af)# exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf)# exit

Configuring a BGP Neighbor

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 router bgp asn-number Enters BGP policy configuration.

Example:
apic1(config-leaf)# router bgp 100

Step 4 vrf member tenant tenant-name vrf vrf-name Specifies the VRF instance to associate with
subsequent policy configuration mode
Example:
commands.
apic1(config-bgp)# vrf member tenant
exampleCorp vrf v100

Step 5 (Optional) aggregate-address Configures a summary address for a range of

ip-address/masklength [as-set] addresses and creates an aggregate entry in a
BGP database. The address can be either IPv4
Example:
or IPv6. The as-set option generates
apic1(config-leaf-bgp-vrf)# autonomous system set path information.
aggregate-address 192.0.10.0/24 as-set

Step 6 neighbor ip-address [/masklength] Specifies the IP address of the neighbor.

Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

204
Configuring Layer 3 External Connectivity
Configuring a BGP Neighbor

Command or Action Purpose

apic1(config-leaf-bgp-vrf)# neighbor
192.0.2.229/32

Step 7 address-family {ipv4 | ipv6} unicast Declares neighbors with whom we want to
exchange normal IPv4 unicast routes.
Example:
apic1(config-leaf-bgp-vrf-neighbor)#
address-family ipv4 unicast

Step 8 [no] maximum-prefix count [action {log | Sets the maximum number of prefixes from
shut | restart [restart-time minutes]}] this neighbor. the range is 1 to 300000
[threshold percent] prefixes. Other optional settings are:
Example: • action — The action to be performed
apic1(config-leaf-bgp-vrf-neighbor-af)# when the maximum prefix limit is
maximum-prefix 10 threshold 10 action reached. If the action is restart , you
restart restart-time 10 can optionally specify the restart-time
, which is the period of time in minutes
before restarting the peer when the
maximum prefix limit is reached. The
range is 1 to 65535 minutes.
• threshold — The threshold percentage
of the maximum number of prefixes
before a warning is issued. The range is
1 to 100 percent.

Step 9 exit
Example:
apic1(config-leaf-bgp-vrf-neighbor-af)#
exit

Step 10 update-source {loopback ip-address | if the neighbor address is being learned through
ethernet ip-address | vlan vlan-id} OSPF, specify the same loopback address as
being used under OSPF.
Example:
apic1(config-leaf-bgp-vrf-neighbor)#
update-source loopback 192.0.2.230

Step 11 weight number Specifies the weight attribute to select a best

path. A weight can from 0 to 65,535. Routes
Example:
with a higher weight value have preference
apic1(config-leaf-bgp-vrf-neighbor)# when there are multiple routes to the same
weight 2000
destination.

Step 12 private-as-control {remove-exclusive | Removes private autonomous system numbers

remove-exclusive-all | from the autonomous system path. Private AS
remove-exclusive-all-replace-as} numbers can be removed from the AS path on
a per peer basis and can only be used for eBGP
Example:
peers according to the following three possible
apic1(config-leaf-bgp-vrf-neighbor)# variations:
private-as-control

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

205
 Configuring Layer 3 External Connectivity
Configuring a BGP Neighbor

Command or Action Purpose

• remove-exclusive —Remove if AS path
has only private AS numbers.
• remove-exclusive-all —Remove if AS
path has both private and public AS
numbers.
• remove-exclusive-all-replace-as
—Replaces private AS numbers with the
router’s local AS number.

This command is shown as an example. At this

point you can configure any of the neighbor
settings shown in the table below.

The following table shows the interface settings that can be configured at this point.

Command Purpose

allow-self-as Accept as-path with my AS present in it

allowed-self-as-count count The number of occurrences of a local access service

network

disable-connected-check Disable check for directly connected peer

disable-peer-as-check Disable checking of peer AS-number while advertising

ebgp-multihop count Specify multihop TTL for remote peer

local-as asn Local Autonomous System Configuration for a BGP

Peer

next-hop-self Set our peering address as nexthop

password password Configure a password for neighbor

private-as-control Removes private ASNs from the AS path

remote-as asn Specify Autonomous System Number of the neighbor

route-map name {in | out} Apply route-map to neighbor

send-community [extended] Send Community attribute to this neighbor

update-source vlan vlan-id Source Vlan Interface

update-source ethernet slot/port Source Ethernet Interface

update-source loopback ip-address Source Loopback Interface

weight number BGP weight for the routing table

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

206
Configuring Layer 3 External Connectivity
Configuring a BGP Neighbor

Examples
This example shows how to configure an IPv4 BGP neighbor.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# router bgp 100
apic1(config-bgp)# vrf member tenant exampleCorp vrf v100
apic1(config-leaf-bgp-vrf)# aggregate-address 192.0.10.0/24 as-set
apic1(config-leaf-bgp-vrf)# neighbor 192.0.2.229/32
apic1(config-leaf-bgp-vrf-neighbor)# address-family ipv4 unicast
apic1(config-leaf-bgp-vrf-neighbor-af)# maximum-prefix 10 threshold 10 action restart
restart-time 10
apic1(config-leaf-bgp-vrf-neighbor-af)# exit
apic1(config-leaf-bgp-vrf-neighbor)# allow-self-as
apic1(config-leaf-bgp-vrf-neighbor)# allowed-self-as-count 2
apic1(config-leaf-bgp-vrf-neighbor)# disable-connected-check
apic1(config-leaf-bgp-vrf-neighbor)# disable-peer-as-check
apic1(config-leaf-bgp-vrf-neighbor)# ebgp-multihop 4
apic1(config-leaf-bgp-vrf-neighbor)# local-as 100
apic1(config-leaf-bgp-vrf-neighbor)# next-hop-self
apic1(config-leaf-bgp-vrf-neighbor)# password abcdef
apic1(config-leaf-bgp-vrf-neighbor)# remote-as 200
apic1(config-leaf-bgp-vrf-neighbor)# send-community extended
apic1(config-leaf-bgp-vrf-neighbor)# update-source vlan 601
apic1(config-leaf-bgp-vrf-neighbor)# update-source ethernet 1/15
apic1(config-leaf-bgp-vrf-neighbor)# update-source loopback 192.0.2.230
Warning: BGP Configuration changed. Please re-configure BGP Password if it was enabled
apic1(config-leaf-bgp-vrf-neighbor)# local-as 100 no-prepend replace-as dual-as
apic1(config-leaf-bgp-vrf-neighbor)# route-map rMapT3 out
apic1(config-leaf-bgp-vrf-neighbor)# weight 2000
apic1(config-leaf-bgp-vrf-neighbor)# private-as-control
apic1(config-leaf-bgp-vrf-neighbor)# exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf)# exit

This example shows how to configure an IPv6 BGP neighbor.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# router bgp 100
apic1(config-bgp)# vrf member tenant exampleCorp vrf v100
apic1(config-leaf-bgp-vrf)# neighbor 2001:80:1:2::229
apic1(config-leaf-bgp-vrf-neighbor)# address-family ipv6 unicast
apic1(config-leaf-bgp-vrf-neighbor-af)# maximum-prefix 100
apic1(config-leaf-bgp-vrf-neighbor-af)# exit
apic1(config-leaf-bgp-vrf-neighbor)# allow-self-as
apic1(config-leaf-bgp-vrf-neighbor)# allowed-self-as-count 2
apic1(config-leaf-bgp-vrf-neighbor)# disable-connected-check
apic1(config-leaf-bgp-vrf-neighbor)# disable-peer-as-check
apic1(config-leaf-bgp-vrf-neighbor)# ebgp-multihop 4
apic1(config-leaf-bgp-vrf-neighbor)# local-as 100
apic1(config-leaf-bgp-vrf-neighbor)# next-hop-self
apic1(config-leaf-bgp-vrf-neighbor)# password abcdef
apic1(config-leaf-bgp-vrf-neighbor)# remote-as 200
apic1(config-leaf-bgp-vrf-neighbor)# send-community extended
apic1(config-leaf-bgp-vrf-neighbor)# update-source vlan 601
apic1(config-leaf-bgp-vrf-neighbor)# update-source ethernet 1/15
apic1(config-leaf-bgp-vrf-neighbor)# update-source loopback 2001:80:1:2::230/128

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

207
 Configuring Layer 3 External Connectivity
Configuring a Per VRF Per Node BGP Timer Policy Using the NX-OS Style CLI

Warning: BGP Configuration changed. Please re-configure BGP Password if it was enabled
apic1(config-leaf-bgp-vrf-neighbor)# local-as 100 no-prepend replace-as dual-as
apic1(config-leaf-bgp-vrf-neighbor)# route-map rMapT3 out
apic1(config-leaf-bgp-vrf-neighbor)# weight 2000
apic1(config-leaf-bgp-vrf-neighbor)# private-as-control
apic1(config-leaf-bgp-vrf-neighbor)# exit
apic1(config-leaf-bgp-vrf-af)# exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf)# exit

Configuring a Per VRF Per Node BGP Timer Policy Using the NX-OS Style CLI
Procedure

Command or Action Purpose

Step 1 Configure BGP ASN and the route reflector
before creating a timer policy.
Example:
apic1(config)#
apic1(config)# bgp-fabric
apic1(config-bgp-fabric)# route-reflector
spine 102
apic1(config-bgp-fabric)# asn 42
apic1(config-bgp-fabric)# exit
apic1(config)# exit
apic1#

Step 2 Create a timer policy. The specific values are provided as examples
only.
Example:
apic1# config
apic1(config)# leaf 101
apic1(config-leaf)# template bgp timers
pol7 tenant tn1
This template will be available on all
nodes where tenant tn1 has a VRF
deployment
apic1(config-bgp-timers)# timers bgp 120
240
apic1(config-bgp-timers)#
graceful-restart stalepath-time 500
apic1(config-bgp-timers)# maxas-limit
300
apic1(config-bgp-timers)# exit
apic1(config-leaf)# exit
apic1(config)# exit
apic1#

Step 3 Display the configured BGP policy.

Example:

apic1# show run leaf 101 template bgp

timers pol7
# Command: show running-config leaf 101

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

208
 Configuring Layer 3 External Connectivity
Configuring BGP Max Path

Command or Action Purpose

template bgp timers pol7
leaf 101
template bgp timers pol7 tenant tn1
timers bgp 120 240
graceful-restart stalepath-time
500
maxas-limit 300
exit
exit

Step 4 Refer to a specific policy at a node.

Example:
apic1# config
apic1(config)# leaf 101
apic1(config-leaf)# router bgp 42
apic1(config-leaf-bgp)# vrf member tenant
tn1 vrf ctx1
apic1(config-leaf-bgp-vrf)# inherit
node-only bgp timer pol7
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf-bgp)# exit
apic1(config-leaf)# exit
apic1(config)# exit
apic1#

Step 5 Display the node specific BGP timer policy.

Example:

apic1# show run leaf 101 router bgp 42

vrf member tenant tn1 vrf ctx1
# Command: show running-config leaf 101
router bgp 42 vrf member tenant tn1 vrf
ctx1
leaf 101
router bgp 42
vrf member tenant tn1 vrf ctx1
inherit node-only bgp timer pol7

exit
exit
exit
apic1#

Configuring BGP Max Path

Before you begin
The appropriate tenant and the BGP external routed network are created and available.
The following feature enables you to add the maximum number of paths to the route table to enable equal
cost, multipath load balancing.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

209
 Configuring Layer 3 External Connectivity
Configuring AS Path Prepend

The two properties which enable you to configure more paths are maxEcmp and maxEcmpIbgp in the
bgpCtxAfPol object. After you configure these two properties, they are propagated to the rest of your
implementation.
Use the following commands when logged in to BGP:
maximum-paths [ibgp]
no maximum-paths [ibgp]

Example Configuration:

Procedure

Example:
apic1(config)# leaf 101
apic1(config-leaf)# template bgp address-family newAf tenant t1
This template will be available on all nodes where tenant t1 has a VRF deployment
apic1(config-bgp-af)# maximum-paths ?
<1-16> Maximum number of equal-cost paths for load sharing. The default is 16.
ibgp Configure multipath for IBGP paths
apic1(config-bgp-af)# maximum-paths 10
apic1(config-bgp-af)# maximum-paths ibpg 8
apic1(config-bgp-af)# end
apic1#
no maximum-paths [ibgp]

Configuring AS Path Prepend

A BGP peer can influence the best-path selection by a remote peer by increasing the length of the AS-Path
attribute. AS-Path Prepend provides a mechanism that can be used to increase the length of the AS-Path
attribute by prepending a specified number of AS numbers to it.
AS-Path prepending can only be applied in the outbound direction using route-maps. AS Path prepending
does not work in iBGP sessions.
The AS Path Prepend feature enables modification as follows:

Prepend Appends the specified AS number to the AS path of the route matched by
the route map.
Note • You can configure more than one AS number.
• 4 byte AS numbers are supported.
• You can prepend a total 32 AS numbers. You must specify
the order in which the AS Number is inserted into the AS
Path attribute.

Prepend-last-as Prepends the last AS numbers to the AS path with a range between 1 and 10.

The following table describes the selection criteria for implementation of AS Path Prepend:

Prepend 1 Prepend the specified AS number.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

210
 Configuring Layer 3 External Connectivity
Configuring AS Path Prepend Using the NX-OS Style CLI

Prepend-last-as 2 Prepend the last AS numbers to the AS path.

DEFAULT Prepend(1) Prepend the specified AS number.

Configuring AS Path Prepend Using the NX-OS Style CLI

This section provides information on how to configure the AS Path Prepend feature using the NX-OS style
command line interface (CLI).

Before you begin

A configured tenant.

Procedure

To modify the autonomous system path (AS Path) for Border Gateway Protocol (BGP) routes, you can use
the set as-path command. The set as-path command takes the form of
apic1(config-leaf-vrf-template-route-profile)# set as-path {'prepend as-num [ ,... as-num ]
| prepend-last-as num}

Example:
apic1(config)# leaf 103
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# template route-profile rp1
apic1(config-leaf-vrf-template-route-profile)# set as-path ?
prepend Prepend to the AS-Path
prepend-last-as Prepend last AS to the as-path
apic1(config-leaf-vrf-template-route-profile)# set as-path prepend 100, 101, 102, 103
apic1(config-leaf-vrf-template-route-profile)# set as-path prepend-last-as 8
apic1(config-leaf-vrf-template-route-profile)# exit
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# exit

What to do next
To disable AS Path prepend, use the no form of the shown command:
apic1(config-leaf-vrf-template-route-profile)# [no] set
as-path { prepend as-num [ ,... as-num ] | prepend-last-as num}

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

211
 Configuring Layer 3 External Connectivity
Route Distribution Into BGP

Route Distribution Into BGP

Configuring a Route-Profile with Tenant Scope

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 template route-profile profile-name tenant Creates a route-profile template under tenant
tenant-name for BGP dampening and route redistribution.
Example:
apic1(config-leaf)# template
route-profile map_eigrp tenant
exampleCorp

Step 4 Required: [no] set tag name Sets the tag value. The name parameter is an
unsigned integer.
Example:
apic1(config-leaf-template-route-profile)#
set tag 200

Step 5 Required: exit Returns to leaf configuration mode.

Example:
apic1(config-leaf-template-route-profile)#
exit

Step 6 template route-profile profile-name tenant Creates a route-profile template under tenant
tenant-name for BGP dampening and route redistribution.
Example:
apic1(config-leaf)# template
route-profile map_ospf tenant exampleCorp

Step 7 Required: [no] set tag name Sets the tag value. The name parameter is an
unsigned integer.
Example:
apic1(config-leaf-template-route-profile)#
set tag 100

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

212
 Configuring Layer 3 External Connectivity
Configuring a Redistribute Route-Profile

Example

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# template route-profile map_eigrp tenant exampleCorp
apic1(config-leaf-template-route-profile)# set tag 200
apic1(config-leaf-template-route-profile)# exit
apic1(config-leaf)# template route-profile map_ospf tenant exampleCorp
apic1(config-leaf-template-route-profile)# set tag 100
apic1(config-leaf-template-route-profile)# exit

What to do next
Configure a redistribute route-profile under BGP for OSPF and EIGRP using one of the route-profiles created
in this procedure.

Configuring a Redistribute Route-Profile

Before you begin

Create a route-profile template under tenant for route redistribution.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 router bgp asn-number Enters BGP policy configuration.

Example:
apic1(config-leaf)# router bgp 100

Step 4 vrf member tenant tenant-name vrf vrf-name Specifies the VRF instance to associate with
subsequent policy configuration mode
Example:
commands.
apic1(config-bgp)# vrf member tenant
exampleCorp vrf v100

Step 5 Required: redistribute {ospf | eigrp}

route-map map-name
Example:
apic1(config-leaf-bgp-vrf)# redistribute
ospf route-map map_ospf

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

213
 Configuring Layer 3 External Connectivity
Configuring BGP Route Dampening

Example
This example configures a redistribute route-profile under BGP for OSPF and EIGRP using the
route-profiles created in the example in Creating a Route-Profile with Tenant Scope. The redistribute
route-map allows (permits) all routes and applies the route-profile for the route-control actions. In
this example, all EIGRP learned routes will be redistributed into BGP with tag 200 and OSPF routes
will be redistributed into BGP with tag 100.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# router bgp 100
apic1(config-bgp)# vrf member tenant exampleCorp vrf v1
apic1(config-leaf-bgp-vrf)# redistribute eigrp route-map map_eigrp
apic1(config-leaf-bgp-vrf)# redistribute ospf route-map map_ospf

Configuring BGP Route Dampening

BGP route dampening minimizes propagation into the fabric of flapping eBGP routes received from external
routers connected to border leaf switches (BLs). Frequently flapping routes from external routers are suppressed
on BLs based on configured criteria and prohibited from redistribution to iBGP peers (ACI spine switches).
Suppressed routes are reused after a configured time criteria. Each flap penalizes the eBGP route with a penalty
of 1000. When the flap penalty reaches a defined suppress-limit threshold (default 2000) the eBGP route is
marked as dampened. Dampened routes are not advertised to other BGP peers. The penalty is decremented
to half after every half-life interval (the default is 15 minutes). A dampened route is reused if the penalty falls
below a specified reuse-limit (the default is 750). A dampened route is suppressed at most for a specified
maximum suppress time (maximum of 45 minutes).

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 template route-profile profile-name tenant Creates a route-profile template under tenant
tenant-name for BGP dampening and route redistribution.
Example:
apic1(config-leaf)# template
route-profile damp_rp tenant exampleCorp

Step 4 Required: [no] set dampening half-life reuse Configures route flap dampening behavior.
suppress max-suppress-time The parameters are:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

214
Configuring Layer 3 External Connectivity
Configuring BGP Route Dampening

Command or Action Purpose

Example: • half-life—Decay half life, which is the
apic1(config-leaf-template-route-profile)# time in minutes after which a penalty is
set dampening 15 750 2000 60 decreased. Once the route has been
assigned a penalty, the penalty is
decreased by half after the half life period.
The range is 1 to 60 minutes; the default
is 15 minutes.
• reuse—A route is unsuppressed (reused)
if the penalty for a flapping route
decreases enough to fall below this value.
The range is 1 to 20000; the default is
750.
• suppress—A route is suppressed when
its penalty exceeds this limit. The range
is 1 to 20000; the default is 2000.
• max-suppress-time—The maximum time
in minutes that a stable route can be
suppressed. The range is 1 to 255.

Step 5 Required: exit Returns to leaf configuration mode.

Example:
apic1(config-leaf-template-route-profile)#
exit

Step 6 router bgp asn-number Enters BGP policy configuration.

Example:
apic1(config-leaf)# router bgp 100

Step 7 vrf member tenant tenant-name vrf vrf-name Specifies the VRF instance to associate with
subsequent policy configuration mode
Example:
commands.
apic1(config-bgp)# vrf member tenant
exampleCorp vrf v100

Step 8 neighbor ip-address [/masklength] Specifies the IP address of the neighbor. The
mask length must be 32.
Example:
apic1(config-leaf-bgp-vrf)# neighbor
192.0.2.229/32

Step 9 address-family {ipv4 | ipv6} unicast Declares neighbors with whom we want to
exchange normal IPv4 unicast routes.
Example:
apic1(config-leaf-bgp-vrf-neighbor)#
address-family ipv4 unicast

Step 10 inherit bgp dampening profile-name

Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

215
 Configuring Layer 3 External Connectivity
Configuring BGP Route Dampening

Command or Action Purpose

apic1(config-leaf-bgp-vrf-neighbor-af)#
inherit bgp dampening damp_rp

Step 11 exit
Example:
apic1(config-leaf-bgp-vrf-neighbor-af)#
exit

Step 12 exit
Example:
apic1(config-leaf-bgp-vrf-neighbor)#
exit

Step 13 address-family {ipv4 | ipv6} unicast Declares neighbors with whom we want to
exchange normal IPv4 unicast routes.
Example:
apic1(config-leaf-bgp-vrf)#
address-family ipv4 unicast

Step 14 inherit bgp dampening profile-name

Example:
apic1(config-leaf-bgp-vrf-af)# inherit
bgp dampening damp_rp

Step 15 exit
Example:
apic1(config-leaf-bgp-vrf-af)# exit

Example

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# template route-profile damp_rp tenant exampleCorp
apic1(config-leaf-template-route-profile)# set dampening 15 750 2000 60
apic1(config-leaf-template-route-profile)# exit
apic1(config-leaf)# router bgp 100
apic1(config-bgp)# vrf member tenant exampleCorp vrf v100
apic1(config-leaf-bgp-vrf)# neighbor 192.0.2.229/32
apic1(config-leaf-bgp-vrf-neighbor)# address-family ipv4 unicast
apic1(config-leaf-bgp-vrf-neighbor-af)# inherit bgp dampening damp_rp
apic1(config-leaf-bgp-vrf-neighbor-af)# exit
apic1(config-leaf-bgp-vrf)# address-family ipv6 unicast
apic1(config-leaf-bgp-vrf-af)# inherit bgp dampening damp_rp
apic1(config-leaf-bgp-vrf-af)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

216
 Configuring Layer 3 External Connectivity
EIGRP Configuration

EIGRP Configuration
Creating EIGRP VRF and Interface Templates
Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 template eigrp vrf-policy vrf-policy-name Creates the EIGRP VRF policy template under
tenant tenant-name the specified tenant.
Example:
apic1(config-leaf)# template eigrp
vrf-policy vrfTemplate3 tenant
exampleCorp
This template will be available on all
leaves where tenant exampleCorp has a
VRF deployment

Step 4 distance internal external Sets EIGRP administrative distance preference

for internal and external routes. The distances
Example:
can be 1 to 255.
apic1(config-template-eigrp-vrf-pol)#
distance 2 5

Step 5 maximum-paths limit Sets EIGRP Maximum Path Limit for the VRF
policy template. The limit can be 1 to 32.
Example:
apic1(config-template-eigrp-vrf-pol)#
maximum-paths 8

Step 6 metric version 64bit Sets EIGRP metric style to wide metric (64
bits).
Example:
apic1(config-template-eigrp-vrf-pol)#
metric version 64bit

Step 7 timers active-time minutes Sets EIGRP active timer interval. The range
is 1 to 65535 minutes.
Example:
apic1(config-template-eigrp-vrf-pol)#
timers active-time 1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

217
 Configuring Layer 3 External Connectivity
Creating EIGRP VRF and Interface Templates

Command or Action Purpose

Step 8 template eigrp interface-policy Creates the EIGRP interface policy template
if-policy-name tenant tenant-name under the specified tenant.
Example:
apic1(config-leaf)# template eigrp
interface-policy ifTemplate5 tenant
exampleCorp
This template will be available on all
leaves where tenant exampleCorp has a
VRF deployment

Step 9 ip hello-interval eigrp default seconds Sets EIGRP hello interval time. The range is
1 to 65535 seconds.
Example:
apic1(config-template-eigrp-if-pol)# ip
hello-interval eigrp default 10

Step 10 ip hold-interval eigrp default seconds Sets EIGRP hold interval time. The range is 1
to 65535 seconds.
Example:
apic1(config-template-eigrp-if-pol)# ip
hold-interval eigrp default 10

Step 11 ip next-hop-self eigrp default Sets EIGRP next-hop-self flag.

Example:
apic1(config-template-eigrp-if-pol)# ip
next-hop-self eigrp default

Step 12 ip passive-interface eigrp default Set EIGRP passive-interface flag.

Example:
apic1(config-template-eigrp-if-pol)# ip
passive-interface eigrp default

Step 13 ip split-horizon eigrp default Sets EIGRP split-horizon flag.

Example:
apic1(config-template-eigrp-if-pol)# ip
split-horizon eigrp default

Step 14 exit Returns to leaf configuration mode.

Example:
apic1(config-template-eigrp-if-pol)#
exit

Examples

apic1# configure
apic1(config)# leaf 101

# CONFIGURING THE VRF TEMPLATE:

apic1(config-leaf)# template eigrp vrf-policy vrfTemplate3 tenant exampleCorp

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

218
 Configuring Layer 3 External Connectivity
Configuring EIGRP Address Family and Counters

This template will be available on all leaves where tenant exampleCorp has a VRF deployment
apic1(config-template-eigrp-vrf-pol)# distance 2 5
apic1(config-template-eigrp-vrf-pol)# maximum-paths 8
apic1(config-template-eigrp-vrf-pol)# metric version 64bit
apic1(config-template-eigrp-vrf-pol)# timers active-time 1
apic1(config-template-eigrp-vrf-pol)# exit

# CONFIGURING THE INTERFACE TEMPLATE:

apic1(config-leaf)# template eigrp interface-policy ifTemplate5 tenant exampleCorp
This template will be available on all leaves where tenant exampleCorp has a VRF deployment
apic1(config-template-eigrp-if-pol)# ip hello-interval eigrp default 5
apic1(config-template-eigrp-if-pol)# ip hold-interval eigrp default 10
apic1(config-template-eigrp-if-pol)# ip next-hop-self eigrp default
apic1(config-template-eigrp-if-pol)# ip passive-interface eigrp default
apic1(config-template-eigrp-if-pol)# ip split-horizon eigrp default
apic1(config-template-eigrp-if-pol)# exit
apic1(config-leaf)# exit
apic1(config)# exit

What to do next
Configure EIGRP address family and counters.

Configuring EIGRP Address Family and Counters

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 router eigrp default Enters EIGRP policy configuration.

Example:
apic1(config-leaf)# router eigrp default

Step 4 vrf member tenant tenant-name vrf vrf-name Specifies the VRF instance to associate with
subsequent address family configuration mode
Example:
commands.
apic1(config-eigrp)# vrf member tenant
exampleCorp vrf v100

Step 5 autonomous-system asn Enters Autonomous System configuration for

EIGRP.
Example:
apic1(config-eigrp-vrf)#
autonomous-system 300

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

219
 Configuring Layer 3 External Connectivity
Configuring EIGRP Address Family and Counters

Command or Action Purpose

Step 6 address-family {ipv4 | ipv6} unicast Configures an EIGRP policy address family.
Example:
apic1(config-eigrp-vrf)# address-family
ipv4 unicast

Step 7 distance internal external Sets EIGRP administrative distance preference

for internal and external routes. The distances
Example:
can be 1 to 255.
apic1(config-address-family)# distance
2 5

Step 8 maximum-paths limit Sets EIGRP Maximum Path Limit for the VRF
policy template. The limit can be 1 to 32.
Example:
apic1(config-address-family)#
maximum-paths 8

Step 9 metric version 64bit Sets EIGRP metric style to wide metric (64
bits).
Example:
apic1(config-address-family)# metric
version 64bit

Step 10 timers active-time minutes Sets EIGRP active timer interval. The range
is 1 to 65535 minutes.
Example:
apic1(config-address-family)# timers
active-time 1

Step 11 inherit eigrp vrf-policy vrf-policy-name Applies an EIGRP VRF policy to this address
family.
Example:
apic1(config-address-family)# inherit
eigrp vrf-policy vrfTemplate3

Examples
This example shows how to configure an EIGRP address family and inherit an EIGRP VRF policy.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# router eigrp default
apic1(config-eigrp)# vrf member tenant exampleCorp vrf v100
apic1(config-eigrp-vrf)# autonomous-system 300
apic1(config-eigrp-vrf)# address-family ipv4 unicast
This configuration will affect all leaves where VRF v100 has been deployed
apic1(config-address-family)# distance 2 5
This configuration will affect all leaves where VRF v100 has been deployed
apic1(config-address-family)# maximum-paths 8
This configuration will affect all leaves where VRF v100 has been deployed
apic1(config-address-family)# metric version 64bit
This configuration will affect all leaves where VRF v100 has been deployed
apic1(config-address-family)# timers active-time 1
This configuration will affect all leaves where VRF v100 has been deployed

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

220
 Configuring Layer 3 External Connectivity
Configuring an EIGRP Interface

apic1(config-address-family)# inherit eigrp vrf-policy vrfTemplate3

This template will be inherited on all leaves where VRF v100 has been deployed
apic1(config-address-family)# exit
apic1(config-eigrp-vrf)# exit
apic1(config-eigrp)# exit

Configuring an EIGRP Interface

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 interface ethernet slot/port Specifies the interface to be configured.

Example:
apic1(config-leaf)# interface ethernet
1/21

Step 4 [no] switchport slot/port By default, a port is in Layer 2 trunk mode. If

the port is in Layer 3 mode, it must be
Example:
converted to Layer 2 trunk mode using this
apic1(config-leaf-if)# no switchport command.

Step 5 [no] vlan-domain member vlan-id Creates and enters the configuration mode for
the VLAN domain.
Example:
apic1(config-leaf-if)# vlan-domain
member dom1

Step 6 [no] vrf member tenant exampleCorp vrf Associates the interface with a VRF.
vrf-name
Example:
apic1(config-leaf-if)# vrf member tenant
exampleCorp vrf v100

Step 7 [no] {ip | ipv6} address Sets an IP address for the interface.
ip-address/mask-length
Example:
apic1(config-leaf-if)# ip address
181.12.12.1/24

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

221
 Configuring Layer 3 External Connectivity
Configuring an EIGRP Interface

Command or Action Purpose

Step 8 [no] {ip | ipv6} router eigrp default Sets router EIGRP policies to default.
Example:
apic1(config-leaf-if)# ip router eigrp
default

Step 9 [no] {ip | ipv6} distribute-list eigrp default EIGRP advertises routes that are matched in
route-map map-name out the route-map specified in the distribute-list
command. The route prefixes mentioned in the
Example:
prefix-list in the route-map can be learned from
apic1(config-leaf-if)# ip other protocol sources like BGP, OSPF, Static,
distribute-list eigrp default route-map
rMapT5 out
Connected. Redistribute route-maps are
automatically created based on the
distribute-list command. Note that prefixes
learned from an EIGRP session running on an
another interface on the same switch will not
be filtered by the distribute-list and will always
be advertised out.

Step 10 [no] {ip | ipv6} hello-interval eigrp default Sets EIGRP hello interval time. The range is
seconds 1 to 65535 seconds.
Example:
apic1(config-leaf-if)# ip hello-interval
eigrp default 10

Step 11 [no] {ip | ipv6} hold-interval eigrp default Sets EIGRP hold interval time. The range is 1
seconds to 65535 seconds.
Example:
apic1(config-leaf-if)# ip hold-interval
eigrp default 10

Step 12 [no] {ip | ipv6} next-hop-self eigrp default Sets EIGRP next-hop-self flag.
Example:
apic1(config-leaf-if)# ip next-hop-self
eigrp default

Step 13 [no] {ip | ipv6} passive-interface eigrp Set EIGRP passive-interface flag.
default
Example:
apic1(config-leaf-if)# ip
passive-interface eigrp default

Step 14 [no] {ip | ipv6} split-horizon eigrp default Sets EIGRP split-horizon flag.
Example:
apic1(config-leaf-if)# ip split-horizon
eigrp default

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

222
Configuring Layer 3 External Connectivity
Configuring an EIGRP Interface

Command or Action Purpose

Step 15 [no] inherit eigrp ip interface-policy Applies an EIGRP interface policy to this
if-policy-name interface.
Example:
apic1(config-leaf-if)# inherit eigrp ip
interface-policy ifTemplate5

Step 16 [no] ip summary-address eigrp default Configures route summarization for EIGRP.
ip-prefix A summary address can be configured to
advertise an aggregated prefix on an EIGRP
Example:
session.
apic1(config-leaf-if)# ip
summary-address eigrp default Note A summary address enabled on one
172.10.1.0/24 interface will also be applied on
apic1(config-leaf-if)# ip other EIGRP enabled interfaces on
summary-address eigrp default 2001::/64 the same VRF on the switch.

Examples
This example shows how to configure an EIGRP interface.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/21
apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v100
apic1(config-leaf-if)# ip address 181.12.12.1/24
apic1(config-leaf-if)# ip router eigrp default
apic1(config-leaf-if)# ip distribute-list eigrp default route-map rMapT5 out
distribute list will be updated on all EIGRP interfaces on node 1021 VRF exampleCorp/v100
apic1(config-leaf-if)# ip hello-interval eigrp default 5
apic1(config-leaf-if)# ip hold-interval eigrp default 10
apic1(config-leaf-if)# ip next-hop-self eigrp default
apic1(config-leaf-if)# ip passive-interface eigrp default
apic1(config-leaf-if)# ip split-horizon eigrp default
apic1(config-leaf-if)# inherit eigrp ip interface-policy ifTemplate5
apic1(config-leaf-if)# ip summary-address eigrp default 172.10.1.0/24
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit
apic1(config)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

223
 Configuring Layer 3 External Connectivity
Configuring Route-Maps

Configuring Route-Maps
Configuring Templates
About Route Profiles
A route profile specifies the route-control set actions used in import, export, and redistribute route-maps.
Route profile templates can be defined either under the tenant or under the tenant VRF.

Configuring a Tenant-Scoped Route Profile

This procedure creates a tenant-scoped route profile that is used to configure BGP dampening and route
redistribution.

Before you begin

• Configure a tenant and VRF.
• Enable VRF on a leaf.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 [no] template route-profile profile-name Creates a tenant-scoped route profile.

tenant tenant-name
Example:
apic1(config-leaf)# template
route-profile rp1 tenant exampleCorp

Step 4 Required: [no] set community {regular | Sets the BGP community attribute.
extended} value {none | replace | additive}
Example:
apic1(config-leaf-template-route-profile)#
set community extended 20:22 additive

Step 5 Required: [no] set dampening half-life reuse Configures route flap dampening behavior.
suppress max-suppress-time The parameters are:
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

224
Configuring Layer 3 External Connectivity
Configuring a Tenant-Scoped Route Profile

Command or Action Purpose

apic1(config-leaf-template-route-profile)# • half-life—Decay half life, which is the
set dampening 15 750 2000 60
time in minutes after which a penalty is
decreased. Once the route has been
assigned a penalty, the penalty is
decreased by half after the half life period.
The range is 1 to 60 minutes.
• reuse—A route is unsuppressed (reused)
if the penalty for a flapping route
decreases enough to fall below this value.
The range is 1 to 20000.
• suppress—A route is suppressed when
its penalty exceeds this limit. The range
is 1 to 20000.
• max-suppress-time—The maximum time
in minutes that a stable route can be
suppressed. The range is 1 to 255.

Step 6 Required: [no] set local-preference value Sets the BGP local preference value. The range
is from 0 to 4294967295.
Example:
apic1(config-leaf-template-route-profile)#
set local-preference 64

Step 7 Required: [no] set metric value Sets the metric for the destination routing
protocol.
Example:
apic1(config-leaf-template-route-profile)#
set metric 128

Step 8 Required: [no] set metric-type {type-1 | The options are as follows:
type-2}
• type-1 —OSPF external type 1 metric
Example:
• type-2 —OSPF external type 2 metric
apic1(config-leaf-template-route-profile)#
set metric-type type-2

Step 9 Required: [no] set tag name Sets the tag value for the destination routing
protocol. The name parameter is an unsigned
Example:
integer.
apic1(config-leaf-template-route-profile)#
set tag 1111

Step 10 Required: [no] set weight weight Sets the tag value for the destination routing
protocol. The weight parameter is an unsigned
Example:
integer.
apic1(config-leaf-template-route-profile)#
set weight 20

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

225
 Configuring Layer 3 External Connectivity
Configuring a VRF-Scoped Route Profile

Examples
This example shows how to configure a tenant-scoped route profile.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# template route-profile rp1 tenant exampleCorp
This template will be available on all leaves where tenant exampleCorp has a VRF deployment
apic1(config-leaf-template-route-profile)# set community extended 20:22 additive
apic1(config-leaf-template-route-profile)# set dampening 15 750 2000 60
apic1(config-leaf-template-route-profile)# set local-preference 64
apic1(config-leaf-template-route-profile)# set metric 128
apic1(config-leaf-template-route-profile)# set metric-type type-2
apic1(config-leaf-template-route-profile)# set tag 1111
apic1(config-leaf-template-route-profile)# set weight 20

Configuring a VRF-Scoped Route Profile

This procedure creates a VRF-scoped route profile including ‘default-export’ and ‘default-import’. This route
profile can be attached to a bridge domain (BD) while ‘matching’ a bridge-domain inside a route map through
the inherit route-profile command.

Note VRF-scoped route profiles name default-export and default-import values, which are automatically
applied on the match statements on the respective export/import route-maps used in the same tenant VRF.

Before you begin

• Configure a tenant and VRF.
• Enable VRF on a leaf.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 [no] vrf context tenant tenant-name vrf Enables VRF on the leaf and enters VRF
vrf-name configuration mode.
Example:
apic1(config-leaf)# vrf context tenant
exampleCorp vrf vrf1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

226
Configuring Layer 3 External Connectivity
Configuring a VRF-Scoped Route Profile

Command or Action Purpose

Step 4 [no] template route-profile profile-name Creates a VRF-scoped route profile.
Example:
apic1(config-leaf-vrf)# template
route-profile default-export

Step 5 Required: [no] set community {regular | Sets the BGP community attribute.
extended} {no-advertise| no-export|value
{none | replace | additive}
Example:
apic1(config-leaf-vrf-template-route-profile)#
set community extended 20:22 additive

Step 6 Required: [no] set local-preference value Sets the BGP local preference value. The range
is from 0 to 4294967295.
Example:
apic1(config-tenant-vrf-route-profile)#
set local-preference 64

Step 7 Required: [no] set metric value Sets the metric for the destination routing
protocol.
Example:
apic1(config-tenant-vrf-route-profile)#
set metric 128

Step 8 Required: [no] set metric-type {type-1 | The options are as follows:
type-2}
• type-1 —OSPF external type 1 metric
Example:
• type-2 —OSPF external type 2 metric
apic1(config-tenant-vrf-route-profile)#
set metric-type type-2

Step 9 Required: [no] set tag name Sets the tag value for the destination routing
protocol. The name parameter is an unsigned
Example:
integer.
apic1(config-tenant-vrf-route-profile)#
set tag 1111

Step 10 Required: [no] set weight weight Sets the tag value for the destination routing
protocol. The weight parameter is an unsigned
Example:
integer.
apic1(config-tenant-vrf-route-profile)#
set weight 20

Examples
This example shows how to configure a VRF-scoped route profile.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant exampleCorp vrf vrf1
apic1(config-leaf-vrf)# template route-profile default-export
apic1(config-leaf-vrf-template-route-profile)# set community extended 20:22 additive

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

227
 Configuring Layer 3 External Connectivity
Creating a Route-Map

apic1(config-leaf-vrf-template-route-profile)# set local-preference 64

apic1(config-leaf-vrf-template-route-profile)# set metric 128
apic1(config-leaf-vrf-template-route-profile)# set metric-type type-2
apic1(config-leaf-vrf-template-route-profile)# set tag 1111
apic1(config-leaf-vrf-template-route-profile)# set weight 20

Creating a Route-Map
Route-maps are created with a prefix-list on a per-tenant basis to indicate the bridge domain public subnets
to be advertised to external routers. In addition, a prefix-list must be created to allow all transit routes to be
advertised to an external router. The prefix-list for transit routes are configured by an administrator. The
default behavior is to deny all transit route advertisement to an external router.

Before you begin

Configure a tenant and VRF.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 [no] vrf context tenant tenant-name vrf Configures a tenant VRF on the node.
vrf-name
Example:
apic1(config-leaf)# vrf context tenant
exampleCorp vrf v1

Step 4 (Optional) [no] router-id ipv4-address Assigns a router ID for routing protocols
running on the VRF. If you do not assign a
Example:
router ID, an ID is generated internally that is
apic1(config-leaf-vrf)# router-id unique to each leaf switch.
1.2.3.4

Step 5 Required: [no] route-map name Creates a route-map and enters route-map
configuration.
Example:
apic1(config-leaf-vrf)# route-map bgpMap

Step 6 Required: [no] ip prefix-list list-name permit Creates a prefix-list under the route-map.
prefix/masklen [le {32 | 128}]
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

228
Configuring Layer 3 External Connectivity
Creating a Route-Map

Command or Action Purpose

apic1(config-leaf-vrf-route-map)# ip
prefix-list list1 permit 13.13.13.0/24

Step 7 Required: [no] match prefix-list list-name Matches a prefix-list that has already been
created and enters the match mode to configure
Example:
the route-control profile for the prefix-list.
apic1(config-leaf-vrf-route-map)# match
prefix-list list1

Step 8 Required: [no] set metric value Sets the metric for the destination routing
protocol.
Example:
apic1(config-leaf-vrf-route-map-match)#
set metric 128

Step 9 Required: [no] set metric-type {type-1 | The options are as follows:
type-2}
• type-1 —OSPF external type 1 metric
Example:
• type-2 —OSPF external type 2 metric
apic1(config-leaf-vrf-route-map-match)#
set metric-type type-2

Step 10 Required: [no] set local-preference value Sets the BGP local preference value. The range
is from 0 to 4294967295.
Example:
apic1(config-leaf-vrf-route-map-match)#
set local-preference 64

Step 11 Required: [no] set community {regular | Sets the community attribute for a BGP route
extended} value {none | replace | additive} update. Specify the community-value in
aa:nn format. Specify the action as one of the
Example:
following:
apic1(config-leaf-vrf-route-map-match)#
set community extended 20:22 additive • additive —Add to existing community
• replace —Replace existing community
• none —Do not change community

Step 12 Required: [no] set tag name Sets the tag value for the destination routing
protocol. The name parameter is an unsigned
Example:
integer.
apic1(config-leaf-vrf-route-map-match)#
set tag 1111

Step 13 Required: [no] set weight value Specifies the BGP weight for the routing table.
Example:
apic1(config-leaf-vrf-route-map-match)#
set weight 32

Step 14 Required: [no] contract {provider| consumer Add contract, required to leak routes (matching
} contract-name [imported] this prefix list) from the VRF.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

229
 Configuring Layer 3 External Connectivity
Creating a Route-Map

Command or Action Purpose

apic1(config-leaf-vrf-route-map-match)#
contract provider prov 1

Step 15 Required: [no] match route group Matches a route group that has already been
group-name [order number ] created and enters the match mode to configure
the route-map.
Example:
apic1(config-leaf-vrf-route-map)# match Repeat the steps 8-13 or only step 18 to
route group g1 order 1 configure the route map for the route group.
See step 17 to inherit the route map instead of
inline set actions.

Step 16 Required: [no] match bridge-domain Matches a bridge domain in order to export its
bd-name public subnets through the protocol.
Example:
apic1(config-leaf-vrf-route-map)#
bridge-domain bd1

Step 17 Required: [no] inherit route-profile Configures route map for bridge domain.
profile-name
Note The route map was already created
Example: using the command template
apic1(config-leaf-vrf-route-map-match)# route-profile.
inherit route-profile rp1

Step 18 Required: [no] bridge-domain-match Configures route map for bridge domain.
Example: Note Disables the bridge domain (BD)
apic1(config-leaf-vrf-route-map)# no match in a route map, eliminating
bridge-domain-match the need to delete the BD
configuration from the route map.
This is required if there are BDs
matched in a route map, and the
route map is used to filter out the
BD subnets using route
group/explicit prefix list.

Examples
This example shows how to create a route-map and add/match a prefix-list, a community-list, and a
bridge-domain.

# CREATE A ROUTE-MAP
apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant exampleCorp vrf v1
apic1(config-leaf-vrf)# route-map bgpMap

# CREATE A PREFIX-LIST
apic1(config-leaf-vrf-route-map)# ip prefix-list list1 permit 13.13.13.0/24
apic1(config-leaf-vrf-route-map)# ip prefix-list list1 permit 14.14.14.0/24

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

230
Configuring Layer 3 External Connectivity
Creating a Route-Map

# MATCH THE PREFIX-LIST

apic1(config-leaf-vrf-route-map)# match prefix-list list1

# CONFIGURE A ROUTE-PROFILE FOR THE PREFIX-LIST

apic1(config-leaf-vrf-route-map-match)# set metric 128
apic1(config-leaf-vrf-route-map-match)# set metric-type type-2
apic1(config-leaf-vrf-route-map-match)# set local-preference 64
apic1(config-leaf-vrf-route-map-match)# set community extended 20:22 additive
apic1(config-leaf-vrf-route-map-match)# set tag 1111
apic1(config-leaf-vrf-route-map-match)# set weight 32
apic1(config-leaf-vrf-route-map-match)# contract provider prov 1

# CREATE COMMUNITY LIST

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# template community-list standard CL_1 65536:20 tenant exampleCorp

# CREATE ROUTE GROUP

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# template route group g1 tenant exampleCorp
apic1(config-route-group)# ip prefix permit 15.15.15.0/24
apic1(config-route-group)# community-list standard 65535:20

# MATCH ROUTE GROUP

apic1(config-leaf)# vrf context tenant exampleCorp vrf v1
apic1(config-leaf-vrf)# route-map bgpMap
apic1(config-leaf-vrf-route-map)# match route group g1 order 1

# CONFIGURE ROUTE PROFILE FOR COMMUNITY-LIST

apic1(config-leaf-vrf-route-map-match)# set metric 128
apic1(config-leaf-vrf-route-map-match)# set metric-type type-2
apic1(config-leaf-vrf-route-map-match)# set local-preference 64
apic1(config-leaf-vrf-route-map-match)# set community extended 20:22 additive
apic1(config-leaf-vrf-route-map-match)# set tag 1111
apic1(config-leaf-vrf-route-map-match)# set weight 32

# CONFIGURE ROUTE PROFILE ROUTE GROUP

apic1(config-leaf-vrf-route-map-match)# set metric 128
apic1(config-leaf-vrf-route-map-match)# set metric-type type-2
apic1(config-leaf-vrf-route-map-match)# set local-preference 64
apic1(config-leaf-vrf-route-map-match)# set community extended 20:22 additive
apic1(config-leaf-vrf-route-map-match)# set tag 1111
apic1(config-leaf-vrf-route-map-match)# set weight 32

# Or CREATE A ROUTE PROFILE TEMPLATE AND INHERIT IT FOR ROUTE GROUP

apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant exampleCorp vrf v1
apic1(config-leaf-vrf)# template route-profile rp1
apic1(config-leaf-vrf-template-route-profile)# set metric 128
apic1(config-leaf-vrf-template-route-profile)# set metric-type type-2
apic1(config-leaf-vrf-template-route-profile)# set local-preference 64
apic1(config-leaf-vrf-template-route-profile)# set community extended 20:22 additive
apic1(config-leaf-vrf-template-route-profile)# set tag 1111
apic1(config-leaf-vrf-template-route-profile)# set weight 32
apic1(config-leaf-vrf-template-route-profile)# exit

apic1(config-leaf-vrf)# route-map bgpMap

apic1(config-leaf-vrf-route-map)# match route group g1 order 1
apic1(config-leaf-vrf-route-map-match)# inherit route-profile rp1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

231
 Configuring Layer 3 External Connectivity
Configuring Route-Maps in Routing Protocols

# CREATE A BRIDGE-DOMAIN
apic1# configure
apic1(config)# tenant exampleCorp
apic1(config-tenant)# vrf context v1
apic1(config-tenant-vrf)# exit
apic1(config-tenant)# bridge-domain bd1
apic1(config-tenant-bd)# vrf member v1
apic1(config-tenant-bd)# exit
apic1(config-tenant)# interface bridge-domain bd1
apic1(config-tenant-interface)# ip address 13.13.13.1/24 scope public
apic1(config-tenant-interface)# exit
apic1(config-tenant)# exit

# CREATE A ROUTE-PROFILE FOR THE BRIDGE-DOMAIN

apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant exampleCorp vrf v1
apic1(config-leaf-vrf)# template route-profile default-export
apic1(config-leaf-vrf-template-route-profile)# set metric 128
apic1(config-leaf-vrf-template-route-profile)# set metric-type type-2
apic1(config-leaf-vrf-template-route-profile)# set local-preference 64
apic1(config-leaf-vrf-template-route-profile)# set community extended 20:22 additive
apic1(config-leaf-vrf-template-route-profile)# set tag 1111
apic1(config-leaf-vrf-template-route-profile)# set weight 20
apic1(config-leaf-vrf-template-route-profile)# exit

# MATCH THE BRIDGE-DOMAIN

apic1(config-leaf-vrf)# route-map bgpMap
apic1(config-leaf-vrf-route-map)# match bridge-domain bd1

# CONFIGURE A ROUTE-PROFILE FOR THE BRIDGE-DOMAIN

apic1(config-leaf-vrf-route-map-match)# inherit route-profile default-export

Configuring Route-Maps in Routing Protocols

The OSPF, BGP, and EIGRP routing protocols use route-maps to filter routes for import and export. For the
general steps required to configure these protocols, see the documentation sections for each. To configure
route-maps in these protocols, use the following commands and see the examples.

Protocol Route-Map Command

BGP [no] route-map map-name {in | out}

OSPF [no] area area-id route-map map-name {in |out }

EIGRP [no] ip distribute list default route-map map-name out

Examples
This example shows how to configure a route-map in BGP, OSPF and EIGRP.

# BGP
apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# router bgp 100
apic1(config-bgp)# vrf member tenant exampleCorp vrf v1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

232
 Configuring Layer 3 External Connectivity
Configuring an Export Map (Inter-VRF Route Leak)

apic1(config-leaf-bgp-vrf)# neighbor 3.3.3.3

apic1(config-leaf-bgp-vrf-neighbor)# route-map map1 out
apic1(config-leaf-bgp-vrf-neighbor)# route-map map2 in
apic1(config-leaf-bgp-vrf-neighbor)# exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-bgp)# exit
apic1(config-leaf)# exit

# OSPF
apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# router ospf default
apic1(config-leaf-ospf)# vrf member tenant exampleCorp vrf v1
apic1(config-leaf-ospf-vrf)# area 0.0.0.1 route-map map1 out
apic1(config-leaf-ospf-vrf)# area 0.0.0.1 route-map map2 in

apic1(config-leaf-ospf-vrf)# exit
apic1(config-leaf-ospf)# exit
apic1(config-leaf)# exit

#EIGRP
apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet 1/3
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1
apic1(config-leaf-if)# ip address 13.13.13.13/24
apic1(config-leaf-if)# ip router eigrp default
apic1(config-leaf-if)# ip distribute-list eigrp default route-map map1 out
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit

Configuring an Export Map (Inter-VRF Route Leak)

Before you begin
• Create a route-map.
• Add prefix-list(s) to the route-map containing prefixes matching routes that need to be leaked.
• Match the prefix-list(s) and add the contract(s) to enable the route leak.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

233
 Configuring Layer 3 External Connectivity
Configuring Bi-Directional Route Forwarding (BFD)

Command or Action Purpose

Step 3 [no] vrf context tenant tenant-name vrf Configures a tenant VRF on the node.
vrf-name
Example:
apic1(config-leaf)# vrf context tenant
exampleCorp vrf v1

Step 4 [no] export map map-name Configures route-map in this VRF to export
(leak) routes from this VRF into consumer
Example:
VRFs.
apic1(config-leaf-vrf)# export map
shared-route-map1

Examples
This example shows how to create and export a route-map.

# CREATE A ROUTE-MAP
apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant exampleCorp vrf v1
apic1(config-leaf-vrf)# router-id 1.2.3.4
apic1(config-leaf-vrf)# route-map shared-route-map1
apic1(config-leaf-vrf-route-map)# ip prefix-list list1 permit 13.13.13.0/24
apic1(config-leaf-vrf-route-map)# match prefix-list list1
apic1(config-leaf-vrf-route-map-match)# contract provider prov1
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# exit

# EXPORT THE ROUTE-MAP

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant exampleCorp vrf v1
apic1(config-leaf-vrf)# export map shared-route-map1

Configuring Bi-Directional Route Forwarding (BFD)

About BFD
Bidirectional Forwarding Detection (BFD) is a detection protocol designed to provide fast forwarding-path
failure detection times for media types, encapsulations, topologies, and routing protocols. You can use BFD
to detect forwarding path failures at a uniform rate, rather than the variable rates for different protocol hello
mechanisms. BFD makes network profiling and planning easier and reconvergence time consistent and
predictable.
Use Bidirectional Forwarding Detection (BFD) to provide sub-second failure detection times in the forwarding
path between ACI fabric border leaf switches configured to support peering router connections.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

234
 Configuring Layer 3 External Connectivity
Configuring BFD Globally

Configuring BFD Globally

You can configure the BFD session parameters for all BFD sessions on the device. The BFD session parameters
are negotiated between the BFD peers in a three-way handshake.
To configure BFD globally, perform the following procedures:
• Configure the BFD global configuration settings
• Configure an access leaf policy group and inherit the previously created BFD global policies
• Associate the previously created leaf policy group onto a leaf switch or group of leaf switches

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 [no] template bfd {ip | ipv6} Creates a BFD policy template.
global-policy-name
Example:
apic1(config)# template bfd ip
bfd_global

Step 3 [no] echo-address ip-address Specifies the IP address to use as the source
address for BFD echo packets.
Example:
apic1(config-bfd)# echo-address
192.0.20.123
apic1(config-bfd)# echo-address 34::1/64

Step 4 [no] slow-timer milliseconds Configures the slow timer used in the echo
function. This value determines how fast BFD
Example:
starts up a new sessions and at what speed the
apic1(config-bfd)# slow-timer 2000 asynchrounous sessions use for BFD control
packets when the echo function is enabled. The
slow-timer value is used as the new control
packet interval, while the echo packets use the
configured BFD intervals. The echo packets
are used for link failure detection, while the
control packets at the slower rate maintain the
BFD session. The range is from 1000 to 30000
milliseconds.

Step 5 [no] min-tx milliseconds Specifies the interval at which this device
wants to send BFD hello messages. The range
Example:
is 50 to 999 milliseconds.
apic1(config-bfd)# min-tx 100

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

235
 Configuring Layer 3 External Connectivity
Configuring BFD Globally

Command or Action Purpose

Step 6 [no] min-rx milliseconds Specifies the minimum interval at which this
device can accept BFD hello messages from
Example:
another BFD device. The range is 50 to 999
apic1(config-bfd)# min-rx 70 milliseconds.

Step 7 [no] multiplier policy-name Specifies the number of missing BFD hello
messages from another BFD device before this
Example:
local device detects a fault in the forwarding
apic1(config-bfd)# multiplier 3 path. The range is 1 to 50.

Step 8 [no] echo-rx-interval policy-name Specifies the minimum interval between

received BFD echo packets that this system is
Example:
capable of supporting. The range is 50 to 999
apic1(config-bfd)# echo-rx-interval 500 milliseconds.

Step 9 exit Returns to global configuration mode.

Example:
apic1(config-bfd)# exit

Step 10 [no] template leaf-policy-group Configures an access leaf policy group.

leaf-policy-name
Example:
apic1(config)# template
leaf-policy-group leaf_pg1

Step 11 [no] inherit bfd {ip | ipv6} Inherits the previously created BFD global
global-policy-name policies.
Example:
apic1(config-leaf-policy-group)# inherit
bfd ip bfd_global

Step 12 exit Returns to global configuration mode.

Example:
apic1(config-leaf-policy-group)# exit

Step 13 [no] leaf-profile leaf-profile-name Configures a leaf profile.

Example:
apic1(config)# leaf-profile
leaf_profile1

Step 14 [no] leaf-group leaf-group-name Creates or specifies a group of leaf switches.

Example:
apic1(config-leaf-profile)# leaf-group
leaf_group1

Step 15 [no] leaf-policy-group leaf-policy-name Specifies the previously created leaf policy
group to be associated to the leaf switches.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

236
 Configuring Layer 3 External Connectivity
Configuring BFD Globally on Leaf Switch Using the NX-OS Style CLI

Command or Action Purpose

apic1(config-leaf-group)#
leaf-policy-group leaf_pg1

Step 16 [no] leaf leaf-range Adds one or more leaf switches to the leaf
switch group.
Example:
apic1(config-leaf-group)# leaf 101-102

Examples
This example shows how to configure BFD globally and apply it to a group of leaf switches.

# CONFIGURE BFD GLOBAL POLICIES

apic1# configure
apic1(config)# template bfd ip bfd_global
apic1(config-bfd)# echo-address 192.0.20.123
apic1(config-bfd)# slow-timer 2000
apic1(config-bfd)# min-tx 100
apic1(config-bfd)# min-rx 70
apic1(config-bfd)# multiplier 3
apic1(config-bfd)# echo-rx-interval 500
apic1(config-bfd)# exit

# CONFIGURE AN ACCESS LEAF POLICY GROUP AND INHERIT BFD GLOBAL POLICIES
apic1(config)# template leaf-policy-group leaf_pg1
apic1(config-leaf-policy-group)# inherit bfd ip bfd_global
apic1(config-leaf-policy-group)# exit

# CONFIGURE A LEAF GROUP AND ASSOCIATE THE LEAF POLICY GROUP

apic1(config)# leaf-profile leaf_profile1
apic1(config-leaf-profile)# leaf-group leaf_group1
apic1(config-leaf-group)# leaf-policy-group leaf_pg1
apic1(config-leaf-group)# leaf 101-102

Configuring BFD Globally on Leaf Switch Using the NX-OS Style CLI
Procedure

Step 1 To configure the BFD IPV4 global configuration (bfdIpv4InstPol) using the NX-OS CLI:
Example:

apic1# configure
apic1(config)# template bfd ip bfd_ipv4_global_policy
apic1(config-bfd)# [no] echo-address 1.2.3.4
apic1(config-bfd)# [no] slow-timer 2500
apic1(config-bfd)# [no] min-tx 100
apic1(config-bfd)# [no] min-rx 70
apic1(config-bfd)# [no] multiplier 3
apic1(config-bfd)# [no] echo-rx-interval 500
apic1(config-bfd)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

237
 Configuring Layer 3 External Connectivity
Configuring BFD Globally on Spine Switch Using the NX-OS Style CLI

Step 2 To configure the BFD IPV6 global configuration (bfdIpv6InstPol) using the NX-OS CLI:
Example:

apic1# configure
apic1(config)# template bfd ipv6 bfd_ipv6_global_policy
apic1(config-bfd)# [no] echo-address 34::1/64
apic1(config-bfd)# [no] slow-timer 2500
apic1(config-bfd)# [no] min-tx 100
apic1(config-bfd)# [no] min-rx 70
apic1(config-bfd)# [no] multiplier 3
apic1(config-bfd)# [no] echo-rx-interval 500
apic1(config-bfd)# exit

Step 3 To configure access leaf policy group (infraAccNodePGrp) and inherit the previously created BFD global
policies using the NX-OS CLI:
Example:

apic1# configure
apic1(config)# template leaf-policy-group test_leaf_policy_group
apic1(config-leaf-policy-group)# [no] inherit bfd ip bfd_ipv4_global_policy
apic1(config-leaf-policy-group)# [no] inherit bfd ipv6 bfd_ipv6_global_policy
apic1(config-leaf-policy-group)# exit

Step 4 To associate the previously created leaf policy group onto a leaf using the NX-OS CLI:
Example:

apic1(config)# leaf-profile test_leaf_profile

apic1(config-leaf-profile)# leaf-group test_leaf_group
apic1(config-leaf-group)# leaf-policy-group test_leaf_policy_group
apic1(config-leaf-group)# leaf 101-102
apic1(config-leaf-group)# exit

Configuring BFD Globally on Spine Switch Using the NX-OS Style CLI
Use this procedure to configure BFD globally on spine switch using the NX-OS style CLI.

Procedure

Step 1 To configure the BFD IPV4 global configuration (bfdIpv4InstPol) using the NX-OS CLI:
Example:

apic1# configure
apic1(config)# template bfd ip bfd_ipv4_global_policy
apic1(config-bfd)# [no] echo-address 1.2.3.4
apic1(config-bfd)# [no] slow-timer 2500
apic1(config-bfd)# [no] min-tx 100
apic1(config-bfd)# [no] min-rx 70
apic1(config-bfd)# [no] multiplier 3
apic1(config-bfd)# [no] echo-rx-interval 500
apic1(config-bfd)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

238
 Configuring Layer 3 External Connectivity
Overriding Global BFD Settings

Step 2 To configure the BFD IPV6 global configuration (bfdIpv6InstPol) using the NX-OS CLI:
Example:

apic1# configure
apic1(config)# template bfd ipv6 bfd_ipv6_global_policy
apic1(config-bfd)# [no] echo-address 34::1/64
apic1(config-bfd)# [no] slow-timer 2500
apic1(config-bfd)# [no] min-tx 100
apic1(config-bfd)# [no] min-rx 70
apic1(config-bfd)# [no] multiplier 3
apic1(config-bfd)# [no] echo-rx-interval 500
apic1(config-bfd)# exit

Step 3 To configure spine policy group and inherit the previously created BFD global policies using the NX-OS CLI:
Example:

apic1# configure
apic1(config)# template spine-policy-group test_spine_policy_group
apic1(config-spine-policy-group)# [no] inherit bfd ip bfd_ipv4_global_policy
apic1(config-spine-policy-group)# [no] inherit bfd ipv6 bfd_ipv6_global_policy
apic1(config-spine-policy-group)# exit

Step 4 To associate the previously created spine policy group onto a spine switch using the NX-OS CLI:
Example:

apic1# configure
apic1(config)# spine-profile test_spine_profile
apic1(config-spine-profile)# spine-group test_spine_group
apic1(config-spine-group)# spine-policy-group test_spine_policy_group
apic1(config-spine-group)# spine 103-104
apic1(config-leaf-group)# exit

Overriding Global BFD Settings

Configuring BFD Interface Override Policy
There are three supported interfaces (routed L3 interfaces, the external SVI interface, and the routed
sub-interfaces) on which you can configure an explicit BFD configuration. If you don't want to use the global
configuration, yet you want to have an explicit configuration on a given interface, you can create your own
global configuration, which gets applied to all the interfaces on a specific switch or set of switches. This
interface override configuration should be used if you want more granularity on a specific switch on a specific
interface.

Before you begin

A tenant has already been created.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

239
 Configuring Layer 3 External Connectivity
Configuring BFD Interface Override Policy

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 tenant tenant-name Specifies the tenant to be configured.

Example:
apic1(config)# tenant exampleCorp

Step 3 vrf context vrf-name Associates a VRF with the tenant.

Example:
apic1(config-tenant)# vrf context vrf1

Step 4 exit Returns to tenant configuration mode.

Example:
apic1(config-tenant-vrf)# exit

Step 5 exit Returns to global configuration mode.

Example:
apic1(config-tenant)# exit

Step 6 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 7 [no] vrf context tenant tenant-name vrf Configures a tenant VRF on the node.
vrf-name
Example:
apic1(config-leaf)# vrf context tenant
exampleCorp vrf vrf1

Step 8 exit Returns to leaf configuration mode.

Example:
apic1(config-leaf-vrf)# exit

Step 9 [no] interface type Enters interface configuration mode.

Example:
apic1(config-leaf)# interface eth 1/18

Step 10 [no] vrf member tenant tenant-name vrf

vrf-name
Example:
apic1(config-leaf-if)# vrf member tenant
exampleCorp vrf vrf1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

240
Configuring Layer 3 External Connectivity
Configuring BFD Interface Override Policy

Command or Action Purpose

Step 11 exit Returns to leaf configuration mode.
Example:
apic1(config-leaf-if)# exit

Step 12 [no] template bfd template-name tenant Configures a BFD interface policy.
tenant-name
Example:
apic1(config-leaf)# template bfd
bfdIfPol1 tenant exampleCorp

Step 13 [no] echo-mode enable Enables or disables the sending of BFD echo
packets in addition to BFD control packets.
Example:
apic1(config-template-bfd-pol)#
echo-mode enable

Step 14 [no] echo-rx-interval policy-name Specifies the minimum interval between

received BFD echo packets that this system is
Example:
capable of supporting. The range is 50 to 999
apic1(config-template-bfd-pol)# milliseconds.
echo-rx-interval 500

Step 15 [no] min-tx milliseconds Specifies the interval at which this device
sends BFD hello messages. The range is 50 to
Example:
999 milliseconds.
apic1(config-template-bfd-pol)# min-tx
100

Step 16 [no] min-rx milliseconds Specifies the minimum interval at which this
device can accept BFD hello messages from
Example:
another BFD device. The range is 50 to 999
apic1(config-template-bfd-pol)# min-rx milliseconds.
70

Step 17 [no] multiplier policy-name Specifies the number of missing BFD hello
messages from another BFD device before this
Example:
local device detects a fault in the forwarding
apic1(config-template-bfd-pol)# path. The range is 1 to 50.
multiplier 5

Step 18 [no] optimize subinterface Enables or disables sub-interface optimization.

BFD creates sessions for all configured
Example:
subinterfaces. BFD sets the subinterface with
apic1(config-template-bfd-pol)# optimize the lowest configured VLAN ID as the master
subinterface
subinterface and that subinterface uses the
BFD session parameters of the parent interface.
The remaining subinterfaces use the slow
timer. If the optimized subinterface session
detects an error, BFD marks all subinterfaces
on that physical interface as down.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

241
 Configuring Layer 3 External Connectivity
Applying the BFD Interface Override Policy to Interfaces

Examples
This example shows how to configure a BFD override policy and apply it to an interface.

apic1# configure
apic1(config)# tenant exampleCorp
apic1(config-tenant)# vrf context vrf1
apic1(config-tenant-vrf)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant exampleCorp vrf vrf1
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# interface eth 1/18
apic1(config-leaf-if)# vrf member tenant exampleCorp vrf vrf1
apic1(config-leaf-if)# exit

# CONFIGURE BFD INTERFACE OVERRIDE POLICY

apic1(config-leaf)# template bfd bfdIfPol1 tenant exampleCorp
apic1(config-template-bfd-pol)# echo-mode enable
apic1(config-template-bfd-pol)# echo-rx-interval 500
apic1(config-template-bfd-pol)# min-tx 100
apic1(config-template-bfd-pol)# min-rx 70
apic1(config-template-bfd-pol)# multiplier 5
apic1(config-template-bfd-pol)# optimize subinterface

Applying the BFD Interface Override Policy to Interfaces

You can apply a BFD interface override policy to routed L3 interfaces, the external SVI interface, and the
routed sub-interfaces.

Before you begin

A BFD interface override policy has already been created.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 [no] interface type Enters interface configuration mode. Supported

interfaces are routed L3 interfaces, the external
Example:
SVI interface, and the routed sub-interfaces.
apic1(config-leaf)# interface Ethernet
1/15

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

242
Configuring Layer 3 External Connectivity
Applying the BFD Interface Override Policy to Interfaces

Command or Action Purpose

Step 4 [no] ipv6 address ipv6-address [preferred] Specifies an IP address to be the default source
address for traffic from the interface.
Example:
apic1(config-leaf-if)# ipv6 address Note This command is used only if the
2001::10:1/64 preferred interface is an IPv6 interface.

Step 5 [no] vrf member tenant tenant-name vrf Attaches the interface to the tenant VRF.
vrf-name
Note This command is used only if the
Example: interface is a VLAN interface.
apic1(config-leaf-if)# vrf member tenant
exampleCorp vrf vrf1

Step 6 bfd {ip | ipv6} tenant mode Enables BFD tenant mode.
Example:
apic1(config-leaf-if)# bfd ip tenant mode

Step 7 bfd {ip | ipv6} inherit interface-policy Inherits the specified BFD interface template
policy-name policy.
Example:
apic1(config-leaf-if)# bfd ip inherit
interface-policy bfdIfPol1

Step 8 bfd {ip | ipv6} authentication keyed-sha1 Configures BFD authentication as keyed
keyid keyid key key SHA-1.
Example:
apic1(config-leaf-if)# bfd ip
authentication keyed-sha1 key 10 key
password

Examples
This example shows how to inherit the previously created BFD interface policy onto a L3 interface
with an IPv4 address.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface eth 1/15
apic1(config-leaf-if)# bfd ip tenant mode
apic1(config-leaf-if)# bfd ip inherit interface-policy bfdIfPol1
apic1(config-leaf-if)# bfd ip authentication keyed-sha1 key 10 key password

This example shows how to inherit the previously created BFD interface policy onto a L3 interface
with an IPv6 address.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface eth 1/15

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

243
 Configuring Layer 3 External Connectivity
Enabling BFD on Consumer Protocols

apic1(config-leaf-if)# ipv6 address 2001::10:1/64 preferred

apic1(config-leaf-if)# bfd ip tenant mode
apic1(config-leaf-if)# bfd ip inherit interface-policy bfdIfPol1
apic1(config-leaf-if)# bfd ip authentication keyed-sha1 key 10 key password

This example shows how to configure BFD on a VLAN interface with an IPv4 address.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface vlan 15
apic1(config-leaf-if)# vrf member tenant exampleCorp vrf vrf1
apic1(config-leaf-if)# bfd ip tenant mode
apic1(config-leaf-if)# bfd ip inherit interface-policy bfdIfPol1
apic1(config-leaf-if)# bfd ip authentication keyed-sha1 key 10 key password

This example shows how to configure BFD on a VLAN interface with an IPv6 address.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface vlan 15
apic1(config-leaf-if)# ipv6 address 2001::10:1/64 preferred
apic1(config-leaf-if)# vrf member tenant exampleCorp vrf vrf1
apic1(config-leaf-if)# bfd ip tenant mode
apic1(config-leaf-if)# bfd ip inherit interface-policy bfdIfPol1
apic1(config-leaf-if)# bfd ip authentication keyed-sha1 key 10 key password

Enabling BFD on Consumer Protocols

These procedures provide the steps to enable BFD in the four consumer protocols (BGP, EIGRP, OSPF, and
Static Routes), which are consumers of the BFD feature.

Enabling BFD on the BGP Consumer Protocol

Before you begin

A tenant has already been created.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 bgp-fabric Enters BGP configuration mode for the fabric.

Example:
apic1(config-bgp-fabric)# bgp-fabric

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

244
Configuring Layer 3 External Connectivity
Enabling BFD on the BGP Consumer Protocol

Command or Action Purpose

Step 3 asn asn-number Specifies the BGP autonomous system number
(ASN).
Example:
apic1(config-bgp-fabric)# asn 200

Step 4 exit Returns to global configuration mode.

Example:
apic1(config-bgp-fabric)# exit

Step 5 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 6 router bgp asn-number Enters BGP policy configuration.

Example:
apic1(config-leaf)# router bgp 200

Step 7 vrf member tenant tenant-name vrf vrf-name Specifies the VRF instance to associate with
subsequent policy configuration mode
Example:
commands.
apic1(config-bgp)# vrf member tenant
exampleCorp vrf v100

Step 8 neighbor ip-address [/masklength] Specifies the IP address of the neighbor. The
mask length must be 32.
Example:
apic1(config-leaf-bgp-vrf)# neighbor
1.2.3.4

Step 9 [no] bfd enable Enables or disables BFD on the BGP consumer
protocol.
Example:
apic1(config-leaf-bgp-vrf-neighbor)# bfd
enable

Examples
This example shows how to enable BFD on the BGP consumer protocol.

apic1# configure
apic1(config)# bgp-fabric
apic1(config-bgp-fabric)# asn 200
apic1(config-bgp-fabric)# exit
apic1(config)# leaf 101
apic1(config-leaf)# router bgp 200
apic1(config-bgp)# vrf member tenant exampleCorp vrf v100
apic1(config-leaf-bgp-vrf)# neighbor 1.2.3.4
apic1(config-leaf-bgp-vrf-neighbor)# bfd enable

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

245
 Configuring Layer 3 External Connectivity
Enabling BFD on the EIGRP Consumer Protocol

Enabling BFD on the EIGRP Consumer Protocol

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 [no] interface type Enters interface configuration mode.

Example:
apic1(config-leaf)# interface Ethernet
1/15

Step 4 [no] {ip | ipv6} bfd eigrp enable Enables or disables BFD on the EIGRP
consumer protocol.
Example:
apic1(config-leaf-if)# ip bfd eigrp
enable

Examples
This example shows how to enable BFD on the EIGRP consumer protocol.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface eth 1/15
apic1(config-leaf-if)# ip bfd eigrp enable

Enabling BFD on the OSPF Consumer Protocol

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

246
 Configuring Layer 3 External Connectivity
Enabling BFD on the Static Route Consumer Protocol

Command or Action Purpose

Step 3 [no] interface type Enters interface configuration mode.
Example:
apic1(config-leaf)# interface vlan 123

Step 4 [no] ip ospf bfd enable Enables or disables BFD on the OSPF consumer
protocol.
Example:
apic1(config-leaf-if)# ip ospf bfd enable

Examples
This example shows how to enable BFD on the OSPF consumer protocol.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface vlan 123
apic1(config-leaf-if)# ip ospf bfd enable

Enabling BFD on the Static Route Consumer Protocol

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 [no] vrf context tenant tenant-name vrf Configures a tenant VRF on the node.
vrf-name
Example:
apic1(config-leaf)# vrf context tenant
exampleCorp vrf vrf1

Step 4 [no] {ip | ipv6} route ip-prefix/masklen Enables or disables BFD on the static route
next-hop-address bfd consumer protocol.
Example:
apic1(config-leaf-vrf)# ip route
10.0.0.1/16 10.0.0.5 bfd

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

247
 Configuring Layer 3 External Connectivity
Configuring BFD Consumer Protocols Using the NX-OS Style CLI

Examples
This example shows how to enable BFD on the static route consumer protocol.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant exampleCorp vrf vrf1
apic1(config-leaf-vrf)# ip route 10.0.0.1/16 10.0.0.5 bfd

Configuring BFD Consumer Protocols Using the NX-OS Style CLI

Procedure

Step 1 To enable BFD on the BGP consumer protocol using the NX-OS CLI:
Example:

apic1# configure
apic1(config)# bgp-fabric
apic1(config-bgp-fabric)# asn 200
apic1(config-bgp-fabric)# exit
apic1(config)# leaf 101
apic1(config-leaf)# router bgp 200
apic1(config-bgp)# vrf member tenant t0 vrf v0
apic1(config-leaf-bgp-vrf)# neighbor 1.2.3.4
apic1(config-leaf-bgp-vrf-neighbor)# [no] bfd enable

Step 2 To enable BFD on the EIGRP consumer protocol using the NX-OS CLI:
Example:

apic1(config-leaf-if)# [no] ip bfd eigrp enable

Step 3 To enable BFD on the OSPF consumer protocol using the NX-OS CLI:
Example:

apic1(config-leaf-if)# [no] ip ospf bfd enable

apic1# configure
apic1(config)# spine 103
apic1(config-spine)# interface ethernet 5/3.4
apic1(config-spine-if)# [no] ip ospf bfd enable

Step 4 To enable BFD on the Static Route consumer protocol using the NX-OS CLI:
Example:

apic1(config-leaf-vrf)# [no] ip route 10.0.0.1/16 10.0.0.5 bfd

apic1(config)# spine 103

apic1(config-spine)# vrf context tenant infra vrf overlay-1
apic1(config-spine-vrf)# [no] ip route 21.1.1.1/32 32.1.1.1 bfd

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

248
 Configuring Layer 3 External Connectivity
Configuring Layer 3 Multicast

Step 5 To enable BFD on IS-IS consumer protocol using the NX-OS CLI:
Example:

apic1(config)# leaf 101

apic1(config-spine)# interface ethernet 1/49
apic1(config-spine-if)# isis bfd enabled
apic1(config-spine-if)# exit
apic1(config-spine)# exit

apic1(config)# spine 103

apic1(config-spine)# interface ethernet 5/2
apic1(config-spine-if)# isis bfd enabled
apic1(config-spine-if)# exit
apic1(config-spine)# exit

Configuring Layer 3 Multicast

Layer 3 Multicast
In the ACI fabric, most unicast and multicast routing operate together on the same border leaf switches, with
the multicast protocol operating over the unicast routing protocols.
In this architecture, only the border leaf switches run the full Protocol Independent Multicast (PIM) protocol.
Non-border leaf switches run PIM in a passive mode on the interfaces. They do not peer with any other PIM
routers. The border leaf switches peer with other PIM routers connected to them over L3 Outs and also with
each other.
The following figure shows the border leaf (BL) switches (BL1 and BL2) connecting to routers (R1 and R2)
in the multicast cloud. Each virtual routing and forwarding (VRF) in the fabric that requires multicast routing
will peer separately with external multicast routers.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

249
 Configuring Layer 3 External Connectivity
Guidelines and Restrictions for Configuring Layer 3 Multicast

Figure 22: Overview of Multicast Cloud

Guidelines and Restrictions for Configuring Layer 3 Multicast

See the following guidelines and restrictions:
• Custom QoS policy is not supported for Layer 3 multicast traffic sourced from outside the ACI fabric
(received from L3Out).
• Enabling PIMv4 (Protocol-Independent Multicast, version 4) and Advertise Host routes on a BD is not
supported.
• If the border leaf switches in your ACI fabric are running multicast and you disable multicast on the
L3Out while you still have unicast reachability, you will experience traffic loss if the external peer is a
Cisco Nexus 9000 switch. This impacts cases where traffic is destined towards the fabric (where the
sources are outside the fabric but the receivers are inside the fabric) or transiting through the fabric (where
the source and receivers are outside the fabric, but the fabric is transit).
• If the (s, g) entry is installed on a border leaf switch, you might see drops in unicast traffic that comes
from the fabric to this source outside the fabric when the following conditions are met:
• Preferred group is used on the L3Out EPG
• Unicast routing table for the source is using the default route 0.0.0.0/0

This behavior is expected.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

250
Configuring Layer 3 External Connectivity
Guidelines and Restrictions for Configuring Layer 3 Multicast

• The Layer 3 multicast configuration is done at the VRF level so protocols function within the VRF and
multicast is enabled in a VRF, and each multicast VRF can be turned on or off independently.
• Once a VRF is enabled for multicast, the individual bridge domains (BDs) and L3 Outs under the enabled
VRF can be enabled for multicast configuration. By default, multicast is disabled in all BDs and Layer
3 Outs.
• Layer 3 multicast is not currently supported on VRFs that are configured with a shared L3 Out.
• Any Source Multicast (ASM) and Source-Specific Multicast (SSM) are supported.
• Bidirectional PIM, Rendezvous Point (RP) within the ACI fabric, and PIM IPv6 are currently not supported
• IGMP snooping cannot be disabled on pervasive bridge domains with multicast routing enabled.
• Multicast routers are not supported in pervasive bridge domains.
• The Layer 3 multicast feature is supported on the following leaf switches:
• EX models:
• N9K-93108TC-EX
• N9K-93180LC-EX
• N9K-93180YC-EX

• FX models:
• N9K-93108TC-FX
• N9K-93180YC-FX
• N9K-C9348GC-FXP

• FX2 models:
• N9K-93240YC-FX2
• N9K-C9336C-FX2

• PIM is supported on Layer 3 Out routed interfaces and routed subinterfaces including Layer 3 port-channel
interfaces. PIM is not supported on Layer 3 Out SVI interfaces.
• Enabling PIM on an L3Out causes an implicit external network to be configured. This action results in
the L3Out being deployed and protocols potentially coming up even if you have not defined an external
network.
• For Layer 3 multicast support, when the ingress leaf switch receives a packet from a source that is attached
on a bridge domain, and the bridge domain is enabled for multicast routing, the ingress leaf switch sends
only a routed VRF copy to the fabric (routed implies that the TTL is decremented by 1, and the source-mac
is rewritten with a pervasive subnet MAC). The egress leaf switch also routes the packet into receivers
in all the relevant bridge domains. Therefore, if a receiver is on the same bridge domain as the source,
but on a different leaf switch than the source, that receiver continues to get a routed copy, although it is
in the same bridge domain. This also applies if the source and receiver are on the same bridge domain
and on the same leaf switch, if PIM is enabled on this bridge domain.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

251
 Configuring Layer 3 External Connectivity
Configuration Steps for Layer 3 Multicast

For more information, see details about Layer 3 multicast support for multipod that leverages existing
Layer 2 design, at the following link Adding Pods.
• Starting with Release 3.1(1x), Layer 3 multicast is supported with FEX. Multicast sources or receivers
that are connected to FEX ports are supported. For further details about how to add FEX in your testbed,
see Configure a Fabric Extender with Application Centric Infrastructure at this URL:
https://www.cisco.com/c/en/us/support/docs/cloud-systems-management/
application-policy-infrastructure-controller-apic/200529-Configure-a-Fabric-Extender-with-Applica.html.
For releases preceeding Release 3.1(1x), Layer 3 multicast is not supported with FEX. Multicast sources
or receivers that are connected to FEX ports are not supported.

Note Cisco ACI does not support IP fragmentation. Therefore, when you configure Layer 3 Outside (L3Out)
connections to external routers, or multipod connections through an Inter-Pod Network (IPN), it is critical
that the interface MTU is set appropriately on both ends of a link. On some platforms, such as Cisco ACI,
Cisco NX-OS, and Cisco IOS, the configurable MTU value does not take into account the Ethernet headers
(matching IP MTU, and excluding the 14-18 Ethernet header size), while other platforms, such as IOS-XR,
include the Ethernet header in the configured MTU value. A configured value of 9000 results in a max IP
packet size of 9000 bytes in Cisco ACI, Cisco NX-OS, and Cisco IOS, but results in a maximum IP packet
size of 8986 bytes for an IOS-XR untagged interface.
For the appropriate MTU values for each platform, see the relevant configuration guides.
We highly recommend that you test the MTU using CLI-based commands. For example, on the Cisco NX-OS
CLI, use a command such as ping 1.1.1.1 df-bit packet-size 9000 source-interface ethernet 1/1.

Configuration Steps for Layer 3 Multicast

The following sections show the configuration steps for layer 3 Multicast. The steps are as follows:
1. Configure PIM options on the tenant VRF.
2. Configure IGMP options for the VRF.
3. Configure an L3 Out for the tenant, enable PIM, and configure the leaf interface.
4. Enable PIM in the desired bridge domains.

Configuring PIM Options for Layer 3 Multicast

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 tenant tenant-name Specifies the tenant to be configured.

Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

252
Configuring Layer 3 External Connectivity
Configuring PIM Options for Layer 3 Multicast

Command or Action Purpose

apic1(config)# tenant exampleCorp

Step 3 vrf context vrf-name Associates a VRF with the tenant.

Example:
apic1(config-tenant)# vrf context
exampleCorp_vrf1

Step 4 [no] ip pim Configures Protocol Independent Multicast

(PIM).
Example:
apic1(config-tenant-vrf)# ip pim

Step 5 (Optional) [no] ip pim auto-rp {forward Configures PIM auto-RP (Rendezvous Point)
[listen] | listen | mapping-agent-policy options. Auto-RP automates the distribution
mapping-agent-policy-name } of group-to-RP mappings in a PIM network.
You can choose to forward auto-RP messages,
Example:
listen to auto-RP messages, or associate a
apic1(config-tenant-vrf)# ip pim auto-rp route-map policy for filtering mapping agent
forward listen
messages.

Step 6 (Optional) [no] ip pim bsr {forward [listen] Configures PIM bootstrap router (BSR)
| listen | bsr-policy options. BSR performs similarly to auto-RP
mapping-agent-policy-name } in that it uses candidate routers for the RP
function and for relaying the RP information
Example:
for a group. RP information is distributed
apic1(config-tenant-vrf)# ip pim bsr through BSR messages, which are carried
forward listen
within PIM messages. You can choose to
forward Bootstrap/Candidate-RP messages,
listen to Bootstrap/Candidate-RP messages, or
associate a route-map policy for filtering BSR
messages.

Step 7 (Optional) [no] ip pim fast-convergence Enables the PIM fast convergence feature,
which allows the switch to discover
Example:
unresponsive neighbors more quickly.
apic1(config-tenant-vrf)# ip pim
fast-convergence

Step 8 (Optional) [no] ip pim mtu mtu-size Configures the maximum size of a PIM
message. The range is 1500 to 65536 bytes.
Example:
apic1(config-tenant-vrf)# ip pim mtu
1500

Step 9 (Optional) [no] ip pim register-policy Specifies the name of a policy for filtering
register-policy-name register messages.
Example:
apic1(config-tenant-vrf)# ip pim
register-policy regPolicy1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

253
 Configuring Layer 3 External Connectivity
Configuring PIM Options for Layer 3 Multicast

Command or Action Purpose

Step 10 (Optional) [no] ip pim register-rate-limit Specifies a rate limit for PIM data registers.
mtu-size The range is 0 to 65535 packets per second.
Example:
apic1(config-tenant-vrf)# ip pim
register-rate-limit 1024

Step 11 (Optional) [no] ip pim register-source Configures a source IP address for PIM
ip-address messages.
Example:
apic1(config-tenant-vrf)# ip pim
register-source 192.0.20.123

Step 12 (Optional) [no] ip pim rp-address ip-address Configures a static route processor (RP)
[route-map route-map-name] address for a multicast group range.
Example:
apic1(config-tenant-vrf)# ip pim
rp-address 192.0.20.99

Step 13 (Optional) [no] ip pim sg-expiry-timer Configures the (S, G) expiry timer interval for
ip-address [sg-list route-map-name] PIM sparse mode (PIM-SM) (S, G) multicast
routes. The range is 180 to 604801 seconds.
Example:
The optional sg-list parameter specifies S,G
apic1(config-tenant-vrf)# ip pim values to which the timer applies. The default
sg-expiry-timer 4096
is 4096.

Step 14 (Optional) [no] ip pim ssm route-map Configures Source Specific Multicast (SSM),
route-map-name which is an extension of IP multicast in which
datagram traffic is forwarded to receivers from
Example:
only those multicast sources that the receivers
apic1(config-tenant-vrf)# ip pim ssm have explicitly joined. The route-map policy
route-map SSMRtMap
lists the group prefixes.

Step 15 (Optional) [no] ip pim state-limit max-entries Configures a maximum number of PIM state
[reserved route-map-name entries in the current VRF instance. The range
[maximum-reserve-state-entries]] is 0 to 4294967295 maximum state entries.
You can optionally specify a number of state
Example:
entries to be reserved for the routes specified
apic1(config-tenant-vrf)# ip pim in a policy map and you can specify the
state-limit 100000 reserved
myReservedPolicy 40000
maximum reserved (*, G) and (S, G) entries
allowed in this VRF. This number must be less
than or equal to the maximum states allowed.
The range is from 1 to 4294967295.

Step 16 (Optional) [no] ip pim use-shared-tree-only Creates the PIM (*, G) state only (where no
group-list policy-name source state is created). The policy defines the
group prefixes where this feature is applied.
Example:
apic1(config-tenant-vrf)# ip pim
use-shared-tree-only group-list myGroup1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

254
 Configuring Layer 3 External Connectivity
Configuring IGMP Options on the VRF for Layer 3 Multicast

Command or Action Purpose

Step 17 exit Returns to tenant configuration mode.
Example:
apic1(config-tenant-vrf)# exit

What to do next
Configure IGMP options for the VRF.

Configuring IGMP Options on the VRF for Layer 3 Multicast

Before you begin
Configure PIM options on the tenant VRF.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 tenant tenant-name Specifies the tenant to be configured.

Example:
apic1(config)# tenant exampleCorp

Step 3 vrf context vrf-name Associates a VRF with the tenant.

Example:
apic1(config-tenant)# vrf context vrf1

Step 4 [no] ip igmp Enables Internet Group Management Protocol

(IGMP).
Example:
apic1(config-tenant-vrf)# ip igmp

Step 5 exit Returns to tenant configuration mode.

Example:
apic1(config-tenant-vrf)# exit

Step 6 interface bridge-domain bd-name Enters tenant interface configuration mode to

configure the bridge domain.
Example:
apic1(config-tenant)# interface
bridge-domain exampleCorp_bd1

Step 7 [no] ip multicast Enables IP multicast routing on the interface.

Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

255
 Configuring Layer 3 External Connectivity
Configuring IGMP Options on the VRF for Layer 3 Multicast

Command or Action Purpose

apic1(config-tenant-interface)# ip
multicast

Step 8 [no] ip igmp allow-v3-asm Allows filtering for source addresses in

IGMPv3 reports for Any Source Multicast
Example:
(ASM) groups.
apic1(config-tenant-interface)# ip igmp
allow-v3-asm

Step 9 [no] ip igmp fast-leave Enables IP IGMP snooping fast leave

processing. This feature supports IGMPv2
Example:
hosts that cannot be explicitly tracked because
apic1(config-tenant-interface)# ip igmp of the host report suppression mechanism of
fast-leave
the IGMPv2 protocol. When you enable fast
leave, the IGMP software assumes that no
more than one host is present on each port.

Step 10 [no] ip igmp group-timeout seconds Sets the group membership timeout for
IGMPv2. The range is 3 to 65535 seconds. The
Example:
default is 260 seconds.
apic1(config-tenant-interface)# ip igmp
group-timeout 260

Step 11 [no] ip igmp inherit interface-policy Associates a IGMP interface policy to this
policy-name interface.
Example:
apic1(config-tenant-interface)# ip igmp
inherit interface-policy MyIfPolicy

Step 12 [no] ip igmp join-group route-map Statically binds one or more multicast groups
route-map-name to the interface. The route-map policy lists the
group prefixes, group ranges, and source
Example:
prefixes.
apic1(config-tenant-interface)# ip igmp
join-group route-map MyGroupsRMap

Step 13 [no] ip igmp last-member-query-count count Sets the number of times that the software
sends an IGMP query in response to a host
Example:
leave message. The range is 1 to 5 queries. The
apic1(config-tenant-interface)# ip igmp default is 2 queries.
last-member-query-count 2

Step 14 [no] ip igmp Sets the query interval waited after sending
last-member-query-response-time seconds membership reports before the software deletes
the group state. The range is 1 to 25 seconds.
Example:
The default is 1 second.
apic1(config-tenant-interface)# ip igmp
last-member-query-response-time 1

Step 15 [no] ip igmp querier-timeout seconds Sets the number of seconds that the software
waits after the previous querier has stopped
Example:
querying and before it takes over as the querier.
apic1(config-tenant-interface)# ip igmp The range is 1 to 65535 seconds. The default
querier-timeout 255
is 255 seconds.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

256
Configuring Layer 3 External Connectivity
Configuring IGMP Options on the VRF for Layer 3 Multicast

Command or Action Purpose

Step 16 [no] ip igmp query-interval seconds Sets the frequency at which the software sends
IGMP host query messages. You can tune the
Example:
number of IGMP messages on the network by
apic1(config-tenant-interface)# ip igmp setting a larger value so that the software sends
query-interval 125
IGMP queries less often. The range is 1 to
18000 seconds. The default is 125 seconds.

Step 17 [no] ip igmp query-max-response-time Sets the response time advertised in IGMP
seconds queries. You can tune the burstiness of IGMP
messages on the network by setting a larger
Example:
value so that host responses are spread out over
apic1(config-tenant-interface)# ip igmp a longer time. This value must be less than the
query-max-response-time 10
query interval. The range is 1 to 25 seconds.
The default is 10 seconds.

Step 18 [no] ip igmp report-link-local-groups Enables sending reports for groups in

224.0.0.0/24. Link local addresses are used
Example:
only by protocols on the local network. Reports
apic1(config-tenant-interface)# ip igmp are always sent for nonlink local groups. By
report-link-local-groups
default, reports are not sent for link local
groups.

Step 19 [no] ip igmp report-policy policy-name Configures an access policy for IGMP reports
that is based on a route-map policy.
Example:
apic1(config-tenant-interface)# ip igmp
report-policy MyReportPolicy

Step 20 [no] ip igmp robustness-variable value Sets the robustness variable to compensate for
packet loss on a congested network. The
Example:
robustness value is used by the IGMP software
apic1(config-tenant-interface)# ip igmp to determine the number of times to send
robustness-variable 2
messages. You can use a larger value for a
lossy network. The range is 1 to 7. The default
is 2.

Step 21 [no] ip igmp snooping Enables IGMP snooping for the interface.
Example:
apic1(config-tenant-interface)# ip igmp
snooping

Step 22 [no] ip igmp snooping fast-leave Enables the software to remove the group state
when it receives an IGMP Leave report without
Example:
sending an IGMP query message. This
apic1(config-tenant-interface)# ip igmp parameter is used for IGMPv2 hosts when no
snooping fast-leave
more than one host is present on each port.

Step 23 [no] ip igmp snooping Sets a time interval in seconds after which the
last-member-query-interval group is removed from the associated port if
no hosts respond to an IGMP query message.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

257
 Configuring Layer 3 External Connectivity
Configuring IGMP Options on the VRF for Layer 3 Multicast

Command or Action Purpose

apic1(config-tenant-interface)# ip igmp The range is 1 to 25 seconds. The default is 5
snooping last-member-query-interval 5 seconds.

Step 24 [no] ip igmp snooping policy policy-name Associates the bridge domain with an IGMP
snooping policy.
Example:
apic1(config-tenant-interface)# ip igmp
snooping policy MySnoopingPolicy

Step 25 [no] ip igmp snooping querier Enables an IP IGMP snooping querier, which
sends out periodic IGMP queries that trigger
Example:
IGMP report messages from hosts who want
apic1(config-tenant-interface)# ip igmp to receive IP multicast traffic. IGMP snooping
snooping querier
listens to these IGMP reports to establish
appropriate forwarding.

Step 26 [no] ip igmp snooping query-interval Configures a snooping query interval when
seconds you do not enable PIM because multicast
traffic does not need to be routed. The range
Example:
is 1 to 18000 seconds. The default is 125
apic1(config-tenant-interface)# ip igmp seconds.
snooping query-interval 125

Step 27 [no] ip igmp snooping Configures a snooping maximum response

query-max-response-time seconds time for query messages when you do not
enable PIM because multicast traffic does not
Example:
need to be routed. The range is 1 to 25 seconds.
apic1(config-tenant-interface)# ip igmp The default is 10 seconds.
snooping query-max-response-time 10

Step 28 [no] ip igmp snooping startup-query-count Configures snooping for a number of queries
count sent at startup when you do not enable PIM
because multicast traffic does not need to be
Example:
routed. The range is 1 to 10 queries. The
apic1(config-tenant-interface)# ip igmp default is 5 queries.
snooping startup-query-count 5

Step 29 [no] ip igmp snooping Configures a snooping query interval at startup

startup-query-interval seconds when you do not enable PIM because multicast
traffic does not need to be routed. The range
Example:
is 1 to 18000 seconds. The default is 15000
apic1(config-tenant-interface)# ip igmp seconds.
snooping startup-query-interval 15000

Step 30 [no] ip igmp startup-query-count count Sets the number of queries sent at startup that
are separated by the startup query interval. The
Example:
range is 1 to 10 queries. The default is 2
apic1(config-tenant-interface)# ip igmp queries.
startup-query-count 2

Step 31 [no] ip igmp startup-query-interval seconds Sets the query interval used when the software
starts up. By default, this interval is shorter
Example:
than the query interval so that the software can
apic1(config-tenant-interface)# ip igmp establish the group state as quickly as possible.
startup-query-interval 31

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

258
 Configuring Layer 3 External Connectivity
Configuring an L3 Out for Layer 3 Multicast

Command or Action Purpose

The range is 1 to 18000 seconds. The default
is 260 seconds. The default is 31 seconds.

Step 32 [no] ip igmp state-limit max-states [reserved Configures a per interface limit on the number
route-map-name [max-reserved-gsg-entries]] of mroutes states created as a result of IGMP
membership reports (IGMP joins). The range
Example:
of states allowed is 1 to 4294967295 states.
apic1(config-tenant-interface)# ip igmp You can optionally specify a number of state
state-limit 100000 reserved
myReservedPolicy 40000
entries to be reserved for the routes specified
in a policy map and you can specify the
maximum reserved (*, G) and (S, G) entries
allowed on the interface. The number of
reserved states must be less than or equal to
the maximum states allowed. The range is from
1 to 4294967295.

Step 33 [no] ip igmp static-oif route-map Statically binds a multicast group to the
route-map-name outgoing interface (OIF), which is handled by
the device hardware. The route map defines
Example:
the group prefixes where this feature is applied.
apic1(config-tenant-interface)# ip igmp
static-oif route-map MyOifMap

Step 34 [no] ip igmp version {v1 | v2 | v3} Configures the IGMP version number for the
interface. The default version is v2.
Example:
apic1(config-tenant-interface)# ip igmp
version v3

Step 35 exit Returns to tenant configuration mode.

Example:
apic1(config-tenant-interface)# exit

What to do next
Configure an L3 Out for the tenant, enable PIM, and configure the leaf interface.

Configuring an L3 Out for Layer 3 Multicast

Before you begin
• Configure PIM options on the tenant VRF.
• Configure IGMP on the tenant VRF.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

259
 Configuring Layer 3 External Connectivity
Configuring an L3 Out for Layer 3 Multicast

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 tenant tenant-name Specifies the tenant to be configured.

Example:
apic1(config)# tenant exampleCorp

Step 3 l3out l3out-name Configures an L3 Out interface on the tenant.

Example:
apic1(config-tenant)# l3out
exampleCorp_l3out

Step 4 ip pim Enables PIM on the interface.

Example:
apic1(config-tenant-l3out)# ip pim

Step 5 exit Returns to tenant configuration mode.

Example:
apic1(config-tenant-l3out)#

Step 6 exit Returns to global configuration mode.

Example:
apic1(config-tenant)# exit

Step 7 leaf node-id Enters leaf configuration mode.

Example:
apic1(config)# leaf 101

Step 8 interface ethernet slot/port Specifies the interface to be configured.

Example:
apic1(config-leaf)# interface ethernet
1/3

Step 9 [no] ip igmp allow-v3-asm Allows filtering for source addresses in

IGMPv3 reports for Any Source Multicast
Example:
(ASM) groups.
apic1(config-leaf-if)# ip igmp
allow-v3-asm

Step 10 [no] ip igmp fast-leave Enables IP IGMP snooping fast leave

processing. This feature supports IGMPv2
Example:
hosts that cannot be explicitly tracked because
apic1(config-leaf-if)# ip igmp of the host report suppression mechanism of
fast-leave
the IGMPv2 protocol. When you enable fast

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

260
Configuring Layer 3 External Connectivity
Configuring an L3 Out for Layer 3 Multicast

Command or Action Purpose

leave, the IGMP software assumes that no
more than one host is present on each port.

Step 11 [no] ip igmp group-timeout seconds Sets the group membership timeout for
IGMPv2. The range is 3 to 65535 seconds. The
Example:
default is 260 seconds.
apic1(config-leaf-if)# ip igmp
group-timeout 260

Step 12 [no] ip igmp inherit interface-policy Associates a IGMP interface policy to this
policy-name interface.
Example:
apic1(config-leaf-if)# ip igmp inherit
interface-policy MyIfPolicy

Step 13 [no] ip igmp join-group route-map Statically binds one or more multicast groups
route-map-name to the interface. The route-map policy lists the
group prefixes, group ranges, and source
Example:
prefixes.
apic1(config-leaf-if)# ip igmp
join-group route-map MyGroupsRMap

Step 14 [no] ip igmp last-member-query-count count Sets the number of times that the software
sends an IGMP query in response to a host
Example:
leave message. The range is 1 to 5 queries. The
apic1(config-leaf-if)# ip igmp default is 2 queries.
last-member-query-count 2

Step 15 [no] ip igmp Sets the query interval waited after sending
last-member-query-response-time seconds membership reports before the software deletes
the group state. The range is 1 to 25 seconds.
Example:
The default is 1 second.
apic1(config-leaf-if)# ip igmp
last-member-query-response-time 1

Step 16 [no] ip igmp querier-timeout seconds Sets the number of seconds that the software
waits after the previous querier has stopped
Example:
querying and before it takes over as the querier.
apic1(config-leaf-if)# ip igmp The range is 1 to 65535 seconds. The default
querier-timeout 255
is 255 seconds.

Step 17 [no] ip igmp query-interval seconds Sets the frequency at which the software sends
IGMP host query messages. You can tune the
Example:
number of IGMP messages on the network by
apic1(config-leaf-if)# ip igmp setting a larger value so that the software sends
query-interval 125
IGMP queries less often. The range is 1 to
18000 seconds. The default is 125 seconds.

Step 18 [no] ip igmp query-max-response-time Sets the response time advertised in IGMP
seconds queries. You can tune the burstiness of IGMP
messages on the network by setting a larger
Example:
value so that host responses are spread out over

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

261
 Configuring Layer 3 External Connectivity
Configuring an L3 Out for Layer 3 Multicast

Command or Action Purpose

apic1(config-leaf-if)# ip igmp a longer time. This value must be less than the
query-max-response-time 10 query interval. The range is 1 to 25 seconds.
The default is 10 seconds.

Step 19 [no] ip igmp report-link-local-groups Enables sending reports for groups in

224.0.0.0/24. Link local addresses are used
Example:
only by protocols on the local network. Reports
apic1(config-leaf-if)# ip igmp are always sent for nonlink local groups. By
report-link-local-groups
default, reports are not sent for link local
groups.

Step 20 [no] ip igmp report-policy policy-name Configures an access policy for IGMP reports
that is based on a route-map policy.
Example:
apic1(config-leaf-if)# ip igmp
report-policy MyReportPolicy

Step 21 [no] ip igmp robustness-variable value Sets the robustness variable to compensate for
packet loss on a congested network. The
Example:
robustness value is used by the IGMP software
apic1(config-leaf-if)# ip igmp to determine the number of times to send
robustness-variable 2
messages. You can use a larger value for a
lossy network. The range is 1 to 7. The default
is 2.

Step 22 [no] ip igmp startup-query-count count Sets the number of queries sent at startup that
are separated by the startup query interval. The
Example:
range is 1 to 10 queries. The default is 2
apic1(config-leaf-if)# ip igmp queries.
startup-query-count 2

Step 23 [no] ip igmp startup-query-interval seconds Sets the query interval used when the software
starts up. By default, this interval is shorter
Example:
than the query interval so that the software can
apic1(config-leaf-if)# ip igmp establish the group state as quickly as possible.
startup-query-interval 31
The range is 1 to 18000 seconds. The default
is 260 seconds. The default is 31 seconds.

Step 24 [no] ip igmp state-limit max-states [reserved Configures a per interface limit on the number
route-map-name [max-reserved-gsg-entries]] of mroutes states created as a result of IGMP
membership reports (IGMP joins). The range
Example:
of states allowed is 1 to 4294967295 states.
apic1(config-leaf-if)# ip igmp You can optionally specify a number of state
state-limit 100000 reserved
myReservedPolicy 40000
entries to be reserved for the routes specified
in a policy map and you can specify the
maximum reserved (*, G) and (S, G) entries
allowed on the interface. The number of
reserved states must be less than or equal to
the maximum states allowed. The range is from
1 to 4294967295.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

262
 Configuring Layer 3 External Connectivity
Example: Configuring Layer 3 Multicast

Command or Action Purpose

Step 25 [no] ip igmp static-oif route-map Statically binds a multicast group to the
route-map-name outgoing interface (OIF), which is handled by
the device hardware. The route map defines
Example:
the group prefixes where this feature is applied.
apic1(config-leaf-if)# ip igmp
static-oif route-map MyOifMap

Step 26 [no] ip igmp version {v1 | v2 | v3} Configures the IGMP version number for the
interface. The default version is v2.
Example:
apic1(config-leaf-if)# ip igmp version
v3

Step 27 exit Returns to tenant configuration mode.

Example:
apic1(config-leaf-if)# exit

Example: Configuring Layer 3 Multicast

# CONFIGURE PIM OPTIONS ON A TENANT VRF

apic1# configure
apic1(config)# tenant exampleCorp
apic1(config-tenant)# vrf context exampleCorp_vrf1
apic1(config-tenant-vrf)# ip pim
apic1(config-tenant-vrf)# ip pim fast-convergence
apic1(config-tenant-vrf)# ip pim bsr forward

# ENABLE AND CONFIGURE IGMP ON THE TENANT VRF AND BRIDGE DOMAIN

apic1(config-tenant-vrf)# ip igmp
apic1(config-tenant-vrf)# exit
apic1(config-tenant)# interface bridge-domain exampleCorp_bd
apic1(config-tenant-interface)# ip multicast
apic1(config-tenant-interface)# ip igmp allow-v3-asm
apic1(config-tenant-interface)# ip igmp fast-leave
apic1(config-tenant-interface)# exit

# CREATE AN L3OUT AND CONFIGURE PIM

apic1(config-tenant)# l3out exampleCorp_l3out

apic1(config-tenant-l3out)# ip pim
apic1(config-tenant-l3out)# exit
apic1(config-tenant)# exit

# CONFIGURE AN EXTERNAL INTERFACE AND CONFIGURE IGMP ON THE INTERFACE

apic1(config)# leaf 101

apic1(config-leaf)# interface ethernet 1/125
apic1(config-leaf-if)# ip igmp fast-leave
apic1(config-leaf-if)# ip-igmp join-group

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

263
 Configuring Layer 3 External Connectivity
Configuring External-L3 EPGs

Configuring External-L3 EPGs

External-L3 EPGs are classified under a tenant VRF. In the CLI, an external-l3 EPG is defined in the tenant
mode and is deployed to individual nodes. You have the flexibility to place external-l3 EPGs in a select set
of nodes instead of all nodes in a VRF.
Each external-l3 EPG can be a producer/consumer of multiple contracts, and each external-l3 EPG has its
own QoS policy for DSCP marking and queuing priority within the fabric.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 tenant tenant-name Enters the tenant configuration mode.

Example:
apic1(config)# tenant exampleCorp

Step 3 external-l3 epg epg-name Enters the external-l3 EPG configuration

mode.
Example:
apic1(config-tenant)# external-l3 epg
epgExtern1

Step 4 vrf member vrf-name Associates the EPG with a VRF.

Example:
apic1(config-tenant-l3ext-epg)# vrf
member v1

Step 5 match {ip | ipv6} ip-address/masklength Creates a rule to match a subnet.

Example:

apic1(config-tenant-l3ext-epg)# match
ip 192.0.20.0/24
apic1(config-tenant-l3ext-epg)# match
ipv6 2001::1/64

Step 6 set qos-class class Specifies the QOS level for the EPG.
Example:
apic1(config-tenant-l3ext-epg)# set
qos-class level1

Step 7 set dscp dscp-value Specifies the DSCP value for the EPG.
Example:
apic1(config-tenant-l3ext-epg)# set dscp
af31

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

264
Configuring Layer 3 External Connectivity
Configuring External-L3 EPGs

Command or Action Purpose

Step 8 contract consumer contract-name Specifies the consumer contract for the EPG.
Example:
apic1(config-tenant-l3ext-epg)# contract
consumer cConsumer1

Step 9 contract provider contract-name Specifies the provider contract for the EPG.
Example:
apic1(config-tenant-l3ext-epg)# contract
provider cProvider1

Step 10 contract deny contract-name Specifies a deny contract for the EPG.
Example:
apic1(config-tenant-l3ext-epg)# contract
deny cDeny1

Step 11 exit
Example:
apic1(config-tenant-l3ext-epg)# exit

Step 12 exit
Example:
apic1(config-tenant)# exit

Step 13 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 14 vrf context tenant tenant-name vrf vrf-name Configures a tenant VRF on the node.
Example:
apic1(config-leaf)# vrf context tenant
exampleCorp vrf v1

Step 15 external-l3 epg epg-name Associates the external layer 3 EPG on the
VRF.
Example:
apic1(config-leaf-vrf)# external-l3 epg
epgExtern1

Examples
This example shows how to configure an external layer 3 EPG and to deploy the EPG on a leaf.

apic1# configure
apic1(config)# tenant exampleCorp

# CONFIGURE EXTERNAL L3 EPG

apic1(config-tenant)# external-l3 epg epgExtern1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

265
 Configuring Layer 3 External Connectivity
Configuring Layer 3 External Connectivity Using the Named Mode

apic1(config-tenant-l3ext-epg)# vrf member v1

apic1(config-tenant-l3ext-epg)# match ip 192.0.20.0/24
apic1(config-tenant-l3ext-epg)# match ipv6 2001::1/64
apic1(config-tenant-l3ext-epg)# set qos-class level1
apic1(config-tenant-l3ext-epg)# set dscp af31
apic1(config-tenant-l3ext-epg)# contract consumer cConsumer1
apic1(config-tenant-l3ext-epg)# contract provider cProvider1
apic1(config-tenant-l3ext-epg)# contract deny cDeny1
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# exit

# DEPLOY EXTERNAL L3 EPG ON A LEAF

apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant exampleCorp vrf v1
apic1(config-leaf-vrf)# external-l3 epg epgExtern1

Configuring Layer 3 External Connectivity Using the Named

Mode
Creating a Named L3Out
Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 tenant tenant-name Enters the tenant configuration mode.

Example:
apic1(config)# tenant exampleCorp

Step 3 vrf context vrf-name Associates the tenant with a VRF.

Example:
apic1(config-tenant)# vrf context v1

Step 4 l3out l3out-name Creates a named L3Out.

Example:
apic1(config-tenant)# l3out out1

Step 5 vrf member vrf-name Associates the L3Out with the tenant VRF.
Example:
apic1(config-tenant-l3out)# vrf member
v1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

266
Configuring Layer 3 External Connectivity
Creating a Named L3Out

Command or Action Purpose

Step 6 exit Returns to tenant configuration mode.
Example:
apic1(config-tenant-l3out)# exit

Step 7 exit Returns to global configuration mode.

Example:
apic1(config-tenant)# exit

Step 8 leaf node-id node

Example:
apic1(config)# leaf 101

Step 9 vrf context tenant tenant-name vrf vrf-name Configures a tenant VRF on the node.
l3out l3out-name
Example:
apic1(config-leaf)# vrf context tenant
exampleCorp vrf v1 l3out out1

Step 10 Required: [no] router-id ipv4-address Assigns a router ID for routing protocols
running on the VRF.
Example:
apic1(config-leaf-vrf)# router-id
1.2.3.4

Step 11 [no] {ip | ipv6} route ip-prefix/masklen Configures static route information for the
next-hop-address [preferred] VRF.
Example:

apic1(config-leaf-vrf)# ip route
21.1.1.1/32 32.1.1.1
apic1(config-leaf-vrf)# ipv6 route
5001::1/128 6002::1

Examples
This example shows how to create a named L3Out under the tenant, assign it to the tenant VRF, and
deploy it on the border leaf switch.

apic1# configure
apic1(config)# tenant exampleCorp
apic1(config-tenant)# vrf context v1
apic1(config-tenant)# l3out out1
apic1(config-tenant-l3out)# vrf member v1
apic1(config-tenant-l3out)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant exampleCorp vrf v1 l3out out1
apic1(config-leaf-vrf)# router-id 1.2.3.4

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

267
 Configuring Layer 3 External Connectivity
Configuring Layer 3 Interfaces for a Named L3Out

apic1(config-leaf-vrf)# ip route 21.1.1.1/32 32.1.1.1

What to do next
Configure layer 3 interfaces for the named L3Out.

Configuring Layer 3 Interfaces for a Named L3Out

This procedure shows how to configure a layer 3 port interface to a named L3Out. The examples show how
to configure a subinterface or SVI to a named L3Out.
• A given interface can be added to multiple L3Outs by providing multiple L3Out names after the l3out
keyword.
• An SVI can be configured using the switchport trunk allowed vlan command under any of the following
interface types:
• interface Ethernet
• interface port-channel
• interface vpc

Before you begin

Create a named L3Out.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 interface type Specifies a port for the external interface.

Example:
apic1(config-leaf)# interface eth 1/20

Step 4 no switchport Configures the interface as a layer 3 interface,

exposing the layer 3 commands in the
Example:
configuration options.
apic1(config-leaf-if)# no switchport

Step 5 vrf member tenant tenant-name vrf vrf-name Attaches the interface to the tenant VRF.
l3out l3out-name

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

268
Configuring Layer 3 External Connectivity
Configuring Layer 3 Interfaces for a Named L3Out

Command or Action Purpose

Example:
apic1(config-leaf-if)# vrf member tenant
exampleCorp vrf v1 l3out out1

Step 6 [no] {ip | ipv6} address ip-prefix/masklen Configures IP addresses on the interface. The
[eui64] [secondary] [preferred] specified address can be declared as either:
Example: • preferred —The default source address
for traffic from the interface.
apic1(config-leaf-if)# ip address
10.1.1.1/24 • secondary —The secondary address of
apic1(config-leaf-if)# ipv6 address the interface.
2001::1/64 preferred
With the optional eui64 keyword, the host can
assign itself a 64-bit Extended Unique Identifier
(EUI).
In this mode, you can also configure ipv6
link-local , mac address , mtu , and other
layer 3 properties on the interface.

Examples
This example shows how to assign a layer 3 port to a named L3Out.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface eth 1/20
apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1 l3out out1
apic1(config-leaf-if)# ip address 10.1.1.1/24
apic1(config-leaf-if)# ipv6 address 2001::1/64 preferred

This example shows how to assign a layer 3 subinterface to a named L3Out.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface eth 1/5
apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# vlan-domain member d1
apic1(config-leaf-if)# exit
apic1(config-leaf)# interface ethernet 1/5.1000
apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1 l3out out1
apic1(config-leaf-if)# ip address 10.1.1.1/24
apic1(config-leaf-if)# ipv6 address 2001::1/64 preferred

This example shows how to assign a layer 3 SVI to a named L3Out.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# interface vlan 200
apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

269
 Configuring Layer 3 External Connectivity
Configuring Route Maps for a Named L3Out

apic1(config-leaf-if)# ip address 10.1.1.1/24

apic1(config-leaf-if)# exit
apic1(config-leaf)# interface ethernet 1/4
apic1(config-leaf-if)# vlan-domain member d1
apic1(config-leaf-if)# switchport trunk allowed vlan 200 tenant t1 external-svi l3out out1

Configuring Route Maps for a Named L3Out

• Route-maps are configured under the leaf, VRF mode.
• The following route-maps are created for every named L3Out :
• Export—Route-map for routes advertised out of a routing protocol enabled on the L3Out. By default,
no routes are exported out until you explicitly enable them in the route-map through one or more
of match bridge-domain , match prefix-list and match community-list statements.
• Import—Route-map for routes imported into the routing protocol on the L3Out. By default, all
routes are imported. You can control specific routes to be imported by using one or more match
prefix-list or match community-list statements.
• Shared—Route-map that contains the routes and the contract provider/consumer policy that will be
used for leaking the routes from this VRF to any other VRF that has the contract association.

These route-maps are created when you associate a leaf to the L3Out through the vrf context tenant
tenant-name vrf vrf-name l3out l3out-name command.
• The scope of the route-maps under the named L3Out is always global and is applicable on all nodes
where the Named L3Out is deployed.
• All commands under the route-map (such as match prefix-list , match community-list , match
bridge-domain ) are the same as the route-map configuration for the Basic Mode discussed in the previous
sections.

Before you begin

Create a named L3Out.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 [no] vrf context tenant tenant-name vrf Configures a tenant VRF on the node.
vrf-name l3out l3out-name
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

270
Configuring Layer 3 External Connectivity
Configuring Route Maps for a Named L3Out

Command or Action Purpose

apic1(config-leaf)# vrf context tenant
exampleCorp vrf v1 l3out out1

Step 4 Required: [no] route-map name Creates a route-map and enters route-map
configuration. This will be the import
Example:
route-map.
apic1(config-leaf-vrf)# route-map
out1_in

Step 5 Required: [no] ip prefix-list list-name permit Creates a prefix-list under the route-map.
prefix/masklen [le {32 | 128}]
Example:
apic1(config-leaf-vrf-route-map)# ip
prefix-list p1 permit 15.1.1.0/24

Step 6 Required: [no] match prefix-list list-name Matches a prefix-list that has already been
created and enters the match mode to configure
Example:
the route-control profile for the prefix-list.
apic1(config-leaf-vrf-route-map)# match
prefix-list p1

Step 7 Required: exit Returns to route-map configuration mode.

Example:
apic1(config-leaf-vrf-route-map-match)#
exit

Step 8 Required: exit Returns to leaf VRF configuration mode.

Example:
apic1(config-leaf-vrf-route-map)# exit

Step 9 Required: [no] route-map name Creates a route-map and enters route-map
configuration. This will be the export
Example:
route-map.
apic1(config-leaf-vrf)# route-map
out1_out

Step 10 Required: [no] ip prefix-list list-name permit Creates a prefix-list under the route-map.
prefix/masklen [le {32 | 128}]
Example:
apic1(config-leaf-vrf-route-map)# ip
prefix-list p2 permit 16.1.1.0/24

Step 11 Required: [no] match prefix-list list-name Matches a prefix-list that has already been
created and enters the match mode to configure
Example:
the route-control profile for the prefix-list.
apic1(config-leaf-vrf-route-map)# match
prefix-list p2

Step 12 Required: set tag name Sets the tag value. The name parameter is an
unsigned integer.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

271
 Configuring Layer 3 External Connectivity
Configuring Route Maps for a Named L3Out

Command or Action Purpose

apic1(config-leaf-vrf-route-map-match)#
set tag 100

Step 13 Required: exit Returns to route-map configuration mode.

Example:
apic1(config-leaf-vrf-route-map-match)#
exit

Step 14 Required: [no] match bridge-domain Matches a bridge domain in order to export its
list-name public subnets through the protocol.
Example:
apic1(config-leaf-vrf-route-map)# match
bridge-domain bd1

Step 15 Required: exit Returns to route-map configuration mode.

Example:
apic1(config-leaf-vrf-route-map-match)#
exit

Step 16 Required: [no] route-map name Creates a route-map and enters route-map
configuration. This will be the shared
Example:
route-map.
apic1(config-leaf-vrf)# route-map
out1_shared

Step 17 Required: [no] ip prefix-list list-name permit Creates a prefix-list under the route-map.
prefix/masklen [le {32 | 128}]
Example:
apic1(config-leaf-vrf-route-map)# ip
prefix-list p3 permit 16.10.1.0/24

Step 18 Required: [no] match prefix-list list-name Matches a prefix-list that has already been
created and enters the match mode to configure
Example:
the route-control profile for the prefix-list.
apic1(config-leaf-vrf-route-map)# match
prefix-list p3

Step 19 Required: contract provider name Adds contract, required to leak routes
(matching this prefix-list) from this VRF.
Example:
apic1(config-leaf-vrf-route-map-match)#
contract provider default

Examples
This example shows how to configure route maps for a named L3Out.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant exampleCorp vrf v1 l3out out1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

272
 Configuring Layer 3 External Connectivity
Configuring Routing Protocols for a Named L3Out

# CREATE IMPORT ROUTE-MAP

apic1(config-leaf-vrf)# route-map out1_in
apic1(config-leaf-vrf-route-map)# ip prefix-list p1 permit 15.1.1.0/24
apic1(config-leaf-vrf-route-map)# match prefix-list p1
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit

# CREATE EXPORT ROUTE-MAP

apic1(config-leaf-vrf)# route-map out1_out
apic1(config-leaf-vrf-route-map)# ip prefix-list p2 permit 16.1.1.0/24
apic1(config-leaf-vrf-route-map)# match prefix-list p2
apic1(config-leaf-vrf-route-map-match)# set tag 100
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# match bridge-domain bd1
apic1(config-leaf-vrf-route-map-match)# exit

# CREATE SHARED ROUTE-MAP

apic1(config-leaf-vrf)# route-map out1_shared
apic1(config-leaf-vrf-route-map)# ip prefix-list p3 permit 16.10.1.0/24
apic1(config-leaf-vrf-route-map)# match prefix-list p3
apic1(config-leaf-vrf-route-map-match)# contract provider default

Configuring Routing Protocols for a Named L3Out

Configuring BGP for a Named L3Out
• All commands under the BGP neighbor with the exception of route-map are identical to those in the
Basic Mode of L3Out configuration. The BGP template configuration and the inheritance of the template
are identical to the Basic Mode.
• In the Named Mode of L3Out configuration, the route-map is applied at the L3Out level. By associating
a neighbor with an L3Out, the route-map is automatically applied on the protocols on the L3Out. For
this reason, the route-map option is not applicable and is not available under the BGP Neighbor. For the
same reason, the route-map option is not available for OSPF Area and the distribute-list EIGRP option
is not available under the interface.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 router bgp asn-number Enters BGP policy configuration.

Example:
apic1(config-leaf)# router bgp 100

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

273
 Configuring Layer 3 External Connectivity
Configuring OSPF for a Named L3Out

Command or Action Purpose

Step 4 vrf member tenant tenant-name vrf vrf-name Specifies the VRF instance to associate with
subsequent policy configuration mode
Example:
commands.
apic1(config-bgp)# vrf member tenant
exampleCorp vrf v100

Step 5 neighbor ip-address [/masklength] l3out Specifies the IP address of the neighbor.
l3out-name
Example:
apic1(config-leaf-bgp-vrf)# neighbor
192.0.2.229 l3out out1

Step 6 remote-as asn Specifies Autonomous System Number of the

neighbor.
Example:
apic1(config-leaf-bgp-vrf-neighbor)#
remote-as 300

Step 7 allow-self-as-count count The count can be 1 to 10. The default is 3.

Example:
apic1(config-leaf-bgp-vrf-neighbor)#
allow-self-as-count 5

Step 8 update-source ethernet interface-range Update the Source IP for BGP Packets to one
of loopback, physical, sub-interface or SVI
Example:
interfaces..
apic1(config-leaf-bgp-vrf-neighbor)#
update-source ethernet 1/3

Examples
This example shows how to configure BGP routing protocol for a named L3Out.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# router bgp 100
apic1(config-bgp)# vrf member tenant exampleCorp vrf v1
apic1(config-leaf-bgp-vrf)# neighbor 192.0.2.229 l3out out1
apic1(config-leaf-bgp-vrf-neighbor)# remote-as 300
apic1(config-leaf-bgp-vrf-neighbor)# allow-self-as-count 5
apic1(config-leaf-bgp-vrf-neighbor)# update-source ethernet 1/3

Configuring OSPF for a Named L3Out

All commands under the router ospf default command, with the exception of area area-id route-map
map-name out , are identical to those in the Basic Mode of L3Out configuration. The OSPF commands under
the interface and the OSPF template inherit commands are also identical to the Basic Mode.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

274
Configuring Layer 3 External Connectivity
Configuring OSPF for a Named L3Out

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 router ospf default Creates an OSPF routing process and enters
OSPF policy configuration.
Example:
apic1(config-leaf)# router ospf default

Step 4 vrf member tenant tenant-name vrf vrf-name Enables a VRF in the OSPF session.
Example:
apic1(config-leaf-ospf)# vrf member
tenant exampleCorp vrf v100

Step 5 area area-id l3out l3out-name Enables OSPF in the L3Out.

Example:
apic1(config-leaf-ospf-vrf)# area
0.0.0.1 l3out out1

Step 6 area area-id loopback loopback-address When OSPF is used as a connectivity protocol
for BGP, OSPF advertises the loopback
Example:
address which is used as the source of the BGP
apic1(config-leaf-ospf-vrf)# area session. Note that the loopback IP address and
0.0.0.1 loopback 192.0.20.11
not the loopback ID is used. In this case, a
BGP session relying on OSPF will use the
same loopback IP address in its update-source
command.

Step 7 area area-id nssa [no-redistribution] Defines a not-so-stubby area (NSSA).

[default-information-originate]
Example:
apic1(config-leaf-ospf-vrf)# area
0.0.0.1 nssa

Step 8 exit Returns to the OSPF configuration mode.

Example:
apic1(config-leaf-ospf-vrf)# exit

Step 9 exit Returns to leaf configuration mode.

Example:
apic1(config-leaf-ospf)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

275
 Configuring Layer 3 External Connectivity
Configuring OSPF for a Named L3Out

Command or Action Purpose

Step 10 interface type Specifies a port for the external interface.
Example:
apic1(config-leaf)# interface eth 1/20

Step 11 vlan-domain member domain-name Assign a VLAN domain to the interface. The
VLAN domain must have already been created
Example:
using the vlan-domain command in the
apic1(config-leaf-if)# vlan-domain global configuration mode.
member dom1

Step 12 no switchport Configures the interface as a layer 3 interface,

exposing the layer 3 commands in the
Example:
configuration options.
apic1(config-leaf-if)# no switchport

Step 13 vrf member tenant tenant-name vrf vrf-name Attaches the interface to the tenant VRF.
l3out l3out-name
Example:
apic1(config-leaf-if)# vrf member tenant
exampleCorp vrf v1 l3out out1

Step 14 [no] {ip | ipv6} address ip-prefix/masklen Configures IP addresses on the interface. The
[eui64] [secondary] [preferred] specified address can be declared as either:
Example: • preferred —The default source address
for traffic from the interface.
apic1(config-leaf-if)# ip address
10.1.1.1/24 • secondary —The secondary address of
apic1(config-leaf-if)# ipv6 address the interface.
2001::1/64 preferred
With the optional eui64 keyword, the host
can assign itself a 64-bit Extended Unique
Identifier (EUI).
In this mode, you can also configure ipv6
link-local , mac address , mtu , and other
layer 3 properties on the interface.

Step 15 {ip | ipv6} router ospf default area area-id Creates an OSPF routing process and enters
OSPF policy configuration.
Example:
apic1(config-leaf-if)# ip router ospf
default area 0.0.0.1

Examples
This example shows how to configure OSPF routing protocol for a named L3Out.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# router ospf default

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

276
 Configuring Layer 3 External Connectivity
Configuring EIGRP for a Named L3Out

apic1(config-leaf-ospf)# vrf member tenant exampleCorp vrf v1

apic1(config-leaf-ospf-vrf)# area 0.0.0.1 l3out out1
apic1(config-leaf-ospf-vrf)# area 0.0.0.1 loopback 192.0.20.11
apic1(config-leaf-ospf-vrf)# area 0.0.0.1 nssa
apic1(config-leaf-ospf-vrf)# exit
apic1(config-leaf-ospf)# exit
apic1(config-leaf)# interface eth 1/20
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1 l3out out1
apic1(config-leaf-if)# ip address 10.1.1.1/24
apic1(config-leaf-if)# ip router ospf default area 0.0.0.1

Configuring EIGRP for a Named L3Out

All EIGRP commands under vrf mode and interface mode, with the exception of ip distribute-list , are
identical to those in the Basic Mode of L3Out configuration. This includes the EIGRP template and inherit
commands. The ip distribute-list commands are not applicable to the Named Mode of L3Out configuration,
as the route-maps are defined at the L3Out level and by associating an interface with the L3Out, the route-map
distribute-list is automatically associated. For this reason, ip distribute-list is not available in the CLI as a
option.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 leaf node-id Specifies the leaf to be configured.

Example:
apic1(config)# leaf 101

Step 3 router eigrp default Enters EIGRP policy configuration.

Example:
apic1(config-leaf)# router eigrp default

Step 4 vrf member tenant tenant-name vrf vrf-name Specifies the VRF instance to associate with
subsequent configuration mode commands.
Example:
apic1(config-eigrp)# vrf member tenant
exampleCorp vrf v100

Step 5 autonomous-system asn l3out l3out-name Enters Autonomous System configuration for
EIGRP.
Example:
apic1(config-eigrp-vrf)#
autonomous-system 500 l3out out1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

277
 Configuring Layer 3 External Connectivity
Configuring EIGRP for a Named L3Out

Command or Action Purpose

Step 6 exit Returns to the EIGRP configuration mode.
Example:
apic1(config-eigrp-vrf)# exit

Step 7 exit Returns to leaf configuration mode.

Example:
apic1(config-eigrp)# exit

Step 8 interface type Specifies a port for the external interface.

Example:
apic1(config-leaf)# interface eth 1/5

Step 9 vlan-domain member domain-name Assign a VLAN domain to the interface. The
VLAN domain must have already been created
Example:
using the vlan-domain command in the
apic1(config-leaf-if)# vlan-domain global configuration mode.
member dom1

Step 10 no switchport Configures the interface as a layer 3 interface,

exposing the layer 3 commands in the
Example:
configuration options.
apic1(config-leaf-if)# no switchport

Step 11 vrf member tenant tenant-name vrf vrf-name Attaches the interface to the tenant VRF.
l3out l3out-name
Example:
apic1(config-leaf-if)# vrf member tenant
exampleCorp vrf v1 l3out out1

Step 12 [no] {ip | ipv6} address ip-prefix/masklen Configures IP addresses on the interface. The
[eui64] [secondary] [preferred] specified address can be declared as either:
Example: • preferred —The default source address
for traffic from the interface.
apic1(config-leaf-if)# ip address
10.1.1.1/24 • secondary —The secondary address of
apic1(config-leaf-if)# ipv6 address the interface.
2001::1/64 preferred
With the optional eui64 keyword, the host
can assign itself a 64-bit Extended Unique
Identifier (EUI).
In this mode, you can also configure ipv6
link-local , mac address , mtu , and other
layer 3 properties on the interface.

Step 13 {ip | ipv6} router eigrp default Sets EIGRP policies to default.
Example:
apic1(config-leaf-if)# ip router eigrp
default

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

278
 Configuring Layer 3 External Connectivity
Configuring External-L3 EPGs for a Named L3Out

Examples
This example shows how to configure EIGRP routing protocol for a named L3Out.

apic1# configure
apic1(config)# leaf 101
apic1(config-leaf)# router eigrp default
apic1(config-eigrp)# vrf member tenant exampleCorp vrf v1
apic1(config-eigrp-vrf)# autonomous-system 500 l3out out1
apic1(config-eigrp-vrf)# exit
apic1(config-eigrp)# exit
apic1(config-leaf)# interface eth 1/5
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# vrf member tenant exampleCorp vrf v1 l3out out1
apic1(config-leaf-if)# ip address 10.1.1.1/24
apic1(config-leaf-if)# ip router eigrp default

Configuring External-L3 EPGs for a Named L3Out

External-L3 EPGs are classified under a tenant VRF.
All commands under the config-tenant-l3ext-epg mode are identical to those in the Basic Mode of L3Out
configuration with the following differences:
• The VRF is automatically associated with the EPG. The L3Out associates with a VRF and the EPG
associates with the L3Out.
• The external-l3 epg command is not available under the leaf vrf context tenant tenant-name vrf
vrf-name l3out l3out-name command, as this configuration is not applicable for Named L3Outs. The
external-l3 epg is automatically deployed on the leaf, when the external-l3 epg is created within a
named L3Out and a leaf is associated with the same L3Out through the vrf context tenant tenant-name
vrf vrf-name l3out l3out-name command.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 tenant tenant-name Enters the tenant configuration mode.

Example:
apic1(config)# tenant exampleCorp

Step 3 external-l3 epg epg-name l3out l3out-name Enters the external-l3 EPG configuration mode.
Example:
apic1(config-tenant)# external-l3 epg
epg1 l3out out1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

279
 Configuring Layer 3 External Connectivity
IPv6 Neighbor Discovery

Command or Action Purpose

Step 4 match {ip | ipv6} ip-address/masklength Creates a rule to match a subnet.
Example:

apic1(config-tenant-l3ext-epg)# match ip
192.0.20.0/24
apic1(config-tenant-l3ext-epg)# match
ipv6 2001::1/64

Step 5 contract consumer contract-name Specifies the consumer contract for the EPG.
Example:
apic1(config-tenant-l3ext-epg)# contract
consumer cConsumer1

Step 6 contract provider contract-name Specifies the provider contract for the EPG.
Example:
apic1(config-tenant-l3ext-epg)# contract
provider cProvider1

Examples
This example shows how to configure an external layer 3 EPG for a named L3Out.

apic1# configure
apic1(config)# tenant exampleCorp
apic1(config-tenant)# external-l3 epg epg1 l3out out1
apic1(config-tenant-l3ext-epg)# match ip 192.0.20.0/24
apic1(config-tenant-l3ext-epg)# match ipv6 2001::1/64
apic1(config-tenant-l3ext-epg)# contract consumer cConsumer1
apic1(config-tenant-l3ext-epg)# contract provider cProvider1

IPv6 Neighbor Discovery

Neighbor Discovery
The IPv6 Neighbor Discovery (ND) protocol is responsible for the address auto configuration of nodes,
discovery of other nodes on the link, determining the link-layer addresses of other nodes, duplicate address
detection, finding available routers and DNS servers, address prefix discovery, and maintaining reachability
information about the paths to other active neighbor nodes.
ND-specific Neighbor Solicitation or Neighbor Advertisement (NS or NA) and Router Solicitation or Router
Advertisement (RS or RA) packet types are supported on all ACI fabric Layer 3 interfaces, including physical,
Layer 3 sub interface, and SVI (external and pervasive). Up to APIC release 3.1(1x), RS/RA packets are used
for auto configuration for all Layer 3 interfaces but are only configurable for pervasive SVIs.
Starting with APIC release 3.1(2x), RS/RA packets are used for auto configuration and are configurable on
Layer 3 interfaces including routed interface, Layer 3 sub interface, and SVI (external and pervasive).

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

280
 Configuring Layer 3 External Connectivity
Configuring a Tenant, VRF, and Bridge Domain with IPv6 Neighbor Discovery on the Bridge Domain Using the NX-OS Style CLI

ACI bridge domain ND always operates in flood mode; unicast mode is not supported.
The ACI fabric ND support includes the following:
• Interface policies (nd:IfPol) control ND timers and behavior for NS/NA messages.
• ND prefix policies (nd:PfxPol) control RA messages.
• Configuration of IPv6 subnets for ND (fv:Subnet).
• ND interface policies for external networks.
• Configurable ND subnets for external networks, and arbitrary subnet configurations for pervasive bridge
domains are not supported.

Configuration options include the following:

• Adjacencies
• Configurable Static Adjacencies: (<vrf, L3Iface, ipv6 address> --> mac address)
• Dynamic Adjacencies: Learned via exchange of NS/NA packets

• Per Interface
• Control of ND packets (NS/NA)
• Neighbor Solicitation Interval
• Neighbor Solicitation Retry count

• Control of RA packets
• Suppress RA
• Suppress RA MTU
• RA Interval, RA Interval minimum, Retransmit time

• Per Prefix (advertised in RAs) control

• Lifetime, preferred lifetime
• Prefix Control (auto configuration, on link)

• Neighbor Discovery Duplicate Address Detection (DAD)

Configuring a Tenant, VRF, and Bridge Domain with IPv6 Neighbor Discovery
on the Bridge Domain Using the NX-OS Style CLI
Procedure

Step 1 Configure an IPv6 neighbor discovery interface policy and assign it to a bridge domain:
a) Create an IPv6 neighbor discovery interface policy:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

281
 Configuring Layer 3 External Connectivity
Guidelines and Limitations

Example:

apic1(config)# tenant ExampleCorp

apic1(config-tenant)# template ipv6 nd policy NDPol001
apic1(config-tenant-template-ipv6-nd)# ipv6 nd mtu 1500

b) Create a VRF and bridge domain:

Example:

apic1(config-tenant)# vrf context pvn1

apic1(config-tenant-vrf)# exit
apic1(config-tenant)# bridge-domain bd1
apic1(config-tenant-bd)# vrf member pvn1
apic1(config-tenant-bd)# exit

c) Assign an IPv6 neighbor discovery policy to the bridge domain:

Example:

apic1(config-tenant)# interface bridge-domain bd1

apic1(config-tenant-interface)# ipv6 nd policy NDPol001
apic1(config-tenant-interface)#exit

Step 2 Configure an IPV6 bridge domain subnet and neighbor discovery prefix policy on the subnet:
Example:

apic1(config-tenant)# interface bridge-domain bd1

apic1(config-tenant-interface)# ipv6 address 34::1/64
apic1(config-tenant-interface)# ipv6 address 33::1/64
apic1(config-tenant-interface)# ipv6 nd prefix 34::1/64 1000 1000
apic1(config-tenant-interface)# ipv6 nd prefix 33::1/64 4294967295 4294967295

Guidelines and Limitations

The following guidelines and limitations apply to Neighbor Discovery Router Advertisement (ND RA) Prefixes
for Layer 3 Interfaces:
• An ND RA configuration applies only to IPv6 Prefixes. Any attempt to configure an ND policy on IPv4
Prefixes will fail to apply.

Configuring an IPv6 Neighbor Discovery Interface Policy with RA on a Layer

3 Interface Using the NX-OS Style CLI
This example configures an IPv6 neighbor discovery interface policy, and assigns it to a Layer 3 interface.
Next, it configures an IPv6 Layer 3 Out interface, neighbor discovery prefix policy, and associates the neighbor
discovery policy to the interface.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

282
Configuring Layer 3 External Connectivity
Configuring an IPv6 Neighbor Discovery Interface Policy with RA on a Layer 3 Interface Using the NX-OS Style CLI

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 tenant tenant_name Creates a tenant and enters the tenant mode.
Example:

apic1(config)# tenant ExampleCorp

apic1(config-tenant)#

Step 3 template ipv6 nd policy policy_name Creates an IPv6 ND policy.

Example:

apic1(config-tenant)# template ipv6 nd

policy NDPol001

Step 4 ipv6 nd mtu mtu value Assigns an MTU value to the IPv6 ND policy.
Example:

apic1(config-tenant-template-ipv6-nd)#
ipv6 nd mtu 1500
apic1(config-tenant-template-ipv6)# exit
apic1(config-tenant-template)# exit
apic1(config-tenant)#

Step 5 vrf context VRF_name Creates a VRF.

Example:

apic1(config-tenant)# vrf context pvn1

apic1(config-tenant-vrf)# exit

Step 6 l3out VRF_name Creates a Layer 3 Out.

Example:

apic1(config-tenant)# l3out l3extOut001

Step 7 vrf member VRF_name Associates the VRF with the Layer 3 Out.
Example:

apic1(config-tenant-l3out)# vrf member

pvn1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

283
 Configuring Layer 3 External Connectivity
Configuring an IPv6 Neighbor Discovery Interface Policy with RA on a Layer 3 Interface Using the NX-OS Style CLI

Command or Action Purpose

apic1(config-tenant-l3out)# exit

Step 8 external-l3 epg instp l3out l3extOut001 Assigns the Layer 3 Out and the VRF to a
Layer 3 interface.
Example:

apic1(config-tenant)# external-l3 epg

instp l3out l3extOut001
apic1(config-tenant-l3ext-epg)# vrf
member pvn1
apic1(config-tenant-l3ext-epg)# exit

Step 9 leaf 2011 Enters the leaf switch mode.

Example:

apic1(config)# leaf 2011

Step 10 vrf context tenant ExampleCorp vrf pvn1 Associates the VRF to the leaf switch.
l3out l3extOut001
Example:

apic1(config-leaf)# vrf context tenant

ExampleCorp vrf pvn1 l3out l3extOut001

apic1(config-leaf-vrf)# exit

Step 11 int eth 1/1 Enters the interface mode.

Example:

apic1(config-leaf)# int eth 1/1

apic1(config-leaf-if)#

Step 12 vrf member tenant ExampleCorp vrf pvn1 Specifies the associated Tenant, VRF, Layer
l3out l3extOut001 3 Out in the interface.
Example:

apic1(config-leaf-if)# vrf member tenant

ExampleCorp vrf pvn1 l3out l3extOut001

Step 13 ipv6 address 2001:20:21:22::2/64 preferred Specifies the primary or preferred IPv6
address.
Example:

apic1(config-leaf-if)# ipv6 address

2001:20:21:22::2/64 preferred

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

284
 Configuring Layer 3 External Connectivity
Microsoft NLB

Command or Action Purpose

Step 14 ipv6 nd prefix 2001:20:21:22::2/64 1000 Configures the IPv6 ND prefix policy under
1000 the Layer 3 interface.
Example:

apic1(config-leaf-if)# ipv6 nd prefix

2001:20:21:22::2/64 1000 1000

Step 15 inherit ipv6 nd NDPol001 Configures the ND policy under the Layer 3
interface.
Example:

apic1(config-leaf-if)# inherit ipv6 nd

NDPol001
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit

The configuration is complete.

Microsoft NLB
Configuring Microsoft NLB in Unicast Mode Using the NX-OS Style CLI
This task configures Microsoft NLB to flood all of the ports in the bridge domain.

Before you begin

Have the following information available before proceeding with these procedures:
• Microsoft NLB cluster VIP
• Microsoft NLB cluster MAC address

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 tenant tenant-name Creates a tenant if it does not exist or enters

tenant configuration mode.
Example:
apic1 (config)# tenant tenant1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

285
 Configuring Layer 3 External Connectivity
Configuring Microsoft NLB in Multicast Mode Using the NX-OS Style CLI

Command or Action Purpose

Step 3 application app-profile-name Creates an application profile if it doesn't exist
or enters application profile configuration mode.
Example:
apic1 (config-tenant)# application app1

Step 4 epg epg-name Creates an EPG if it doesn't exist or enters EPG

configuration mode.
Example:
apic1 (config-tenant-app)# epg epg1

Step 5 [no] endpoint {ip | ipv6} ip-address epnlb Configures Microsoft NLB in unicast mode,
mode mode-uc mac mac-address where:
Example: • ip-address is the Microsoft NLB cluster
apic1 (config-tenant-app-epg)# endpoint VIP.
ip 192.0.2.2/32 epnlb mode mode-uc mac
03:BF:01:02:03:04 • mac-address is the Microsoft NLB cluster
MAC address.

Configuring Microsoft NLB in Multicast Mode Using the NX-OS Style CLI
This task configures Microsoft NLB to flood only on certain ports in the bridge domain.

Before you begin

Have the following information available before proceeding with these procedures:
• Microsoft NLB cluster VIP
• Microsoft NLB cluster MAC address

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 tenant tenant-name Creates a tenant if it does not exist or enters

tenant configuration mode.
Example:
apic1 (config)# tenant tenant1

Step 3 application app-profile-name Creates an application profile if it doesn't exist

or enters application profile configuration mode.
Example:
apic1 (config-tenant)# application app1

Step 4 epg epg-name Creates an EPG if it doesn't exist or enters EPG

configuration mode.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

286
 Configuring Layer 3 External Connectivity
Configuring Microsoft NLB in IGMP Mode Using the NX-OS Style CLI

Command or Action Purpose

apic1 (config-tenant-app)# epg epg1

Step 5 [no] endpoint {ip | ipv6} ip-address epnlb Configures Microsoft NLB in static multicast
mode mode-mcast--static mac mac-address mode, where:
Example: • ip-address is the Microsoft NLB cluster
apic1 (config-tenant-app-epg)# endpoint VIP.
ip 192.0.2.2/32 epnlb mode
mode-mcast--static mac 03:BF:01:02:03:04 • mac-address is the Microsoft NLB cluster
MAC address.

Step 6 [no] nld static-group mac-address leaf Adds Microsoft NLB multicast VMAC to the
leaf-num interface {ethernet slot/port | EPG ports where the Microsoft NLB servers
port-channel port-channel-name } vlan are connected, where:
portEncapVlan
• mac-address is the Microsoft NLB cluster
Example: MAC address that you entered in Step 5,
apic1 (config-tenant-app-epg)# nlb on page 287.
static-group 03:BF:01:02:03:04 leaf 102
interface ethernet 1/12 vlan 19 • leaf-num is the leaf switch that contains
the interface to be added or removed.
• port-channel-name is the name of the
port-channel, when the port-channel option
is used.
• portEncapVlan is the encapsulation VLAN
for the static member of the application
EPG.

Configuring Microsoft NLB in IGMP Mode Using the NX-OS Style CLI
This task configures Microsoft NLB to flood only on certain ports in the bridge domain.

Before you begin

Have the following information available before proceeding with these procedures:
• Microsoft NLB cluster VIP
• Microsoft NLB cluster MAC address

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

287
 Configuring Layer 3 External Connectivity
MLD Snooping

Command or Action Purpose

Step 2 tenant tenant-name Creates a tenant if it does not exist or enters
tenant configuration mode.
Example:
apic1 (config)# tenant tenant1

Step 3 application app-profile-name Creates an application profile if it doesn't exist

or enters application profile configuration mode.
Example:
apic1 (config-tenant)# application app1

Step 4 epg epg-name Creates an EPG if it doesn't exist or enters EPG

configuration mode.
Example:
apic1 (config-tenant-app)# epg epg1

Step 5 [no] endpoint {ip | ipv6} ip-address epnlb Configures Microsoft NLB in IGMP mode,
mode mode-mcast-igmp group where:
multicast-IP-address
• ip-address is the Microsoft NLB cluster
Example: VIP.
apic1 (config-tenant-app-epg)# endpoint
ip 192.0.2.2/32 epnlb mode
• multicast-IP-address is the multicast IP
mode-mcast-igmp group 1.3.5.7 for the NLB endpoint group.

MLD Snooping
Configuring and Assigning an MLD Snooping Policy to a Bridge Domain using
the NX-OS Style CLI
Before you begin
• Create the tenant that will consume the MLD Snooping policy.
• Create the bridge domain for the tenant, where you will attach the MLD Snooping policy.

Procedure

Command or Action Purpose

Step 1 configure terminal Enters configuration mode.
Example:

apic1# configure terminal

apic1(config)#

Step 2 tenant tenant-name Creates a tenant or enters tenant configuration

mode.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

288
Configuring Layer 3 External Connectivity
Configuring and Assigning an MLD Snooping Policy to a Bridge Domain using the NX-OS Style CLI

Command or Action Purpose

apic1(config)# tenant tn1

apic1(config-tenant)#

Step 3 template ipv6 mld snooping policy Creates an MLD snooping policy. The example
policy-name NX-OS style CLI sequence creates an MLD
snooping policy named mldPolicy1.
Example:

apic1(config-tenant)# template ipv6 mld

snooping policy mldPolicy1
apic1(config-tenant-template-ip-mld-snooping)#

Step 4 [no] ipv6 mld snooping Enables or disables the admin state of the MLD
snoop policy. The default state is disabled.
Example:

apic1(config-tenant-template-ip-mld-snooping)#
ipv6 mld snooping
apic1(config-tenant-template-ip-mld-snooping)#
no ipv6 mld snooping

Step 5 [no] ipv6 mld snooping fast-leave Enables or disables IPv6 MLD snooping
fast-leave processing.
Example:

apic1(config-tenant-template-ip-mld-snooping)#
ipv6 mld snooping fast-leave
apic1(config-tenant-template-ip-mld-snooping)#
no ipv6 mld snooping fast-leave

Step 6 [no] ipv6 mld snooping querier Enables or disables IPv6 MLD snooping
querier processing. For the enabling querier
Example:
option to be effectively enabled on the assigned
policy, you must also enable the querier option
apic1(config-tenant-template-ip-mld-snooping)#
ipv6 mld snooping querier in the subnets assigned to the bridge domains
apic1(config-tenant-template-ip-mld-snooping)# to which the policy is applied, as described in
no ipv6 mld snooping querier Step 14, on page 290.

Step 7 ipv6 mld snooping Changes the IPv6 MLD snooping last member
last-member-query-interval parameter query interval parameter. The example NX-OS
style CLI sequence changes the IPv6 MLD
Example:
snooping last member query interval parameter
to 25 seconds. Valid options are 1-25. The
apic1(config-tenant-template-ip-mld-snooping)#
ipv6 mld snooping default is 1 second.
last-member-query-interval 25

Step 8 ipv6 mld snooping query-interval parameter Changes the IPv6 MLD snooping query
interval parameter. The example NX-OS style
Example:
CLI sequence changes the IPv6 MLD snooping
query interval parameter to 300 seconds. Valid
apic1(config-tenant-template-ip-mld-snooping)#
ipv6 mld snooping query-interval 300 options are 1-18000. The default is 125
seconds.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

289
 Configuring Layer 3 External Connectivity
Configuring and Assigning an MLD Snooping Policy to a Bridge Domain using the NX-OS Style CLI

Command or Action Purpose

Step 9 ipv6 mld snooping query-max-response-time Changes the IPv6 MLD snooping max query
parameter response time. The example NX-OS style CLI
sequence changes the IPv6 MLD snooping
Example:
max query response time to 25 seconds. Valid
options are 1-25. The default is 10 seconds.
apic1(config-tenant-template-ip-mld-snooping)#
ipv6 mld snooping
query-max-response-time 25

Step 10 ipv6 mld snooping startup-query-count Changes the IPv6 MLD snooping number of
parameter initial queries to send. The example NX-OS
style CLI sequence changes the IPv6 MLD
Example:
snooping number of initial queries to send to
10. Valid options are 1-10. The default is 2.
apic1(config-tenant-template-ip-mld-snooping)#
ipv6 mld snooping startup-query-count
10

Step 11 ipv6 mld snooping startup-query-interval Changes the IPv6 MLD snooping time for
parameter sending initial queries. The example NX-OS
style CLI sequence changes the IPv6 MLD
Example:
snooping time for sending initial queries to
300 seconds. Valid options are 1-18000. The
apic1(config-tenant-template-ip-mld-snooping)#
ipv6 mld snooping default is 31 seconds.
startup-query-interval 300

Step 12 exit Returns to configure mode.

Example:

apic1(config-tenant-template-ip-mld-snooping)#
exit
apic1(config-tenant)#

Step 13 interface bridge-domain bridge-domain-name Configures the interface bridge-domain. The

example NX-OS style CLI sequence configures
Example:
the interface bridge-domain named bd1.
apic1(config-tenant)# interface
bridge-domain bd1
apic1(config-tenant-interface)#

Step 14 ipv6 address sub-bits/prefix-length Configures the bridge domain as

snooping-querier switch-querier. This will enable the querier
option in the subnet assigned to the bridge
Example:
domain where the policy is applied.
apic1(config-tenant-interface)# ipv6
address 2000::5/64 snooping-querier

Step 15 ipv6 mld snooping policy policy-name Associates the bridge domain with an MLD
snooping policy. The example NX-OS style
Example:
CLI sequence associates the bridge domain

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

290
 Configuring Layer 3 External Connectivity
Configuring HSRP

Command or Action Purpose

with an MLD snooping policy named
apic1(config-tenant-interface)# ipv6
mldPolicy1.
mld snooping policy mldPolicy1

Step 16 exit Returns to configure mode.

Example:

apic1(config-tenant-interface)# exit
apic1(config-tenant)#

Configuring HSRP
Configuring HSRP in Cisco APIC Using Inline Parameters in NX-OS Style CLI
HSRP is enabled when the leaf switch is configured.

Before you begin

• The tenant and VRF configured.
• VLAN pools must be configured with the appropriate VLAN range defined and the appropriate Layer
3 domain created and attached to the VLAN pool.
• The Attach Entity Profile must also be associated with the Layer 3 domain.
• The interface profile for the leaf switches must be configured as required.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 Configure HSRP by creating inline parameters.

Example:
apic1(config)# leaf 101
apic1(config-leaf)# interface ethernet
1/17
apic1(config-leaf-if)# hsrp version 1
apic1(config-leaf-if)# hsrp use-bia
apic1(config-leaf-if)# hsrp delay minimum
30
apic1(config-leaf-if)# hsrp delay reload
30
apic1(config-leaf-if)# hsrp 10 ipv4
apic1(config-if-hsrp)# ip 182.16.1.2
apic1(config-if-hsrp)# ip 182.16.1.3

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

291
 Configuring Layer 3 External Connectivity
Configuring HSRP in Cisco APIC Using Template and Policy in NX-OS Style CLI

Command or Action Purpose

secondary
apic1(config-if-hsrp)# ip 182.16.1.4
secondary
apic1(config-if-hsrp)# mac-address
5000.1000.1060
apic1(config-if-hsrp)# timers 5 18
apic1(config-if-hsrp)# priority 100
apic1(config-if-hsrp)# preempt
apic1(config-if-hsrp)# preempt delay
minimum 60
apic1(config-if-hsrp)# preempt delay
reload 60
apic1(config-if-hsrp)# preempt delay sync
60
apic1(config-if-hsrp)# authentication
none
apic1(config-if-hsrp)# authentication
simple
apic1(config-if-hsrp)# authentication
md5
apic1(config-if-hsrp)# authentication-key
<mypassword>
apic1(config-if-hsrp)#
authentication-key-timeout <timeout>

Configuring HSRP in Cisco APIC Using Template and Policy in NX-OS Style
CLI
HSRP is enabled when the leaf switch is configured.

Before you begin

• The tenant and VRF configured.
• VLAN pools must be configured with the appropriate VLAN range defined and the appropriate Layer
3 domain created and attached to the VLAN pool.
• The Attach Entity Profile must also be associated with the Layer 3 domain.
• The interface profile for the leaf switches must be configured as required.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 Configure HSRP policy templates.

Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

292
Configuring Layer 3 External Connectivity
Configuring HSRP in Cisco APIC Using Template and Policy in NX-OS Style CLI

Command or Action Purpose

apic1(config)# leaf 101

apic1(config-leaf)# template hsrp
interface-policy hsrp-intfPol1 tenant t9
apic1(config-template-hsrp-if-pol)# hsrp
use-bia
apic1(config-template-hsrp-if-pol)# hsrp
delay minimum 30
apic1(config-template-hsrp-if-pol)# hsrp
delay reload 30

apic1(config)# leaf 101

apic1(config-leaf)# template hsrp
group-policy hsrp-groupPol1 tenant t9
apic1(config-template-hsrp-group-pol)#
timers 5 18
apic1(config-template-hsrp-group-pol)#
priority 100
apic1(config-template-hsrp-group-pol)#
preempt
apic1(config-template-hsrp-group-pol)#
preempt delay minimum 60
apic1(config-template-hsrp-group-pol)#
preempt delay reload 60
apic1(config-template-hsrp-group-pol)#
preempt delay sync 60

Step 3 Use the configured policy templates

Example:

apic1(config)# leaf 101

apic1(config-leaf)# interface ethernet
1/17
apic1(config-leaf-if)# hsrp version 1
apic1(config-leaf-if)# inherit hsrp
interface-policy hsrp-intfPol1
apic1(config-leaf-if)# hsrp 10 ipv4
apic1(config-if-hsrp)# ip 182.16.1.2
apic1(config-if-hsrp)# ip 182.16.1.3
secondary
apic1(config-if-hsrp)# ip 182.16.1.4
secondary
apic1(config-if-hsrp)# mac-address
5000.1000.1060
apic1(config-if-hsrp)# inherit hsrp
group-policy hsrp-groupPol1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

293
 Configuring Layer 3 External Connectivity
Cisco ACI GOLF

Cisco ACI GOLF

Cisco ACI GOLF
The Cisco ACI GOLF feature (also known as Layer 3 EVPN Services for Fabric WAN) enables much more
efficient and scalable ACI fabric WAN connectivity. It uses the BGP EVPN protocol over OSPF for WAN
routers that are connected to spine switches.
Figure 23: Cisco ACI GOLF Topology

All tenant WAN connections use a single session on the spine switches where the WAN routers are connected.
This aggregation of tenant BGP sessions towards the Data Center Interconnect Gateway (DCIG) improves
control plane scale by reducing the number of tenant BGP sessions and the amount of configuration required
for all of them. The network is extended out using Layer 3 subinterfaces configured on spine fabric ports.
Transit routing with shared services using GOLF is not supported.
A Layer 3 external outside network (L3extOut) for GOLF physical connectivity for a spine switch is specified
under the infra tenant, and includes the following:
• LNodeP (l3extInstP is not required within the L3Out in the infra tenant. )
• A provider label for the L3extOut for GOLF in the infra tenant.
• OSPF protocol policies
• BGP protocol policies

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

294
Configuring Layer 3 External Connectivity
Cisco ACI GOLF

All regular tenants use the above-defined physical connectivity. The L3extOut defined in regular tenants
requires the following:
• An l3extInstP (EPG) with subnets and contracts. The scope of the subnet is used to control import/export
route control and security policies. The bridge domain subnet must be set to advertise externally and it
must be in the same VRF as the application EPG and the GOLF L3Out EPG.
• Communication between the application EPG and the GOLF L3Out EPG is governed by explicit contracts
(not Contract Preferred Groups).
• An l3extConsLbl consumer label that must be matched with the same provider label of an L3Out for
GOLF in the infra tenant. Label matching enables application EPGs in other tenants to consume the
LNodeP external L3Out EPG.

• The BGP EVPN session in the matching provider L3extOut in the infra tenant advertises the tenant
routes defined in this L3Out.

Guidelines and Limitations

Observe the following GOLF guidelines and limitations:
• GOLF routers must advertise at least one route to Cisco ACI in order to accept traffic. No tunnel is
created between leaf switches and the external routers until Cisco ACI receives a route from the external
routers.
• All Cisco Nexus 9000 Series ACI-mode switches and all of the Cisco Nexus 9500 platform ACI-mode
switch line cards and fabric modules support GOLF. With Cisco APIC, release 3.1(x) and higher, this
includes the N9K-C9364C switch.
• At this time, only a single GOLF provider policy can be deployed on spine switch interfaces for the
whole fabric.
• Up to APIC release 2.0(2), GOLF is not supported with multipod. In release 2.0 (2) the two features are
supported in the same fabric only over Cisco Nexus N9000K switches without “EX” on the end of the
switch name; for example, N9K-9312TX. Since the 2.1(1) release, the two features can be deployed
together over all the switches used in the multipod and EVPN topologies.
• When configuring GOLF on a spine switch, wait for the control plane to converge before configuring
GOLF on another spine switch.
• A spine switch can be added to multiple provider GOLF outside networks (GOLF L3Outs), but the
provider labels have to be different for each GOLF L3Out. Also, in this case, the OSPF Area has to be
different on each of the L3extOuts and use different loopback addresses.
• The BGP EVPN session in the matching provider L3Out in the infra tenant advertises the tenant routes
defined in this L3extOut.
• When deploying three GOLF Outs, if only 1 has a provider/consumer label for GOLF, and 0/0 export
aggregation, APIC will export all routes. This is the same as existing L3extOut on leaf switches for
tenants.
• If there is direct peering between a spine switch and a data center interconnect (DCI) router, the transit
routes from leaf switches to the ASR have the next hop as the PTEP of the leaf switch. In this case, define
a static route on the ASR for the TEP range of that ACI pod. Also, if the DCI is dual-homed to the same
pod, then the precedence (administrative distance) of the static route should be the same as the route
received through the other link.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

295
 Configuring Layer 3 External Connectivity
Configuration Tasks to Configure Cisco ACI GOLF Services Using the NX-OS Style CLI

• The default bgpPeerPfxPol policy restricts routes to 20, 000. For ACI WAN Interconnect peers, increase
this as needed.
• In a deployment scenario where there are two L3extOuts on one spine switch, and one of them has the
provider label prov1 and peers with the DCI 1, the second L3extOut peers with DCI 2 with provider
label prov2. If the tenant VRF has a consumer label pointing to any 1 of the provider labels (either prov1
or prov2), the tenant route will be sent out both DCI 1 and DCI 2.
• When aggregating GOLF OpFlex VRFs, the leaking of routes cannot occur in the ACI fabric or on the
GOLF device between the GOLF OpFlex VRF and any other VRF in the system. An external device
(not the GOLF router) must be used for the VRF leaking.

Note Cisco ACI does not support IP fragmentation. Therefore, when you configure Layer 3 Outside (L3Out)
connections to external routers, or multipod connections through an Inter-Pod Network (IPN), it is critical
that the interface MTU is set appropriately on both ends of a link. On some platforms, such as Cisco ACI,
Cisco NX-OS, and Cisco IOS, the configurable MTU value does not take into account the Ethernet headers
(matching IP MTU, and excluding the 14-18 Ethernet header size), while other platforms, such as IOS-XR,
include the Ethernet header in the configured MTU value. A configured value of 9000 results in a max IP
packet size of 9000 bytes in Cisco ACI, Cisco NX-OS, and Cisco IOS, but results in a maximum IP packet
size of 8986 bytes for an IOS-XR untagged interface.
For the appropriate MTU values for each platform, see the relevant configuration guides.
We highly recommend that you test the MTU using CLI-based commands. For example, on the Cisco NX-OS
CLI, use a command such as ping 1.1.1.1 df-bit packet-size 9000 source-interface ethernet 1/1.

Configuration Tasks to Configure Cisco ACI GOLF Services Using the NX-OS
Style CLI
Perform the following tasks to configure GOLF services (using the BGP EVPN protocol), with the NX-OS
style CLI:
• Configure the infra tenant for BGP EVPN, including the VLAN domain, VRF, Interface IP addressing,
and OSPF.
• Configure BGP on the spine node to support BGP EVPN.
• Configure a tenant for BGP EVPN.
• Configure the BGP EVPN route target, route map, and prefix-epg for the tenant.
• Configure BGP address-families to enable distributing BGP EVPN type-2 (MAC-IP) host routes to the
DCIG, with the host-rt-enable command .

Configuring a Spine and the Infra Tenant for BGP EVPN, Using the NX-OS Style
CLI
This task describes how to configure the infra tenant for BGP EVPN, including the VLAN domain, VRF,
Interface IP addressing, and OSPF in the following steps:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

296
Configuring Layer 3 External Connectivity
Configuring a Spine and the Infra Tenant for BGP EVPN, Using the NX-OS Style CLI

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
apic1# configure

Step 2 vlan-domain vlan-domain-name dynamic Creates a VLAN domain.

apic1(config)# vlan-domain evpn-dom
dynamic

Step 3 spine spine-name Creates the spine or enters spine configuration

mode.
apic1(config)# spine 111

Step 4 vrf context tenant tenant-name vrf vrf-name Associates the VRF with the tenant.
apic1(config-spine)# vrf context tenant
infra vrf overlay-1

Step 5 router-id A.B.C.D Configures the router ID for the VRF.

apic1(config-spine-vrf)# router-id
10.10.3.3

Step 6 exit Returns to spine configuration mode.

apic1(config-spine-vrf)# exit

Step 7 interface ethernet slot/port Configures an interface for a spine node.

apic1(config-spine)# interface ethernet
1/33

Step 8 vlan-domain member vlan-domain-name Associates the interface with the VLAN
domain.
apic1(config-spine-if)# vlan-domain
member evpn-dom

Step 9 exit Returns to spine configuration mode.

apic1(config-spine-if)# exit

Step 10 interface ethernet sub-interface-id Creates a sub-interface.

apic1(config-spine)# interface ethernet
1/33.4

Step 11 vrf member tenant tenant-name vrf vrf-name Associates the interface with the overlay-1
VRF and the infra tenant.
apic1(config-spine-if)# vrf member
tenant infra vrf overlay-1

Step 12 mtu mtu-value Sets the maximum transmission unit (MTU)

for the interface.
apic1(config-spine-if)# mtu 1500

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

297
 Configuring Layer 3 External Connectivity
Configuring a Spine and the Infra Tenant for BGP EVPN, Using the NX-OS Style CLI

Command or Action Purpose

Step 13 ip address A.B.C.D/LEN Sets the IP address for the interface.
apic1(config-spine-if)# ip address
5.0.0.1/24

Step 14 ip router ospf default area ospf-area-id Sets the default OSPF area ID for the interface.
apic1(config-spine-if)# ip router ospf
default area 0.0.0.150

Step 15 exit Returns to spine configuration mode.

apic1(config-spine-if)# exit

Step 16 interface ethernet slot/port Configures an interface for a spine node.

apic1(config-spine)# interface ethernet
1/34

Step 17 vlan-domain member vlan-domain-name Associates the interface with the VLAN
domain.
apic1(config-spine-if)# vlan-domain
member evpn-dom

Step 18 exit Returns to spine configuration mode.

apic1(config-spine-if)# exit

Step 19 interface ethernet sub-interface-id Creates a sub-interface.

apic1(config-spine)# interface ethernet
1/34.4

Step 20 vrf member tenant tenant-name vrf vrf-name Associates the interface with the overlay-1
VRF and the infra tenant.
apic1(config-spine-if)# vrf member
tenant infra vrf overlay-1

Step 21 mtu mtu-value Sets the maximum transmission unit (MTU)

for the interface.
apic1(config-spine-if)# mtu 1500

Step 22 ip address A.B.C.D/LEN Sets the IP address for the interface.

apic1(config-spine-if)# ip address
2.0.0.1/24

Step 23 ip router ospf default area ospf-area-id Sets the default OSPF area ID for the interface.
apic1(config-spine-if)# ip router ospf
default area 0.0.0.200

Step 24 exit Returns to spine configuration mode.

apic1(config-spine-if)# exit

Step 25 router ospf default Configures OSPF for the spine.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

298
 Configuring Layer 3 External Connectivity
APIC GOLF Connections Shared by Multi-Site Sites

Command or Action Purpose

apic1(config-spine)# router ospf default

Step 26 vrf member tenant tenant-name vrf vrf-name Associates the Router OSPF policy with the
overlay-1 VRF and infra tenant.
apic1(config-spine-ospf)# vrf member
tenant infra vrf overlay-1

Step 27 area area-id loopback loopback-ip-address Configure an OSPF area for the OSPF policy.
apic1(config-spine-ospf-vrf)# area
0.0.0.150 loopback 10.10.5.3

Step 28 area area-id loopback loopback-ip-address Configure another OSPF area for the OSPF
policy.
apic1(config-spine-ospf-vrf)# area
0.0.0.200 loopback 10.10.4.3

Step 29 exit Returns to spine OSPF configuration mode.

apic1(config-spine-ospf-vrf)# exit

Step 30 exit Returns to spine configuration mode.

apic1(config-spine-ospf)# exit

APIC GOLF Connections Shared by Multi-Site Sites

For APIC Sites in a Multi-Site topology, if stretched VRFs share GOLF connections, follow these guidelines
to avoid the risk of cross-VRF traffic issues.

Route Target Configuration between the Spine Switches and the DCI
There are two ways to configure EVPN route targets (RTs) for the GOLF VRFs: Manual RT and Auto RT.
The route target is synchronized between ACI spines and DCIs through OpFlex. Auto RT for GOLF VRFs
has the Fabric ID embedded in the format: – ASN: [FabricID] VNID
If two sites have VRFs deployed as in the following diagram, traffic between the VRFs can be mixed.

Site 1 Site 2

ASN: 100, Fabric ID: 1 ASN: 100, Fabric ID: 1

VRF A: VNID 1000 VRF A: VNID 2000

Import/Export Route Target: 100: [1] 1000 Import/Export Route Target: 100: [1] 2000

VRF B: VNID 2000 VRF B: VNID 1000

Import/Export Route Target: 100: [1] 2000 Import/Export Route Target: 100: [1] 1000

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

299
 Configuring Layer 3 External Connectivity
Recommended Shared GOLF Configuration Using the NX-OS Style CLI

Route Maps Required on the DCI

Since tunnels are not created across sites when transit routes are leaked through the DCI, the churn in the
control plane must be reduced as well. EVPN type-5 and type-2 routes sent from GOLF spine in one site
towards the DCI should not be sent to GOLF spine in another site. This can happen when the DCI to spine
switches have the following types of BGP sessions:
Site1 — IBGP ---- DCI ---- EBGP ---- Site2
Site1 — EBGP ---- DCI ---- IBGP ---- Site2
Site1 — EBGP ---- DCI ---- EBGP ---- Site2
Site1 — IBGP RR client ---- DCI (RR)---- IBGP ---- Site2
To avoid this happening on the DCI, route maps are used with different BGP communities on the inbound
and outbound peer policies.
When routes are received from the GOLF spine at one site, the outbound peer policy towards the GOLF spine
at another site filters the routes based on the community in the inbound peer policy. A different outbound peer
policy strips off the community towards the WAN. All the route-maps are at peer level.

Recommended Shared GOLF Configuration Using the NX-OS Style CLI

Use the following steps to configure route maps and BGP to avoid cross-VRF traffic issues when sharing
GOLF connections with a DCI between multiple APIC sites that are managed by Multi-Site.

Procedure

Step 1 Configure the inbound route map

Example:

Inbound peer policy to attach community:

route-map multi-site-in permit 10

set community 1:1 additive

Step 2 Configure the outbound peer policy to filter routes based on the community in the inbound peer policy.
Example:
ip community-list standard test-com permit 1:1

route-map multi-site-out deny 10

match community test-com exact-match

route-map multi-site-out permit 11

Step 3 Configure the outbound peer policy to filter the community towards the WAN.
Example:
ip community-list standard test-com permit 1:1

route-map multi-site-wan-out permit 11

set comm-list test-com delete

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

300
 Configuring Layer 3 External Connectivity
Configuring BGP to Support BGP EVPN on a Spine, Using the NX-OS Style CLI

Step 4 Configure BGP.

Example:
router bgp 1

address-family l2vpn evpn

neighbor 11.11.11.11 remote-as 1

update-source loopback0

address-family l2vpn evpn

send-community both

route-map multi-site-in in

neighbor 13.0.0.2 remote-as 2

address-family l2vpn evpn

send-community both

route-map multi-site-out out

Configuring BGP to Support BGP EVPN on a Spine, Using the NX-OS Style CLI
This task shows how to configure BGP on the spine to support BGP EVPN in the following steps:

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
apic1# configure

Step 2 spine spine-name Creates the spine or enters spine configuration

mode.
apic1(config)# spine 111

Step 3 router bgp AS-number Configures BGP for the spine node.
apic1(config-spine)# router bgp 100

Step 4 vrf context tenant tenant-name vrf vrf-name Associates the Router BGP policy with the
infra tenant and the overlay-1 VRF.
apic1(config-spine-bgp)# vrf context
tenant infra vrf overlay-1

Step 5 vrf context tenant tenant-name vrf vrf-name Associates the Router BGP policy with the
infra tenant and the overlay-1 VRF.
apic1(config-spine-bgp-vrf)# vrf context
tenant infra vrf overlay-1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

301
 Configuring Layer 3 External Connectivity
Configuring a Tenant for BGP EVPN Using the NX-OS Style CLI

Command or Action Purpose

Step 6 neighbor neighbor-ip-address evpn Configures the IP address for an EVPN BGP
neighbor.
apic1(config-spine-bgp-vrf)# neighbor
10.10.4.1 evpn

Step 7 label label-name Assigns a label to the neighbor.

apic1(config-spine-bgp-vrf-neighbor)#
label evpn-aci

Step 8 update-source loopback loopback-ip-address Sets the update source to be the neighbor
vrf vrf-name loopback IP address.
apic1(config-spine-bgp-vrf-neighbor)#
update-source loopback 10.10.4.3

Step 9 remote-as AS-number Specifies the autonomous system (AS) number

of the neighbor. The valid value can be from
1 to 4294967295.
apic1(config-spine-bgp-vrf-neighbor)#
remote-as 100

Step 10 exit Returns to BGP VRF configuration mode.

apic1(config-spine-bgp-vrf-neighbor)#
exit

Step 11 neighbor neighbor-ip-address evpn Configures the IP address for an EVPN BGP
neighbor.
apic1(config-spine-bgp-vrf)# neighbor
10.10.5.1 evpn

Step 12 label label-name Assigns a label to the neighbor.

apic1(config-spine-bgp-vrf-neighbor)#
label evpn-aci2

Step 13 update-source loopback loopback-ip-address Sets the update source to be the neighbor
vrf vrf-name loopback IP address.
apic1(config-spine-bgp-vrf-neighbor)#
update-source loopback 10.10.5.3

Step 14 remote-as AS-number Specifies the autonomous system (AS) number

of the neighbor. The valid value can be from
1 to 4294967295.
apic1(config-spine-bgp-vrf-neighbor)#
remote-as 100

Configuring a Tenant for BGP EVPN Using the NX-OS Style CLI
This task shows how to configure a tenant for BGP EVPN in the following steps:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

302
Configuring Layer 3 External Connectivity
Configuring a Tenant for BGP EVPN Using the NX-OS Style CLI

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
apic1# configure

Step 2 tenant tenant-name Creates the tenant or enters tenant

configuration mode.
apic1(config)# tenant sky

Step 3 vrf context vrf-name Creates a VRF for the tenant.

apic1(config-tenant)# vrf context
vrf-sky

Step 4 exit Returns to tenant configuration mode.

apic1(config-tenant-vrf)# exit

Step 5 bridge-domain bd-name Creates a bridge domain

apic1(config-tenant)# bridge-domain
bd-sky

Step 6 vrf member vrf-name Associates the bridge domain with the VRF
and tenant.
apic1(config-tenant-bd)# vrf member
vrf-sky

Step 7 exit Returns to tenant configuration mode.

apic1(config-tenant-bd)# exit

Step 8 interface bridge-domain bd-name Creates an interface for a bridge domain.

apic1(config-tenant)# interface
bridge-domain bd_sky

Step 9 ip address A.B.C.D/LEN Assigns an IP address and length to the

bridge-domain interface.
apic1(config-tenant-interface)# ip
address 59.10.1.1/24

Step 10 exit Returns to tenant configuration mode.

apic1(config-tenant-interface)# exit

Step 11 bridge-domain bd-name Creates a bridge domain

apic1(config-tenant)# bridge-domain
bd-sky2

Step 12 vrf member vrf-name Associates the bridge domain with the VRF
and tenant.
apic1(config-tenant-bd)# vrf member
vrf-sky

Step 13 exit Returns to tenant configuration mode.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

303
 Configuring Layer 3 External Connectivity
Configuring a Route Map

Command or Action Purpose

apic1(config-tenant-bd)# exit

Step 14 interface bridge-domain bd-name Creates an interface for a bridge domain.

apic1(config-tenant)# interface
bridge-domain bd_sky2

Step 15 ip address A.B.C.D/LEN Assigns an IP address and length to the

bridge-domain interface.
apic1(config-tenant-interface)# ip
address 59.11.1.1/24

Step 16 exit Returns to tenant configuration mode.

apic1(config-tenant-interface)# exit

Configuring a Route Map

This task shows how to configure a route map to advertise bridge-domain subnets through BGP EVPN. Each
bridge domain is advertised through a different BGP EVPN session on the spine, with a unique provider label.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
apic1# configure

Step 2 spine spine-name Creates a spine or enters spine configuration

mode.
apic1(config)# spine 111

Step 3 vrf context tenant tenant-name vrf vrf-name Enters creates a VRF or enters VRF
configuration mode.
apic1(config-spine)# vrf context tenant
sky vrf vrf_sky

Step 4 address-family { ipv4 | ipv6 } unicast Sets IPv4 or IPv6 unicast address family for
the VRF.
apic1(config-spine-vrf)# address-family
ipv4 unicast

Step 5 route-target mode Assigns an export route target to the address

extended-community-number family.
apic1(config-spine-vrf-af)# route-target
export 100:1

Step 6 route-target mode Assigns an import route target to the address

extended-community-number family.
apic1(config-spine-vrf-af)# route-target
import 100:1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

304
Configuring Layer 3 External Connectivity
Configuring a Route Map

Command or Action Purpose

Step 7 exit Returns to spine VRF configuration mode.
apic1(config-spine-vrf-af)# exit

Step 8 route-map route-map-name Creates a route map for EVPN (with prefix
learned from a transit network).
apic1(config-spine-vrf)# route map rmap

Step 9 ip prefix-list ip-pl-name permit A.B.C.D/LEN Adds an IP prefix list to the route map to
permit traffic from the specified subnet.
apic1(config-spine-vrf-route-map)# ip
prefix-list pl permit 11.10.10.0/24

Step 10 match bridge-domain bd-name Configures the route-map to match traffic

belonging to the bridge domain.
apic1(config-spine-vrf-route-map)# match
bridge-domain bd_sky

Step 11 exit Returns to spine VRF route-map configuration

mode.
apic1(config-spine-vrf-route-map-match)#
exit

Step 12 match prefix-list pl-name Sets the route-map to match the specified
prefix-list.
apic1(config-spine-vrf-route-map)# match
prefix-list pl

Step 13 exit Returns to spine VRF route-map configuration

mode.
apic1(config-spine-vrf-route-map-match)#
exit

Step 14 exit Returns to spine VRF configuration mode.

apic1(config-spine-vrf-route-map)# exit

Step 15 evpn export map route-map-name label Assigns a consumer label to the VRF.
consumer-label-name apic1(config-spine-vrf)# evpn export
map rmap label evpn-aci

Step 16 route-map route-map-name Creates a route map for EVPN (with prefix
learned from a transit network).
apic1(config-spine-vrf)# route map rmap2

Step 17 match bridge-domain bd-name Configures the route-map to match traffic

belonging to the bridge domain.
apic1(config-spine-vrf-route-map)# match
bridge-domain bd_sky

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

305
 Configuring Layer 3 External Connectivity
Enabling Distributing BGP EVPN Type-2 Host Routes to a DCIG Using the NX-OS Style CLI

Command or Action Purpose

Step 18 exit Returns to spine VRF route-map configuration
mode.
apic1(config-spine-vrf-route-map-match)#
exit

Step 19 match prefix-list pl-name Sets the route-map to match the specified
prefix-list.
apic1(config-spine-vrf-route-map)# match
prefix-list pl

Step 20 exit Returns to spine VRF route-map configuration

mode.
apic1(config-spine-vrf-route-map-match)#
exit

Step 21 exit Returns to spine VRF configuration mode.

apic1(config-spine-vrf-route-map)# exit

Step 22 evpn export map route-map-name label Assigns a consumer label to the VRF.
consumer-label-name apic1(config-spine-vrf)# evpn export
map rmap label evpn-aci2

Step 23 external-l3 epg epg-name apic1(config-spine-vrf)# external-l3

epg l3_sky

Step 24 vrf member vrf-name apic1(config-spine-vrf-l3ext-epg)# vrf

member vrf_sky

Step 25 match ip A.B.C.D/LEN Configure the subnet that identifies hosts as

being part of the EPG.
apic1(config-spine-vrf-l3ext-epg)# match
ip 80.10.1.0/24

Enabling Distributing BGP EVPN Type-2 Host Routes to a DCIG Using the NX-OS
Style CLI
Procedure

Command or Action Purpose

Step 1 Configure distributing EVPN type-2 host routes This template will be available on all nodes
to a DCIG with the following commands in the where tenant bgp_t1 has a VRF deployment.
BGP address family configuration mode. To disable distributing EVPN type-2 host
routes, enter the no host-rt-enable command.
Example:
apic1(config)# leaf
101
template bgp
apic1(config-leaf)#
address-family bgpAf1 tenant

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

306
 Configuring Layer 3 External Connectivity
Cisco ACI GOLF Configuration Example, Using the NX-OS Style CLI

Command or Action Purpose

bgp_t1
apic1(config-bgp-af)# distance 250
240 230
apic1(config-bgp-af)# host-rt-enable

apic1(config-bgp-af)# exit

Cisco ACI GOLF Configuration Example, Using the NX-OS Style CLI
These examples show the CLI commands to configure GOLF Services, which uses the BGP EVPN protocol
over OSPF for WAN routers that are connected to spine switches.

Configuring the infra Tenant for BGP EVPN

The following example shows how to configure the infra tenant for BGP EVPN, including the VLAN domain,
VRF, Interface IP addressing, and OSPF:

configure
vlan-domain evpn-dom dynamic
exit
spine 111
# Configure Tenant Infra VRF overlay-1 on the spine.
vrf context tenant infra vrf overlay-1
router-id 10.10.3.3
exit

interface ethernet 1/33

vlan-domain member golf_dom
exit
interface ethernet 1/33.4
vrf member tenant infra vrf overlay-1
mtu 1500
ip address 5.0.0.1/24
ip router ospf default area 0.0.0.150
exit
interface ethernet 1/34
vlan-domain member golf_dom
exit
interface ethernet 1/34.4
vrf member tenant infra vrf overlay-1
mtu 1500
ip address 2.0.0.1/24
ip router ospf default area 0.0.0.200
exit

router ospf default

vrf member tenant infra vrf overlay-1
area 0.0.0.150 loopback 10.10.5.3
area 0.0.0.200 loopback 10.10.4.3
exit
exit

Configuring BGP on the Spine Node

The following example shows how to configure BGP to support BGP EVPN:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

307
 Configuring Layer 3 External Connectivity
Cisco ACI GOLF Configuration Example, Using the NX-OS Style CLI

Configure
spine 111
router bgp 100
vrf member tenant infra vrf overlay- 1
neighbor 10.10.4.1 evpn
label golf_aci
update-source loopback 10.10.4.3
remote-as 100
exit
neighbor 10.10.5.1 evpn
label golf_aci2
update-source loopback 10.10.5.3
remote-as 100
exit
exit
exit

Configuring a Tenant for BGP EVPN

The following example shows how to configure a tenant for BGP EVPN, including a gateway subnet which
will be advertised through a BGP EVPN session:

configure
tenant sky
vrf context vrf_sky
exit
bridge-domain bd_sky
vrf member vrf_sky
exit
interface bridge-domain bd_sky
ip address 59.10.1.1/24
exit
bridge-domain bd_sky2
vrf member vrf_sky
exit
interface bridge-domain bd_sky2
ip address 59.11.1.1/24
exit
exit

Configuring the BGP EVPN Route Target, Route Map, and Prefix EPG for the Tenant
The following example shows how to configure a route map to advertise bridge-domain subnets through BGP
EVPN.

configure
spine 111
vrf context tenant sky vrf vrf_sky
address-family ipv4 unicast
route-target export 100:1
route-target import 100:1
exit

route-map rmap
ip prefix-list p1 permit 11.10.10.0/24
match bridge-domain bd_sky
exit
match prefix-list p1
exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

308
 Configuring Layer 3 External Connectivity
Troubleshooting EVPN Type-2 Route Distribution to a DCIG

evpn export map rmap label golf_aci

route-map rmap2
match bridge-domain bd_sky
exit
match prefix-list p1
exit
exit

evpn export map rmap label golf_aci2

external-l3 epg l3_sky

vrf member vrf_sky
match ip 80.10.1.0/24
exit

Troubleshooting EVPN Type-2 Route Distribution to a DCIG

For optimal traffic forwarding in an EVPN topology, you can enable fabric spines to distribute host routes to
a Data Center Interconnect Gateway (DCIG) using EVPN type-2 (MAC-IP) routes along with the public BD
subnets in the form of BGP EVPN type-5 (IP Prefix) routes. This is enabled using the HostLeak object. If
you encounter problems with route distribution, use the steps in this topic to troubleshoot.

Procedure

Step 1 Verify that HostLeak object is enabled under the VRF-AF in question, by entering a command such as the
following in the spine-switch CLI:
Example:
spine1# ls /mit/sys/bgp/inst/dom-apple/af-ipv4-ucast/
ctrl-l2vpn-evpn ctrl-vpnv4-ucast hostleak summary

Step 2 Verify that the config-MO has been successfully processed by BGP, by entering a command such as the
following in the spine-switch CLI:
Example:
spine1# show bgp process vrf apple
Look for output similar to the following:
Information for address family IPv4 Unicast in VRF apple
Table Id : 0
Table state : UP
Table refcount : 3
Peers Active-peers Routes Paths Networks Aggregates
0 0 0 0 0 0

Redistribution
None

Wait for IGP convergence is not configured

GOLF EVPN MAC-IP route is enabled
EVPN network next-hop 192.41.1.1
EVPN network route-map map_pfxleakctrl_v4
Import route-map rtctrlmap-apple-v4
EVPN import route-map rtctrlmap-evpn-apple-v4

Step 3 Verify that the public BD-subnet has been advertised to DCIG as an EVPN type-5 route:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

309
 Configuring Layer 3 External Connectivity
Troubleshooting EVPN Type-2 Route Distribution to a DCIG

Example:
spine1# show bgp l2vpn evpn 10.6.0.0 vrf overlay-1
Route Distinguisher: 192.41.1.5:4123 (L3VNI 2097154)
BGP routing table entry for [5]:[0]:[0]:[16]:[10.6.0.0]:[0.0.0.0]/224, version 2088
Paths: (1 available, best #1)
Flags: (0x000002 00000000) on xmit-list, is not in rib/evpn
Multipath: eBGP iBGP

Advertised path-id 1
Path type: local 0x4000008c 0x0 ref 1, path is valid, is best path
AS-Path: NONE, path locally originated
192.41.1.1 (metric 0) from 0.0.0.0 (192.41.1.5)
Origin IGP, MED not set, localpref 100, weight 32768
Received label 2097154
Community: 1234:444
Extcommunity:
RT:1234:5101
4BYTEAS-GENERIC:T:1234:444

Path-id 1 advertised to peers:

50.41.50.1

In the Path type entry, ref 1 indicates that one route was sent.

Step 4 Verify whether the host route advertised to the EVPN peer was an EVPN type-2 MAC-IP route:
Example:
spine1# show bgp l2vpn evpn 10.6.41.1 vrf overlay-1
Route Distinguisher: 10.10.41.2:100 (L2VNI 100)
BGP routing table entry for [2]:[0]:[2097154]:[48]:[0200.0000.0002]:[32]:[10.6.41
.1]/272, version 1146
Shared RD: 192.41.1.5:4123 (L3VNI 2097154)
Paths: (1 available, best #1)
Flags: (0x00010a 00000000) on xmit-list, is not in rib/evpn
Multipath: eBGP iBGP

Advertised path-id 1
Path type: local 0x4000008c 0x0 ref 0, path is valid, is best path
AS-Path: NONE, path locally originated
EVPN network: [5]:[0]:[0]:[16]:[10.6.0.0]:[0.0.0.0] (VRF apple)
10.10.41.2 (metric 0) from 0.0.0.0 (192.41.1.5)
Origin IGP, MED not set, localpref 100, weight 32768
Received label 2097154 2097154
Extcommunity:
RT:1234:16777216

Path-id 1 advertised to peers:

50.41.50.1

The Shared RD line indicates the RD/VNI shared by the EVPN type-2 route and the BD subnet.
The EVPN Network line shows the EVPN type-5 route of the BD-Subnet.
The Path-id advertised to peers indicates the path advertised to EVPN peers.

Step 5 Verify that the EVPN peer (a DCIG) received the correct type-2 MAC-IP route and the host route was
successfully imported into the given VRF, by entering a command such as the following on the DCIG device
(assuming that the DCIG is a Cisco ASR 9000 switch in the example below):
Example:
RP/0/RSP0/CPU0:asr9k#show bgp vrf apple-2887482362-8-1 10.6.41.1
Tue Sep 6 23:38:50.034 UTC

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

310
 Configuring Layer 3 External Connectivity
Multipod_Fabric

BGP routing table entry for 10.6.41.1/32, Route Distinguisher: 44.55.66.77:51

Versions:
Process bRIB/RIB SendTblVer
Speaker 2088 2088
Last Modified: Feb 21 08:30:36.850 for 28w2d
Paths: (1 available, best #1)
Not advertised to any peer
Path #1: Received by speaker 0
Not advertised to any peer
Local
192.41.1.1 (metric 42) from 10.10.41.1 (192.41.1.5)
Received Label 2097154
Origin IGP, localpref 100, valid, internal, best, group-best, import-candidate,
imported
Received Path ID 0, Local Path ID 1, version 2088
Community: 1234:444
Extended community: 0x0204:1234:444 Encapsulation Type:8 Router
MAC:0200.c029.0101 RT:1234:5101
RIB RNH: table_id 0xe0000190, Encap 8, VNI 2097154, MAC Address: 0200.c029.0101,
IP Address: 192.41.1.1, IP table_id 0x00000000
Source AFI: L2VPN EVPN, Source VRF: default,
Source Route Distinguisher: 192.41.1.5:4123

In this output, the received RD, next hop, and attributes are the same for the type-2 route and the BD subnet.

Multipod_Fabric
About Multipod Fabric
Multipod enables provisioning a more fault tolerant fabric comprised of multiple pods with isolated control
plane protocols. Also, multipod provides more flexibility with regard to the full mesh cabling between leaf
and spine switches. For example, if leaf switches are spread across different floors or different buildings,
multipod enables provisioning multiple pods per floor or building and providing connectivity between pods
through spine switches.
Multipod uses MP-BGP EVPN as the control-plane communication protocol between the ACI spines in
different Pods.
WAN routers can be provisioned in the IPN, directly connected to spine switches or connected to border leaf
switches. Multipod uses a single APIC cluster for all the pods; all the pods act as a single fabric. Individual
APIC controllers are placed across the pods but they are all part of a single APIC cluster.

Assigning Switches in a Multipod Fabric

Before you begin
The node group and L3Out policies have already been created.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

311
 Configuring Layer 3 External Connectivity
Configuring Fabric-External Connectivity for a Multipod Fabric

Procedure

Command or Action Purpose

Step 1 configure Enter global configuration mode.
Example:
apic1# configure

Step 2 [no] system switch-id serial-number switch-id For each switch in the multipod fabric, declare
switch-name [pod pod-id] [role {leaf | spine}] the associated pod and the role (leaf or spine)
of the switch. Repeat this command for each
Example:
leaf and spine switch in the multipod fabric.
apic1(config)# system switch-id
SAL1748H56D 201 ifav4-spine1 pod 1 role
spine

Step 3 [no] system pod pod-id tep-pool Configure a tunnel endpoint IP address pool for
ip-prefix/length a pod. Repeat this command for each pod in the
multipod fabric.
Example:
apic1(config)# system pod 1 tep-pool
10.0.0.0/16

Example
This example shows how to assign spine and leaf switches in a two-pod fabric.

apic1# configure
apic1(config)# system switch-id SAL1748H56D 201 ifav4-spine1 pod 1 role spine
apic1(config)# system switch-id SAL1938P7A6 202 ifav4-spine3 pod 1 role spine
apic1(config)# system switch-id SAL1819RXP4 101 ifav4-leaf1 pod 1 role leaf
apic1(config)# system switch-id SAL1803L25H 102 ifav4-leaf2 pod 1 role leaf
apic1(config)# system switch-id SAL1934MNY0 103 ifav4-leaf3 pod 1 role leaf
apic1(config)# system switch-id SAL1934MNY3 104 ifav4-leaf4 pod 1 role leaf
apic1(config)# system switch-id SAL1931LA3B 203 ifav4-spine2 pod 2 role spine
apic1(config)# system switch-id FGE173400A9 204 ifav4-spine4 pod 2 role spine
apic1(config)# system switch-id SAL1938PHBB 105 ifav4-leaf5 pod 2 role leaf
apic1(config)# system switch-id SAL1942R857 106 ifav4-leaf6 pod 2 role leaf
apic1(config)# system pod 1 tep-pool 10.0.0.0/16
apic1(config)# system pod 2 tep-pool 10.1.0.0/16

What to do next
Configure fabric-external connectivity.

Configuring Fabric-External Connectivity for a Multipod Fabric

Before you begin
• The node group and L3Out policies have already been created.
• Switches have been assigned to pods.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

312
Configuring Layer 3 External Connectivity
Configuring Fabric-External Connectivity for a Multipod Fabric

Procedure

Command or Action Purpose

Step 1 configure Enter global configuration mode.
Example:
apic1# configure

Step 2 [no] fabric-external controller-number

Example:
apic1(config)# fabric-external 1

Step 3 [no] bgp evpn peering [password Configure BGP EVPN peering profile. You
peering-password] [type can configure a peering password, and you can
{automatic_with_full_mesh | set the type to be either full mesh or with
automatic_with_rr}] route-reflector.
Example:
apic1(config-fabric-external)# bgp evpn
peering

Step 4 [no] pod pod-id Select a pod for configuring.

Example:
apic1(config-fabric-external)# pod 1

Step 5 [no] interpod data hardware-proxy Configure the anycast hardware-proxy IP

ip-addr/mask address for each pod for inter-pod traffic.
Example:
apic1(config-fabric-external-pod)#
interpod data hardware-proxy
100.11.1.1/32

Step 6 [no] bgp evpn peering [password

peering-password] [type
{automatic_with_full_mesh |
automatic_with_rr}]
Example:
apic1(config-fabric-external-pod)# bgp
evpn peering

Step 7 exit Return to BGP EVPN peering profile

configuration.
Example:
apic1(config-fabric-external-pod)# exit

Step 8 Repeat steps 4 through 7 for each pod in the

multipod fabric.
Step 9 [no] route-map interpod-import Configure a route-map that contains subnets
on the inter-pod network (IPN) that will be
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

313
 Configuring Layer 3 External Connectivity
Configuring Fabric-External Connectivity for a Multipod Fabric

Command or Action Purpose

apic1(config-fabric-external)# route-map allowed into the fabric through the OSPF
interpod-import protocol.

Step 10 [no] ip prefix-list prefix-list-name [permit

ip-address/len
Example:
apic1(config-fabric-external-route-map)#
ip prefix-list default permit 0.0.0.0/0

Step 11 exit Return to fabric-external configuration mode.

Example:
apic1(config-fabric-external-route-map)#
exit

Step 12 [no] route-target extended ASN4:NN Route targets are carried as extended
community attributes. Enter the community
Example:
number in the AA4:NN2 format:
apic1(config-fabric-external)# 1-4294967295: 1-65535.
route-target extended 5:16

Step 13 exit

Example
This example shows how to configure fabric-external connectivity for a multipod fabric.

apic1# configure
apic1(config)# fabric-external 1
apic1(config-fabric-external)# bgp evpn peering
apic1(config-fabric-external)# pod 1
apic1(config-fabric-external-pod)# interpod data hardware-proxy 100.11.1.1/32
apic1(config-fabric-external-pod)# bgp evpn peering
apic1(config-fabric-external-pod)# exit
apic1(config-fabric-external)# pod 2
apic1(config-fabric-external-pod)# interpod data hardware-proxy 200.11.1.1/32
apic1(config-fabric-external-pod)# bgp evpn peering
apic1(config-fabric-external-pod)# exit
apic1(config-fabric-external)# route-map interpod-import
apic1(config-fabric-external-route-map)# ip prefix-list default permit 0.0.0.0/0
apic1(config-fabric-external-route-map)# exit
apic1(config-fabric-external)# route-target extended 5:16
apic1(config-fabric-external)# exit

What to do next
Configure spine interfaces and OSPF.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

314
 Configuring Layer 3 External Connectivity
Configuring Spine Interfaces and OSPF for a Multipod Fabric

Configuring Spine Interfaces and OSPF for a Multipod Fabric

Before you begin
• Switches have been assigned to pods.
• A VLAN domain must exist.

Procedure

Command or Action Purpose

Step 1 configure Enter global configuration mode.
Example:
apic1# configure

Step 2 spine spine-id You can specify the spine switch by an ID

number in the range of 101 to 4000 or by
Example:
name, such as 'spine1.'
apic1(config)# spine 104

Step 3 [no] vrf context tenant infra vrf vrf-name

Example:
apic1(config-spine)# vrf context tenant
infra vrf overlay-1

Step 4 [no] router-id A.B.C.D Configure a router identifier (ID).

Example:
apic1(config-spine-vrf)# router-id
201.201.201.201

Step 5 exit Return to spine configuration mode.

Example:
apic1(config-spine-vrf)# exit

Step 6 [no] interface ethernet slot/port

Example:
apic1(config-spine)# interface ethernet
1/1

Step 7 [no] vlan-domain member domain-name The VLAN domain must already exist, having
been created using the vlan-domain
Example:
domain-name command in the global
apic1(config-spine)# vlan-domain member configuration mode.
l3Dom

Step 8 exit Return to spine configuration mode.

Example:
apic1(config-spine-if)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

315
 Configuring Layer 3 External Connectivity
Configuring Spine Interfaces and OSPF for a Multipod Fabric

Command or Action Purpose

Step 9 [no] interface ethernet type/slot.subinterface Encapsulation for the subinterface must be 4.
Example:
apic1(config-spine)# interface ethernet
1/1.4

Step 10 [no] vrf member tenant infra vrf vrf-name Configure the interface as a member of the
tenant VRF.
Example:
apic1(config-spine-if)# vrf member
tenant infra vrf overlay-1

Step 11 [no] ip address ip-address

Example:
apic1(config-spine-if)# ip address
201.1.1.1/30

Step 12 [no] ip router ospf default area 0.0.0.0

Example:
apic1(config-spine-if)# ip router ospf
default area 0.0.0.0

Step 13 [no] ip ospf cost cost

Example:
apic1(config-spine-if)# ip ospf cost 1

Step 14 exit Return to spine configuration mode.

Example:
apic1(config-spine-if)# exit

Step 15 Repeat steps Step 6, on page 315 through

Step 14, on page 316 to add any additional
interfaces.
Step 16 [no] router ospf default
Example:
apic1(config-spine)# router ospf default

Step 17 [no] vrf member tenant infra vrf vrf-name

Example:
apic1(config-spine-ospf)# vrf member
tenant infra vrf overlay-1

Step 18 [no] area area loopback ip-address Advertise the loopback address through OSPF.
This address is used by BGP EVPN sessions
Example:
for peering.
apic1(config-spine-ospf-vrf)# area
0.0.0.0 loopback 201.201.201.201

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

316
Configuring Layer 3 External Connectivity
Configuring Spine Interfaces and OSPF for a Multipod Fabric

Command or Action Purpose

Step 19 [no] area area interpod peering Enable inter-pod peering on the OSPF area,
which will set up BGP EVPN sessions
Example:
automatically using the loopback address
apic1(config-spine-ospf-vrf)# area advertised by OSPF.
0.0.0.0 interpod peering

Step 20 exit Return to OSPF configuration mode.

Example:
apic1(config-spine-ospf-vrf)# exit

Step 21 exit Return to spine configuration mode.

Example:
apic1(config-spine-ospf)# exit

Step 22 exit Return to global configuration mode.

Example:
apic1(config-spine)# exit

Step 23 Repeat steps Step 2, on page 315 through

Step 22, on page 317 to configure additional
spine switches.

Example

apic1# configure

# CONFIGURE FIRST SPINE

apic1(config)# spine 201

apic1(config-spine)# vrf context tenant infra vrf overlay-1
apic1(config-spine-vrf)# router-id 201.201.201.201
apic1(config-spine-vrf)# exit

apic1(config-spine)# interface ethernet 1/1

apic1(config-spine-if)# vlan-domain member l3Dom
apic1(config-spine-if)# exit
apic1(config-spine)# interface ethernet 1/1.4
apic1(config-spine-if)# vrf member tenant infra vrf overlay-1
apic1(config-spine-if)# ip address 201.1.1.1/30
apic1(config-spine-if)# ip router ospf default area 0.0.0.0
apic1(config-spine-if)# ip ospf cost 1
apic1(config-spine-if)# exit

apic1(config-spine)# interface ethernet 1/2

apic1(config-spine-if)# vlan-domain member l3Dom
apic1(config-spine-if)# exit
apic1(config-spine)# interface ethernet 1/2.4
apic1(config-spine-if)# vrf member tenant infra vrf overlay-1
apic1(config-spine-if)# ip address 201.2.1.1/30
apic1(config-spine-if)# ip router ospf default area 0.0.0.0
apic1(config-spine-if)# ip ospf cost 1
apic1(config-spine-if)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

317
 Configuring Layer 3 External Connectivity
Remote Leaf Switches

apic1(config-spine)# router ospf default

apic1(config-spine-ospf)# vrf member tenant infra vrf overlay-1
apic1(config-spine-ospf-vrf)# area 0.0.0.0 loopback 201.201.201.201
apic1(config-spine-ospf-vrf)# area 0.0.0.0 interpod peering
apic1(config-spine-ospf-vrf)# exit
apic1(config-spine-ospf)# exit
apic1(config-spine)# exit

# CONFIGURE SECOND SPINE

apic1(config)# spine 202

apic1(config-spine)# vrf context tenant infra vrf overlay-1
apic1(config-spine-vrf)# router-id 202.202.202.202
apic1(config-spine-vrf)# exit

apic1(config-spine)# interface ethernet 1/2

apic1(config-spine-if)# vlan-domain member l3Dom
apic1(config-spine-if)# exit
apic1(config-spine)# interface ethernet 1/2.4
apic1(config-spine-if)# vrf member tenant infra vrf overlay-1
apic1(config-spine-if)# ip address 202.1.1.1/30
apic1(config-spine-if)# ip router ospf default area 0.0.0.0
apic1(config-spine-if)# ip ospf cost 1
apic1(config-spine-if)# exit

apic1(config-spine)# router ospf default

apic1(config-spine-ospf)# vrf member tenant infra vrf overlay-1
apic1(config-spine-ospf-vrf)# area 0.0.0.0 loopback 202.202.202.202
apic1(config-spine-ospf-vrf)# area 0.0.0.0 interpod peering
apic1(config-spine-ospf-vrf)# exit
apic1(config-spine-ospf)# exit
apic1(config-spine)# exit

# CONFIGURE ADDITIONAL SPINES

Remote Leaf Switches

About Remote Leaf Switches in the ACI Fabric
With an ACI fabric deployed, you can extend ACI services and APIC management to remote datacenters with
Cisco ACI leaf switches that have no local spine switch or APIC attached.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

318
 Configuring Layer 3 External Connectivity
Remote Leaf Switch Hardware Requirements

Figure 24: Remote Leaf Topology

The remote leaf switches are added to an existing pod in the fabric. All policies deployed in the main datacenter
are deployed in the remote switches, which behave like local leaf switches belonging to the pod. In this
topology, all unicast traffic is through VXLAN over Layer 3. Layer 2 Broadcast, Unknown Unicast, and
Multicast (BUM) messages are sent using Head End Replication (HER) tunnels without the use of Multicast.
All local traffic on the remote site is switched directly between endpoints, whether physical or virtual. Any
traffic that requires use of the spine switch proxy is forwarded to the main datacenter.
The APIC system discovers the remote leaf switches when they come up. From that time, they can be managed
through APIC, as part of the fabric.

Note • All inter-VRF traffic goes to the spine switch before being forwarded.
• Before decommissioning a remote leaf, you must first delete the vPC.

You can configure Remote Leaf in the APIC GUI, either with and without a wizard, or use the REST API or
the NX-OS style CLI.

Remote Leaf Switch Hardware Requirements

The following switches are supported for the Remote Leaf Switch feature.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

319
 Configuring Layer 3 External Connectivity
Restrictions and Limitations

Fabric Spine Switches

For the spine switch at the ACI Main Datacenter that is connected to the WAN router, the following spine
switches are supported:
• Fixed spine switches Cisco Nexus 9000 series N9K-C9364C and N9K-C9332C
• For modular spine switches, only Cisco Nexus 9000 series switches with names that end in EX, and later
(for example, N9K-X9732C- EX ) are supported.
• Older generation spine switches, such as the fixed spine switch N9K-C9336PQ or modular spine switches
with the N9K-X9736PQ linecard are supported in the Main Datacenter, but only next generation spine
switches are supported to connect to the WAN.

Remote Leaf Switches

• For the remote leaf switches, only Cisco Nexus 9000 series switches with names that end in EX, and
later (for example, N9K-C93180LC-EX) are supported.
• The remote leaf switches must be running a switch image of 13.1.x or later (aci-n9000-dk9.13.1.x.x.bin)
before they can be discovered. This may require manual upgrades on the leaf switches.

Restrictions and Limitations

Note In Cisco APIC Release 3.2(x), the following features are supported that were not previously:
• FEX devices connected to remote leaf switches
• Cisco AVS with VLAN and Cisco AVS with VXLAN
• Cisco ACI Virtual Edge with VLAN and ACI Virtual Edge with VXLAN
• The Cisco Nexus 9336C-FX2 switch is now supported for remote leaf switches

Stretching of L3out SVI between local leaf switches (ACI main data center switches) and remote leaf switches
is not supported.
The following deployments and configurations are not supported with the remote leaf switch feature:
• APIC controllers directly connected to remote leaf switches
• Orphan port-channel or physical ports on remote leaf switches, with a vPC domain (this restriction applies
for releases 3.1 and earlier)
• With and without service node integration, local traffic forwarding within a remote location is only
supported if the consumer, provider, and services nodes are all connected to Remote Leaf switches are
in vPC mode

Full fabric and tenant policies are supported on remote leaf switches, in this release, except for the following
features:
• ACI Multi-Site

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

320
 Configuring Layer 3 External Connectivity
WAN Router and Remote Leaf Switch Configuration Guidelines

• Layer 2 Outside Connections (except Static EPGs)

• 802.1Q Tunnels
• Copy services with vzAny contract
• FCoE connections on remote leaf switches
• Flood in encapsulation for bridge domains or EPGs
• Fast Link Failover policies
• Managed Service Graph-attached devices at remote locations
• Netflow
• PBR Tracking on remote leaf switches (with system-level global GIPo enabled)
• Q-in-Q Encapsulation Mapping for EPGs
• Traffic Storm Control
• Cloud Sec and MacSec Encryption
• First Hop Security
• PTP
• Layer 3 Multicast routing on remote leaf switches
• Openstack and Kubernetes VMM domains
• Maintenance mode
• Troubleshooting Wizard
• Transit L3Out across remote locations, which is when the main Cisco ACI datacenter pod is a transit
between two remote locations (the L3Out in RL location-1 and L3Out in RL location-2 are advertising
prefixes for each other)
• Traffic forwarding directly across two remote leaf vPC pairs in the same remote datacenter or across
datacenters

WAN Router and Remote Leaf Switch Configuration Guidelines

Before a remote leaf is discovered and incorporated in APIC management, you must configure the WAN
router and the remote leaf switches.
Configure the WAN routers that connect to the fabric spine switch external interfaces and the remote leaf
switch ports, with the following requirements:
WAN Routers
• Enable OSPF on the interfaces, with the same details, such as area ID, type, and cost.
• Configure DHCP Relay on the interface leading to each APIC's IP address in the main fabric.
• The interfaces on the WAN routers which connect to the VLAN-5 interfaces on the spine switches must
be on different VRFs than the interfaces connecting to a regular multipod network.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

321
 Configuring Layer 3 External Connectivity
Configure Remote Leaf Switches Using the NX-OS Style CLI

Remote Leaf Switches

• Connect the remote leaf switches to an upstream router by a direct connection from one of the fabric
ports. The following connections to the upstream router are supported:
• 40 Gbps & higher connections
• With a QSFP-to-SFP Adapter, supported 1G/10G SFPs

Bandwidth in the WAN must be a minimum of 100 Mbps and maximum supported latency is 300 msecs.
• It is recommended, but not required to connect the pair of remote leaf switches with a vPC. The switches
on both ends of the vPC must be remote leaf switches at the same remote datacenter.
• Configure the northbound interfaces as Layer 3 sub-interfaces on VLAN-4, with unique IP addresses.
If you connect more than one interface from the remote leaf switch to the router, configure each interface
with a unique IP address.
• Enable OSPF on the interfaces, but do not set the OSPF area type as stub area.
• The IP addresses in the remote leaf switch TEP Pool subnet must not overlap with the pod TEP subnet
pool. The subnet used must be /24 or lower.
• Multipod is supported, but not required, with the Remote Leaf feature.
• When connecting a pod in a single-pod fabric with remote leaf switches, configure an L3Out from a
spine switch to the WAN router and an L3Out from a remote leaf switch to the WAN router, both using
VLAN-4 on the switch interfaces.
• When connecting a pod in a multipod fabric with remote leaf switches, configure an L3Out from a spine
switch to the WAN router and an L3Out from a remote leaf switch to the WAN router, both using VLAN-4
on the switch interfaces. Also configure a multipod-internal L3Out using VLAN-5 to support traffic that
crosses pods destined to a remote leaf switch. The regular multipod and multipod-internal connections
can be configured on the same physical interfaces, as long as they use VLAN-4 and VLAN-5.
• When configuring the Multipod-internal L3Out, use the same router ID as for the regular multipod L3Out,
but deselect the Use Router ID as Loopback Address option for the router-id and configure a different
loopback IP address. This enables ECMP to function.

Configure Remote Leaf Switches Using the NX-OS Style CLI

This example configures a spine switch and a remote leaf switch to enable the leaf switch to communicate
with the main fabric pod.

Before you begin

• The IPN router and remote leaf switches are active and configured; see WAN Router and Remote Leaf
Switch Configuration Guidelines, on page 321.
• The remote leaf switches are running a switch image of 13.1.x or later (aci-n9000-dk9.13.1.x.x.bin).
• The pod in which you plan to add the remote leaf switches is created and configured.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

322
Configuring Layer 3 External Connectivity
Configure Remote Leaf Switches Using the NX-OS Style CLI

Procedure

Step 1 Define the TEP pool for a remote location 5, in pod 2.

The network mask must be /24 or lower.
Use the following new command: system remote-leaf-site site-id pod pod-id tep-pool ip-address-and-netmask
Example:
apic1(config)# system remote-leaf-site 5 pod 2 tep-pool 192.0.0.0/16

Step 2 Add a remote leaf switch to pod 2, remote-leaf-site 5.

Use the following command: system switch-id serial-number node-id leaf-switch-name pod pod-id
remote-leaf-site remote-leaf-site-id node-type remote-leaf-wan
Example:
apic1(config)# system switch-id FDO210805SKD 109 ifav4-leaf9 pod 2
remote-leaf-site 5 node-type remote-leaf-wan

Step 3 Configure a VLAN domain with a VLAN that includes VLAN 4.

Example:
apic1(config)# vlan-domain ospfDom
apic1(config-vlan)# vlan 4-5
apic1(config-vlan)# exit

Step 4 Configure two L3Outs for the infra tenant, one for the remote leaf connections and one for the multipod IPN.
Example:

apic1(config)# tenant infra

apic1(config-tenant)# l3out rl-wan
apic1(config-tenant-l3out)# vrf member overlay-1
apic1(config-tenant-l3out)# exit
apic1(config-tenant)# l3out ipn-multipodInternal
apic1(config-tenant-l3out)# vrf member overlay-1
apic1(config-tenant-l3out)# exit
apic1(config-tenant)# exit
apic1(config)#

Step 5 Configure the spine switch interfaces and sub-interfaces to be used by the L3Outs.
Example:

apic1(config)# spine 201

apic1(config-spine)# vrf context tenant infra vrf overlay-1 l3out rl-wan-test
apic1(config-spine-vrf)# exit
apic1(config-spine)# vrf context tenant infra vrf overlay-1 l3out ipn-multipodInternal
apic1(config-spine-vrf)# exit
apic1(config-spine)#
apic1(config-spine)# interface ethernet 8/36
apic1(config-spine-if)# vlan-domain member ospfDom
apic1(config-spine-if)# exit
apic1(config-spine)# router ospf default
apic1(config-spine-ospf)# vrf member tenant infra vrf overlay-1
apic1(config-spine-ospf-vrf)# area 5 l3out rl-wan-test
apic1(config-spine-ospf-vrf)# exit
apic1(config-spine-ospf)# exit
apic1(config-spine)#

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

323
 Configuring Layer 3 External Connectivity
Configure Remote Leaf Switches Using the NX-OS Style CLI

apic1(config-spine)# interface ethernet 8/36.4

apic1(config-spine-if)# vrf member tenant infra vrf overlay-1 l3out rl-wan-test
apic1(config-spine-if)# ip router ospf default area 5
apic1(config-spine-if)# exit
apic1(config-spine)# router ospf multipod-internal
apic1(config-spine-ospf)# vrf member tenant infra vrf overlay-1
apic1(config-spine-ospf-vrf)# area 5 l3out ipn-multipodInternal
apic1(config-spine-ospf-vrf)# exit
apic1(config-spine-ospf)# exit
apic1(config-spine)#
apic1(config-spine)# interface ethernet 8/36.5
apic1(config-spine-if)# vrf member tenant infra vrf overlay-1 l3out ipn-multipodInternal
apic1(config-spine-if)# ip router ospf multipod-internal area 5
apic1(config-spine-if)# exit
apic1(config-spine)# exit
apic1(config)#

Step 6 Configure the remote leaf switch interface and sub-interface used for communicating with the main fabric
pod.
Example:
(config)# leaf 101
apic1(config-leaf)# vrf context tenant infra vrf overlay-1 l3out rl-wan-test
apic1(config-leaf-vrf)# exit
apic1(config-leaf)#
apic1(config-leaf)# interface ethernet 1/49
apic1(config-leaf-if)# vlan-domain member ospfDom
apic1(config-leaf-if)# exit
apic1(config-leaf)# router ospf default
apic1(config-leaf-ospf)# vrf member tenant infra vrf overlay-1
apic1(config-leaf-ospf-vrf)# area 5 l3out rl-wan-test
apic1(config-leaf-ospf-vrf)# exit
apic1(config-leaf-ospf)# exit
apic1(config-leaf)#
apic1(config-leaf)# interface ethernet 1/49.4
apic1(config-leaf-if)# vrf member tenant infra vrf overlay-1 l3out rl-wan-test
apic1(config-leaf-if)# ip router ospf default area 5
apic1(config-leaf-if)# exit

Example
The following example provides a downloadable configuration:
apic1# configure
apic1(config)# system remote-leaf-site 5 pod 2 tep-pool 192.0.0.0/16
apic1(config)# system switch-id FDO210805SKD 109 ifav4-leaf9 pod 2
remote-leaf-site 5 node-type remote-leaf-wan
apic1(config)# vlan-domain ospfDom
apic1(config-vlan)# vlan 4-5
apic1(config-vlan)# exit
apic1(config)# tenant infra
apic1(config-tenant)# l3out rl-wan-test
apic1(config-tenant-l3out)# vrf member overlay-1
apic1(config-tenant-l3out)# exit
apic1(config-tenant)# l3out ipn-multipodInternal
apic1(config-tenant-l3out)# vrf member overlay-1
apic1(config-tenant-l3out)# exit
apic1(config-tenant)# exit
apic1(config)#
apic1(config)# spine 201

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

324
 Configuring Layer 3 External Connectivity
Transit Routing

apic1(config-spine)# vrf context tenant infra vrf overlay-1 l3out rl-wan-test

apic1(config-spine-vrf)# exit
apic1(config-spine)# vrf context tenant infra vrf overlay-1 l3out ipn-multipodInternal
apic1(config-spine-vrf)# exit
apic1(config-spine)#
apic1(config-spine)# interface ethernet 8/36
apic1(config-spine-if)# vlan-domain member ospfDom
apic1(config-spine-if)# exit
apic1(config-spine)# router ospf default
apic1(config-spine-ospf)# vrf member tenant infra vrf overlay-1
apic1(config-spine-ospf-vrf)# area 5 l3out rl-wan-test
apic1(config-spine-ospf-vrf)# exit
apic1(config-spine-ospf)# exit
apic1(config-spine)#
apic1(config-spine)# interface ethernet 8/36.4
apic1(config-spine-if)# vrf member tenant infra vrf overlay-1 l3out rl-wan-test
apic1(config-spine-if)# ip router ospf default area 5
apic1(config-spine-if)# exit
apic1(config-spine)# router ospf multipod-internal
apic1(config-spine-ospf)# vrf member tenant infra vrf overlay-1
apic1(config-spine-ospf-vrf)# area 5 l3out ipn-multipodInternal
apic1(config-spine-ospf-vrf)# exit
apic1(config-spine-ospf)# exit
apic1(config-spine)#
apic1(config-spine)# interface ethernet 8/36.5
apic1(config-spine-if)# vrf member tenant infra vrf overlay-1 l3out ipn-multipodInternal
apic1(config-spine-if)# ip router ospf multipod-internal area 5
apic1(config-spine-if)# exit
apic1(config-spine)# exit
apic1(config)#
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant infra vrf overlay-1 l3out rl-wan-test
apic1(config-leaf-vrf)# exit
apic1(config-leaf)#
apic1(config-leaf)# interface ethernet 1/49
apic1(config-leaf-if)# vlan-domain member ospfDom
apic1(config-leaf-if)# exit
apic1(config-leaf)# router ospf default
apic1(config-leaf-ospf)# vrf member tenant infra vrf overlay-1
apic1(config-leaf-ospf-vrf)# area 5 l3out rl-wan-test
apic1(config-leaf-ospf-vrf)# exit
apic1(config-leaf-ospf)# exit
apic1(config-leaf)#
apic1(config-leaf)# interface ethernet 1/49.4
apic1(config-leaf-if)# vrf member tenant infra vrf overlay-1 l3out rl-wan-test
apic1(config-leaf-if)# ip router ospf default area 5
apic1(config-leaf-if)# exit

Transit Routing
Transit Routing in the ACI Fabric
The Cisco APIC software supports external Layer 3 connectivity with OSPF (NSSA) and iBGP. The fabric
advertises the tenant bridge domain subnets out to the external routers on the External Layer 3 Outside (L3Out)
connections. The routes that are learned from the external routers are not advertised to other external routers.
The fabric behaves like a stub network that can be used to carry the traffic between the external Layer 3
domains.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

325
 Configuring Layer 3 External Connectivity
Transit Routing Related Topics

Figure 25: Transit Routing in the Fabric

In transit routing, multiple L3Out connections within a single tenant and VRF are supported and the APIC
advertises the routes that are learned from one L3Out connection to another L3Out connection. The external
Layer 3 domains peer with the fabric on the border leaf switches. The fabric is a transit Multiprotocol-Border
Gateway Protocol (MP-BGP) domain between the peers.
The configuration for external L3Out connections is done at the tenant and VRF level. The routes that are
learned from the external peers are imported into MP-BGP at the ingress leaf per VRF. The prefixes that are
learned from the L3Out connections are exported to the leaf switches only where the tenant VRF is present.

Note For cautions and guidelines for configuring transit routing, see Guidelines for Transit Routing, on page 328

Transit Routing Related Topics

For a transit routing overview, use cases, guidelines, and information about transit routing route controls, see
Cisco APIC Layer 3 Networking Configuration Guide.

Transit Routing Overview

This topic provides a typical example of how to configure Transit Routing when using Cisco APIC.
The examples in this chapter use the following topology:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

326
Configuring Layer 3 External Connectivity
Transit Routing Overview

Figure 26:

In the examples in this chapter, the Cisco ACI fabric has 2 leaf switches and two spine switches, that are
controlled by an APIC cluster. The border leaf switches 101 and 102 have L3Outs on them providing
connections to two routers and thus to the Internet. The goal of this example is to enable traffic to flow from
EP 1 to EP 2 on the Internet into and out of the fabric through the two L3Outs.
In this example, the tenant that is associated with both L3Outs is t1, with VRF v1.
Before configuring the L3Outs, configure the nodes, ports, functional profiles, AEPs, and a Layer 3 domain.
You must also configure the spine switches 104 and 105 as BGP route reflectors.
Configuring transit routing includes defining the following components:
1. Tenant and VRF
2. Node and interface on leaf 101 and leaf 102
3. Primary routing protocol on each L3Out (used to exchange routes between border leaf switch and external
routers; in this example, BGP)
4. Connectivity routing protocol on each L3Out (provides reachability information for the primary protocol;
in this example, OSPF)
5. Two external EPGs
6. One route map
7. At least one filter and one contract
8. Associate the contract with the external EPGs

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

327
 Configuring Layer 3 External Connectivity
Guidelines for Transit Routing

Note For transit routing cautions and guidelines, see Guidelines for Transit Routing, on page 328.

The following table lists the names that are used in the examples in this chapter:

Property Names for L3Out1 on Node 101 Names for L3Out2 on Node 102

Tenant t1 t1

VRF v1 v1

Node nodep1 with router ID 11.11.11.103 nodep2 with router ID 22.22.22.203

OSPF Interface ifp1 at eth/1/3 ifp2 at eth/1/3

BGP peer address 15.15.15.2/24 25.25.25.2/24

External EPG extnw1 at 192.168.1.0/24 extnw2 at 192.168.2.0/24

Route map rp1 with ctx1 and route destination rp2 with ctx2 and route destination
192.168.1.0/24 192.168.2.0/24

Filter http-filter http-filter

Contract httpCtrct provided by extnw1 httpCtrct consumed by extnw2

Guidelines for Transit Routing

Use the following guidelines when creating and maintaining transit routing connections:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

328
Configuring Layer 3 External Connectivity
Guidelines for Transit Routing

Topic Caution or Guideline

Transit Routing with a Single L3Out Before APIC, release 2.3(1f), transit routing was not supported
Profile within a single L3Out profile. In APIC, release 2.3(1f) and later,
you can configure transit routing with a single L3Out profile, with
the following limitations:
• If the VRF is unenforced, an external subnet (l3extSubnet)
of 0.0.0.0/0 can be used to allow traffic between the routers
sharing the same L3EPG.
• If the VRF is enforced, an external default subnet (0.0.0.0/0)
cannot be used to match both source and destination prefixes
for traffic within the same Layer 3 EPG. To match all traffic
within the same Layer 3 EPG, the following prefixes are
supported:
• IPv4
• 0.0.0.0/1—with External Subnets for the External
EPG
• 128.0.0.0/1—with External Subnets for the
External EPG
• 0.0.0.0/0—with Import Route Control Subnet,
Aggregate Import

• IPv6
• 0::0/1—with External Subnets for the External
EPG
• 8000::0/1—with External Subnets for the External
EPG
• 0:0/0—with Import Route Control Subnet,
Aggregate Import

• Alternatively, a single default subnet (0.0.0.0/0) can be used

when combined with a VzAny contract. For example:
• Use a VzAny provided contract and an Layer 3 EPG
consumed contract (matching 0.0.0.0/0), or a VzAny
consumed contract and Layer 3 EPG provided contract
(matching 0.0.0.0/0).
• Use the subnet 0.0.0.0/0—with Import/Export Route
Control Subnet, Aggregate Import, and Aggregate
Export.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

329
 Configuring Layer 3 External Connectivity
Guidelines for Transit Routing

Topic Caution or Guideline

Shared Routes: Differences in Hardware Routes shared between VRFs function correctly on Generation 2
Support switches (Cisco Nexus N9K switches with "EX" or "FX" on the
end of the switch model name, or later; for example,
N9K-93108TC-EX). On Generation 1 switches, however, there
may be dropped packets with this configuration, because the
physical ternary content-addressable memory (TCAM) tables that
store routes do not have enough capacity to fully support route
parsing.

OSPF or EIGRP in Back to Back Cisco APIC supports transit routing in export route control policies
Configuration that are configured on the L3Out. These policies control which
transit routes (prefixes) are redistributed into the routing protocols
in the L3Out. When these transit routes are redistributed into
OSPF or EIGRP, they are tagged 4294967295 to prevent routing
loops. The Cisco ACI fabric does not accept routes matching this
tag when learned on an OSPF or EIGRP L3Out. However, in the
following cases, it is necessary to override this behavior:
• When connecting two Cisco ACI fabrics using OSPF or
EIGRP.
• When connecting two different VRFs in the same Cisco ACI
fabric using OSPF or EIGRP.

Where an override is required, you must configure the VRF with

a different tag policy at the following APIC GUI location:
Tenant > Tenant_name > Networking > Protocol Policies >
Route Tag. Apply a different tag.
In addition to creating the new route-tag policy, update the VRF
to use this policy at the following APIC GUI location: Tenant >
Tenant_name > Networking > VRFs > Tenant_VRF . Apply
the route tag policy that you created to the VRF.
Note When multiple L3Outs or multiple interfaces in the
same L3Out are deployed on the same leaf switch and
used for transit routing, the routes are advertised within
the IGP (not redistributed into the IGP). In this case
the route-tag policy does not apply.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

330
Configuring Layer 3 External Connectivity
Guidelines for Transit Routing

Topic Caution or Guideline

Advertising BD Subnets Outside the The import and export route control policies only apply to the
Fabric transit routes (the routes that are learned from other external peers)
and the static routes. The subnets internal to the fabric that are
configured on the tenant BD subnets are not advertised out using
the export policy subnets. The tenant subnets are still permitted
using the IP prefix-lists and the route-maps but they are
implemented using different configuration steps. See the following
configuration steps to advertise the tenant subnets outside the
fabric:
1. Configure the tenant subnet scope as Public Subnet in the
subnet properties window.
2. Optional. Set the Subnet Control as ND RA Prefix in the
subnet properties window.
3. Associate the tenant bridge domain (BD) with the external
Layer 3 Outside (L3Out).
4. Create contract (provider or consumer) association between
the tenant EPG and the external EPG.
Setting the BD subnet to Public scope and associating the BD
to the L3Out creates an IP prefix-list and the route-map
sequence entry on the border leaf for the BD subnet prefix.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

331
 Configuring Layer 3 External Connectivity
Guidelines for Transit Routing

Topic Caution or Guideline

Advertising a Default Route For external connections to the fabric that only require a default
route, there is support for originating a default route for OSPF,
EIGRP, and BGP L3Out connections. If a default route is received
from an external peer, this route can be redistributed out to another
peer following the transit export route control as described earlier
in this article.
A default route can also be advertised out using a Default Route
Leak policy. This policy supports advertising a default route if it
is present in the routing table or it always supports advertising a
default route. The Default Route Leak policy is configured in the
L3Out connection.
When creating a Default Route Leak policy, follow these
guidelines:
• For BGP, the Always property is not applicable.
• For BGP, when configuring the Scope property, choose
Outside.
• For OSPF, the scope value Context creates a type-5 LSA
while the Scope value Outside creates type-7 LSA. Your
choice depends on the area type configured in the L3Out. If
the area type is Regular, set the scope to Context. If the area
type is NSSA, set the scope to Outside.
• For EIGRP, when choosing the Scope property, you must
choose Context.

MTU Cisco ACI does not support IP fragmentation. Therefore, when

you configure Layer 3 Outside (L3Out) connections to external
routers, or multipod connections through an Inter-Pod Network
(IPN), it is critical that the MTU is set appropriately on both sides.
On some platforms, such as ACI, Cisco NX-OS, and Cisco IOS,
the configurable MTU value takes into account the IP headers
(resulting in a max packet size to be set as 9216 bytes for ACI
and 9000 for NX-OS and IOS). However, other platforms such
as IOS-XR configure the MTU value exclusive of packet headers
(resulting in a max packet size of 8986 bytes).
For the appropriate MTU values for each platform, see the relevant
configuration guides.
Cisco highly recommends you test the MTU using CLI-based
commands. For example, on the Cisco NX-OS CLI, use a
command such as ping 1.1.1.1 df-bit packet-size 9000
source-interface ethernet 1/1.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

332
 Configuring Layer 3 External Connectivity
Configure Transit Routing Using the NX-OS Style CLI

Configure Transit Routing Using the NX-OS Style CLI

These steps describe how to configure transit routing for a tenant. This example deploys two L3Outs, in one
VRF, on two border leaf switches, that are each connected to separate routers.

Before you begin

• Configure the node, port, functional profile, AEP, and Layer 3 domain.
• Configure a VLAN domain using the vlan-domain domain and vlan vlan-range commands.
• Configure a BGP route reflector policy to propagate the routed within the fabric.

For an example of the commands for these prerequisites, see NX-OS Style CLI Example: L3Out Prerequisites,
on page 173.

Procedure

Step 1 Configure the tenant and VRF.

This example configures tenant t1 with VRF v1. The VRF is not yet deployed.
Example:
apic1# configure
apic1(config)# tenant t1
apic1(config-tenant)# vrf context v1
apic1(config-tenant-vrf)# exit
apic1(config-tenant)# exit

Step 2 Configure the nodes and interfaces.

This example configures two L3Outs for the tenant t1, on two border leaf switches:
• The first L3Out is on node 101, which is named nodep1. Node 101 is configured with router ID
11.11.11.103. It has a routed interface ifp1 at eth1/3, with the IP address 12.12.12.3/24.

• The second L3Out is on node 102, which is named nodep2. Node 102 is configured with router ID
22.22.22.203. It has a routed interface ifp2 at eth1/3, with the IP address, 23.23.23.1/24.

Example:
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# router-id 11.11.11.103
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# interface ethernet 1/3
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# vrf member tenant t1 vrf v1
apic1(config-leaf-if)# ip address 12.12.12.3/24
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit
apic1(config)# leaf 102
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# router-id 22.22.22.203
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# interface ethernet 1/3
apic1(config-leaf-if)# vlan-domain member dom1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

333
 Configuring Layer 3 External Connectivity
Configure Transit Routing Using the NX-OS Style CLI

apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# vrf member tenant t1 vrf v1
apic1(config-leaf-if)# ip address 23.23.23.3/24
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit

Step 3 Configure the routing protocol for both leaf switches.

This example configures BGP as the primary routing protocol for both the border leaf switches, both with
ASN 100. It also configures Node 101 with BGP peer 15.15.15.2 and node 102 with BGP peer 25.25.25.2.
Example:
apic1(config)# leaf 101
apic1(config-leaf)# router bgp 100
apic1(config-leaf-bgp)# vrf member tenant t1 vrf v1
apic1(config-leaf-bgp-vrf)# neighbor 15.15.15.2
apic1(config-leaf-bgp-vrf-neighbor)# exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf-bgp)# exit
apic1(config-leaf)# exit
apic1(config)# leaf 102
apic1(config-leaf)# router bgp 100
apic1(config-leaf-bgp)# vrf member tenant t1 vrf v1
apic1(config-leaf-bgp-vrf)# neighbor 25.25.25.2
apic1(config-leaf-bgp-vrf-neighbor)# exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf-bgp)# exit
apic1(config-leaf)# exit

Step 4 Configure a connectivity routing protocol.

This example configures OSPF as the communication protocol, for both L3Outs, with regular area ID 0.0.0.0.
Example:

apic1(config)# leaf 101

apic1(config-leaf)# router ospf default
apic1(config-leaf-ospf)# vrf member tenant t1 vrf v1
apic1(config-leaf-ospf-vrf)# area 0.0.0.0 loopback 40.40.40.1
apic1(config-leaf-ospf-vrf)# exit
apic1(config-leaf-ospf)# exit
apic1(config-leaf)# exit
apic1(config)# leaf 102
apic1(config-leaf)# router ospf default
apic1(config-leaf-ospf)# vrf member tenant t1 vrf v1
apic1(config-leaf-ospf-vrf)# area 0.0.0.0 loopback 60.60.60.1
apic1(config-leaf-ospf-vrf)# exit
apic1(config-leaf-ospf)# exit
apic1(config-leaf)# exit

Step 5 Configure the external EPGs.

This example configures the network 192.168.1.0/24 as external network extnw1 on node 101 and the
network 192.168.2.0/24 as external network extnw2 on node 102.
Example:
apic1(config)# tenant t1
apic1(config-tenant)# external-l3 epg extnw1
apic1(config-tenant-l3ext-epg)# vrf member v1
apic1(config-tenant-l3ext-epg)# match ip 192.168.1.0/24
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# external-l3 epg extnw2
apic1(config-tenant-l3ext-epg)# vrf member v1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

334
Configuring Layer 3 External Connectivity
Configure Transit Routing Using the NX-OS Style CLI

apic1(config-tenant-l3ext-epg)# match ip 192.168.2.0/24

apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# exit
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# external-l3 epg extnw1
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# exit
apic1(config)# leaf 102
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# external-l3 epg extnw2
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# exit

Step 6 Optional. Configure the route maps.

This example configures a route map for each BGP peer in the inbound and outbound directions.
Example:
Example:
apic1(config)# leaf 101
apic1(config-leaf)# template route group match-rule1 tenant t1
apic1(config-route-group)# ip prefix permit 192.168.1.0/24
apic1(config-route-group)# exit
apic1(config-leaf)# template route group match-rule2 tenant t1
apic1(config-route-group)# ip prefix permit 192.168.2.0/24
apic1(config-route-group)# exit
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# route-map rp1
apic1(config-leaf-vrf-route-map)# match route group match-rule1 order 0
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# route-map rp2
apic1(config-leaf-vrf-route-map)# match route group match-rule2 order 0
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# router bgp 100
apic1(config-leaf-bgp)# vrf member tenant t1 vrf v1
apic1(config-leaf-bgp-vrf)# neighbor 15.15.15.2
apic1(config-leaf-bgp-vrf-neighbor)# route-map rp1 in
apic1(config-leaf-bgp-vrf-neighbor)# route-map rp2 out
apic1(config-leaf-bgp-vrf-neighbor)# exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf-bgp)# exit
apic1(config-leaf)# exit

apic1(config)# leaf 102

apic1(config-leaf)# template route group match-rule1 tenant t1
apic1(config-route-group)# ip prefix permit 192.168.1.0/24
apic1(config-route-group)# exit
apic1(config-leaf)# template route group match-rule2 tenant t1
apic1(config-route-group)# ip prefix permit 192.168.2.0/24
apic1(config-route-group)# exit
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# route-map rp1
apic1(config-leaf-vrf-route-map)# match route group match-rule2 order 0
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# route-map rp2
apic1(config-leaf-vrf-route-map)# match route group match-rule1 order 0
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

335
 Configuring Layer 3 External Connectivity
Example: Transit Routing

apic1(config-leaf-vrf)# exit
apic1(config-leaf)# router bgp 100
apic1(config-leaf-bgp)# vrf member tenant t1 vrf v1
apic1(config-leaf-bgp-vrf)# neighbor 25.25.25.2
apic1(config-leaf-bgp-vrf-neighbor)# route-map rp2 in
apic1(config-leaf-bgp-vrf-neighbor)# route-map rp1 out
apic1(config-leaf-bgp-vrf-neighbor)# exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf-bgp)# exit
apic1(config-leaf)# exit

Step 7 Create filters (access lists) and contracts to enable the EPGs to communicate.
Example:
apic1(config)# tenant t1
apic1(config-tenant)# access-list http-filter
apic1(config-tenant-acl)# match ip
apic1(config-tenant-acl)# match tcp dest 80
apic1(config-tenant-acl)# exit
apic1(config-tenant)# contract httpCtrct
apic1(config-tenant-contract)# scope vrf
apic1(config-tenant-contract)# subject subj1
apic1(config-tenant-contract-subj)# access-group http-filter both
apic1(config-tenant-contract-subj)# exit
apic1(config-tenant-contract)# exit
apic1(config-tenant)# exit

Step 8 Configure contracts and associate them with EPGs.

Example:
apic1(config)# tenant t1
apic1(config-tenant)# external-l3 epg extnw1
apic1(config-tenant-l3ext-epg)# vrf member v1
apic1(config-tenant-l3ext-epg)# contract provider httpCtrct
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# external-l3 epg extnw2
apic1(config-tenant-l3ext-epg)# vrf member v1
apic1(config-tenant-l3ext-epg)# contract consumer httpCtrct
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# exit
apic1(config)#

Example: Transit Routing

This example provides a merged configuration for transit routing. The configuration is for a single tenant and
VRF, with two L3Outs, on two border leaf switches, that are each connected to separate routers.
apic1# configure
apic1(config)# tenant t1
apic1(config-tenant)# vrf context v1
apic1(config-tenant-vrf)# exit
apic1(config-tenant)# exit

apic1(config)# leaf 101

apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# router-id 11.11.11.103
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# interface ethernet 1/3
apic1(config-leaf-if)# vlan-domain member dom1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

336
Configuring Layer 3 External Connectivity
Example: Transit Routing

apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# vrf member tenant t1 vrf v1
apic1(config-leaf-if)# ip address 12.12.12.3/24
apic1(config-leaf-if)# exit
apic1(config-leaf)# router bgp 100
apic1(config-leaf-bgp)# vrf member tenant t1 vrf v1
apic1(config-leaf-bgp-vrf)# neighbor 15.15.15.2
apic1(config-leaf-bgp-vrf-neighbor)# exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf-bgp)# exit
apic1(config-leaf)# router ospf default
apic1(config-leaf-ospf)# vrf member tenant t1 vrf v1
apic1(config-leaf-ospf-vrf)# area 0.0.0.0 loopback 40.40.40.1
apic1(config-leaf-ospf-vrf)# exit
apic1(config-leaf-ospf)# exit
apic1(config-leaf)# exit

apic1(config)# leaf 102

apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# router-id 22.22.22.203
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# interface ethernet 1/3
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# vrf member tenant t1 vrf v1
apic1(config-leaf-if)# ip address 23.23.23.3/24
apic1(config-leaf-if)# exit
apic1(config-leaf)# router bgp 100
apic1(config-leaf-bgp)# vrf member tenant t1 vrf v1
apic1(config-leaf-bgp-vrf)# neighbor 25.25.25.2/24
apic1(config-leaf-bgp-vrf-neighbor)# exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf-bgp)# exit
apic1(config-leaf)# router ospf default
apic1(config-leaf-ospf)# vrf member tenant t1 vrf v1
apic1(config-leaf-ospf-vrf)# area 0.0.0.0 loopback 60.60.60.3
apic1(config-leaf-ospf-vrf)# exit
apic1(config-leaf-ospf)# exit
apic1(config-leaf)# exit

apic1(config)# tenant t1
apic1(config-tenant)# external-l3 epg extnw1
apic1(config-tenant-l3ext-epg)# vrf member v1
apic1(config-tenant-l3ext-epg)# match ip 192.168.1.0/24
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# external-l3 epg extnw2
apic1(config-tenant-l3ext-epg)# vrf member v1
apic1(config-tenant-l3ext-epg)# match ip 192.168.2.0/24
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# exit

apic1(config)# leaf 101

apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# external-l3 epg extnw1
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# exit
apic1(config)# leaf 102
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# external-l3 epg extnw2
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# exit

apic1(config)# leaf 101

apic1(config-leaf)# template route group match-rule1 tenant t1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

337
 Configuring Layer 3 External Connectivity
Example: Transit Routing

apic1(config-route-group)# ip prefix permit 192.168.1.0/24

apic1(config-route-group)# exit
apic1(config-leaf)# template route group match-rule2 tenant t1
apic1(config-route-group)# ip prefix permit 192.168.2.0/24
apic1(config-route-group)# exit
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# route-map rp1
apic1(config-leaf-vrf-route-map)# match route group match-rule1 order 0
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# route-map rp2
apic1(config-leaf-vrf-route-map)# match route group match-rule2 order 0
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# router bgp 100
apic1(config-leaf-bgp)# vrf member tenant t1 vrf v1
apic1(config-leaf-bgp-vrf)# neighbor 15.15.15.2
apic1(config-leaf-bgp-vrf-neighbor)# route-map rp1 in
apic1(config-leaf-bgp-vrf-neighbor)# route-map rp2 out
apic1(config-leaf-bgp-vrf-neighbor)# exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf-bgp)# exit
apic1(config-leaf)# exit

apic1(config)# leaf 102

apic1(config-leaf)# template route group match-rule1 tenant t1
apic1(config-route-group)# ip prefix permit 192.168.1.0/24
apic1(config-route-group)# exit
apic1(config-leaf)# template route group match-rule2 tenant t1
apic1(config-route-group)# ip prefix permit 192.168.2.0/24
apic1(config-route-group)# exit
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# route-map rp1
apic1(config-leaf-vrf-route-map)# match route group match-rule1 order 0
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# route-map rp2
apic1(config-leaf-vrf-route-map)# match route group match-rule2 order 0
apic1(config-leaf-vrf-route-map-match)# exit
apic1(config-leaf-vrf-route-map)# exit
apic1(config-leaf-vrf)# exit
apic1(config-leaf)# router bgp 100
apic1(config-leaf-bgp)# vrf member tenant t1 vrf v1
apic1(config-leaf-bgp-vrf)# neighbor 25.25.25.2
apic1(config-leaf-bgp-vrf-neighbor)# route-map rp2 in
apic1(config-leaf-bgp-vrf-neighbor)# route-map rp1 out
apic1(config-leaf-bgp-vrf-neighbor)# exit
apic1(config-leaf-bgp-vrf)# exit
apic1(config-leaf-bgp)# exit
apic1(config-leaf)# exit

apic1(config)# tenant t1
apic1(config-tenant)# access-list http-filter
apic1(config-tenant-acl)# match ip
apic1(config-tenant-acl)# match tcp dest 80
apic1(config-tenant-acl)# exit
apic1(config-tenant)# contract httpCtrct
apic1(config-tenant-contract)# scope vrf
apic1(config-tenant-contract)# subject http-subj
apic1(config-tenant-contract-subj)# access-group http-filter both
apic1(config-tenant-contract-subj)# exit
apic1(config-tenant-contract)# exit
apic1(config-tenant)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

338
Configuring Layer 3 External Connectivity
Example: Transit Routing

apic1(config)# tenant t1
apic1(config-tenant)# external-l3 epg extnw1
apic1(config-tenant-l3ext-epg)# vrf member v1
apic1(config-tenant-l3ext-epg)# contract provider httpCtrct
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# external-l3 epg extnw2
apic1(config-tenant-l3ext-epg)# vrf member v1
apic1(config-tenant-l3ext-epg)# contract consumer httpCtrct
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# exit
apic1(config)#

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

339
 Configuring Layer 3 External Connectivity
Example: Transit Routing

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

340
 CHAPTER 8
Configuring Cisco ACI QoS
This chapter contains the following sections:
• QoS for L3Outs, on page 341
• CoS Preservation, on page 343
• Multipod QoS, on page 345
• Translating QoS Ingress Markings to Egress Markings, on page 347

QoS for L3Outs

To configure QoS policies for an L3Out, use the following guidelines:
• To configure the QoS policy to be enforced on the border leaf where the L3Out is located, the VRF
instance must be in egress mode (Policy Control Enforcement Direction must be "Egress").
• To enable the QoS policy to be enforced, the VRF Policy Control Enforcement Preference must be
"Enforced."
• When configuring the contract governing communication between the L3Out and other EPGs, include
the QoS class or target DSCP in the contract or subject.

Note Only configure a QoS class or target DSCP in the contract, not in the external
EPG (l3extInstP).

• When creating a contract subject, you must choose a QoS priority level. You cannot choose Unspecified.

Configuring QoS for L3Outs Using the NX-OS Style CLI

QoS for L3Out is configured as part of the L3Out configuration.

Procedure

Step 1 When configuring the tenant and VRF, to support QoS priority enforcement on the L3Out, configure the VRF
for egress mode and enable policy enforcement, using the following commands:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

341
 Configuring Cisco ACI QoS
Configuring QoS Directly on L3Out Using CLI

Example:
apic1# configure
apic1(config)# tenant t1
apic1(config-tenant)# vrf context v1
apic1(config-tenant-vrf)# contract enforce egress
apic1(config-tenant-vrf)# exit
apic1(congig-tenant)# exit
apic1(config)#

Step 2 When creating filters (access-lists), include the match dscp command, in this example with target DSCP
level EF. When configuring contracts, include the QoS class, for example, level1, for traffic ingressing on the
L3Out. Alternatively, it could define a target DSCP value. QoS policies are supported on either the contract
or the subject.
Example:
apic1(config)# tenant t1
apic1(config-tenant)# access-list http-filter
apic1(config-tenant-acl)# match ip
apic1(config-tenant-acl)# match tcp dest 80
apic1(config-tenant-acl)# match dscp EF
apic1(config-tenant-acl)# exit
apic1(config-tenant)# contract httpCtrct
apic1(config-tenant-contract)# scope vrf
apic1(config-tenant-contract)# qos-class level1
apic1(config-tenant-contract)# subject http-subject
apic1(config-tenant-contract-subj)# access-group http-filter both
apic1(config-tenant-contract-subj)# exit
apic1(config-tenant-contract)# exit
apic1(config-tenant)# exit
apic1(config)#

Configuring QoS Directly on L3Out Using CLI

This section describes how to configure QoS directly on an L3Out. This is the preferred way of configuring
L3Out QoS starting with Cisco APIC Release 4.0(1).
You can configure QoS for L3Out on one of the following objects:
• Switch Virtual Interface (SVI)
• Sub Interface
• Routed Outside

Procedure

Step 1 Configure QoS priorities for a L3Out SVI.

Example:
interface vlan 19
vrf member tenant DT vrf dt-vrf
ip address 107.2.1.252/24
description 'SVI19'
service-policy type qos VrfQos006 // for custom QoS attachment

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

342
 Configuring Cisco ACI QoS
CoS Preservation

set qos-class level6 // for set QoS priority

exit

Step 2 Configure QoS priorities for a sub-interface.

Example:
interface ethernet 1/48.10
vrf member tenant DT vrf inter-tentant-ctx2 l3out L4_E48_inter_tennant
ip address 210.2.0.254/16
service-policy type qos vrfQos002
set qos-class level5

Step 3 Configure QoS priorities for a routed outside.

Example:
interface ethernet 1/37
no switchport
vrf member tenant DT vrf dt-vrf l3out L2E37
ip address 30.1.1.1/24
service-policy type qos vrfQos002
set qos-class level5
exit

CoS Preservation
Preserving 802.1P Class of Service Settings
APIC enables preserving 802.1P class of service (CoS) settings within the fabric. Enable the fabric global
QoS policy dot1p-preserve option to guarantee that the CoS value in packets which enter and transit the
ACI fabric is preserved.
802.1P CoS preservation is supported in single pod and multipod topologies.
In multipod topologies, CoS Preservation can be used where you want to preserve the QoS priority settings
of 802.1P traffic entering POD 1 and egressing out of POD 2, but you are not concerned with preserving the
CoS/DSCP settings in interpod network (IPN) traffic between the pods. To preserve CoS/DSCP settings when
multipod traffic is transitting an IPN, use a DSCP policy. For more information, see Preserving QoS Priority
Settings in a Multipod Fabric, on page 346.
Observe the following 801.1P CoS preservation guidelines and limitations:
• The current release can only preserve the 802.1P value within a VLAN header. The DEI bit is not
preserved.
• For VXLAN encapsulated packets, the current release will not preserve the 802.1P CoS value contained
in the outer header.
• 802.1P is not preserved when the following configuration options are enabled:
• Multipod QoS (using a DSCP policy) is enabled.
• Contracts are configured that include QoS.
• Dynamic packet prioritization is enabled.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

343
 Configuring Cisco ACI QoS
Enable Class Of Service (CoS) Preservation Using NX-OS Style CLI

• The outgoing interface is on a FEX.

• Preserving QoS CoS priority settings is not supported when traffic is flowing from an EPG with
isolation enforced to an EPG without isolation enforced.
• A DSCP QoS policy is configured on a VLAN EPG and the packet has an IP header. DSCP marking
can be set at the filter level on the following with the precedence order from the innermost to the
outermost:
• Contract
• Subject
• In Term
• Out Term

Note When specifying vzAny for a contract, external EPG DSCP values are not honored
because vzAny is a collection of all EPGs in a VRF, and EPG specific
configuration cannot be applied. If EPG specific target DSCP values are required,
then the external EPG should not use vzAny.

Enable Class Of Service (CoS) Preservation Using NX-OS Style CLI

This section describes how to enable CoS preservation to ensure that QoS priority settings are handled the
same for traffic entering and transiting a single-pod fabric as for traffic entering one pod and egressing another
in a multipod fabric.

Note Enabling CoS preservation applies a default CoS-to-DSCP mapping to the various traffic types.

Procedure

Step 1 Enter configuration mode.

Example:

apic1# configure

Step 2 Enables CoS preservation.

Example:

apic1(config)# qos preserve cos

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

344
 Configuring Cisco ACI QoS
Multipod QoS

Multipod QoS
Creating DSCP Translation Policy Using NX-OS Style CLI
This section describes how to create a DSCP translation policy to guarantee QoS Level settings across multiple
PODs connected by an IPN.

Procedure

Step 1 Enters configuration mode.

Example:

apic1# configure

Step 2 Enters tenant configuration mode for the infra tenant.

Example:

apic1(config)# tenant infra

Step 3 Create the DSCP translation map.

Example:

apic1(config-tenant)# qos dscp-map default

Step 4 Configure the DSCP translation mappings.

Note All mappings must be unique within a DSCP translation map and you must not map any QoS level
to CS6.

Example:
apic1(config-qos-cmap# set dscp-code control CS3
apic1(config-qos-cmap# set dscp-code span CS5
apic1(config-qos-cmap# set dscp-code level1 CS0
apic1(config-qos-cmap# set dscp-code level2 CS1
apic1(config-qos-cmap# set dscp-code level3 CS2
apic1(config-qos-cmap# set dscp-code level4 CS3
apic1(config-qos-cmap# set dscp-code level5 CS4
apic1(config-qos-cmap# set dscp-code level6 CS5
apic1(config-qos-cmap# set dscp-code policy CS4
apic1(config-qos-cmap# set dscp-code traceroute CS5

Step 5 Enable the DSCP translation.

Example:

apic1(config-qos-cmap)# no shutdown

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

345
 Configuring Cisco ACI QoS
Preserving QoS Priority Settings in a Multipod Fabric

Preserving QoS Priority Settings in a Multipod Fabric

This topic describes how to guarantee QoS priority settings in a multipod topology, where devices in the
interpod network are not under APIC management, and may modify 802.1P settings in traffic transitting their
network.

Note You can alternatively use CoS Preservation where you want to preserve the QoS priority settings of 802.1P
traffic entering POD 1 and egressing out of POD 2, but you are not concerned with preserving the CoS/DSCP
settings in interpod network (IPN) traffic between the pods. For more information, see Preserving 802.1P
Class of Service Settings, on page 343.

Figure 27: Multipod Topology

As illustrated in this figure, traffic between pods in a multipod topology passes through an IPN, which may
not be under APIC management. When an 802.1P frame is sent from a spine or leaf switch in POD 1, the
devices in the IPN may not preserve the CoS setting in 802.1P frames. In this situation, when the frame reaches
a POD 2 spine or leaf switch, it has the CoS level assigned by the IPN device, instead of the level assigned
at the source in POD 1. Use a DSCP policy to ensure that the QoS priority levels are preserved in this case.
Configure a DSCP policy to preserve the QoS priority settings in a multipod topology, where there is a need
to do deterministic mapping from CoS to DSCP levels for different traffic types, and you want to prevent the
devices in the IPN from changing the configured levels. With a DSCP policy enabled, APIC converts the CoS
level to a DSCP level, according to the mapping you configure. When a frame is sent from POD 1 (with the
PCP level mapped to a DSCP level), when it reaches POD 2, the mapped DSCP level is then mapped back
to the original PCP CoS level.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

346
 Configuring Cisco ACI QoS
Translating QoS Ingress Markings to Egress Markings

Translating QoS Ingress Markings to Egress Markings

Translating QoS Ingress Markings to Egress Markings
APIC enables translating the 802.1P CoS field (Class of Service) based on the ingress DSCP value. 802.1P
CoS translation is supported only if DSCP is present in the IP packet and dot1P is present in the Ethernet
frames.
This functionality enables the ACI Fabric to classify the traffic for devices that classify the traffic based only
on the CoS value. It allows mapping the dot1P CoS value based on the ingress dot1P value. It is mainly
applicable for Layer 2 packets, which do not have an IP header.
Observe the following 802.1P CoS translation guidelines and limitations:
• Enable the fabric global QoS policy dot1p-preserve option.
• 802.1P CoS translation is not supported on external L3 interfaces.
• 802.1P CoS translation is supported only if the egress frame is 802.1Q encapsulated.

802.1P CoS translation is not supported when the following configuration options are enabled:
• Contracts are configured that include QoS.
• The outgoing interface is on a FEX.
• Multipod QoS using a DSCP policy is enabled.
• Dynamic packet prioritization is enabled.
• If an EPG is configured with intra-EPG endpoint isolation enforced.
• If an EPG is configured with allow-microsegmentation enabled.

Creating Custom QoS Policy Using NX-OS Style CLI

This section describes how to create a custom QoS policy and associate it with an EPG using the NX-OS style
CLI.

Before you begin

You must have created the tenant, application, and EPGs that will consume the custom QoS policy.

Procedure

Step 1 Enter configuration mode.

Example:
apic1# configure

Step 2 Enter tenant configuration mode.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

347
 Configuring Cisco ACI QoS
Creating Custom QoS Policy Using NX-OS Style CLI

Example:
apic1(config)# tenant <tenant-name>

Step 3 Create QoS policy.

Example:
apic1(config-tenant)# policy-map type qos <qos-policy-name>

Step 4 Set DCSP range and target QoS priority level.

Example:
Example:
apic1(config-tenant-pmap-qos)# match dscp AF23 AF31 set-cos 6

Step 5 Return to tenant configuration mode.

Example:
apic1(config-tenant-pmap-qos)# exit

Step 6 Create or edit an application profile.

Example:
apic1(config-tenant)# application <application-name>

Step 7 Create or edit an EPG in the application profile.

To create a normal EPG:
Example:
apic1(config-tenant-app)# epg <epg-name>

To create an external Layer-2 EPG:

Example:
apic1(config-tenant)# external-l2 epg <ext-l2-epg-name>

Step 8 Associate the QoS policy with the EPG.

The system prompt may be different depending on whether you create a normal EPG or an external EPG.
Example:
apic1(config-tenant-app-epg)# service-policy <qos-policy-name>

Step 9 Return to the tenant configuration mode.

Example:
apic1(config-tenant-app-epg)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

348
 CHAPTER 9
Configuring Management Interfaces
• Configuring Out-of-Band Management Access, on page 349
• Configuring Inband Management Access, on page 351

Configuring Out-of-Band Management Access

To configure out-of-band (OOB) management access for controllers, leaf switches, or spine switches, these
steps must be performed:
• Configure the OOB management IP address and gateway on the management interface
• Allow access from the necessary external subnets
• Allow the necessary protocols on the management ports

Before you begin

The APIC out-of-band management connection link must be 1 Gbps.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 {controller apic-number-or-range | switch Specifies the controller or switch to be

node-id[-node-id-or-range]} configured. You can enter a range of
controllers or switches using dashes or
Example:
commas.
apic1(config)# controller 1-3

Step 3 interface mgmt0 The mgmt0 interface provides out-of-band

management, which enables you to manage
Example:
the device by its IPv4 address.
apic1(config-controller)# interface
mgmt0

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

349
 Configuring Management Interfaces
Configuring Out-of-Band Management Access

Command or Action Purpose

Step 4 ip address addr/mask gateway addr Configures the IP address and gateway for
OOB management. If you specified more than
Example:
one controller or switch, the command
apic1(config-controller-if)# ip becomes ip address-range and IP addresses
address-range 172.23.48.16/21 gateway
172.23.48.1
are assigned sequentially beginning with the
address specified in this command.
Note The APIC management interface
does not support an IPv6 address
and cannot connect to an external
IPv6 server through this interface.

Step 5 exit
Example:
apic1(config-controller-if)# exit

Step 6 exit
Example:
apic1(config-controller)# exit

Step 7 tenant mgmt System Management policies are configured

under a special tenant called mgmt.
Example:
apic1(config)# tenant mgmt

Step 8 external-l3 epg default oob-mgmt Enters the configuration mode of the
out-of-band management EPG.
Example:
apic1(config-tenant)# external-l3 epg
default oob-mgmt

Step 9 match ip addr/mask Provides access control for out-of-band

management interface to external management
Example:
subnets.
apic1(config-tenant-l3ext-epg)# match
ip 192.0.20.0/24

Step 10 exit
Example:
apic1(config-tenant-l3ext-epg)# exit

Step 11 access-list oob-default Configures the access list filter for the OOB
default policy.
Example:
apic1(config-tenant)# access-list
oob-default

Step 12 match tcp dest 443 Allows access on the management interface
for HTTPS traffic (TCP/443).
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

350
 Configuring Management Interfaces
Configuring Inband Management Access

Command or Action Purpose

apic1(config-tenant-acl)# match tcp dest
443

Step 13 match tcp dest 22 Allows access on the management interface

for SSH traffic (TCP/22).
Example:
apic1(config-tenant-acl)# match tcp dest
22

Examples
This example shows how to configure out-of-band management access for three APIC controllers.
In this example, the three controllers are assigned sequential IP addresses, with controller 1 at
172.23.48.16/21, controller 2 at 172.23.48.17/21, and controller 3 at 172.23.48.18/21.

apic1# configure
apic1(config)# controller 1-3
apic1(config-controller)# interface mgmt0
apic1(config-controller-if)# ip address-range 172.23.48.16/21 gateway 172.23.48.1
apic1(config-controller-if)# exit
apic1(config-controller)# exit
apic1(config)# tenant mgmt
apic1(config-tenant)# external-l3 epg default oob-mgmt
apic1(config-tenant-l3ext-epg)# match ip 192.0.20.0/24
apic1(config-tenant-l3ext-epg)# exit
apic1(config-tenant)# access-list oob-default
apic1(config-tenant-acl)# match tcp dest 443
apic1(config-tenant-acl)# match tcp dest 22

This example shows how to configure out-of-band management access for a leaf or spine switch.

apic1# configure
apic1(config)# switch 101
apic1(config-switch)# interface mgmt0
apic1(config-switch-if)# ip address 172.23.48.101/21 gateway 172.23.48.1

Configuring Inband Management Access

Configuring Inband Management Access to a Switch from an Outside Network
To configure inband (IB) management access for leaf switches or spine switches, these steps must be performed:
• Configure the inband management IP address and gateway on the inband management interface
• Create or specify a VLAN domain for external inband connectivity
• Add the external management station interface to the VLAN domain
• Allow the necessary protocols on the management ports

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

351
 Configuring Management Interfaces
Configuring Inband Management Access to a Switch from an Outside Network

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 switch switch-id-or-range Specifies the switch to be configured. You can

enter a range of switches using dashes or
Example:
commas.
apic1(config)# switch 101

Step 3 interface inband-mgmt0 The inband-mgmt0 interface provides inband

management.
Example:
apic1(config-switch)# interface
inband-mgmt0

Step 4 ip address addr/mask gateway addr Configures the IP address and gateway for
inband management. If you specified more than
Example:
one switch, the command becomes ip
apic1(config-switch-if)# ip address address-range and IP addresses are assigned
10.13.1.1/24 gateway 10.13.1.254
sequentially beginning with the address
specified in this command.

Step 5 exit
Example:
apic1(config-switch-if)# exit

Step 6 exit
Example:
apic1(config-switch)# exit

Examples
This example shows how to configure inband management for a switch from a management station
on an external network..

apic1# configure
apic1(config)# switch 101
apic1(config-switch)# interface inband-mgmt0
apic1(config-switch-if)# ip address 10.13.1.1/24 gateway 10.13.1.254
apic1(config-switch-if)# exit
apic1(config-switch)# exit

What to do next
• Configure inband (IB) management connectivity to the management station.
• Allow the necessary protocols (HTTPS and SSH) on the inbound management port.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

352
 Configuring Management Interfaces
Configuring Inband Management Access to a Controller from an Outside Network

Configuring Inband Management Access to a Controller from an Outside

Network
To configure inband (IB) management access for controllers, these steps must be performed:
• Configure the inband management IP address and gateway on the inband management interface
• Create a VLAN domain for external inband connectivity
• Allow the VLAN on the port connected to the controller

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 controller controller-id-or-range Specifies the controller to be configured. You

can enter a range of controllers using dashes
Example:
or commas.
apic1(config)# controller 1-3

Step 3 interface inband-mgmt0 The inband-mgmt0 interface provides inband

management.
Example:
apic1(config-controller)# interface
inband-mgmt0

Step 4 ip address addr/mask gateway addr Configures the IP address and gateway for
inband management. If you specified more
Example:
than one controller or switch, the command
apic1(config-controller-if)# ip becomes ip address-range and IP addresses
address-range 10.13.1.1/24 gateway
10.13.1.254
are assigned sequentially beginning with the
address specified in this command.

Step 5 vlan vlan-id Assigns a controller VLAN which is enabled

on the port connected to the controller. For
Example:
multiple controllers, all controllers must use
apic1(config-controller-if)# vlan 10 the same VLAN.

Step 6 exit
Example:
apic1(config-controller-if)# exit

Step 7 exit
Example:
apic1(config-controller)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

353
 Configuring Management Interfaces
Configuring Inband Management Access to a Controller from an Outside Network

Command or Action Purpose

Step 8 vlan-domain domain-name Creates and enters the configuration mode for
the VLAN domain.
Example:
apic1(config)# vlan-domain apic-inband

Step 9 vlan vlan-id Assigns the controller VLAN to the VLAN

domain.
Example:
apic1(config-vlan)# vlan 10

Step 10 exit Returns to global configuration mode.

Example:
apic1(config-vlan)# exit

Step 11 leaf node-id Specifies the leaf switch to which the controller
connected.
Example:
apic1(config)# leaf 102

Step 12 interface slot/port Specifies the port to which the controller is

connected.
Example:
apic1(config-leaf)# interface eth 1/1

Step 13 vlan-domain member apic-inband Configures controller connectivity to inband

management.
Example:
apic1(config-leaf-if)# vlan-domain
member apic-inband

Step 14 exit
Example:
apic1(config-leaf-if)# exit

Step 15 exit
Example:
apic1(config-leaf)# exit

Examples
This example shows how to configure inband management for a controller from a management
station on an external network. APIC controller 1 is connected to port Ethernet 1/1 on Leaf 101, and
VLAN 10 is used for the controller's inband connectivity.

apic1# configure
apic1(config)# controller 1-3
apic1(config-controller)# interface inband-mgmt0
apic1(config-controller-if)# ip address-range 10.13.1.1/24 gateway 10.13.1.254
apic1(config-controller-if)# vlan 10
apic1(config-controller-if)# exit
apic1(config-controller)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

354
 Configuring Management Interfaces
Configuring Inband Management Connectivity to the Management Station

# CREATE A VLAN DOMAIN FOR THE APIC INBAND VLAN

apic1(config)# vlan-domain apic-inband
apic1(config-vlan)# vlan 10
apic1(config-vlan)# exit

# ALLOW THE VLAN ON THE PORT CONNECTED TO THE CONTROLLER

apic1(config)# leaf 101
apic1(config-leaf)# interface eth 1/1
apic1(config-leaf-if)# vlan-domain member apic-inband
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit

What to do next
• Configure inband (IB) management connectivity to the management station.
• Allow the necessary protocols (HTTPS and SSH) on the inbound management port.

Configuring Inband Management Connectivity to the Management Station

To configure inband (IB) management connectivity to the management station, these steps must be performed:
• Create or specify a VLAN domain for external inband connectivity
• Add the external management station interface to the VLAN domain

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 vlan-domain domain-name Creates and enters the configuration mode for
the VLAN domain.
Example:
apic1(config)# vlan-domain
external-inband

Step 3 vlan vlan-id Assigns a VLAN to the domain.

Example:
apic1(config-vlan)# vlan 11

Step 4 exit Returns to global configuration mode.

Example:
apic1(config-vlan)# exit

Step 5 leaf node-id Specifies the leaf switch to which the

management station is connected.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

355
 Configuring Management Interfaces
Configuring Inband Management Connectivity to the Management Station

Command or Action Purpose

apic1(config)# leaf 102

Step 6 interface slot/port Specifies the port to which the management

station is connected.
Example:
apic1(config-leaf)# interface eth 1/2

Step 7 vlan-domain member external-inband Configures external layer2 connectivity to

inband management.
Example:
apic1(config-leaf-if)# vlan-domain
member external-inband

Step 8 switchport trunk allowed vlan vlan-id Configures external layer2 connectivity to
inband-mgmt gateway-ip/mask inband management. The specified IP address
is the gateway address used by the external
Example:
management station and the gateway
apic1(config-leaf-if)# switchport trunk functionality is provided by the ACI fabric.
allowed vlan 11 inband-mgmt
179.10.1.254/24

Step 9 exit
Example:
apic1(config-leaf-if)# exit

Step 10 exit
Example:
apic1(config-leaf)# exit

Examples
This example shows how to configure inband management connectivity to the management station.

# CREATE A VLAN DOMAIN FOR EXTERNAL CONNECTIVITY TO INBAND MANAGEMENT

apic1# configure
apic1(config)# vlan-domain external-inband
apic1(config-vlan)# vlan 11
apic1(config-vlan)# exit

# CONFIGURE LAYER 2 CONNECTIVITY FROM THE MANAGEMENT STATION INTERFACE TO INBAND MANAGEMENT
apic1(config)# leaf 102
apic1(config-leaf)# interface eth 1/2
apic1(config-leaf-if)# vlan-domain member external-inband
apic1(config-leaf-if)# switchport trunk allowed vlan 11 inband-mgmt 179.10.1.254/24
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit

What to do next
• Allow the necessary protocols (HTTPS and SSH) on the inbound management port.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

356
 Configuring Management Interfaces
Configuring Inband Management Contract to Open HTTPS/SSH Ports

Configuring Inband Management Contract to Open HTTPS/SSH Ports

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 tenant mgmt System Management policies are configured

under a special tenant called mgmt.
Example:
apic1(config)# tenant mgmt

Step 3 access-list inband-default Configures the access list filter for the inband
default policy.
Example:
apic1(config-tenant)# access-list
inband-default

Step 4 match tcp dest 443 Allows access on the management interface for
HTTPS traffic (TCP/443).
Example:
apic1(config-tenant-acl)# match tcp dest
443

Step 5 match tcp dest 22 Allows access on the management interface for
SSH traffic (TCP/22).
Example:
apic1(config-tenant-acl)# match tcp dest
22

Examples
This example shows how to allow HTTPS and SSH access to the inband management port.

apic1# configure
apic1(config)# tenant mgmt
apic1(config-tenant)# access-list inband-default
apic1(config-tenant-acl)# match tcp dest 443
apic1(config-tenant-acl)# match tcp dest 22
apic1(config-tenant-acl)# exit
apic1(config-tenant)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

357
 Configuring Management Interfaces
Configuring Inband Management Contract to Open HTTPS/SSH Ports

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

358
 CHAPTER 10
Configuring Security
• About Security Configuration, on page 359
• Configuring AAA, on page 360
• Configuring Security Servers, on page 363
• Configuring the Password Policy, on page 370
• Configuring Users, on page 373
• Configuring Public Key Infrastructure, on page 377
• Configuring Communication Policies, on page 382
• Configuring AES Encryption, on page 387
• Configuring Fabric Secure Mode, on page 388
• Configuring COOP Authentication, on page 389
• Configuring FIPS, on page 390
• Configuring Control Plane Policing, on page 392
• Configuring First Hop Security, on page 395
• Configuring 802.1x, on page 403

About Security Configuration

Access control is the way you control who is allowed access to the network server and what services they are
allowed to use once they have access. Authentication, authorization, and accounting (AAA) network security
services provide the primary framework through which you set up access control on APIC.

Overview of the AAA Configuration

To configure security on APIC using AAA, follow this process:
1. To use a separate security server, configure security protocol parameters using the radius-server ,
ldap-server , or tacacs-server configuration commands.
2. Define the method lists for authentication by using an aaa authentication command.
3. Apply the method lists to a particular interface or line, if required.
4. (Optional) Configure authorization using the aaa authentication command.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

359
 Configuring Security
Configuring AAA

Login Authentication Using a Local Password

Use the aaa authentication login command with the method argument to specify that APIC will use the
local username database for authentication. For example, to specify the local username database as the method
of user authentication at login when no other method list has been defined, enter the following commands:

apic1# configure
apic1(config)# aaa authentication login default
apic1(config-default)# realm local

For information about adding users into the local username database, refer to the section “Configuring a
Locally Authenticated User.”

Login Authentication Using a Remote Server

Use the aaa authentication login command with the server radius/tacacs/ldap method to specify
RADIUS/TACACS+/LDAP as the login authentication method. For example, to specify RADIUS as the
method of user authentication at login when no other method list has been defined, enter the following
commands:

apic1# configure
apic1(config)# aaa authentication login default
apic1(config-default)# realm radius

Before you can use RADIUS as the login authentication method, you need to enable communication with the
RADIUS security server, same is true for TACACS+ or LDAP. For more information about establishing
communication with a remote security server, see the appropriate chapter:
• "Configuring a RADIUS Server"
• "Configuring a TACACS+ Server"
• "Configuring an LDAP Server"

Configuring AAA
Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 aaa authentication login console Enters console configuration mode for users
accessing APIC through the console.
Example:
apic1(config)# aaa authentication login
console

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

360
Configuring Security
Configuring AAA

Command or Action Purpose

Step 3 [no] realm {ldap | local | radius | tacacs} Specifies the authentication method.
Example:
apic1(config-console)# realm radius

Step 4 [no] group group-name Specifies an authentication server group.

Example:
apic1(config-console)# group
radiusGroup5

Step 5 exit Returns to global configuration mode.

Example:
apic1(config-console)# exit

Step 6 aaa authentication login default Enters the configuration mode for default login
authentication.
Example:
apic1(config)# aaa authentication login
default

Step 7 [no] realm {ldap | local | radius | tacacs} Specifies the authentication method.
Example:
apic1(config-default)# realm radius

Step 8 [no] group group-name Specifies an authentication server group.

Example:
apic1(config-default)# group radiusGroup

Step 9 exit Returns to global configuration mode.

Example:
apic1(config-default)# exit

Step 10 aaa authentication login domain Enters the configuration mode for default login
{domain-name | fallback} authentication. A login domain specifies the
authentication domain for a user.
Example:
apic1(config)# aaa authentication login
domain cisco

Step 11 [no] realm {ldap | local | none | radius | Specifies the authentication method.
tacacs}
Example:
apic1(config-domain)# realm radius

Step 12 [no] group group-name Specifies an authentication server group.

Example:
apic1(config-domain)# group radiusGroup

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

361
 Configuring Security
Configuring AAA

Command or Action Purpose

Step 13 exit Returns to global configuration mode.
Example:
apic1(config-domain)# exit

Step 14 aaa banner text Specifies the informational banner to be

displayed before the user login. The banner
Example:
must be contained in single quotes.
apic1(config)# aaa banner 'Welcome to
APIC'

Step 15 aaa group {ldap | radius | tacacs} Creates or configures an authentication server
group-name group.
Example:
apic1(config)# aaa group radius
radiusGroup

Step 16 [no] server {ip-address | hostname} priority Adds a server to the authentication server
priority-number group and specifies its priority within the
server group. The priority can be between 0
Example:
and 17.
apic1(config-radius)# server 192.0.20.71
priority 2

Step 17 exit Returns to global configuration mode.

Example:
apic1(config-radius)# exit

Step 18 aaa scvmm-certificate certificate-name Specifies an SCVMM certificate. See the Cisco
ACI Virtualization Guide.
Example:
apic1(config)# aaa scvmm-certificate
myScvmmCert

Step 19 aaa user default-role {assign-default-role | Specifies how to respond when remote users
no-login} who do not have a user role attempt to log in
to APIC. The action can be either of these
Example:
options:
apic1(config)# aaa user default-role
assign-default-role • assign-default-role —Remote users who
do not have a user role are assigned a
default role.
• no-login —Remote users who do not
have a user role cannot log in.

Step 20 show aaa authentication Displays configured AAA methods.

Example:
apic1(config)# show aaa authentication

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

362
 Configuring Security
Configuring Security Servers

Command or Action Purpose

Step 21 show aaa groups Displays configured AAA server groups.
Example:
apic1(config)# show aaa groups

Examples
This example shows how to configure AAA.

apic1# configure terminal

apic1(config)# aaa authentication login console
apic1(config-console)# realm local
apic1(config-console)# exit
apic1(config)# aaa authentication login default
apic1(config-default)# realm radius
apic1(config-default)# group radiusGroup5
apic1(config-default)# exit
apic1(config)# aaa authentication login domain cisco
apic1(config-domain)# realm none
apic1(config-domain)# exit
apic1(config)# aaa banner 'Welcome to APIC'
apic1(config)# aaa group radius radiusGroup
apic1(config-radius)# server 192.0.20.71 priority 2
apic1(config-radius)# exit
apic1(config)# aaa user default-role assign-default-role
apic1(config)# show aaa authentication
Default : radius
Console : local

apic1(config)# show aaa groups

Total number of Groups : 1

RadiusGroups : radiusGroup5
TacacsGroups :
LdapGroups :

Configuring Security Servers

Configuring a RADIUS Server
Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

363
 Configuring Security
Configuring a RADIUS Server

Command or Action Purpose

Step 2 [no] radius-server retries count Specifies how many times APIC transmits each
RADIUS request to the server before giving
Example:
up. The range is 0 to 5.
apic1(config)# radius-server retries 1
In the global configuration mode, this
command applies to all RADIUS servers
unless overridden in the specific RADIUS host
configuration.

Step 3 [no] radius-server timeout seconds Specifies the number of seconds APIC waits
for a reply to a RADIUS request before
Example:
retransmitting the request.
apic1(config)# radius-server timeout 5
In the global configuration mode, this
command applies to all RADIUS servers
unless overridden in the specific RADIUS host
configuration.

Step 4 [no] radius-server host {ip-address | Specifies the IP address or hostname of the
hostname} RADIUS server.
Example:
apic1(config)# radius-server host
192.0.20.71

Step 5 (Optional) [no] retries count For this RADIUS server, specifies how many
times APIC transmits each RADIUS request
Example:
to the server before giving up. The range is 0
apic1(config-host)# retries 2 to 5.
If no retry count is set, the global value is used.

Step 6 (Optional) [no] timeout seconds For this RADIUS server, specifies the number
of seconds APIC waits for a reply to a
Example:
RADIUS request before retransmitting the
apic1(config-host)# timeout 3 request.
If no timeout is set, the global value is used.

Step 7 (Optional) [no] descr text Provides descriptive information about this
RADIUS server. The text can be up to 128
Example:
alphanumeric characters. If the text contains
apic1(config-host)# descr "My primary spaces, it must be enclosed by single or double
RADIUS server"
quotes.

Step 8 [no] key key-value Specifies the shared secret text string used
between APIC and this RADIUS server for
Example:
authentication. The key can be up to 32
apic1(config-host)# key myRaDiUSpassWoRd characters.

Step 9 [no] port port-number Specifies a UDP port on this RADIUS server
to be used solely for authentication.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

364
Configuring Security
Configuring a RADIUS Server

Command or Action Purpose

apic1(config-host)# port 1812

Step 10 [no] protocol {chap | mschap | pap} Specifies the RADIUS server protocol for
authentication.
Example:
apic1(config-host)# protocol pap

Step 11 exit Returns to global configuration mode.

Example:
apic1(config-host)#

Step 12 show radius-server (Optional) Displays the RADIUS server

information.
Example:
apic1(config)# show radius-server

Examples
This example shows how to configure RADIUS settings globally and on one RADIUS server.

apic1# configure
apic1(config)# radius-server retries 1
apic1(config)# radius-server timeout 5
apic1(config)# radius-server host 192.0.20.71
apic1(config-host)# retries 2
apic1(config-host)# timeout 3
apic1(config-host)# descr "My primary RADIUS server"
apic1(config-host)# key myRaDiUSpassWoRd
apic1(config-host)# port 1812
apic1(config-host)# protocol pap
apic1(config-host)# exit
apic1(config)# show radius-server
timeout : 5
retries : 1

Total number of servers : 1

Hostname : 192.0.20.71
Port : 1812
Protocol : pap
Timeout : 3
Retries : 2
User : test
Descr : My primary RADIUS server

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

365
 Configuring Security
Configuring a TACACS+ Server

Configuring a TACACS+ Server

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] tacacs-server retries count Specifies how many times APIC transmits each
TACACS+ request to the server before giving
Example:
up. The range is 0 to 5.
apic1(config)# tacacs-server retries 1
In the global configuration mode, this
command applies to all TACACS+ servers
unless overridden in the specific TACACS+
host configuration.

Step 3 [no] tacacs-server timeout seconds Specifies the number of seconds APIC waits
for a reply to a TACACS+ request before
Example:
retransmitting the request.
apic1(config)# tacacs-server timeout 5
In the global configuration mode, this
command applies to all TACACS+ servers
unless overridden in the specific TACACS+
host configuration.

Step 4 [no] tacacs-server host {ip-address | Specifies the IP address or hostname of the
hostname} TACACS+ server.
Example:
apic1(config)# tacacs-server host
192.0.20.71

Step 5 (Optional) [no] retries count For this TACACS+ server, specifies how many
times APIC transmits each TACACS+ request
Example:
to the server before giving up. The range is 0
apic1(config-host)# retries 2 to 5.
If no retry count is set, the global value is used.

Step 6 [no] key Specifies the shared secret text string used
between APIC and this TACACS+ server for
Example:
authentication. The key can be up to 32
apic1(config-host)# key characters. For increased security, entering the
Enter key: myTacAcSpassWoRd
Enter key again: myTacAcSpassWoRd key value is interactive.

Step 7 [no] port port-number Specifies a UDP port on this TACACS+ server
to be used for TACACS+ accounting
Example:
messages.
apic1(config-host)# port 49

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

366
 Configuring Security
Configuring an LDAP Server

Command or Action Purpose

Step 8 [no] protocol {chap | mschap | pap} Specifies the TACACS+ server protocol for
authentication.
Example:
apic1(config-host)# protocol pap

Step 9 exit Returns to global configuration mode.

Example:
apic1(config-host)#

Step 10 show tacacs-server (Optional) Displays the TACACS+ server

information.
Example:
apic1(config)# show tacacs-server

Examples
This example shows how to configure TACACS+ settings globally and on one TACACS+ server.

apic1# configure
apic1(config)# tacacs-server retries 1
apic1(config)# tacacs-server timeout 5
apic1(config)# tacacs-server host 192.0.20.72
apic1(config-host)# retries 2
apic1(config-host)# timeout 3
apic1(config-host)# key myTaCaCspassWoRd
apic1(config-host)# port 49
apic1(config-host)# protocol pap
apic1(config-host)# exit
apic1(config)# show tacacs-server
timeout : 5
retries : 1

Total number of servers : 1

Hostname : 192.0.20.72
Port : 1812
Protocol : pap
Timeout : 3
Retries : 2
User : test

Configuring an LDAP Server

Some ldap-server commands can be entered in either the global configuration mode or in the configuration
mode for a specific LDAP host. In the global configuration mode, the command applies to all LDAP servers
unless overridden in the specific LDAP host configuration.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

367
 Configuring Security
Configuring an LDAP Server

Procedure

Command or Action Purpose

Step 1 configure terminal Enters global configuration mode.
Example:
switch# configure terminal

Step 2 [no] ldap-server host {ip-address | hostname} Specifies the IP address or hostname of the
LDAP server and enters the configuration
Example:
mode of that server.
apic1(config)# ldap-server host
192.0.20.73

Step 3 [no] ldap-server attribute attribute-name Specifies an LDAP endpoint attribute to be

used as the CiscoAVPair.
Example:
apic1(config-host)# ldap-server In the global configuration mode, this
attribute memberOf command applies to all LDAP servers unless
overridden in the specific LDAP host
configuration.

Step 4 [no] ldap-server basedn Specifies the location in the LDAP hierarchy
where the server should begin searching when
Example:
it receives an authorization request. This can
apic1(config-host)# ldap-server basedn be a string of up to 127 characters. Spaces are
DC=sampledesign,DC=com
not permitted in the string, but other special
characters are allowed.
In the global configuration mode, this
command applies to all LDAP servers unless
overridden in the specific LDAP host
configuration.

Step 5 [no] ldap-server binddn Specifies the distinguished name (DN) for an
LDAP database account that has read and
Example:
search permissions for all objects under the
apic1(config-host)# ldap-server binddn base DN. This can be a string of up to 127

CN=ucsbind,OU=CiscoUsers,DC=sampledesign,DC=com
characters. Spaces are not permitted in the
string, but other special characters are allowed.

Step 6 [no] ldap-server retries count Specifies how many times APIC transmits each
LDAP request to the server before giving up.
Example:
The range is 0 to 5.
apic1(config-host)# ldap-server retries
1 In the global configuration mode, this
command applies to all LDAP servers unless
overridden in the specific LDAP host
configuration.

Step 7 [no] ldap-server timeout seconds Specifies the number of seconds APIC waits
for a reply to a LDAP request before
Example:
retransmitting the request.
apic1(config-host)# ldap-server timeout
30

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

368
Configuring Security
Configuring an LDAP Server

Command or Action Purpose

In the global configuration mode, this
command applies to all LDAP servers unless
overridden in the specific LDAP host
configuration.

Step 8 [no] ldap-server filter filter-expression Specifies a filter to filter the results of LDAP
searches. The filter can contain a maximum of
Example:
63 characters.
apic1(config-host)# ldap-server filter
sAMAccountName=$userid In the global configuration mode, this
command applies to all LDAP servers unless
overridden in the specific LDAP host
configuration.

Step 9 [no] key key-value Specifies the shared secret text string used
between APIC and this LDAP server for
Example:
authentication. The key can be up to 32
apic1(config-host)# key characters.
Enter key: myLdAppassWoRd
Enter key again: myLdAppassWoRd

Step 10 [no] port port-number Specifies the LDAP server port for
authentication.
Example:
apic1(config-host)# port 389

Step 11 (Optional) [no] retries count For this LDAP server, specifies how many
times APIC transmits each LDAP request to
Example:
the server before giving up. The range is 0 to
apic1(config-host)# retries 2 5.
If no retry count is set, the global value is used.

Step 12 [no] enable-ssl Enables an SSL connection with the LDAP

provider.
Example:
apic1(config-host)# enable-ssl

Step 13 [no] ssl-validation-level [permissive | strict] Sets the LDAP Server SSL Certificate
validation level.
Example:
apic1(config-host)# ssl-validation-level
permissive

Step 14 (Optional) [no] timeout seconds For this LDAP server, specifies the number of
seconds APIC waits for a reply to a LDAP
Example:
request before retransmitting the request.
apic1(config-host)# timeout 3
If no timeout is set, the global value is used.

Step 15 exit Returns to global configuration mode.

Example:
apic1(config-host)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

369
 Configuring Security
Configuring the Password Policy

Command or Action Purpose

Step 16 show ldap-server
Example:
apic1(config)# show ldap-server

Examples
This example shows how to configure LDAP server settings globally and on one LDAP server.

apic1# configure
apic1(config)# ldap-server retries 1
apic1(config)# ldap-server timeout 30
apic1(config)# ldap-server host 192.0.20.73
apic1(config-host)# retries 2
apic1(config-host)# timeout 3
apic1(config-host)# filter sAMAccountName=$userid
apic1(config-host)# key myLdAppassWoRd
apic1(config-host)# ssl-validation-level permissive
apic1(config-host)# enable-ssl
apic1(config-host)# port 389
apic1(config-host)# exit
apic1(config)# show ldap-server
timeout : 30
retries : 1
filter : sAMAccountName=$userid

Total number of servers : 1

Hostname : 192.0.20.73
Port : 389
Timeout : 3
Retries : 2
SSL : yes
SSL Level : permissive
User : test

Configuring the Password Policy

The password policy configuration in this topic set the password history and password change interval properties
for all locally authenticated APIC users. You cannot specify different password policies for each locally
authenticated user.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

370
Configuring Security
Configuring the Password Policy

Command or Action Purpose

Step 2 [no] password change-count count Sets the number of password changes allowed
within the change interval. The range is 0 to 10
Example:
changes.
apic1(config)# password change-count 5

Step 3 [no] password change-during-interval Enables or disables restricting the number of

{enable | disable} password changes a locally authenticated user
can make within the change interval.
Example:
apic1(config)# password
change-during-interval enable

Step 4 [no] password change-interval hours When the change-during-interval is enabled,

restricts the number of password changes a
Example:
locally authenticated user can make within a
apic1(config)# password change-interval given number of hours. The range is 1 to 745
300
hours.

Step 5 [no] password no-change-interval hours Sets a minimum period before which a user
cannot change the password again. The range
Example:
is 1 to 745 hours.
apic1(config)# password
no-change-interval 60

Step 6 password expiration-warn-time Sets a warning period before password

expiration to display warning. The range is 0 to
Example:
30 days.
apic1(config)# password
expiration-warn-time 5

Step 7 [no] password history-count count The password history count allows you to
prevent locally authenticated users from reusing
Example:
the same password over and over again. When
apic1(config)# password history-count 10 this property is configured, APIC stores
passwords that were previously used by locally
authenticated users up to a maximum of 15
passwords. The passwords are stored in reverse
chronological order with the most recent
password first to ensure that the only the oldest
password can be reused when the history count
threshold is reached.
A user must create and use the number of
passwords configured in the password history
count before being able to reuse one. For
example, if you set the password history count
to 8, a locally authenticated user cannot reuse
the first password until after the ninth password
has expired.
By default, the password history is set to 0. This
value disables the history count and allows users
to reuse previous passwords at any time. If

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

371
 Configuring Security
Configuring the Password Policy

Command or Action Purpose

necessary, you can clear a user's password
history using the clear-pwd-history command
in the username configuration mode for that
user.

Step 8 [no] password pwd-strength-check Enforces strong passwords for all users.
Example:
apic1(config)# password
pwd-strength-check

Examples
This example shows how to configure global password settings for locally authenticated users.

apic1# configure
apic1(config)# password change-count 5
apic1(config)# password change-during-interval enable
apic1(config)# password change-interval 300
apic1(config)# password no-change-interval 60
apic1(config)# password expiration-warn-time 5
apic1(config)# password history-count 10
apic1(config)# password pwd-strength-check

This example shows how to prevent the password from being changed within 48 hours after a locally
authenticated user changes his or her password.

apic1# configure
apic1(config)# password change-during-interval disable
apic1(config)# password no-change-interval 48

This example shows how to allow the password to be changed a maximum of once within 24 hours
after a locally authenticated user changes his or her password

apic1# configure
apic1(config)# password change-count 1
apic1(config)# password change-during-interval enable
apic1(config)# password change-interval 24

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

372
 Configuring Security
Configuring Users

Configuring Users
Configuring a Locally Authenticated User
Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 username {name | admin} Creates a locally-authenticated user account

or configures an existing user. The name can
Example:
be a maximum of 28 characters.
apic1(config)# username user5

Step 3 [no] first-name first Sets the first name of this user.
Example:
apic1(config-username)# first-name
George

Step 4 [no] last-name last Sets the last name of this user.
Example:
apic1(config-username)# last-name
Washington

Step 5 [no] email email-address Sets the email address of this user.
Example:
apic1(config-username)# email
[email protected]

Step 6 [no] phone phone-number Sets the phone number of this user.
Example:
apic1(config-username)# phone
14085551212

Step 7 [no] account-status {active | inactive | Activates or deactivates this user account.
status}
Example:
apic1(config-username)# account-status
active

Step 8 clear-pwd-history Clears the user's password history list and

allows this user to reuse previous passwords.
Example:
apic1(config-username)#
clear-pwd-history

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

373
 Configuring Security
Configuring a Locally Authenticated User

Command or Action Purpose

Step 9 [no] expires Enables expiration of this user account at the
date and time configured by the expiration
Example:
command.
apic1(config-username)# expires

Step 10 expiration date-time Sets an expiration date and time for this user
account. The format is UTC Date format
Example:
(YYYY-MM-DDThh:mmTZD). You must
apic1(config-username)# expiration also enable expiration by configuring the
2017-12-31T23:59+08:00
expires command.

Step 11 password password Sets the user password.

Example: Note Special characters such as '$' or '!'
apic1(config-username)# password should be escaped with a backslash
c1\$c0123 ('\$') in this command to avoid
misinterpretation by Bash. The
escape backslash is necessary only
when setting the password in this
command; the user does not enter
the backslash when logging in.

Step 12 [no] pwd-lifetime days Sets the lifetime of the user password. The
range is 0 to 3650 days.
Example:
apic1(config-username)# pwd-lifetime 90

Step 13 [no] domain {all | common | mgmt | Specifies or creates the AAA domain to which
domain-name} this user belongs.
Example:
apic1(config-username)# domain
mySecDomain

Step 14 [no] role role Creates the AAA domain role to set privilege
bitmask of a user domain.
Example:
apic1(config-domain)# role tenant-admin

Step 15 [no] priv-type {readPriv | writePriv} Creates the AAA domain role to set privilege
bitmask of a user domain.
Example:
apic1(config-role)# priv-type writePriv

Step 16 exit Returns to domain configuration mode.

Example:
apic1(config-role)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

374
 Configuring Security
Configuring a Certificate and SSH-Key for a Local User

Command or Action Purpose

Step 17 exit Returns to username configuration mode.
Example:
apic1(config-domain)# exit

Step 18 show username name Displays configuration details about this user.
Example:
apic1(config-username)# show username
user5

Examples
This example shows how to configure a local user.

apic1# configure terminal

apic1(config)# username user5
apic1(config-username)# first-name George
apic1(config-username)# last-name Washington
apic1(config-username)# email [email protected]
apic1(config-username)# phone 14085551212
apic1(config-username)# account-status active
apic1(config-username)# domain mySecDomain
apic1(config-username)# clear-pwd-history
apic1(config-username)# expires
apic1(config-username)# expiration 2017-12-31T23:59+08:00
apic1(config-username)# password c1$c0123
apic1(config-username)# pwd-lifetime 90
apic1(config-username)# domain mySecDomain
apic1(config-domain)# role tenant-admin
apic1(config-role)# priv-type writePriv
apic1(config-role)# exit
apic1(config-domain)# exit
apic1(config-username)# show username user5
UserName : user5
First-Name : George
Last-Name : Washington
Email : [email protected]
Acount Status : active
Password strength check : yes

What to do next
To configure an SSH key or certificate for the local user, see "Configuring Certificates and SSH-Keys."

Configuring a Certificate and SSH-Key for a Local User

This topic describes how to configure a certificate or an SSH key so that a local user can log in without being
prompted for a password.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

375
 Configuring Security
Configuring a Certificate and SSH-Key for a Local User

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 username {name | admin} Creates a locally-authenticated user account or

configures an existing user. The name can be a
Example:
maximum of 28 characters.
apic1(config)# username user5

Step 3 [no] certificate certificate-name Enters certificate configuration mode.

Example:
apic1(config-username)# certificate
myCertificate

Step 4 data certificate-data Sets PEM-encoded certificate.

Example:
apic1(config-certificate)# data
-----BEGIN CERTIFICATE-----MIIC4j.....

Step 5 exit Returns to username configuration mode.

Example:
apic1(config-certificate)# exit

Step 6 [no] ssh-key ssh-key-name Sets an SSH key to log in using the SSH client
without being prompted for a password.
Example:
apic1(config-username)# ssh-key mySSHkey

Step 7 data key-data Sets the SSH key. The key can be up to 64
characters.
Example:
apic1(config-ssh-key)# data
AAAAB3NzaC1yc2EAA......

Step 8 exit Returns to username configuration mode.

Example:
apic1(config-ssh-key)# exit

Examples
This example shows how to configure an SSH key and a certificate for a local user.

apic1# configure terminal

apic1(config)# username user5
apic1(config-username)# certificate myCertificate
apic1(config-certificate)# data -----BEGIN CERTIFICATE-----MIIC4j.....

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

376
 Configuring Security
Configuring Public Key Infrastructure

apic1(config-certificate)# exit
apic1(config-username)# ssh-key mySSHkey
apic1(config-ssh-key)# data AAAAB3NzaC1yc2EAA...
apic1(config-ssh-key)# exit

Configuring Public Key Infrastructure

Configuring a Certificate Authority and Chain of Trust
Certificate authorities (CAs) manage certificate requests and issue certificates to participating entities such
as hosts, network devices, or users. APIC locally stores the self-signed root certificate of the trusted CA (or
certificate chain for a subordinate CA). The stored information about a trusted CA is called the trustpoint and
the CA itself is called a trustpoint CA.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] crypto ca trustpoint-name Enters configuration mode for the specified
trustpoint certificate authority (CA).
Example:
apic1(config)# crypto ca myCA

Step 3 [no] cert-chain pem-data Stores the certificate chain in PEM format.
Enter the entire chain of trust from the trustpoint
Example:
to a trusted root authority.
apic1(config-ca)# cert-chain -----BEGIN
CERTIFICATE----- MIIC4jCCAoygAw.....

Examples
This example shows how to configure a CA.

apic1# configure

apic1(config)# crypto ca myCA

apic1(config-ca)# cert-chain -----BEGIN CERTIFICATE----- MIIC4jCCAoygAw.....

Configuring Keys and a Keyring

You can obtain an identity certificate for APIC by generating an RSA key pair and associating the key pair
with a trustpoint CA where APIC intends to enroll. The RSA keys are stored by APIC in a crypto keyring.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

377
 Configuring Security
Configuring Keys and a Keyring

The APIC software allows you to generate an RSA key pair with a configurable key size (or modulus). The
default key size is 512. You can also configure an RSA key-pair label. The default key label is the device
fully qualified domain name (FQDN).

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] crypto keyring {default | keyring-name} Creates or configures a keyring to hold an SSL
certificate.
Example:
apic1(config)# crypto keyring myKeyring

Step 3 regen Forces regeneration of the RSA key pair.

Example:
apic1(config-keyring)# regen

Step 4 [no] cert certificate-data Imports a certificate containing a public key

and signed information. The certificate data
Example:
must be enclosed in quotes.
apic1(config-keyring)# cert "-----BEGIN
CERTIFICATE----- MIIC4jCCAoygAw.....

Step 5 [no] tp certificate-name Sets a third-party certificate from a trusted

source for device identity.
Example:
apic1(config-keyring)# tp myCertificate

Step 6 [no] key key-data Creates the private key of the certificate.
Example:
apic1(config-keyring)# key
XXXXXXXXXXXXXXXXXXXXXXX

Step 7 [no] modulus {mod512 | mod1024 | mod1536 Sets the length of the encryption keys.
| mod2048}
Example:
apic1(config-keyring)# modulus mod1024

Step 8 exit Returns to global configuration mode.

Example:
apic1(config-keyring)# exit

Examples
This example shows how to configure a keyring.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

378
 Configuring Security
Generating a Certificate Signing Request

apic1# configure
apic1(config)# crypto keyring myKeyring
apic1(config-keyring)# cert "-----BEGIN CERTIFICATE----- MIIC4jCCAoygAw.....
apic1(config-keyring)# tp myCertificate
apic1(config-keyring)# key XXXXXXXXXXXXXXXXXXXXXXX
apic1(config-keyring)# modulus mod1024
apic1(config-keyring)# exit

Generating a Certificate Signing Request

A certificate signing request (CSR) is a message that an applicant sends to a CA in order to apply for a digital
identity certificate. Before a CSR is created, the applicant first generates a key pair, which keeps the private
key secret. The CSR contains information that identifies the applicant, such as the public key generated by
the applicant. The corresponding private key is not included in the CSR, but is used to digitally sign the entire
request.

Before you begin

Before generating a certificate signing request (CSR), you must configure a trustpoint certificate authority
(CA) and generate a key pair.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] crypto keyring {default | keyring-name} Creates or configures a keyring to hold an SSL
certificate.
Example:
apic1(config)# crypto keyring default

Step 3 csr Creates a certificate signing request for this

keyring.
Example:
apic1(config-keyring)# csr

Step 4 subj-name name Sets the fully qualified domain name or

distinguished name of the requesting device.
Example:
The name can be up to 64 characters.
apic1(config-csr)# subj-name
www.exampleCorp.com

Step 5 [no] cert certificate-data Imports a certificate containing a public key

and signed information. The certificate data
Example:
must be enclosed in quotes.
apic1(config-csr)# cert "-----BEGIN
CERTIFICATE----- MIIC4jCCAoygAw.....

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

379
 Configuring Security
Generating a Certificate Signing Request

Command or Action Purpose

Step 6 password Sets the new password.
Example:
apic1(config-csr)# password
Enter password: c1$c0123
Enter password again: c1$c0123

Step 7 org-name Sets the full legal name of the organization.

Example:
apic1(config-csr)# org-name ExampleCorp

Step 8 org-unit-name Sets the department or unit name within the

organization.
Example:
apic1(config-csr)# org-unit-name Sales

Step 9 email Sets the email address of the organization

contact person.
Example:
apic1(config-csr)# email
[email protected]

Step 10 locality city-name Sets the city or town of the organization.

Example:
apic1(config-csr)# locality SanJose

Step 11 state state Sets the state or province in which the

organization is located.
Example:
apic1(config-csr)# state CA

Step 12 country country-code Sets the two-letter ISO code for the country
where the organization is located.
Example:
apic1(config-csr)# country US

Step 13 exit Returns to keyring configuration mode.

Example:
apic1(config-csr)# exit

Examples
This example shows how to generate a certificate signing request (CSR).

apic1# configure
apic1(config)# crypto keyring default
apic1(config-keyring)# csr
apic1(config-csr)# subj-name www.exampleCorp.com
apic1(config-csr)# cert "-----BEGIN CERTIFICATE----- MIIC4jCCAoygAw.....
apic1(config-csr)# pwd c1$c0123

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

380
 Configuring Security
Configuring Webtokens

apic1(config-csr)# org-name ExampleCorp

apic1(config-csr)# org-unit-name Sales
apic1(config-csr)# email [email protected]
apic1(config-csr)# locality SanJose
apic1(config-csr)# state CA
apic1(config-csr)# country US
apic1(config-csr)# exit

What to do next
Submit the CSR to a CA.

Configuring Webtokens
Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] crypto webtoken

Example:
apic1(config)# crypto webtoken

Step 3 [no] max-validity-period hours Sets the maximum validity period for a
webtoken. The range is 4 to 24 hours.
Example:
apic1(config-webtoken)#
max-validity-period 10

Step 4 [no] session-record-flags csv-list Enables or disables refresh in the session

records. The session record flags are specified
Example:
as a comma-separated value list of one or more
apic1(config-webtoken)# of the following flags: login , logout , and
session-record-flags login,refresh
refresh .

Step 5 [no] ui-idle-timeout-seconds seconds Sets the maximum GUI idle duration before
requiring login refresh. The range is 60 to 65525
Example:
seconds.
apic1(config-webtoken)#
ui-idle-timeout-seconds 120

Step 6 [no] webtoken-timeout-seconds seconds Sets the webtoken timeout interval. The range
is 600 to 9600 seconds.
Example:
apic1(config-webtoken)#
webtoken-timeout-seconds 1200

Step 7 exit
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

381
 Configuring Security
Configuring Communication Policies

Command or Action Purpose

apic1(config-webtoken)# exit

Examples
This example shows how to configure a webtoken.

apic1# configure
apic1(config)# crypto webtoken
apic1(config-webtoken)# max-validity-period 10
apic1(config-webtoken)# session-record-flags login,refresh
apic1(config-webtoken)# ui-idle-timeout-seconds 120
apic1(config-webtoken)# webtoken-timeout-seconds 1200

Configuring Communication Policies

Configuring the HTTP Policy
Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] comm-policy {default | policy-name} Enters communication policy configuration

mode.
Example:
apic1(config)# comm-policy myCommPolicy

Step 3 http Enters HTTP policy configuration mode.

Example:
apic1(config-comm-policy)# http

Step 4 [no] admin-state-enable Enables HTTP communication service.

Example:
apic1(config-http)# admin-state-enable

Step 5 [no] allow-origin url Specifies the URL to return in the

Access-Control-Allow-Origin HTTP header.
Example:
apic1(config-http)# allow-origin
www.example.com

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

382
 Configuring Security
Configuring the HTTPS Policy

Command or Action Purpose

Step 6 [no] port port-number Sets the port used for HTTP communication
service.
Example:
apic1(config-http)# port 8080

Step 7 [no] redirect Enables HTTP redirection.

Example:
apic1(config-http)# no redirect

Step 8 [no] request-status-count count Sets the maximum count of HTTP requests to
track. The range is 0 to 10240.
Example:
apic1(config-http)# request-status-count
512

Step 9 exit Returns to communications policy configuration

mode.
Example:
apic1(config-http)# exit

Examples
This example shows how to configure HTTP service.

apic1# configure
apic1(config)# comm-policy myCommPolicy
apic1(config-comm-policy)# http
apic1(config-http)# admin-state-enable
apic1(config-http)# allow-origin www.example.com
apic1(config-http)# port 8080
apic1(config-http)# no redirect
apic1(config-http)# request-status-count 512
apic1(config-http)# exit

Configuring the HTTPS Policy

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] comm-policy {default | policy-name} Enters communication policy configuration

mode.
Example:
apic1(config)# comm-policy myCommPolicy

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

383
 Configuring Security
Configuring the HTTPS Policy

Command or Action Purpose

Step 3 https Enters HTTPS policy configuration mode.
Example:
apic1(config-comm-policy)# https

Step 4 [no] admin-state-enable Enables HTTPS communication service.

Example:
apic1(config-https)# admin-state-enable

Step 5 [no] port port-number Sets the port used for HTTPS communication
service.
Example:
apic1(config-https)# port 443

Step 6 [no] request-status-count count Sets the maximum count of HTTPS requests to
track. The range is 0 to 10240.
Example:
apic1(config-https)# request-status-count
512

Step 7 [no] ssl-protocols {TLSv1 | TLSv1.1 | Specifies in a comma-separated list the SSL
TLSv1.2} protocols that are supported. The options are
TLSv1 , TLSv1.1 , and TLSv1.2 .
Example:
apic1(config-https)# ssl-protocols
TLSv1.1,TLSv1.2

Step 8 [no] use-keyring keyring-name Specifies a keyring to use for the HTTPS server
SSL certificate.
Example:
apic1(config-https)# use-keyring
myKeyRing

Step 9 exit Returns to communications policy configuration

mode.
Example:
apic1(config-https)# exit

Examples
This example shows how to configure HTTPS service.

apic1# configure
apic1(config)# comm-policy myCommPolicy
apic1(config-comm-policy)# https
apic1(config-https)# admin-state-enable
apic1(config-https)# port 443
apic1(config-https)# request-status-count 512
apic1(config-https)# ssl-protocols TLSv1.1,TLSv1.2
apic1(config-https)# use-keyring myKeyRing
apic1(config-https)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

384
 Configuring Security
Configuring the SSH Policy

Configuring the SSH Policy

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] comm-policy {default | policy-name} Enters communication policy configuration

mode.
Example:
apic1(config)# comm-policy myCommPolicy

Step 3 ssh-service Enters SSH policy configuration mode.

Example:
apic1(comm-policy)# ssh-service

Step 4 [no] admin-state-enable Enables HTTP communication service.

Example:
apic1(config-ssh-service)#
admin-state-enable

Step 5 [no] port port-number Sets the port used for SSH communication
service.
Example:
apic1(config-ssh-service)# port 22

Step 6 exit Returns to communications policy configuration

mode.
Example:
apic1(config-ssh-service)# exit

Examples
This example shows how to configure SSH service.

apic1# configure
apic1(config)# comm-policy myCommPolicy
apic1(config-comm-policy)# ssh-service
apic1(config-ssh-service)# admin-state-enable
apic1(config-ssh-service)# port 22
apic1(config-ssh-service)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

385
 Configuring Security
Configuring the Telnet Policy

Configuring the Telnet Policy

Before you begin
To allow telnet communications, you must configure an out-of-band contract allowing telnet traffic, which
is normally on TCP and UDP ports 23.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] comm-policy {default | policy-name} Enters communication policy configuration

mode.
Example:
apic1(config)# comm-policy myCommPolicy

Step 3 telnet Enters Telnet policy configuration mode.

Example:
apic1(config-comm-policy)# telnet

Step 4 [no] admin-state-enable Enables Telnet communication service.

Example:
apic1(config-telnet)# admin-state-enable

Step 5 [no] port port-number Sets the port used for Telnet communication
service.
Example:
apic1(config-telnet)# port 23

Step 6 exit Returns to communications policy configuration

mode.
Example:
apic1(config-telnet)# exit

Examples
This example shows how to configure Telnet service.

apic1# configure
apic1(config)# comm-policy myCommPolicy
apic1(config-comm-policy)# telnet
apic1(config-telnet)# admin-state-enable
apic1(config-telnet)# port 23
apic1(config-telnet)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

386
 Configuring Security
Configuring AES Encryption

Configuring AES Encryption

Beginning with Cisco APIC Release 1.1(2), the secure properties of APIC configuration files can be encrypted
by enabling AES-256 encryption. AES encryption is a global configuration option; all secure properties
conform to the AES configuration setting. It is not possible to export a subset of the ACI fabric configuration
such as a tenant configuration with AES encryption while not encrypting the remainder of the fabric
configuration. For a list of secure properties, see "Appendix K: Secure Properties" in Cisco Application Centric
Infrastructure Fundamentals.
The APIC uses a 16 to 32 character passphrase to generate the AES-256 keys. The APIC GUI displays a hash
of the AES passphrase. This hash can be used to see whether the same passphrase is used on two ACI fabrics.
This hash can be copied to a client computer where it can be compared to the passphrase hash of another ACI
fabric to see if they were generated with the same passphrase. The hash cannot be used to reconstruct the
original passphrase or the AES-256 keys.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 crypto aes Enters AES configuration mode.

Example:
apic1(config)# crypto aes

Step 3 (Optional) clear-encryption-key Deletes any existing AES encryption key.

Example:
apic1(config-aes)# clear-encryption-key

Step 4 passphrase Specifies the AES encryption passphrase. The

passphrase can be 16 to 32 characters and must
Example:
be enclosed in quotes. For increased security,
apic1(config-aes)# passphrase entering the passphrase is interactive.
Enter passphrase: "This is my passphrase"
Enter passphrase again: "This is my
passphrase"

Step 5 [no] encryption Enables (or disables) AES encryption.

Example:
apic1(config-aes)# encryption

Examples
This example shows how to enable AES encryption and configure a passphrase.

apic1# configure
apic1(config)# crypto aes

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

387
 Configuring Security
Configuring Fabric Secure Mode

apic1(config-aes)# clear-encryption-key
apic1(config-aes)# passphrase "This is my passphrase"
apic1(config-aes)# encryption

Configuring Fabric Secure Mode

Fabric secure mode prevents parties with physical access to the fabric equipment from adding a switch or
APIC controller to the fabric without manual authorization by an administrator. Starting with Cisco APIC
Release 1.2(1x), the firmware checks that switches and controllers in the fabric have valid serial numbers
associated with a valid Cisco digitally signed certificate. This validation is performed upon upgrade to this
release or during an initial installation of the fabric. The default setting for this feature is permissive mode;
an existing fabric continues to run as it has after an upgrade to Release 1.2(1). An administrator with fabric-wide
access rights must enable strict mode.
Permissive Mode (default) operates as follows:
• Allows an existing fabric to operate normally even though one or more switches have an invalid certificate.
• Does not enforce serial number based authorization.
• Allows auto-discovered controllers and switches to join the fabric without enforcing serial number
authorization.

Strict Mode operates as follows:

• Only switches with a valid Cisco serial number and SSL certificate are allowed.
• Enforces serial number based authorization.
• Requires an administrator to manually authorize controllers and switches to join the fabric.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 system fabric-security-mode {permissive | Specifies the fabric security mode.

strict}
Example:
apic1(config)# system
fabric-security-mode strict

Step 3 system controller-id controller-id {approve | In strict mode, approves or rejects a controller
reject} to join the fabric.
Example:
apic1(config)# system controller-id
FCH1750V025 approve

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

388
 Configuring Security
Configuring COOP Authentication

Examples
This example shows how to change the fabric security mode to strict.

apic1# configure
apic1(config)# system fabric-security-mode strict

This example shows how to approve a controller to join the fabric when strict mode is configured.

apic1# configure
apic1(config)# system controller-id FCH1750V025 approve

Configuring COOP Authentication

About COOP Authentication
Council of Oracles Protocol (COOP) is used to communicate the mapping information (location and identity)
to the spine proxy. A leaf switch will forward endpoint address information to a spine using ZeroMQ (Zero
Message Queue or ZMQ). COOP running on the spine nodes ensures that all spine nodes maintain a consistent
copy of end point address and location information and additionally maintains the distributed hash table (DHT)
repository of endpoint identity to location mapping database.
Without COOP authentication, it is possible for users to send arbitrary COOP messages, which would be
acted on by the fabric nodes. Cisco APIC Release 2.0 adds an MD5 TCP option to provide authentication and
integrity protection to the ZMQ TCP transportation. Two authentication modes are supported:
• Compatible - COOP accepts both MD5 authenticated and non-authenticated ZMQ connections for
message transportation. COOP data path communication gives high priority to transport via secured
connections.
• Strict - COOP allows MD5 authenticated ZMQ connections only.

Changing the configuration of the COOP authentication type has the following effects:
• When the configuration changes from compatible to strict mode, all non-authenticated ZMQ connections
are disconnected.
• When the configuration changes from strict to compatible mode, COOP immediately accepts both
authenticated and non-authenticated ZMQ connections.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

389
 Configuring Security
Configuring COOP Authentication

Configuring COOP Authentication

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 coop-fabric Enters COOP fabric configuration mode.

Example:
apic1(config)# coop-fabric

Step 3 authentication type {compatible | strict} Configures the COOP authentication type as
one of the following:
Example:
apic1(config-coop-fabric)# authentication • compatible - COOP allows MD5
type compatible authenticated and non-authenticated ZMQ
connections.
• strict - allows MD5 authenticated
ZMQ connections only.

Example
This example shows how to configure COOP authentication in compatible mode:

apic1# configure
apic1(config)# coop-fabric
apic1(config-coop-fabric# authentication type compatible

Configuring FIPS
About Federal Information Processing Standards (FIPS)
The Federal Information Processing Standards (FIPS) Publication 140-2, Security Requirements for
Cryptographic Modules, details the U.S. government requirements for cryptographic modules. FIPS 140-2
specifies that a cryptographic module should be a set of hardware, software, firmware, or some combination
that implements cryptographic functions or processes, including cryptographic algorithms and, optionally,
key generation, and is contained within a defined cryptographic boundary.
FIPS specifies certain cryptographic algorithms as secure, and it also identifies which algorithms should be
used if a cryptographic module is to be called FIPS compliant.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

390
 Configuring Security
Guidelines and Limitations

Guidelines and Limitations

Follow these guidelines and limitations:
• When FIPS is enabled, it is applied across Cisco APIC.
• When performing a Cisco APIC software downgrade, you must disable FIPS first.
• Make your passwords a minimum of eight characters in length.
• Disable Telnet. Users should log in using SSH only.
• Delete all SSH Server RSA1 keypairs.
• Disable remote authentication through RADIUS/TACACS+. Only local and LDAP users can be
authenticated.
• Secure Shell (SSH) and SNMP are supported.
• Disable SNMP v1 and v2. Any existing user accounts on the switch that have been configured for SNMPv3
should be configured only with SHA for authentication and AES for privacy.
• Starting with release 2.3(1x), FIPS can be configured at the switch level.
• Starting with release 3.1(1x), when FIPs is enabled, NTP will operate in FIPS mode, Under FIPS mode
NTP supports authentication with HMAC-SHA1 and no authentication.

Configuring FIPS for Cisco APIC Using NX-OS Style CLI

When FIPS is enabled, it is applied across Cisco APIC.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 fips mode enable Enables FIP. The no fips mode enable
command disables FIPS.
Example:
apic1(config)# fips mode enable You must reboot to complete the configuration.
Anytime you change the mode, you must reboot
to complete the configuration.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

391
 Configuring Security
Configuring Control Plane Policing

Configuring Control Plane Policing

Information About CoPP
Control Plane Policing (CoPP) protects the control plane, which ensures network stability, reachability, and
packet delivery.
This feature allows specification of parameters, for each protocol that can reach the control processor to be
rate-limited using a policer. The policing is applied to all traffic destined to any of the IP addresses of the
router or Layer 3 switch. A common attack vector for network devices is the denial-of-service (DoS) attack,
where excessive traffic is directed at the device interfaces.
The Cisco ACI Leaf/Spine NX-OS provides CoPP to prevent DoS attacks from impacting performance. Such
attacks, which can be perpetrated either inadvertently or maliciously, typically involve high rates of traffic
destined to the supervisor module of an ACI Leaf/Spine CPU or CPU itself.
The supervisor module of ACI Leaf/Spine switches divides the traffic that it manages into two functional
components or planes:
• Data plane—Handles all the data traffic. The basic functionality of a Cisco NX-OS device is to forward
packets from one interface to another. The packets that are not meant for the switch itself are called the
transit packets. These packets are handled by the data plane.
• Control plane—Handles all routing protocol control traffic. These protocols, such as the Border Gateway
Protocol (BGP) and the Open Shortest Path First (OSPF) Protocol, send control packets between devices.
These packets are destined to router addresses and are called control plane packets.

The ACI Leaf/Spine supervisor module has a control plane and is critical to the operation of the network. Any
disruption or attacks to the supervisor module will result in serious network outages. For example, excessive
traffic to the supervisor module could overload and slow down the performance of the entire Cisco ACI fabric.
Another example is a DoS attack on the ACI Leaf/Spine supervisor module that could generate IP traffic
streams to the control plane at a very high rate, forcing the control plane to spend a large amount of time in
handling these packets and preventing the control plane from processing genuine traffic.
Examples of DoS attacks are as follows:
• Internet Control Message Protocol (ICMP) echo requests
• IP fragments
• TCP SYN flooding

These attacks can impact the device performance and have the following negative effects:
• Reduced service quality (such as poor voice, video, or critical applications traffic)
• High route processor or switch processor CPU utilization
• Route flaps due to loss of routing protocol updates or keepalives
• Processor resource exhaustion, such as the memory and buffers
• Indiscriminate drops of incoming packets

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

392
Configuring Security
Information About CoPP

Note ACI Leaf/Spines are by default protected by CoPP with default settings. This feature allows for tuning the
parameters on a group of nodes based on customer needs.

Control Plane Protection

To protect the control plane, the Cisco NX-OS running on ACI Leaf/Spines segregates different packets
destined for the control plane into different classes. Once these classes are identified, the Cisco NX-OS device
polices the packets, which ensures that the supervisor module is not overwhelmed.
Control Plane Packet Types:
Different types of packets can reach the control plane:
• Receive Packets—Packets that have the destination address of a router. The destination address can be
a Layer 2 address (such as a router MAC address) or a Layer 3 address (such as the IP address of a router
interface). These packets include router updates and keepalive messages. Multicast packets can also be
in this category where packets are sent to multicast addresses that are used by a router.
• Exception Packets—Packets that need special handling by the supervisor module. For example, if a
destination address is not present in the Forwarding Information Base (FIB) and results in a miss, the
supervisor module sends an ICMP unreachable packet back to the sender. Another example is a packet
with IP options set.
• Redirect Packets—Packets that are redirected to the supervisor module. Features such as Dynamic Host
Configuration Protocol (DHCP) snooping or dynamic Address Resolution Protocol (ARP) inspection
redirect some packets to the supervisor module.
• Glean Packets—If a Layer 2 MAC address for a destination IP address is not present in the FIB, the
supervisor module receives the packet and sends an ARP request to the host.

All of these different packets could be maliciously used to attack the control plane and overwhelm the Cisco
ACI Fabric. CoPP classifies these packets to different classes and provides a mechanism to individually control
the rate at which the ACI Leaf/Spine supervisor module receives these packets.
Classification for CoPP:
For effective protection, the ACI Leaf/Spine NX-OS classifies the packets that reach the supervisor modules
to allow you to apply different rate controlling policies based on the type of the packet. For example, you
might want to be less strict with a protocol packet such as Hello messages but more strict with a packet that
is sent to the supervisor module because the IP option is set.
Rate Controlling Mechanisms:
Once the packets are classified, the ACI Leaf/Spine NX-OS has different mechanisms to control the rate at
which packets arrive at the supervisor module.
You can configure the following parameters for policing:
• Committed information rate (CIR)—Desired bandwidth, specified as a bit rate or a percentage of the
link rate.
• Committed burst (BC)—Size of a traffic burst that can exceed the CIR within a given unit of time and
not impact scheduling.

Default Policing Policies:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

393
 Configuring Security
Guidelines and Limitations for CoPP

When the Cisco ACI Leaf/Spine is bootup, the platform setup pre-defined CoPP parameters for different
protocols are based on the tests done by Cisco.

Guidelines and Limitations for CoPP

CoPP has the following configuration guidelines and limitations:
• We recommend that you use the default CoPP policy initially and then later modify the CoPP policies
based on the data center and application requirements.
• Customizing CoPP is an ongoing process. CoPP must be configured according to the protocols and
features used in your specific environment as well as the supervisor features that are required by the
server environment. As these protocols and features change, CoPP must be modified.
• We recommend that you continuously monitor CoPP. If drops occur, determine if CoPP dropped traffic
unintentionally or in response to a malfunction or attack. In either event, analyze the situation and evaluate
the need to modify the CoPP policies.
• You must ensure that the CoPP policy does not filter critical traffic such as routing protocols or interactive
access to the device. Filtering this traffic could prevent remote access to the Cisco ACI Leaf/Spine and
require a console connection.
• Do not mis-configure CoPP pre-filter entries. CoPP pre-filter entries might impact connectivity to
multi-pod configurations, remote leaf switches, and Cisco ACI Multi-Site deployments.
• You can use the APIC UI to be able to tune the CoPP parameters.
• Per interface per protocol is only supported on Leaf switches.
• FEX ports are not supported on per interface per protocol.
• For per interface per protocol the supported protocols are; ARP, ICMP, CDP, LLDP, LACP, BGP, STP,
BFD, and OSPF.
• The TCAM entry maximum for per interface per protocol is 256. Once the threshold is exceeded a fault
will be raised.

Configuring CoPP Using the Cisco NX-OS CLI

Procedure

Step 1 Configure a CoPP leaf profile:

Example:
# configure copp Leaf Profile
apic1(config)# policy-map type control-plane-leaf leafProfile
apic1(config-pmap-copp-leaf)# profile-type custom
apic1(config-pmap-copp-leaf)# set arpRate 786
# create a policy group to be applied on leaves
apic1(config)# template leaf-policy-group coppForLeaves
apic1(config-leaf-policy-group)# copp-aggr leafProfile
apic1(config-leaf-policy-group)# exit
# apply the leaves policy group on leaves
apic1(config)# leaf-profile applyCopp

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

394
 Configuring Security
Configuring Per Interface Per Protocol CoPP Policy Using the NX-OS Style CLI

apic1(config-leaf-profile)# leaf-group applyCopp

apic1(config-leaf-group)# leaf 101-102
apic1(config-leaf-group)# leaf-policy-group coppForLeaves

Step 2 Configure a CoPP Spine profile:

Example:
# configure copp Spine Profile
apic1(config)# policy-map type control-plane-spine spineProfile
apic1(config-pmap-copp-spine)# profile-type custom
apic1(config-pmap-copp-spine)# set arpRate 786
# create a policy group to be applied on spines
apic1(config)# template leaf-policy-group coppForSpines
apic1(config-spine-policy-group)# copp-aggr spineProfile
apic1(config-spine-policy-group)# exit
# apply the spine policy group on spines
apic1(config)# spine-profile applyCopp
apic1(config-spine-profile)# spine-group applyCopp
apic1(config-spine-group)# spine 201-202
apic1(config-spine-group)# spine-policy-group coppForSpines

Configuring Per Interface Per Protocol CoPP Policy Using the NX-OS Style CLI
Procedure

Step 1 Define the CoPP class map and policy map:

Example:
(config)# policy-map type control-plane-if <name>
(config-pmap-copp)# protocol bgp bps <value>
(config-pmap-copp)# protocol ospf bps <value>

Step 2 Applying the configuration to an interface on the leaf:

Example:
(config)# leaf 101
(config-leaf)# int eth 1/10
(config-leaf-if)# service-policy type control-plane-if output<name>

Configuring First Hop Security

About First Hop Security
First-Hop Security (FHS) features enable a better IPv4 and IPv6 link security and management over the layer
2 links. In a service provider environment, these features closely control address assignment and derived
operations, such as Duplicate Address Detection (DAD) and Address Resolution (AR).

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

395
 Configuring Security
ACI FHS Deployment

The following supported FHS features secure the protocols and help build a secure endpoint database on the
fabric leaf switches, that are used to mitigate security threats such as MIM attacks and IP thefts:
• ARP Inspection—allows a network administrator to intercept, log, and discard ARP packets with invalid
MAC address to IP address bindings.
• ND Inspection—learns and secures bindings for stateless autoconfiguration addresses in Layer 2 neighbor
tables.
• DHCP Inspection—validates DHCP messages received from untrusted sources and filters out invalid
messages.
• RA Guard—allows the network administrator to block or reject unwanted or rogue router advertisement
(RA) guard messages.
• IPv4 and IPv6 Source Guard—blocks any data traffic from an unknown source.
• Trust Control—a trusted source is a device that is under your administrative control. These devices
include the switches, routers, and servers in the Fabric. Any device beyond the firewall or outside the
network is an untrusted source. Generally, host ports are treated as untrusted sources.

FHS features provide the following security measures:

• Role Enforcement—Prevents untrusted hosts from sending messages that are out the scope of their role.
• Binding Enforcement—Prevents address theft.
• DoS Attack Mitigations—Prevents malicious end-points to grow the end-point database to the point
where the database could stop providing operation services.
• Proxy Services—Provides some proxy-services to increase the efficiency of address resolution.

FHS features are enabled on a per tenant bridge domain (BD) basis. As the bridge domain, may be deployed
on a single or across multiple leaf switches, the FHS threat control and mitigation mechanisms cater to a single
switch and multiple switch scenarios.

ACI FHS Deployment

Most FHS features are configured in a two-step fashion: firstly you define a policy which describes the behavior
of the feature, secondly you apply this policy to a "domain" (being the Tenant Bridge Domain or the Tenant
Endpoint Group). Different policies that define different behaviors can be applied to different intersecting
domains. The decision to use a specific policy is taken by the most specific domain to which the policy is
applied.
The policy options can be defined from the Cisco APIC GUI found under the
Tenant_name>Networking>Protocol Policies>First Hop Security tab.

Guidelines and Limitations

Follow these guidelines and limitations:
• Starting with release 3.1(1), FHS is supported with virtual Endpoints (AVS only).
• FHS is supported with both VLAN and VXLAN encapsulation.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

396
 Configuring Security
Configuring FHS Using the NX-OS CLI

• Any secured endpoint entry in the FHS Binding Table Database in DOWN state will get cleared after
18 Hours of timeout. The entry moves to DOWN state when the front panel port where the entry is
learned is link down. During this window of 18 Hours, if the endpoint is moved to a different location
and is seen on a different port, the entry will be gracefully moved out of DOWN state to
REACHABLE/STALE as long as the endpoint is reachable from the other port it is moved from.
• When IP Source Guard is enabled, the IPv6 traffic that is sourced using IPv6 Link Local address as IP
source address is not subject to the IP Source Guard enforcement (i.e. Enforcement of Source Mac <=>
Source IP Bindings secured by IP Inspect Feature). This traffic is permitted by default irrespective of
binding check failures.
• FHS is not supported on L3Out interfaces.
• FHS is not supported N9K-M12PQ based TORs.
• FHS in ACI Multi-Site is a site local capability therefore it can only be enabled in a site from the APIC
cluster. Also, FHS in ACI Multi-Site only works when the BD and EPG is site local and not stretched
across sites. FHS security cannot be enabled for stretched BD or EPGs.
• FHS is not supported on a Layer 2 only bridge domain.
• Enabling FHS feature can disrupt traffic for 50 seconds because the EP in the BD are flushed and EP
Learning in the BD is disabled for 50 seconds.

Configuring FHS Using the NX-OS CLI

Before you begin
• The tenant and Bridge Domain configured.

Procedure

Step 1 configure
Enters configuration mode.
Example:
apic1# configure

Step 2 Configure FHS policy.

Example:
apic1(config)# tenant coke
apic1(config-tenant)# first-hop-security
apic1(config-tenant-fhs)# security-policy pol1
apic1(config-tenant-fhs-secpol)#
apic1(config-tenant-fhs-secpol)# ip-inspection-admin-status enabled-both
apic1(config-tenant-fhs-secpol)# source-guard-admin-status enabled-both
apic1(config-tenant-fhs-secpol)# router-advertisement-guard-admin-status enabled
apic1(config-tenant-fhs-secpol)# router-advertisement-guard
apic1(config-tenant-fhs-raguard)#
apic1(config-tenant-fhs-raguard)# managed-config-check
apic1(config-tenant-fhs-raguard)# managed-config-flag
apic1(config-tenant-fhs-raguard)# other-config-check

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

397
 Configuring Security
Configuring FHS Using the NX-OS CLI

apic1(config-tenant-fhs-raguard)# other-config-flag
apic1(config-tenant-fhs-raguard)# maximum-router-preference low
apic1(config-tenant-fhs-raguard)# minimum-hop-limit 10
apic1(config-tenant-fhs-raguard)# maximum-hop-limit 100
apic1(config-tenant-fhs-raguard)# exit
apic1(config-tenant-fhs-secpol)# exit
apic1(config-tenant-fhs)# trust-control tcpol1
pic1(config-tenant-fhs-trustctrl)# arp
apic1(config-tenant-fhs-trustctrl)# dhcpv4-server
apic1(config-tenant-fhs-trustctrl)# dhcpv6-server
apic1(config-tenant-fhs-trustctrl)# ipv6-router
apic1(config-tenant-fhs-trustctrl)# router-advertisement
apic1(config-tenant-fhs-trustctrl)# neighbor-discovery
apic1(config-tenant-fhs-trustctrl)# exit
apic1(config-tenant-fhs)# exit
apic1(config-tenant)# bridge-domain bd1
apic1(config-tenant-bd)# first-hop-security security-policy pol1
apic1(config-tenant-bd)# exit
apic1(config-tenant)# application ap1
apic1(config-tenant-app)# epg epg1
apic1(config-tenant-app-epg)# first-hop-security trust-control tcpol1

Step 3 Show FHS configuration example:

Example:
leaf4# show fhs bt all

Legend:
TR : trusted-access UNRES : unresolved Age
: Age since creation
UNTR : untrusted-access UNDTR : undetermined-trust CRTNG
: creating
UNKNW : unknown TENTV : tentative INV
: invalid
NDP : Neighbor Discovery Protocol STA : static-authenticated REACH
: reachable
INCMP : incomplete VERFY : verify INTF
: Interface
TimeLeft : Remaining time since last refresh LM : lla-mac-match DHCP
: dhcp-assigned

EPG-Mode:
U : unknown M : mac V : vlan I : ip

BD-VNID BD-Vlan BD-Name

15630220 3 t0:bd200

---------------------------------------------------------------------------------------------------------------------
| Origin | IP | MAC | INTF | EPG(sclass)(mode) | Trust-lvl |
State | Age | TimeLeft |
---------------------------------------------------------------------------------------------------------------------
| ARP | 192.0.200.12 | D0:72:DC:A0:3D:4F | eth1/1 | epg300(49154)(V) | LM,TR |
STALE | 00:04:49 | 18:08:13 |
| ARP | 172.29.205.232 | D0:72:DC:A0:3D:4F | eth1/1 | epg300(49154)(V) | LM,TR |
STALE | 00:03:55 | 18:08:21 |
| ARP | 192.0.200.21 | D0:72:DC:A0:3D:4F | eth1/1 | epg300(49154)(V) | LM,TR |
REACH | 00:03:36 | 00:00:02 |
| LOCAL | 192.0.200.1 | 00:22:BD:F8:19:FF | vlan3 | LOCAL(16387)(I) | STA |
REACH | 04:49:41 | N/A |
| LOCAL | fe80::200 | 00:22:BD:F8:19:FF | vlan3 | LOCAL(16387)(I) | STA |
REACH | 04:49:40 | N/A |
| LOCAL | 2001:0:0:200::1 | 00:22:BD:F8:19:FF | vlan3 | LOCAL(16387)(I) | STA |

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

398
Configuring Security
Configuring FHS Using the NX-OS CLI

REACH | 04:49:39 | N/A |

---------------------------------------------------------------------------------------------------------------------

The trust levels are:

• TR— Trusted. Displayed when the endpoint is learned from an EPG where the trust configuration is
enabled.
• UNTR— Untrusted. Displayed when the endpoint is learned from an EPG where the trust configuration
is not enabled.
• UNDTR— Undetermined. Displayed in the case of a DHCP relay topology where the DHCP server
bridge domain (BD) is on a remote leaf and the DHCP clients are on a local leaf. In this situation, the
local leaf will not know whether the DHCP server BD has trust DHCP enabled.

Step 4 Show violations with the different types and reasons example:
Example:
leaf4# show fhs violations all

Violation-Type:
POL : policy THR : address-theft-remote
ROLE : role TH : address-theft
INT : internal

Violation-Reason:
IP-MAC-TH : ip-mac-theft OCFG_CHK : ra-other-cfg-check-fail
ANC-COL : anchor-collision
PRF-LVL-CHK : ra-rtr-pref-level-check-fail INT-ERR : internal-error
TRUST-CHK : trust-check-fail
SRV-ROL-CHK : srv-role-check-fail ST-EP-COL : static-ep-collision
LCL-EP-COL : local-ep-collision
MAC-TH : mac-theft EP-LIM : ep-limit-reached
MCFG-CHK : ra-managed-cfg-check-fail
HOP-LMT-CHK : ra-hoplimit-check-fail MOV-COL : competing-move-collision
RTR-ROL-CHK : rtr-role-check-fail
IP-TH : ip-theft

EPG-Mode:
U : unknown M : mac V : vlan I : ip

BD-VNID BD-Vlan BD-Name

15630220 3 t0:bd200
-----------------------------------------------------------------------------------------------------
| Type | Last-Reason | Proto | IP | MAC | Port | EPG(sclass)(mode)
| Count |
-----------------------------------------------------------------------------------------------------
| THR | IP-TH | ARP | 192.0.200.21 | D0:72:DC:A0:3D:4F | tunnel5 | epg300(49154)(V)
| 21 |
-----------------------------------------------------------------------------------------------------
Table Count: 1

Step 5 Show FHS configuration:

Example:
swtb23-ifc1# show tenant t0 bridge-domain bd200 first-hop-security binding-table

Pod/Node Type Family IP Address MAC Address Interface Level

State
-------- ------ ------ -------------------- ----------------- ------------
--------------- -----
1/102 local ipv4 192.0.200.1 00:22:BD:F8:19:FF vlan3 static-

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

399
 Configuring Security
Configuring FHS Using the NX-OS CLI

reach

authenticated able
1/102 local ipv6 fe80::200 00:22:BD:F8:19:FF vlan3 static-
reach

authenticated able
1/102 local ipv6 2001:0:0:200::1 00:22:BD:F8:19:FF vlan3 static-
reach

authenticated able
1/101 arp ipv4 192.0.200.23 D0:72:DC:A0:02:61 eth1/2 lla-mac-match
stale
,untrusted-

access
1/101 local ipv4 192.0.200.1 00:22:BD:F8:19:FF vlan3 static-
reach

authenticated able
1/101 nd ipv6 fe80::d272:dcff:fea0 D0:72:DC:A0:02:61 eth1/2 lla-mac-match
reach
:261 ,untrusted-
able
access
1/101 nd ipv6 2001:0:0:200::20 D0:72:DC:A0:02:61 eth1/2 lla-mac-match
stale
,untrusted-

access
1/101 nd ipv6 2001::200:d272:dcff: D0:72:DC:A0:02:61 eth1/2 lla-mac-match
stale
fea0:261 ,untrusted-

access
1/101 local ipv6 fe80::200 00:22:BD:F8:19:FF vlan3 static-
reach

authenticated able
1/101 local ipv6 2001:0:0:200::1 00:22:BD:F8:19:FF vlan3 static-
reach

authenticated able
1/103 local ipv4 192.0.200.1 00:22:BD:F8:19:FF vlan4 static-
reach

authenticated able
1/103 local ipv6 fe80::200 00:22:BD:F8:19:FF vlan4 static-
reach

authenticated able
1/103 local ipv6 2001:0:0:200::1 00:22:BD:F8:19:FF vlan4 static-
reach

authenticated able
1/104 arp ipv4 192.0.200.10 F8:72:EA:AD:C4:7C eth1/1 lla-mac-match
stale

,trusted-access
1/104 arp ipv4 172.29.207.222 D0:72:DC:A0:3D:4C eth1/1 lla-mac-match
stale

,trusted-access
1/104 local ipv4 192.0.200.1 00:22:BD:F8:19:FF vlan4 static-

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

400
Configuring Security
Configuring FHS Using the NX-OS CLI

reach

authenticated able
1/104 nd ipv6 fe80::fa72:eaff:fead F8:72:EA:AD:C4:7C eth1/1 lla-mac-match
stale
:c47c
,trusted-access
1/104 nd ipv6 2001:0:0:200::10 F8:72:EA:AD:C4:7C eth1/1 lla-mac-match
stale

,trusted-access
1/104 local ipv6 fe80::200 00:22:BD:F8:19:FF vlan4 static-
reach

authenticated able
1/104 local ipv6 2001:0:0:200::1 00:22:BD:F8:19:FF vlan4 static-
reach

authenticated able

Pod/Node Type IP Address Creation TS Last Refresh TS

Lease Period
-------- ------ -------------------- -----------------------------
----------------------------- ------------
1/102 local 192.0.200.1 2017-07-20T04:22:38.000+00:00
2017-07-20T04:22:38.000+00:00
1/102 local fe80::200 2017-07-20T04:22:56.000+00:00
2017-07-20T04:22:56.000+00:00
1/102 local 2001:0:0:200::1 2017-07-20T04:22:57.000+00:00
2017-07-20T04:22:57.000+00:00
1/101 arp 192.0.200.23 2017-07-27T10:55:20.000+00:00
2017-07-27T16:07:24.000+00:00
1/101 local 192.0.200.1 2017-07-27T10:48:09.000+00:00
2017-07-27T10:48:09.000+00:00
1/101 nd fe80::d272:dcff:fea0 2017-07-27T10:52:16.000+00:00
2017-07-27T16:04:29.000+00:00
:261
1/101 nd 2001:0:0:200::20 2017-07-27T10:57:32.000+00:00
2017-07-27T16:07:24.000+00:00
1/101 nd 2001::200:d272:dcff: 2017-07-27T11:21:45.000+00:00
2017-07-27T16:07:24.000+00:00
fea0:261
1/101 local fe80::200 2017-07-27T10:48:10.000+00:00
2017-07-27T10:48:10.000+00:00
1/101 local 2001:0:0:200::1 2017-07-27T10:48:11.000+00:00
2017-07-27T10:48:11.000+00:00
1/103 local 192.0.200.1 2017-07-26T22:03:56.000+00:00
2017-07-26T22:03:56.000+00:00
1/103 local fe80::200 2017-07-26T22:03:57.000+00:00
2017-07-26T22:03:57.000+00:00
1/103 local 2001:0:0:200::1 2017-07-26T22:03:58.000+00:00
2017-07-26T22:03:58.000+00:00
1/104 arp 192.0.200.10 2017-07-27T11:21:13.000+00:00
2017-07-27T16:05:48.000+00:00
1/104 arp 172.29.207.222 2017-07-27T11:54:48.000+00:00
2017-07-27T16:06:38.000+00:00
1/104 local 192.0.200.1 2017-07-27T10:49:13.000+00:00
2017-07-27T10:49:13.000+00:00
1/104 nd fe80::fa72:eaff:fead 2017-07-27T11:21:13.000+00:00
2017-07-27T16:06:43.000+00:00
:c47c
1/104 nd 2001:0:0:200::10 2017-07-27T11:21:13.000+00:00
2017-07-27T16:06:19.000+00:00

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

401
 Configuring Security
Configuring FHS Using the NX-OS CLI

1/104 local fe80::200 2017-07-27T10:49:14.000+00:00

2017-07-27T10:49:14.000+00:00
1/104 local 2001:0:0:200::1 2017-07-27T10:49:15.000+00:00
2017-07-27T10:49:15.000+00:00

swtb23-ifc1#

swtb23-ifc1# show tenant t0 bridge-domain bd200 first-hop-security statistics arp

Pod/Node : 1/101
Request Received : 4
Request Switched : 2
Request Dropped : 2
Reply Received : 257
Reply Switched : 257
Reply Dropped : 0

Pod/Node : 1/104
Request Received : 6
Request Switched : 6
Request Dropped : 0
Reply Received : 954
Reply Switched : 954
Reply Dropped : 0

swtb23-ifc1# show tenant t0 bridge-domain bd200 first-hop-security statistics dhcpv4

Pod/Node : 1/102
Discovery Received : 5
Discovery Switched : 5
Discovery Dropped : 0
Offer Received : 0
Offer Switched : 0
Offer Dropped : 0
Request Received : 0
Request Switched : 0
Request Dropped : 0
Ack Received : 0
Ack Switched : 0
Ack Dropped : 0
Nack Received : 0
Nack Switched : 0
Nack Dropped : 0
Decline Received : 0
Decline Switched : 0
Decline Dropped : 0
Release Received : 0
Release Switched : 0
Release Dropped : 0
Information Received : 0
Information Switched : 0
Information Dropped : 0
Lease Query Received : 0
Lease Query Switched : 0
Lease Query Dropped : 0
Lease Active Received : 0
Lease Active Switched : 0
Lease Active Dropped : 0
Lease Unassignment Received : 0
Lease Unassignment Switched : 0
Lease Unassignment Dropped : 0
Lease Unknown Received : 0
Lease Unknown Switched : 0
Lease Unknown Dropped : 0

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

402
 Configuring Security
Configuring 802.1x

swtb23-ifc1# show tenant t0 bridge-domain bd200 first-hop-security statistics

neighbor-discovery
Pod/Node : 1/101
Neighbor Solicitation Received : 125
Neighbor Solicitation Switched : 121
Neighbor Solicitation Dropped : 4
Neighbor Advertisement Received : 519
Neighbor Advertisement Switched : 519
Neighbor Advertisement Drop : 0
Router Solicitation Received : 4
Router Solicitation Switched : 4
Router Solicitation Dropped : 0
Router Adv Received : 0
Router Adv Switched : 0
Router Adv Dropped : 0
Redirect Received : 0
Redirect Switched : 0
Redirect Dropped : 0

Pod/Node : 1/104
Neighbor Solicitation Received : 123
Neighbor Solicitation Switched : 47
Neighbor Solicitation Dropped : 76
Neighbor Advertisement Received : 252
Neighbor Advertisement Switched : 228
Neighbor Advertisement Drop : 24
Router Solicitation Received : 0
Router Solicitation Switched : 0
Router Solicitation Dropped : 0
Router Adv Received : 53
Router Adv Switched : 6
Router Adv Dropped : 47
Redirect Received : 0
Redirect Switched : 0
Redirect Dropped : 0

Configuring 802.1x
802.1X Overview
802.1X defines a client-server based access control and authentication protocol that restricts unauthorized
clients from connecting to a LAN through publicly accessible ports. The authentication server authenticates
each client connected to a Cisco NX-OS device port.
Until the client is authenticated, 802.1X access control allows only Extensible Authentication Protocol over
LAN (EAPOL) traffic through the port to which the client is connected. After authentication is successful,
normal traffic can pass through the port.
The RADIUS distributed client/server system allows you to secure networks against unauthorized access. In
the Cisco ACI implementation, RADIUS clients run on the ToRs and send authentication and accounting
requests to a central RADIUS server that contains all user authentication and network service access information.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

403
 Configuring Security
Host Support

Host Support
The 802.1X feature can restrict traffic on a port with the following modes:
• Single-host Mode—Allows traffic from only one endpoint device on the 802.1X port. Once the endpoint
device is authenticated, the APIC puts the port in the authorized state. When the endpoint device leaves
the port, the APIC put the port back into the unauthorized state. A security violation in 802.1X is defined
as a detection of frames sourced from any MAC address other than the single MAC address authorized
as a result of successful authentication. In this case, the interface on which this security association
violation is detected (EAPOL frame from the other MAC address) will be disabled. Single host mode is
applicable only for host-to-switch topology and when a single host is connected to the Layer 2 (Ethernet
access port) or Layer 3 port (routed port) of the APIC.
• Multi-host Mode—Allows multiple hosts per port but only the first one gets authenticated. The port is
moved to the authorized state after the successful authorization of the first host. Subsequent hosts are
not required to be authorized to gain network access once the port is in the authorized state. If the port
becomes unauthorized when reauthentication fails or an EAPOL logoff message is received, all attached
hosts are denied access to the network. The capability of the interface to shut down upon security
association violation is disabled in multiple host mode. This mode is applicable for both switch-to-switch
and host-to-switch topologies
• Multi-Auth Mode—Allows multiple hosts and all hosts are authenticated separately.

Note Each host must have the same EPG/VLAN information.

• Multi-Domain Mode—For separate data and voice domain. For use with IP-Phones.

Authentication Modes
ACI 802.1X supports the following authentication modes:
• EAP—The authenticator then sends an EAP-request/identity frame to the supplicant to request its identity
(typically, the authenticator sends an initial identity/request frame followed by one or more requests for
authentication information). When the supplicant receives the frame, it responds with an
EAP-response/identity frame.
• MAB—MAC Authentication Bypass (MAB) is supported as the fallback authentication mode. MAB
enables port-based access control using the MAC address of the endpoint. A MAB-enabled port can be
dynamically enabled or disabled based on the MAC address of the device that connects to it. Prior to
MAB, the endpoint's identity is unknown and all traffic is blocked. The switch examines a single packet
to learn and authenticate the source MAC address. After MAB succeeds, the endpoint's identity is known
and all traffic from that endpoint is allowed. The switch performs source MAC address filtering to help
ensure that only the MAB-authenticated endpoint is allowed to send traffic.

Guidelines and Limitations

802.1X port-based authentication has the following configuration guidelines and limitations:
• The Cisco ACI supports 802.1X authentication only on physical ports.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

404
 Configuring Security
Configuration Overview

• The Cisco ACI does not support 802.1X authentication on port channels or subinterfaces.
• The Cisco ACI supports 802.1X authentication on member ports of a port channel but not on the port
channel itself.
• Member ports with and without 802.1X configuration can coexist in a port channel. However, you must
ensure the identical 802.1X configuration on all the member ports in order for channeling to operate with
802.1X
• When you enable 802.1X authentication, supplicants are authenticated before any other Layer 2 or Layer
3 features are enabled on an Ethernet interface.
• 802.1X is supported only on a leaf chassis that is EX or FX type.
• 802.1X is only supported Fabric Access Ports. 802.1X is not supported on Port-Channels, or
Virtual-Port-Channels.
• IPv6 is not supported for dot1x clients in the 3.2(1) release.
• While downgrading to earlier releases especially in cases where certain interface config (host mode and
auth type) is unsupported in that release, dot1x authentication type defaults to none. Host-mode would
need to be manually re-configured to either single host/multi host depending on whatever is desired. This
is to ensure that the user configures only the supported modes/auth-types in that release and doesn’t run
into unsupported scenarios.
• Multi-Auth supports 1 voice client and multiple data clients (all belonging to same data vlan/epg).
• Fail-epg/vlan under 802.1X node authentication policy is a mandatory configuration.
• Multi-domain more than 1 voice and 1 data client puts the port in security disabled state.
• The following platforms are not supported for 802.1X:
• N9K-C9396PX
• N9K-M12PQ
• N9K-C93128TX
• N9K-M12PQ

Configuration Overview
The 802.1X and RADIUS processes are started only when enabled by APIC. Internally, this means dot1x
process is started when 802.1X Inst MO is created and radius process is created when radius entity is created.
Dot1x based authentication must be enabled on each interface for authenticating users connected on that
interface otherwise the behavior is unchanged.
RADIUS server configuration is done separately from dot1x configuration. RADIUS configuration defines
a list of RADIUS servers and a way to reach them. Dot1x configuration contains a reference to RADIUS
group (or default group) to use for authentication.
Both 802.1X and RADIUS configuration must be done for successful authentication. Order of configuration
is not important but if there is no RADIUS configuration then 802.1X authentication cannot be successful.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

405
 Configuring Security
Configuring 802.1X Node Authentication Using NX-OS Style CLI

Configuring 802.1X Node Authentication Using NX-OS Style CLI

Procedure

Step 1 Configure the radius authentication group:

Example:
apic1# configure
apic1(config)#
apic1(config)# aaa group server radius myradiusgrp
apic1(config-radius)#server 192.168.0.100 priority 1
apic1(config-radius)#exit

Step 2 Configure node level port authentication policy:

Example:

apic1(config)# policy-map type port-authentication mydot1x

apic1(config-pmap-port-authentication)#radius-provider-group myradiusgrp
apic1(config-pmap-port-authentication)#fail-auth-vlan 2001
apic1(config-pmap-port-authentication)#fail-auth-epg tenant tn1 application ap1 epg epg256
apic1(config)# exit

Step 3 Configure policy group and specify port authentication policy in the group:
Example:
apic1(config)#template leaf-policy-group lpg2
apic1(config-leaf-policy-group)# port-authentication mydot1x
apic1(config-leaf-policy-group)#exit

Step 4 Configure the leaf switch profile:

Example:
apic1(config)# leaf-profile mylp2
apic1(config-leaf-profile)#leaf-group mylg2
apic1(config-leaf-group)# leaf-policy-group lpg2
apic1(config-leaf-group)#exit

Configuring 802.1X Port Authentication Using the NX-OS Style CLI

Procedure

Step 1 Configure a Policy Group:

Example:
apic1# configure
apic1(config)#
apic1(config)# template policy-group mypol
apic1(config-pol-grp-if)# switchport port-authentication mydot1x
apic1(config-port-authentication)# host-mode multi-host
apic1(config-port-authentication)# no shutdown

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

406
Configuring Security
Configuring 802.1X Port Authentication Using the NX-OS Style CLI

apic1(config-port-authentication)# exit
apic1(config-pol-grp-if)# exit

Step 2 Configure the leaf interface profile:

Example:
apic1(config)#
apic1(config)#leaf-interface-profile myprofile
apic1(config-leaf-if-profile)#leaf-interface-group mygroup
apic1(config-leaf-if-group)# interface ethernet 1/10-12
apic1(config-leaf-if-group)# policy-group mypol
apic1(config-leaf-if-group)# exit
apic1(config-leaf-if-profile)# exit

Step 3 Configure the leaf profile:

Example:

apic1(config)#
apic1(config)# leaf-profile myleafprofile
apic1(config-leaf-profile)# leaf-group myleafgrp
apic1(config-leaf-group)# leaf 101
apic1(config-leaf-group)# exit

Step 4 Apply an interface policy on the leaf switch profile:

Example:
apic1(config-leaf-profile)# leaf-interface-profile myprofile
apic1(config-leaf-group)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

407
 Configuring Security
Configuring 802.1X Port Authentication Using the NX-OS Style CLI

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

408
 CHAPTER 11
Configuring Anycast Services
This chapter contains the following sections:
• About Anycast Services, on page 409
• Configuring Anycast Services Using the NX-OS Style CLI, on page 410

About Anycast Services

Anycast services are supported in the Cisco ACI fabric. A typical use case is to support Cisco Adaptive
Security Appliance (ASA) firewalls in the pods of a multipod fabric, but Anycast could be used to enable
other services, such as DNS servers or printing services. In the ASA use case, a firewall is installed in every
pod and Anycast is enabled, so the firewall can be offered as an Anycast service. One instance of a firewall
going down does not affect clients, as the requests are routed to the next, nearest instance available. You install
ASA firewalls in each pod, then enable Anycast and configure the IP address and MAC addresses to be used.
Anycast is supported on Cisco Nexus 9000 series switches with names that end in EX, and later (for example,
N9K-C93180LC-EX).
Anycast can be configured on application EPGs or through Layer 4 to Layer 7 Services (with or without
Policy-Based Redirect (PBR)).
Up to 2000 Anycast services are supported per fabric.
A service node is used for Anycast services in the pod where the policy is applied.
APIC deploys the configuration of the Anycast MAC and IP addresses to the leaf switches where the VRF is
deployed or where there is a contract to allow an Anycast EPG.
Initially, each leaf switch installs the Anycast MAC and IP addresses as a proxy route to the spine switch.
When the first packet from the Anycast Service is received, the destination information for the service is
installed on the leaf switch behind which the service is installed. All other leaf switches continue to point to
the spine proxy. When the Anycast service has been learned, located behind a leaf in a pod, COOP installs
the entry on the spine switch to point to the service that is local to the pod.
When the Anycast service is running in one pod, the spine receives the route information for the Anycast
service present in the pod through BGP-EVPN. If the Anycast service is already locally present, then COOP
caches the Anycast service information of the remote pod. This route through the remote pod is only installed
when the local instance of the service goes down.
Anycast services are not supported with the following features and options:
• Multi-Site management

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

409
 Configuring Anycast Services
Configuring Anycast Services Using the NX-OS Style CLI

• Remote leaf switches

• Two firewalls in an Active/Standby relationship (in this scenario, the Anycast service is active in only
one pod and all traffic is sent using the active service)
• Firewalls that are deployed on two port channels (PCs)
• Firewalls that are deployed on a single PC with redundant links
• ECMP
• Symmetric policy-based redirect
• Pod ID Aware Redirection
• IP SLA Monitoring Policies
• Redirect Health Groups
• DAD enabled on external devices, when Anycast IPv6 addresses are used
• For remote IP address learning, to prevent IP address moves across the instances of services, remote
learning of the Anycast service MAC and IP addresses is turned off.
• Anycast services behind L3Outs
• Using the MAC and IP addresses of an existing static endpoint as Anycast addresses.

Note If you configure an Anycast MAC and IP address using the addresses for an existing static endpoint, the
configuration is pushed from the APIC to the switch and no fault is generated, but the switch does not install
the Anycast addresses in the hardware. Deleting the static endpoint does not resolve the problem. You must
delete both the static endpoint and the Anycast configurations and reconfigure the Anycast addresses.

Configuring Anycast Services Using the NX-OS Style CLI

These examples show how to configure Anycast services in three methods, using the NX-OS style CLI:
• Behind an EPG.
• As part of a Layer 4 to Layer 7 Service Graph with Policy Based Redirect (PBR)
• As part of a Layer 4 to Layer 7 Service Graph without PBR

Before you begin

• The tenant, application profile, and application EPG have been created.
• The node group and L3Out policies have already been created.
• The Interpod Network (IPN) is already configured.
• Multipod is configured.
• In each pod, the spine switch used to connect to the IPN is also connected to at least one leaf switch.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

410
Configuring Anycast Services
Configuring Anycast Services Using the NX-OS Style CLI

• ASA firewalls are installed in each pod.

Procedure

Step 1 Configure Anycast services behind an EPG subnet, using the following commands:
a) configure
Enters configuration mode.
Example:
apic1# configure

b) tenant tenant-name
Creates a tenant if it does not exist or enters tenant configuration mode.
Example:
apic1(config)# tenant anycast1-it

c) application app-name
Creates an application profile if it doesn't exist and enters application profile configuration mode.
Example:
apic1(config-tenant)# application AP0

d) epg epg-name
Creates an EPG if it doesn't exist and enters EPG configuration mode.
Example:
apic1(config-tenant-app)# epg epg1

e) endpoint ip ip-address anycast mac-address

Configures the Anycast IP address with netmask and MAC address for the Anycast service behind the
EPG. The Anycast subnet must have a /32 or /128 netmask.
Example:
apic1(config-tenant-app-epg)# endpoint ip 1.2.3.4/32 anycast 00:11:22:33:44:55

Step 2 Configure Anycast for Layer 4 to Layer 7 services with PBR, using the following commands:
a) configure
Enters configuration mode.
Example:
apic1# configure

b) tenant name
Creates a tenant if it does not exist or enters tenant configuration mode.
apic1(config)# tenant t1

c) svcredir-pol name
Enters service-redirect policy mode and creates a service redirection policy.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

411
 Configuring Anycast Services
Configuring Anycast Services Using the NX-OS Style CLI

Example:
apic1(config-tenant)# svcredir-pol N1Ext

d) anycast enable
Enables Anycast for the service redirection policy. Use the no form of the command to disable Anycast
for the policy.
Example:
apic1(svcredir-pol)# anycast enable

e) redir-dest ip-addr mac-addr

Defines the Anycast IP and MAC addresses for the Layer 4 to Layer 7 service redirection policy.
Example:
apic1(svcredir-pol)# redir-dest 2000::25 00:00:00:00:00:07

Step 3 Configure Anycast for Layer 4 to Layer 7 services without PBR, with the following commands:
a) configure
Enters configuration mode.
Example:
apic1# configure

b) tenant name
Creates a tenant if it does not exist or enters tenant configuration mode.
apic1(config)# tenant t1

c) l4l7 graph connector-name contract name

Creates a Layer 4 to Layer 7 service graph associated with a contract.
Example:
apic1(config-tenant)# l4l7 graph WebGraph contract default

d) service device-cluster-name
Defines the service with a device cluster.
Example:
apic1(config-graph)# service N1

e) connector name [cluster-interface cluster-interface-name]

Enters connector configuration mode and defines the device cluster interface.
Example:
apic1(config-service)# connector provider

f) subnet-ip IP-addr_with_netmask subnet-ctrl no-default-gateway

Defines the Anycast IP address (with /32 netmask and the subnet control, no-default-gateway). To remove
it, use the no form of the command.
Example:
apic1(config-connector)# subnet-ip 50.50.50.50/32 subnet-ctrl no-default-gateway

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

412
Configuring Anycast Services
Configuring Anycast Services Using the NX-OS Style CLI

g) mac-address mac-address
Defines the Anycast MAC address. To remove it, use the no form of the command.
Example:
apic1(config-subnet-ip)# mac-address 00.00.00.00.00.50

Example
The following example configures Anycast services behind EPG1:
apic1# configure
apic1(config)# tenant anycast1-it
apic1(config-tenant)# application AP0
apic1(config-tenant-app)# epg epg-1
apic1(config-tenant-app-epg)# endpoint ip 1.2.3.4/32 anycast 00:11:22:33:44:55

The following example configures Anycast services in a Layer 4 to Layer 7 service redirection policy:
apic1# configure
apic1(config)# tenant t1
apic1(config-tenant)# svcredir-pol N1Ext
apic1(svcredir-pol)# anycast enable
apic1(svcredir-pol)# redir-dest 2000::25 00:00:00:00:00:07

The following example configures Anycast services in a Layer 4 to Layer 7 service without PBR:
apic1# configure
apic1(config)# tenant t1
apic1(config-tenant)# l4l7 graph WebGraph contract default
apic1(config-graph)# service N1
apic1(config-service)# connector provider
apic1(config-connector)# subnet-ip 50.50.50.50/32 subnet-ctrl no-default-gateway
apic1(config-subnet-ip)# mac-address 00.00.00.00.00.50

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

413
 Configuring Anycast Services
Configuring Anycast Services Using the NX-OS Style CLI

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

414
 CHAPTER 12
Configuring VMM
• Configuring VMM, on page 415

Configuring VMM
For information about configuring virtual machine management using the NX-OS style CLI for APIC, see
the Cisco ACI Virtualization Guide.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

415
 Configuring VMM
Configuring VMM

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

416
 CHAPTER 13
Configuring Layer 4 to Layer 7 Services
• Configuring Layer 4 to Layer 7 Services, on page 417

Configuring Layer 4 to Layer 7 Services

For information about configuring Layer 4 to Layer 7 services using the NX-OS-style CLI for Cisco Application
Policy Infrastructure Controller (Cisco APIC), see the Cisco APIC Layer 4 to Layer 7 Services Deployment
Guide.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

417
 Configuring Layer 4 to Layer 7 Services
Configuring Layer 4 to Layer 7 Services

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

418
 CHAPTER 14
Configuring Global Policies
• About Global Policies, on page 419
• Configuring Out-of-Band Management NTP, on page 419
• Configuring the System Clock, on page 422
• Configuring Error Disable Recovery, on page 423
• Configuring Link Level Discovery Protocol, on page 424
• Configuring Miscabling Protocol, on page 424
• Configuring the Endpoint Loop Protection Policy, on page 426
• Configuring the Rogue Endpoint Control Policy, on page 427
• Configuring IP Aging, on page 429
• Configuring the Dynamic Load Balancer, on page 429
• Configuring Spanning Tree Protocol, on page 431
• Configuring IS-IS, on page 432
• Configuring BGP Route Reflectors, on page 435
• Decommissioning a Node, on page 436
• Configuring Power Management, on page 436
• Configuring a Scheduler, on page 438
• Configuring System MTU, on page 440
• About PTP, on page 441

About Global Policies

The APIC fabric has many fabric level configurations, which are applied to the entire fabric components
(switches and ports). In some cases, lower level policies (switch or interface level) exist to override these
policies. For example, while MCP policy can enable the MCP feature globally, an interface level MCP policy
exists to enable or disable MCP on an individual interface.

Configuring Out-of-Band Management NTP

When an ACI fabric is deployed with out-of-band management, each node of the fabric is managed from
outside the ACI fabric. You can configure an out-of-band management NTP server so that each node can
individually query the same NTP server as a consistent clock source.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

419
 Configuring Global Policies
Configuring Out-of-Band Management NTP

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
apic1# configure

Step 2 template ntp-fabric ntp-fabric-template-name Specifies the NTP template (policy) for the
fabric.
Example:
apic1(config)# template ntp-fabric pol1

Step 3 [no] server dns-name-or-ipaddress [prefer] Configures an NTP server for the active NTP
[use-vrf {inband-mgmt | oob-default}] [key policy. To make this server the preferred server
key-value] for the active NTP policy, include the prefer
keyword. If NTP authentication is enabled,
Example:
specify a reference key ID. To specify the
apic1(config-template-ntp-fabric)# in-band or out-of-band management access
server 192.0.20.123 prefer use-vrf
oob-mgmt VRF, include the use-vrf keyword with the
inb-default or oob-default keyword.

Step 4 [no] authenticate Enables (or disables) NTP authentication.

Example:
apic1(config-template-ntp-fabric)# no
authenticate

Step 5 [no] authentication-key key-value Configures an authentication NTP

authentication. The range is 1 to 65535.
Example:
apic1(config-template-ntp-fabric)#
authentication-key 12345

Step 6 [no] trusted-key key-value Configures a trusted NTP authentication. The

range is 1 to 65535.
Example:
apic1(config-template-ntp-fabric)#
trusted-key 54321

Step 7 exit Returns to global configuration mode

Example:
apic1(config-template-ntp-fabric)# exit

Step 8 template pod-group Configures a pod-group template (policy).

pod-group-template-name
Example:
apic1(config)# template pod-group
allPods

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

420
Configuring Global Policies
Configuring Out-of-Band Management NTP

Command or Action Purpose

Step 9 inherit ntp-fabric ntp-fabric-template-name Configures the NTP fabric pod-group to use
the previously configured NTP fabric template
Example:
(policy).
apic1(config-pod-group)# inherit
ntp-fabric pol1

Step 10 exit Returns to global configuration mode

Example:
apic1(config-template-pod-group)# exit

Step 11 pod-profile pod-profile-name Configures a pod profile.

Example:
apic1(config)# pod-profile all

Step 12 pods {pod-range-1-255 | all} Configures a set of pods.

Example:
apic1(config-pod-profile)# pods all

Step 13 inherit pod-group pod-group-name Associates the pod-profile with the previously
configured pod group.
Example:
apic1(config-pod-profile-pods)# inherit
pod-group allPods

Step 14 end Returns to EXEC mode.

Example:
apic1(config-pod-profile-pods)# end

Examples
This example shows how to configure a preferred out-of-band NTP server and how to verify the
configuration and deployment.
apic1# configure t
apic1(config)# template ntp-fabric pol1
apic1(config-template-ntp-fabric)# server 192.0.20.123 use-vrf oob-default
apic1(config-template-ntp-fabric)# no authenticate
apic1(config-template-ntp-fabric)# authentication-key 12345
apic1(config-template-ntp-fabric)# trusted-key 12345
apic1(config-template-ntp-fabric)# exit
apic1(config)# template pod-group allPods
apic1(config-pod-group)# inherit ntp-fabric pol1
apic1(config-pod-group)# exit
apic1(config)# pod-profile all
apic1(config-pod-profile)# pods all
apic1(config-pod-profile-pods)# inherit pod-group allPods
apic1(config-pod-profile-pods)# end
apic1#

apic1# show ntpq

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

421
 Configuring Global Policies
Configuring the System Clock

nodeid remote refid st t when poll reach delay offset jitter

------ - ------------ ------ ---- -- ----- ----- ----- ------ ------ ------
1 * 192.0.20.123 .GPS. u 27 64 377 76.427 0.087 0.067

2 * 192.0.20.123 .GPS. u 3 64 377 75.932 0.001 0.021

3 * 192.0.20.123 .GPS. u 3 64 377 75.932 0.001 0.021

Configuring the System Clock

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] clock display-format {local | utc} Sets the clock date time format to either local
or UTC time.
Example:
apic1(config)# clock display-format local

Step 3 [no] clock show-offset enable Enables (or disables) the display of the offfset
from UTC. This setting is valid only when the
Example:
display-format is local.
apic1(config)# clock show-offset enable

Step 4 [no] clock timezone timezone-code Specifies the local time zone. The default is
p0_utc.
Example:
apic1(config)# clock timezone
n420_America-Los_Angeles

Step 5 show clock Specifies the delay time for LLDP to initialize
on any interface . The range is 1 to 10 seconds;
Example:
the default is 2 seconds.
apic1(config)# show clock

Examples
This example shows how to configure the system clock for local time in the Los Angeles timezone.

apic1# configure terminal

apic1(config)# clock display-format local
apic1(config)# clock show-offset enable
apic1(config)# clock timezone n420_America-Los_Angeles
apic1(config)# show clock
Time : 20:47:37.038 UTC-08:00 Sun Nov 08 2015

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

422
 Configuring Global Policies
Configuring Error Disable Recovery

Configuring Error Disable Recovery

The error disabled recovery (EDR) policy is a fabric level policy that can enable ports that loop detection and
BPDU policies disabled after an interval that the administrator can configure.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] errdisable recovery interval seconds Specifies the interval for an interface to recover
from the error-disabled state. The range is from
Example:
30 to 65535 seconds
apic1(config)# errdisable recovery
interval 300

Step 3 [no] errdisable recovery cause {bpduguard Specifies a condition under which the interface
| ep-move | mcp-loop} automatically recovers from the error-disabled
state, and the device retries bringing the
Example:
interface up. The default is disabled. The
apic1(config)# errdisable recovery cause condition options are:
mcp-loop
• bpduguard —Enable timer to recover
from a BPDU guard error disable.
• ep-move —Enable timer to recover from
an endpoint move error disable.
• mcp-loop —Enable timer to recover from
an MCP loop error disable.
• storm-control-recovery —Enable timer
to recover from a storm control recovery
error disable.

Examples
This example shows how to configure EDR to recover from an MCP loop error disable.

apic1# configure terminal

apic1(config)# errdisable recovery interval 300
apic1(config)# errdisable recovery cause mcp-loop

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

423
 Configuring Global Policies
Configuring Link Level Discovery Protocol

Configuring Link Level Discovery Protocol

The Link Layer Discovery Protocol (LLDP) is a device discovery protocol that allows network devices to
advertise information about themselves to other devices on the network. LLDP determines the layer 2
connectivity between switches.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] lldp holdtime seconds Specifies the hold time to be sent in LLDP
packets.
Example:
apic1(config)# lldp holdtime

Step 3 [no] lldp holdtime seconds Specifies the hold time to be sent in LLDP
packets. The range is 10 to 255 seconds; the
Example:
default is 120 seconds.
apic1(config)# lldp holdtime 120

Step 4 [no] lldp reinit seconds Specifies the delay time for LLDP to initialize
on any interface . The range is 1 to 10 seconds;
Example:
the default is 2 seconds.
apic1(config)# lldp reinit 2

Step 5 [no] lldp timer seconds Specifies the transmission frequency seconds
of LLDP updates in seconds. The range is 5 to
Example:
254 seconds; the default is 30 seconds.
apic1(config)# lldp timer 30

Examples
This example shows how to configure LLDP.

apic1# configure terminal

apic1(config)# lldp holdtime 120
apic1(config)# lldp reinit 2
apic1(config)# lldp timer 30

Configuring Miscabling Protocol

The ACI fabric provides loop detection policies that can detect loops in Layer 2 network segments that are
connected to ACI access ports. The ACI fabric implements the mis-cabling protocol (MCP), a fabric level
policy that allows provisioning of MCP parameters as well as determining the port behavior if mis-cabling is

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

424
Configuring Global Policies
Configuring Miscabling Protocol

detected. MCP works in a complementary manner with STP that is running on external Layer 2 networks,
and handles Bridge Protocol Data Unit (BPDU) packets that access ports receive.
A fabric administrator provides a key that MCP uses to identify which MCP packets are initiated by the ACI
fabric. The administrator can choose how the MCP policies identify loops and how to act upon the loops:
syslog only, or disable the port.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] mcp action port-disable Specifies whether a port should be place in a
disabled state if mis-cabling is detected.
Example:
apic1(config)# mcp action port-disable

Step 3 [no] mcp enable [key key-value] Allows enabling or disabling of the MCP
protocol globally for the entire fabric. A
Example:
password (key) is required when enabling the
apic1(config)# mcp enable key policy but not when disabling.
0123456789abcdef

Step 4 [no] mcp factor number Sets the loop detection multiplication factor,
which is used while sending MCP packets. The
Example:
range is 1 to 255.
apic1(config)# mcp factor 64

Step 5 [no] mcp init-delay seconds Specifies the initial delay time. The range is 0
to 1800 seconds; the default is 180.
Example:
apic1(config)# mcp init-delay 180

Step 6 [no] mcp transmit-frequency frequency Sets the frequency of transmission of MCP
packets to detect mis-cabling. The range is 100
Example:
milliseconds to 300 seconds; the default is 2
apic1(config)# mcp transmit-frequency 2 seconds.

Examples
This example shows how to configure MCP for a transmit frequency of 2 seconds.

apic1# configure terminal

apic1(config)# mcp action port-disable
apic1(config)# mcp enable key 0123456789abcdef
apic1(config)# mcp factor 64
apic1(config)# mcp init-delay 180
apic1(config)# mcp transmit-frequency 2

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

425
 Configuring Global Policies
Configuring the Endpoint Loop Protection Policy

This example shows how to configure MCP for a transmit frequency of 2 seconds and 300
milliseconds.

apic1# configure terminal

apic1(config)# mcp action port-disable
apic1(config)# mcp enable key 0123456789abcdef
apic1(config)# mcp factor 64
apic1(config)# mcp init-delay 180
apic1(config)# mcp transmit-frequency 2 300

Configuring the Endpoint Loop Protection Policy

The endpoint loop protection policy is a fabric level policy used in detection of frequent endpoint (host) moves
from one fabric port to another. The policy configures what action is to be taken if such an event is detected.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] endpoint loop-detect action Specifies the action to perform when an
{bd-learn-disable | port-disable} endpoint loop is detected. The options are:
Example: • bd-learn-disable —Disable MAC address
apic1(config)# endpoint loop-detect learning on the bridge domain.
action port-disable
• port-disable —Disable the port.

Step 3 [no] endpoint loop-detect enable Allows enabling or disabling of the endpoint
loop protection protocol globally for the entire
Example:
fabric.
apic1(config)# endpoint loop-detect
enable

Step 4 [no] endpoint loop-detect factor number Sets the loop detection multiplication factor.
The range is 1 to 255.
Example:
apic1(config)# endpoint loop-detect
factor 64

Step 5 [no] endpoint loop-detect interval seconds Specifies the loop detection interval. The range
is 30 to 300 seconds.
Example:
apic1(config)# endpoint loop-detect
interval 60

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

426
 Configuring Global Policies
Configuring the Rogue Endpoint Control Policy

Examples
This example shows how to configure the endpoint loop protection policy.

apic1# configure terminal

apic1(config)# endpoint loop-detect action port-disable
apic1(config)# endpoint loop-detect enable
apic1(config)# endpoint loop-detect factor 64
apic1(config)# endpoint loop-detect interval 60

Configuring the Rogue Endpoint Control Policy

About the Rogue Endpoint Control Policy
A rogue endpoint attacks top of rack (ToR) switches through frequently, repeatedly injecting packets on
different ToR ports and changing 802.1Q tags (thus, emulating endpoint moves) causing learned class and
EPG port changes. Misconfigurations can also cause frequent IP and MAC address changes (moves).
Such rapid movement in the fabric causes significant network instability, high CPU usage, and in rare instances,
endpoint mapper (EPM) and EPM client (EPMC) crashes due to significant and prolonged messaging and
transaction service (MTS) buffer consumption. Also, such frequent moves may result in the EPM and EPMC
logs rolling over very quickly, hampering debugging for unrelated endpoints.
The rogue endpoint control feature addresses this vulnerability by quickly:
• Identifying such rapidly moving MAC and IP endpoints.
• Stopping the movement by temporarily making endpoints static (thus, quarantining the endpoint).
• Prior to 3.2(6) release: Keeping the endpoint static for the Rogue EP Detection Interval and dropping
the traffic to and from the rogue endpoint. After this time expires, deleting the unauthorized MAC or IP
address.
• In the 3.2(6) release and later: Keeping the endpoint static for the Rogue EP Detection Interval (this
feature no longer drops the traffic). After this time expires, deleting the unauthorized MAC or IP address.
• Generating a host tracking packet to enable the system to re-learn the impacted MAC or IP address.
• Raising a fault, to enable corrective action.

The rogue endpoint control policy is configured globally and, unlike other loop prevention methods, functions
at the level of individual endpoints (IP and MAC addresses). It does not distinguish between local or remote
moves; any type of interface change is considered a move in determining if an endpoint should be quarantined.
The rogue endpoint control feature is disabled by default.

Configure Rogue Endpoint Control Using the NX-OS Style CLI

You can configure the Rogue EP Control policy for the fabric, to detect and delete unauthorized endpoints,
using the NX-OS style CLI.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

427
 Configuring Global Policies
Configure Rogue Endpoint Control Using the NX-OS Style CLI

Procedure

Step 1 configure
Enters global configuration mode.
Example:
apic1# configure

Step 2 endpoint rogue-detect enable

Enables the global Rogue Endpoint Control policy.
Example:
apic1(config)# endpoint rogue-detect enable

Step 3 endpoint rogue-detect hold-interval hold_interval

Sets the hold interval in seconds after the endpoint is declared rogue, where it is kept static so learning is
prevented, and the traffic to and from the rogue endpoint is dropped. After this interval, the endpoint is deleted.
Valid values are from 1800 to 3600 seconds. The default is 1800.
Example:
apic1(config)# endpoint rogue-detect hold-interval 1800

Step 4 endpoint rogue-detect interval interval

Sets the rogue detection interval in seconds, which specifies the time to detect rogue endpoints. Valid values
are from 0 to 65535 seconds. The default is 60.
Example:
apic1(config)# endpoint rogue-detect interval 60

Step 5 endpoint rogue-detect factor factor

Specifies the multiplication factor for determining if an endpoint is unauthorized. If the endpoint moves more
times during the interval, the EP is declared rogue. Valid values are from 2 to 10. The default is 6.
Example:
apic1# endpoint rogue-detect factor 6

Step 6 This example configures a Rogue Endpoint Control policy.

Example:
apic1# cconfigure
apic1(config)# endpoint rogue-detect enable
apic1(config)# endpoint rogue-detect hold-interval 1800
apic1(config)# endpoint rogue-detect interval 60
apic1(config)# endpoint rogue-detect factor 6

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

428
 Configuring Global Policies
Configuring IP Aging

Configuring IP Aging
Overview
The IP Aging policy tracks and ages unused IP addresses on an endpoint. Tracking is performed using the
endpoint retention policy configured for the bridge domain to send ARP requests (for IPv4) and neighbor
solicitations (for IPv6) at 75% of the local endpoint aging interval. When no response is received from an IP
address, that IP address is aged out.
This document explains how to configure the IP Aging policy.

Configuring the IP Aging Policy Using the NX-OS-Style CLI

This section explains how to enable and disable the IP Aging policy using the CLI.

Procedure

Step 1 To enable the IP aging policy:

Example:
ifc1(config)# endpoint ip aging

Step 2 To disable the IP aging policy:

Example:
ifav9-ifc1(config)# no endpoint ip aging

What to do next
To specify the interval used for tracking IP addresses on endpoints, create an Endpoint Retention policy.

Configuring the Dynamic Load Balancer

Dynamic load balancing (DLB) adjusts the traffic allocations according to congestion levels. DLB measures
the congestion across the available paths and places the flows on the least congested paths, which results in
an optimal or near optimal placement of the data.
DLB can be configured to place traffic on the available uplinks using the granularity of flows or flowlets.
Flowlets are bursts of packets from a flow that are separated by suitably large gaps in time. If the idle interval
between two bursts of packets is larger than the maximum difference in latency among available paths, the
second burst (or flowlet) can be sent along a different path than the first without reordering packets. This idle
interval is measured with a timer called the flowlet timer. Flowlets provide a higher granular alternative to
flows for load balancing without causing packet reordering.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

429
 Configuring Global Policies
Configuring the Dynamic Load Balancer

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] system dynamic-load-balance mode Specifies the mode of operation of the load
{dynamic-aggressive | dynamic-conservative balancer. The modes are:
| link-failure-resiliency |
• dynamic-aggressive —The flowlet
packet-prioritization}
timeout is a relatively small value. This
very fine-grained dynamic load balancing
is optimal for the distribution of traffic,
but some packet reordering might occur.
However, the overall benefit to application
performance is equal to or better than the
conservative mode.
• dynamic-conservative —The flowlet
timeout is a larger value that guarantees
packets are not to be re-ordered. The
tradeoff is less granular load balancing
because new flowlet opportunities are less
frequent.
• link-failure-resiliency —Static load
balancing gives a distribution of flows
across the available links that is roughly
even.
• packet-prioritization —Dynamic Packet
Prioritization (DPP) prioritizes short flows
higher than long flows; a short flow is less
than approximately 15 packets. Because
short flows are more sensitive to latency
than long ones, DPP can improve overall
application performance.

apic1(config)# system
dynamic-load-balance mode
packet-prioritization

Examples
This example shows how to configure dynamic load balancing with packet prioritization.

apic1# configure terminal

apic1(config)# system dynamic-load-balance mode packet-prioritization

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

430
 Configuring Global Policies
Configuring Spanning Tree Protocol

Configuring Spanning Tree Protocol

Multiple spanning-tree (MST) enables multiple VLANs to be mapped to the same spanning-tree instance,
reducing the number of spanning-tree instances needed to support a large number of VLANs.

Note Multiple Spanning Tree (MST) is not supported on interfaces configured with the Per Port VLAN feature
(configuring multiple EPGs on a leaf switch using the same VLAN ID with localPort scope).

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 spanning-tree mst configuration Enters global configuration mode.

Example:
apic1(config)# spanning-tree mst
configuration

Step 3 [no] bpdu-filter Enters global configuration mode.

Example:
apic1(config-stp)# bpdu-filter

Step 4 [no] region region-name For switches to participate in multiple

spanning-tree (MST) instances, you must
Example:
consistently configure the switches with the
apic1(config-stp)# region region1 same MST configuration information. A
collection of interconnected switches that have
the same MST configuration comprises an MST
region. Each region can support up to 65
spanning-tree instances.

Step 5 [no] instance instance-id vlan vlan-range Maps VLANs to an MST instance. You can
assign a VLAN to only one spanning-tree
Example:
instance at a time. The instance ID range is 1
apic1(config-stp-region)# instance 2 vlan to 4094. To specify a VLAN range, use a
1-63
hyphen.

Step 6 revision number Specifies the configuration revision number.

The range is 0 to 65535.
Example:
apic1(config-stp-region)# revision 16

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

431
 Configuring Global Policies
Configuring IS-IS

Examples
This example shows how to configure an MST spanning-tree policy.

apic1# configure terminal

apic1(config)# spanning-tree mst configuration
apic1(config-stp)# bpdu-filter
apic1(config-stp)# region region1
apic1(config-stp-region)# instance 2 vlan 1-63
apic1(config-stp-region)# revision 16

Configuring IS-IS
Intermediate System-to-Intermediate System (IS-IS) is a dynamic link-state routing protocol that can detect
changes in the network topology and calculate loop-free routes to other nodes in the network.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 template isis-fabric isis-fabric-template-name Enters Intermediate System-to-Intermediate

System (IS-IS) configuration mode and creates
Example:
an IS-IS fabric template (policy).
apic1(config)# template isis-fabric
polIsIs

Step 3 [no] lsp-fast-flood Enables the fast-flood feature, which improves

convergence time when new link-state packets
Example:
(LSPs) are generated in the network and
apic1(config-template-isis-fabric)# shortest path first (SPF) is triggered by the new
lsp-fast-flood
LSPs.
We recommend that you enable the
fast-flooding of LSPs before the router runs
the SPF computation, to ensure that the whole
network achieves a faster convergence time.

Step 4 [no] lsp-gen-interval level-1 lsp-max-wait Configures the IS-IS throttle for LSP
[lsp-initial-wait lsp-second-wait] generation. The parameters are as follows:
Example: • lsp-max-wait —The maximum wait
apic1(config-template-isis-fabric)# between the trigger and LSP generation.
lsp-gen-interval level-1 500 500 500
• lsp-initial-wait —The initial wait between
the trigger and LSP generation.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

432
Configuring Global Policies
Configuring IS-IS

Command or Action Purpose

• lsp-second-wait —The second wait used
for LSP throttle during backoff.

The lsp-max-wait parameter is required. The

other two parameters are optional but must
appear together. The range for each is 50 to
120000 milliseconds.

Step 5 [no] lsp-mtu mtu Sets the maximum transmission unit (MTU)
size of IS-IS hello packets. The range is 256
Example:
to 4352.
apic1(config-template-isis-fabric)#
lsp-mtu 2048 IS-IS hello packets are used to discover and
maintain adjacencies. By default, the hello
packets are padded to the full maximum
transmission unit (MTU) size to allow for early
detection of errors due to transmission
problems with large frames or due to
mismatched MTUs on adjacent interfaces.
However, IS-IS adjacency formation may fail
due to MTU mismatch on a link, requiring the
adjustment of the MTU size.

Step 6 [no] spf-interval level-1 spf-max-wait Configures the interval between LSA arrivals.
[spf-initial-wait spf-second-wait] The parameters are as follows:
Example: • spf-max-wait —The maximum wait
apic1(config-template-isis-fabric)# between the trigger and SPF computation.
spf-interval level-1 500 500 500
• spf-initial-wait —The initial wait between
the trigger and SPF computation.
• spf-second-wait —The second wait used
for SPF computation during backoff.

The spf-max-wait parameter is required. The

other two parameters are optional but must
appear together. The range for each is 50 to
120000 milliseconds.

Step 7 exit Returns to global configuration mode.

Example:
apic1(config-template-isis-fabric)# exit

Step 8 template pod-group Creates a pod group template (policy).

pod-group-template-name
Example:
apic1(config)# template pod-group
allPods

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

433
 Configuring Global Policies
Configuring IS-IS

Command or Action Purpose

Step 9 inherit pod-group pod-group-name Configures the template pod-group to use the
previously configured isis-fabric template
Example:
(policy).
apic1(config-pod-group)# inherit
isis-fabric polIsIs

Step 10 exit Returns to global configuration mode.

Example:
apic1(config-pod-group)# exit

Step 11 pod-profile pod-profile-name Configures a pod profile.

Example:
apic1(config)# pod-profile all

Step 12 pods {pod-range-1-255 | all} Configures a set of pods.

Example:
apic1(config-pod-profile)# pods all

Step 13 inherit pod-group pod-group-name Configures the pod-profile to use the

previously configured pod group.
Example:
apic1(config-pod-profile-pods)# inherit
pod-group allPods

Step 14 end Returns to EXEC mode.

Example:
apic1(config-pod-profile-pods)# end

Examples
This example shows how to configure IS-IS.

aapic1# configure
apic1(config)# template isis-fabric polIsIs
apic1(config-template-isis-fabric)# lsp-fast-flood
apic1(config-template-isis-fabric)# lsp-gen-interval level-1 500 500 500
apic1(config-template-isis-fabric)# lsp-mtu 2048
apic1(config-template-isis-fabric)# spf-interval level-1 500 500 500
apic1(config-template-isis-fabric)# exit
apic1(config)# template pod-group allPods
apic1(config-pod-group)# inherit isis-fabric polIsIs
apic1(config-pod-group)# exit
apic1(config)# pod-profile all
apic1(config-pod-profile)# pods all
apic1(config-pod-profile-pods)# inherit pod-group allPods
apic1(config-pod-profile-pods)# end
apic1#

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

434
 Configuring Global Policies
Configuring BGP Route Reflectors

Configuring BGP Route Reflectors

The ACI fabric route reflectors use multiprotocol Border Gateway Protocol (MP-BGP) to distribute external
routes within the fabric. To enable route reflectors in the ACI fabric, the fabric administrator must select the
spine switches that will be the route reflectors, and provide the autonomous system (AS) number. For
redundancy purposes, more than one spine is configured as a router reflector node (one primary and one
secondary reflector).

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 bgp-fabric Enters BGP configuration mode for the fabric.

Example:
apic1(config)# bgp-fabric

Step 3 asn asn-value Configures the BGP Autonomous System

number (ASN), which uniquely identifies an
Example:
autonomous system. The ASN is between 1 and
apic1(config-bgp-fabric)# asn 123456789 4294967295.
We recommend that you enable the
fast-flooding of LSPs before the router runs the
SPF computation, to ensure that the whole
network achieves a faster convergence time.

Step 4 [no] route-reflector spine list Configure up to two spine nodes as route
reflectors. For redundancy ,you should
Example:
configure primary and secondary route
apic1(config-bgp-fabric)# route-reflector reflectors.
spine spine1,spine2

Examples
This example shows how to configure spine1 and spine2 as BGP route reflectors.
apic1# configure
apic1(config)# bgp-fabric
apic1(config-bgp-fabric)# asn 123456789
apic1(config-bgp-fabric)# route-reflector spine spine1,spine2
apic1(config-bgp-fabric)# exit
apic1(config)#

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

435
 Configuring Global Policies
Decommissioning a Node

Decommissioning a Node
Two levels of decommissioning are supported:
• Regular—Similar to disabling the node. After being decommissioned, the node cannot rejoin the fabric
until the no decommission command is executed.
• Complete—When the node is decommissioned, all fabric configuration related to the node is cleared.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] decommission {controller | switch} Decommissions the specified node. Note that
node-id [remove-from-controller] controller node ID numbers are between 1 and
100, while switch node ID numbers are between
Example:
101 and 4000.
apic1(config)# decommission switch 104
remove-from-controller

Examples
This example shows how to perform a complete decommissioning of node 104 (a switch) and
recommission node 5 (a controller), which was decommissioned with the regular level.

apic1# configure
apic1(config)# decommission switch 104 remove-from-controller
apic1(config)# no decommission controller 5

Configuring Power Management

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] power redundancy-policy policy-name Creates or configures a power supply

redundancy policy.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

436
Configuring Global Policies
Configuring Power Management

Command or Action Purpose

apic1(config)# power redundancy-policy
myPowerPolicy

Step 3 [no] description text Adds a description for this power supply
redundancy policy. If the text includes spaces,
Example:
it must be enclosed in single quotes.
apic1(config-power)# description 'This
is my power redundancy policy'

Step 4 [no] redundancy-mode {combined | Specifies power supply redundancy mode.

ps-redundant | redundant}
• combined — This mode does not provide
Example: power redundancy. The available power
apic1(config-power)# redundancy-mode is the total power capacity of all power
ps-redundant supplies.
• ps-redundant —This mode provides an
extra power supply in case an active power
supply goes down. The power supply that
can supply the most power operates in
standby mode. The other one or two power
supplies are active. The available power
is the amount of power provided by the
active power supply units.
• redundant —This mode combines power
supply redundancy and input source
redundancy, which means that the chassis
has an extra power supply and each half
of each power supply is connected to one
electrical grid while the other half of each
power supply is connected to the other
electrical grid. The available power is the
lesser of the available power for power
supply mode and input source mode.

Examples
This example shows how to configure a power supply redundancy policy for the ps-redundant mode.

apic1# configure
apic1(config)# power redundancy-policy myPowerPolicy
apic1(config-pod)# isis fabric
apic1(config-power)# description 'This is my power redundancy policy'
apic1(config-power)# redundancy-mode ps-redundant

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

437
 Configuring Global Policies
Configuring a Scheduler

Configuring a Scheduler
A schedule allows operations, such as configuration import/export or tech support collection, to occur during
one or more specified windows of time.
A schedule contains a set of time windows (occurrences). These windows can be one time only or can recur
at a specified time and day each week. The options defined in the window, such as the duration or the maximum
number of tasks to be run, determine when a scheduled task will execute. For example, if a change cannot be
deployed during a given maintenance window because the maximum duration or number of tasks has been
reached, that deployment is carried over to the next maintenance window.
Each schedule checks periodically to see whether the APIC has entered one or more maintenance windows.
If it has, the schedule executes the deployments that are eligible according to the constraints specified in the
maintenance policy.
A schedule contains one or more occurrences, which determine the maintenance windows associated with
that schedule. An occurrence can be one of the following:
• Absolute (One Time) Window—An absolute window defines a schedule that will occur only once. This
window continues until the maximum duration of the window or the maximum number of tasks that can
be run in the window has been reached.
• Recurring Window—A recurring window defines a repeating schedule. This window continues until the
maximum number of tasks or the end of the day specified in the window has been reached.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] scheduler schedule-name Creates a new scheduler or configures an

existing scheduler.
Example:
apic1(config)# scheduler controller
schedule myScheduler

Step 3 [no] description text Adds a description for this scheduler. If the
text includes spaces, it must be enclosed in
Example:
single quotes.
apic1(config-scheduler)# description
'This is my scheduler'

Step 4 [no] absolute window window-name Creates an absolute (one time) window
schedule.
Example:
apic1(config-scheduler)# absolute window
myAbsoluteWindow

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

438
Configuring Global Policies
Configuring a Scheduler

Command or Action Purpose

Step 5 [no] max concurrent nodes count Sets the maximum number of nodes (tasks)
that can be processed concurrently. The range
Example:
is 0 to 65535. Set to 0 for unlimited nodes.
apic1(config-scheduler-absolute)# max
concurrent nodes 300

Step 6 [no] max running time time Sets the maximum running time for tasks in
the format dd:hh:mm:ss. The range is 0 to
Example:
65535. Set to 0 for no time limit.
apic1(config-scheduler-absolute)# max
running time 00:01:30:00

Step 7 [no] time start time Sets the starting time in the format
[[[yyyy:]mmm:]dd:]HH:MM.
Example:
apic1(config-scheduler-absolute)# time
start 2016:jan:01:12:01

Step 8 exit Returns to scheduler configuration mode.

Example:
apic1(config-scheduler-absolute)# exit

Step 9 [no] recurring window window-name Creates a recurring window schedule.

Example:
apic1(config-scheduler)# recurring
window myRecurringWindow

Step 10 [no] max concurrent nodes count Sets the maximum number of nodes (tasks)
that can be processed concurrently. The range
Example:
is 0 to 65535. Set to 0 for unlimited nodes.
apic1(config-scheduler-recurring)# max
concurrent nodes 300

Step 11 [no] max running time time Sets the maximum running time for tasks in
the format dd:hh:mm:ss. The range is 0 to
Example:
65535. Set to 0 for no time limit.
apic1(config-scheduler-recurring)# max
running time 00:01:30:00

Step 12 [no] time start {daily HH:MM | weekly (See Sets the period (daily or weekly) and starting
usage) HH:MM} time. If weekly is selected, choose from these
options:
Example:
apic1(config-scheduler-recurring)# time • monday
start weekly wednesday 12:30
• tuesday
• wednesday
• thursday
• friday
• saturday

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

439
 Configuring Global Policies
Configuring System MTU

Command or Action Purpose

• sunday
• even-day
• odd-day
• every-day

Examples
This example shows how to configure a recurring scheduler to run every Wednesday.

apic1# configure
apic1(config)# scheduler controller schedule myScheduler
apic1(config-scheduler)# description 'This is my scheduler'
apic1(config-scheduler)# recurring window myRecurringWindow
apic1(config-scheduler-recurring)# max concurrent nodes 300
apic1(config-scheduler-recurring)# max running time 00:01:30:00
apic1(config-scheduler-recurring)# time start weekly wednesday 12:30

Configuring System MTU

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] system jumbomtu size Sets the maximum transmit unit (MTU) for host
facing ports. Up to Cisco APIC Release 3.1(2),
Example:
the range is 576 to 9000 bytes. From release
apic1(config)# system jumbomtu 9000 3.1(2), and later, the maximum MTU value is
9216. The default has not changed from 9000.

Examples
This example shows how to configure the system MTU size.

apic1# configure terminal

apic1(config)# system jumbomtu 9000

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

440
 Configuring Global Policies
About PTP

About PTP
Precision Time Protocol (PTP) is a time synchronization protocol defined in IEEE 1588 for nodes distributed
across a network. With PTP, it is possible to synchronize distributed clocks with an accuracy of less than 1
microsecond via Ethernet networks. PTP’s accuracy comes from the hardware support for PTP in the ACI
fabric spines and leafs. It allows the protocol to accurately compensate for message delays and variation across
the network.
PTP is a distributed protocol that specifies how real-time PTP clocks in the system synchronize with each
other. These clocks are organized into a master-slave synchronization hierarchy with the grandmaster clock,
which is the clock at the top of the hierarchy, determining the reference time for the entire system.
Synchronization is achieved by exchanging PTP timing messages, with the members using the timing
information to adjust their clocks to the time of their master in the hierarchy. PTP operates within a logical
scope called a PTP domain.
The PTP process consists of two phases: establishing the master-slave hierarchy and synchronizing the clocks.
Within a PTP domain, each port of an ordinary or boundary clock follows this process to determine its state:
• Examines the contents of all received announce messages (issued by ports in the master state).
• Compares the data sets of the foreign master (in the announce message) and the local clock for priority,
clock class, accuracy, and so on.
• Determines its own state as either master or slave.

After the master-slave hierarchy has been established, the clocks are synchronized as follows:
• The master sends a synchronization message to the slave and notes the time it was sent.
• The slave receives the synchronization message and notes the time that it was received. For every
synchronization message, there is a follow-up message. Hence, the number of sync messages should be
equal to the number of follow-up messages.
• The slave sends a delay-request message to the master and notes the time it was sent.
• The master receives the delay-request message and notes the time it was received.
• The master sends a delay-response message to the slave. The number of delay request messages should
be equal to the number of delay response messages.
• The slave uses these timestamps to adjust its clock to the time of its master.

In ACI fabric, when PTP feature is globally enabled in APIC, the software automatically enables PTP on
specific interfaces of all the supported spines and leafs. This auto-configuration ensures that PTP is optimally
enabled on all the supported nodes. In the absence of an external grandmaster clock, one of the spine switch
is chosen as the grandmaster. The master spine is given a different PTP priority as compared to the other
spines and leaf switches so that they will act as PTP slaves. This way we ensure that all the leaf switches in
the fabric synchronize to the PTP clock of the master spine.
If an external Grandmaster clock is connected to the spines, the spine syncs to the external GM and in turn
acts as a master to the leaf nodes.

PTP Default Settings

The following table lists the default settings for PTP parameters.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

441
 Configuring Global Policies
Guidelines and Limitations

Parameters Default

PTP device type Boundary clock

PTP clock type Two-step clock

PTP domain 0

PTP priority 1 value when advertising the clock 255

PTP priority 2 value when advertising the clock 255

PTP announce interval 1 log second

PTP announce timeout 3 announce intervals

PTP delay-request interval 0 log seconds

PTP sync interval -2 log seconds

PTP VLAN 1

Note PTP operates only in boundary clock mode. Cisco recommends deployment of a Grand Master Clock (10
MHz) upstream, with servers containing clocks requiring synchronization connected to the switch.

PTP Verification

Command Purpose

show ptp brief Displays the PTP status.

show ptp clock Displays the properties of the local clock, including
clock identity.

show ptp clock foreign-masters record interface Displays the state of foreign masters known to the
ethernet slot/port PTP process. For each foreign master, the output
displays the clock identity, basic clock properties, and
whether the clock is being used as a grandmaster.

show ptp corrections Displays the last few PTP corrections.

show ptp counters [all |interface Ethernet slot/port] Displays the PTP packet counters for all interfaces or
for a specified interface.

show ptp parent Displays the properties of the PTP parent.

Guidelines and Limitations

Follow these guidelines and limitations:
• Latency requires all the nodes in the fabric to be synchronized using Precision Time Protocol (PTP).

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

442
Configuring Global Policies
Guidelines and Limitations

• Latency measurement and PTP are only supported on the following switches:
• N9K-C93108TC-EX
• N9K-C93108TC-FX
• N9K-C93180LC-EX
• N9K-C93180YC-EX
• N9K-C93180YC-FX
• N9K-C9364C
• N9K-X9732C-EX
• N9K-X9736C-EX
• N9K-X9736C-FX

• Latency measurement is supported only for the packets that ingress, egress, and transit through EX or
FX-based TORs.
• All the spine nodes in the fabric should have EX or FX-based line cards to support PTP.
• PTP and the latency feature is not supported on any N9K-C93128TX, N9K-C9396PX, and N9K-C9396TX
TORs or spine switches. In the presence of non-EX/FX TORs in the fabric, we recommend that you have
the external GM connectivity to all the spine switches to ensure that the PTP time is synced across all
the supported TORs.
• External Grandmaster (GM) clock is not mandatory for PTP in a single Pod. If there is no external GM
connected to the ACI fabric, one of the spine nodes acts as the GM. This spine switch has a PTP priority1
value as 254. All the other spine switches and leaf switches in the fabric will synchronize their clock to
this Master spine switch clock. If the external GM is connected later to the spine switch, it should have
a priority value less than 254 for it to act as the GM for the entire fabric.
• External Grandmaster clock is mandatory for PTP in a multipod scenario. In addition, external GM needs
to be connected to the IPN such that the Grandmaster clock is the master to the spine switches in different
PODs. The spine switches connected to IPN will act as the boundary clock and all the nodes within the
POD will sync their clock this spine switch.
• PTP operates only in boundary clock mode. End-to-end transparent clock and peer-to-peer transparent
clock modes are not supported.
• PTP supports transport over User Datagram Protocol (UDP). Transport over Ethernet is not supported.
• PTP supports multicast communication only; unicast mode is not supported.
• Beginning with release 4.0(1), support is added for changing the resolution factor to 11 which then can
measure up to 214 milliseconds with an accuracy of 204ns.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

443
 Configuring Global Policies
Configuring PTP Using the NX-OS CLI

Configuring PTP Using the NX-OS CLI

Procedure

Step 1 Enable PTP.

Example:
Enable ptp:
========
apic# configure terminal
apic(config)# ptp
Disable ptp:
========
apic# configure terminal
apic(config)# no ptp

Step 2 To verify PTP on ACI switches:

Example:
leaf1# show ptp brief
PTP port status
-----------------------
Port State
------- --------------
Eth1/49 Slave

leaf1#
leaf1#
leaf1# show ptp clock
PTP Device Type: Boundary clock
Clock Identity : 0c:75:bd:ff:fe:03:1d:10
Clock Domain: 0
Number of PTP ports: 1
Priority1 : 255
Priority2 : 255
Clock Quality:
Class : 248
Accuracy : 254
Offset (log variance) : 65535
Offset From Master : 32
Mean Path Delay : 128
Steps removed : 1
Local clock time:Thu Jul 27 19:43:42 2017

leaf1#
leaf1# show ptp clock foreign-masters record interface ethernet 1/49

P1=Priority1, P2=Priority2, C=Class, A=Accuracy,

OSLV=Offset-Scaled-Log-Variance, SR=Steps-Removed
GM=Is grandmaster

--------- ----------------------- --- ---- ---- --- ----- --------

Interface Clock-ID P1 P2 C A OSLV SR
--------- ----------------------- --- ---- ---- --- ----- --------

Eth1/49 d4:6d:50:ff:fe:e6:4d:3f 254 255 248 254 65535 0 GM

leaf1#

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

444
Configuring Global Policies
Configuring PTP Using the NX-OS CLI

leaf1#
leaf1# show ptp corrections

PTP past corrections

-----------------------------------------------------------------------------------
Slave Port SUP Time Correction(ns) MeanPath Delay(ns)
---------- ------------------------------- ------------------ ------------------
Eth1/49 Thu Jul 27 19:44:11 2017 364281 36 152
Eth1/49 Thu Jul 27 19:44:11 2017 114565 16 132
Eth1/49 Thu Jul 27 19:44:10 2017 862912 8 132
Eth1/49 Thu Jul 27 19:44:10 2017 610823 8 132
Eth1/49 Thu Jul 27 19:44:10 2017 359557 16 132
Eth1/49 Thu Jul 27 19:44:10 2017 109937 8 132
Eth1/49 Thu Jul 27 19:44:09 2017 858113 16 132
Eth1/49 Thu Jul 27 19:44:09 2017 606536 16 132
Eth1/49 Thu Jul 27 19:44:09 2017 354837 -16 132
Eth1/49 Thu Jul 27 19:44:09 2017 104226 24 148
Eth1/49 Thu Jul 27 19:44:08 2017 853263 24 148
Eth1/49 Thu Jul 27 19:44:08 2017 601780 16 148
Eth1/49 Thu Jul 27 19:44:08 2017 349639 -4 148
Eth1/49 Thu Jul 27 19:44:08 2017 99970 16 144
Eth1/49 Thu Jul 27 19:44:07 2017 848507 0 144
Eth1/49 Thu Jul 27 19:44:07 2017 596143 24 144
Eth1/49 Thu Jul 27 19:44:07 2017 344808 4 144
Eth1/49 Thu Jul 27 19:44:07 2017 93156 -16 140
Eth1/49 Thu Jul 27 19:44:06 2017 843263 24 140
Eth1/49 Thu Jul 27 19:44:06 2017 590189 8 140
leaf1#
leaf1#
leaf1# show ptp counters all

PTP Packet Counters of Interface Eth1/49:

----------------------------------------------------------------
Packet Type TX RX
---------------- -------------------- --------------------
Announce 56 5424
Sync 441 43322
FollowUp 441 43321
Delay Request 7002 0
Delay Response 0 7002
PDelay Request 0 0
PDelay Response 0 0
PDelay Followup 0 0
Management 0 0
----------------------------------------------------------------

leaf1#
leaf1#
leaf1# show ptp parent

PTP PARENT PROPERTIES

Parent Clock:
Parent Clock Identity: d4:6d:50:ff:fe:e6:4d:3f
Parent Port Number: 258
Observed Parent Offset (log variance): N/A
Observed Parent Clock Phase Change Rate: N/A

Grandmaster Clock:
Grandmaster Clock Identity: d4:6d:50:ff:fe:e6:4d:3f
Grandmaster Clock Quality:
Class: 248
Accuracy: 254
Offset (log variance): 65535

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

445
 Configuring Global Policies
Configuring PTP Using the NX-OS CLI

Priority1: 254
Priority2: 255

leaf1#

Step 3 To verify troubleshooting steps:

Example:
apic1# show troubleshoot eptoep session eptoep latency

Source --> Destination

Last Collection(30 seconds)
+--------------------+-------------------------------+--------------+
| Average (microsec) | Standard Deviation (microsec) | Packet Count |
+--------------------+-------------------------------+--------------+
| 18 | 24 | 1086 |
| | | |
+--------------------+-------------------------------+--------------+

Cumulative
+--------------------+----------------+--------------------+
| Average (microsec) | Max (microsec) | Total Packet Count |
+--------------------+----------------+--------------------+
| 18 | 202 | 6117438 |
| | | |
+--------------------+----------------+--------------------+

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

446
 CHAPTER 15
Configuring Cisco Tetration Analytics
• Overview, on page 447
• Configuring Cisco Tetration Analytics Using the NX-OS Style CLI, on page 447

Overview
This article provides examples of how to configure Cisco Tetration when using the Cisco APIC. The following
information applies when configuring Cisco Tetration.
• An inband management IP address must be configured on each leaf where the Cisco Tetration agent is
active.
• Define an analytics policy and specify the destination IP address of the Cisco Tetration server.
• Create a switch profile and include the policy group created in the previous step.

Configuring Cisco Tetration Analytics Using the NX-OS Style

CLI
Procedure

Step 1 configure terminal

Enters global configuration mode.
Example:
apic1# configure terminal

Step 2 analytics cluster cluster_name

Create the analytics policy.
Example:
apic1(config)# analytics cluster cluster1

Step 3 flow-exporter server_name

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

447
 Configuring Cisco Tetration Analytics
Configuring Cisco Tetration Analytics Using the NX-OS Style CLI

Configure external analytics information.

Example:
apic1(config-analytics)# flow-exporter server1

Step 4 destination ip_address

Configure the destination port.
Example:
apic1(config-analytics-cluster-exporter)# destination 192.0.2.1

Step 5 exit
Exit command mode.
Example:
# apic1(config-analytics-cluster-exporter)# exit

Step 6 exit
Exit command mode.
Example:
apic1(config-analytics)# exit

Step 7 fabric-internal
Enters fabric internal configuration mode.
Example:
apic1(config)# fabric-internal

Step 8 template leaf-policy-group leaf_group_name

Define leaf policy group.
Example:
apic1(config-fabric-internal)# template leaf-policy-group lpg1

Step 9 inherit analytics-policy cluster cluster_name server server_name

Associate analytics policy to leaf policy group.
Example:
apic1(config-leaf-policy-group)# inherit analytics-policy cluster cluster1 server server1

Step 10 exit
Exit command mode.
Example:
apic1(config-leaf-policy-group)# exit

Step 11 leaf-profile lleaf_profile_name

Define leaf profile.
Example:
apic1(config-fabric-internal)# leaf-profile lp1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

448
Configuring Cisco Tetration Analytics
Configuring Cisco Tetration Analytics Using the NX-OS Style CLI

Step 12 leaf-group leaf_group_name

Define leaf group.
Example:
apic1(config-leaf-profile)# leaf-group lg1

Step 13 leaf-policy-group leaf_policy_group_name

Associate leaf policy group to leaf group.
Example:
apic1(config-leaf-group)# leaf-policy-group lpg1

Step 14 leaf leaf_group_number

Add nodes to leaf group.
Example:
apic1(config-leaf-group)# leaf 101

Step 15 show analytics

Display analytics.
Note The destination port is not configurable. UDP port 5640 is always used for leaf switches and UDP
port 5641 is always used for spine switches.
The DSCP is not configurable. VA (Voice Admit) is always used.

Example:
apic1# show analytics
Cluster : cluster1
Config Server Name : server1
Destination IP : 192.0.2.1
Destination Port : unspecified
DSCP : VA

Step 16 show running-config analytics

Display running configuration analytics.
Example:
apic1# show running-config analytics
# Command: show running-config analytics
# Time: Wed May 25 21:14:43 2016
analytics cluster cluster1
flow-exporter server1
destination 192.0.2.1
destination-port unspecified
dscp VA
ip-filter-action deny
exit
exit
apic1#

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

449
 Configuring Cisco Tetration Analytics
Configuring Cisco Tetration Analytics Using the NX-OS Style CLI

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

450
 CHAPTER 16
Configuring NetFlow
• About NetFlow, on page 451
• Configuring a NetFlow Exporter Policy for Virtual Machine Networking Using the NX-OS-Style CLI,
on page 452
• Configuring NetFlow and Tetration Analytics Feature Priority Through Node Control Policy Using
NX-OS-Style CLI, on page 452
• Configuring NetFlow Node Policy Using the NX-OS-Style CLI, on page 453
• Configuring NetFlow Infra Selectors Using the NX-OS-Style CLI, on page 453
• Configuring NetFlow Overrides Using the NX-OS-Style CLI, on page 456
• Configuring NetFlow Tenant Hierarchy Using the NX-OS-Style CLI, on page 456
• Consuming a NetFlow Exporter Policy Under a VMM Domain Using the NX-OS-Style CLI for VMware
VDS, on page 460
• Enabling or Disabling NetFlow on an Endpoint Group Using the NX-OS-Style CLI for VMware VDS,
on page 460

About NetFlow
The NetFlow technology provides the metering base for a key set of applications, including network traffic
accounting, usage-based network billing, network planning, as well as denial of services monitoring, network
monitoring, outbound marketing, and data mining for both service providers and enterprise customers. Cisco
provides a set of NetFlow applications to collect NetFlow export data, perform data volume reduction, perform
post-processing, and provide end-user applications with easy access to NetFlow data. If you have enabled
NetFlow monitoring of the traffic flowing through your datacenters, this feature enables you to perform the
same level of monitoring of the traffic flowing through the Cisco Application Centric Infrastructure (Cisco
ACI) fabric.
Instead of hardware directly exporting the records to a collector, the records are processed in the supervisor
engine and are exported to standard NetFlow collectors in the required format.
For information about configuring NetFlow with virtual machine networking, see the Cisco ACI Virtualization
Guide.

Note NetFlow is only supported on EX switches. See the Cisco NX-OS Release Notes for Cisco Nexus 9000 Series
ACI-Mode Switches document for the release that you have installed for a list of the supported EX switches.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

451
 Configuring NetFlow
Configuring a NetFlow Exporter Policy for Virtual Machine Networking Using the NX-OS-Style CLI

Configuring a NetFlow Exporter Policy for Virtual Machine

Networking Using the NX-OS-Style CLI
The following example procedure uses the NX-OS-style CLI to configure a NetFlow exporter policy for
virtual machine networking.

Procedure

Step 1 Enter the configuration mode.

Example:
apic1# config

Step 2 Configure the exporter policy.

Example:
apic1(config)# flow vm-exporter vmExporter1 destination address 2.2.2.2 transport udp 1234
apic1(config-flow-vm-exporter)# source address 4.4.4.4
apic1(config-flow-vm-exporter)# exit
apic1(config)# exit

Configuring NetFlow and Tetration Analytics Feature Priority

Through Node Control Policy Using NX-OS-Style CLI
The following example procedure uses the NX-OS-style CLI to configure the NetFlow and Tetration Analytics
feature priority through a node control policy:

Procedure

Step 1 Enter the configuration mode.

Example:
apic1# config

Step 2 Create a node control policy.

Example:
apic1(config)# node-control policy pol1

Step 3 Set NetFlow as the priority feature.

Example:
apic1(config-node)# feature netflow

Step 4 Exit the node control policy configuration.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

452
 Configuring NetFlow
Configuring NetFlow Node Policy Using the NX-OS-Style CLI

Example:
apic1(config-node)# end

Step 5 Deploy the policy to node 101 and node 102.

Example:
ifav-isim15-ifc1(config)# fabric-internal
ifav-isim15-ifc1(config-fabric-internal)# template leaf-policy-group lpg1
ifav-isim15-ifc1(config-leaf-policy-group)# inherit node-control-policy pol1
ifav-isim15-ifc1(config-leaf-policy-group)# exit
ifav-isim15-ifc1(config-fabric-internal)# leaf-profile leafProfile1
ifav-isim15-ifc1(config-leaf-profile)# leaf-group leafgrp1
ifav-isim15-ifc1(config-leaf-group)# leaf 101
ifav-isim15-ifc1(config-leaf-group)# leaf 102
ifav-isim15-ifc1(config-leaf-group)# leaf-policy-group lpg1
ifav-isim15-ifc1(config-leaf-group)# end

Configuring NetFlow Node Policy Using the NX-OS-Style CLI

The following example procedure uses the NX-OS-style CLI to configure a NetFlow node policy:

Procedure

Step 1 Enter the configuration mode.

Example:
apic1# config

Step 2 Configure the node policy.

Example:
apic1(config)# flow node-policy nodePol
apic1(config-flow-node-pol)# flow timeout collection 100
apic1(config-flow-node-pol)# flow timeout template 123
apic1(config-flow-node-pol)# exit

Configuring NetFlow Infra Selectors Using the NX-OS-Style CLI

You can use the NX-OS-style CLI to configure NetFlow infra selectors. The infra selectors are used for
attaching a Netflow monitor to a PHY, port channel, virtual port channel, fabric extender (FEX), or port
channel fabric extender (FEXPC) interface.
The following example CLI commands show how to configure NetFlow infra selectors using the NX-OS-style
CLI:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

453
 Configuring NetFlow
Configuring NetFlow Infra Selectors Using the NX-OS-Style CLI

Procedure

Step 1 Enter the configuration mode.

Example:
apic1# config

Step 2 Create a NetFlow exporter policy.

Example:
In the following commands, the destination endpoint group is the endpoint group that the exporter sits behind.
This endpoint group can also be an external Layer 3 endpoint group.
apic1(config)# flow exporter infraExporter1 destination address 1.2.3.4 transpo udp 1234
apic1(config-flow-exporter)# destination epg tenant tn2 application ap2 epg epg2
apic1(config-flow-exporter)# vrf member tenant tn2 vrf vrf2
apic1(config-flow-exporter)# version v9
apic1(config-flow-exporter)# source address 1.1.1.1
apic1(config-flow-exporter)# exit

Step 3 Create a second NetFlow exporter policy.

Example:
In the following commands, the destination endpoint group is the endpoint group that the exporter sits behind,
which in this case is an external Layer 3 endpoint group.
apic1(config)# flow exporter infraExporter2
apic1(config-flow-exporter)# transport udp 9990
apic1(config-flow-exporter)# destination address 2001:db5:a0c:1f0::2
apic1(config-flow-exporter)# destination external-l3 epg tenant tn2 vrf v2 epg accounting-inst
apic1(config-flow-exporter)# vrf member tenant tn2 vrf vrf2
apic1(config-flow-exporter)# version v5
apic1(config-flow-exporter)# source address 2001:db8:a0b:12f0::1
apic1(config-flow-exporter)# exit

Step 4 Create a NetFlow record policy.

Example:
apic1(config)# flow record infraRecord1
apic1(config-flow-record)# match dst-ip
apic1(config-flow-record)# match dst-ipv4
apic1(config-flow-record)# match dst-ipv6
apic1(config-flow-record)# match dst-mac
apic1(config-flow-record)# match dst-port
apic1(config-flow-record)# match ethertype
apic1(config-flow-record)# match proto
apic1(config-flow-record)# match src-ip
apic1(config-flow-record)# match src-ipv4
apic1(config-flow-record)# match src-ipv6
apic1(config-flow-record)# match src-mac
apic1(config-flow-record)# match src-port
apic1(config-flow-record)# match tos
apic1(config-flow-record)# match vlan
apic1(config-flow-record)# collect count-bytes
apic1(config-flow-record)# collect count-pkts
apic1(config-flow-record)# collect pkt-disp
apic1(config-flow-record)# collect sampler-id
apic1(config-flow-record)# collect src-intf
apic1(config-flow-record)# collect tcp-flags

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

454
Configuring NetFlow
Configuring NetFlow Infra Selectors Using the NX-OS-Style CLI

apic1(config-flow-record)# collect ts-first

apic1(config-flow-record)# collect ts-recent
apic1(config-flow-record)# exit

Step 5 Create a NetFlow monitor policy.

Example:
apic1(config)# flow monitor infraMonitor1
apic1(config-flow-monitor)# record infraRecord1
apic1(config-flow-monitor)# exporter infraExporter1
apic1(config-flow-monitor)# exporter infraExporter2
apic1(config-flow-monitor)# exit

You can attach a maximum of two exporters.

Step 6 Create an interface policy group (AccPortGrp).

Example:
apic1(config)# template policy-group pg1
apic1(config-pol-grp-if)# ip flow monitor infraMonitor1
apic1(config-pol-grp-if)# ipv6 flow monitor infraMonitor2
apic1(config-pol-grp-if)# exit

You can have one monitor policy per address family (IPv4 and IPv6).

Step 7 Create a node profile and infra selectors.

Example:
apic1(config)# leaf-profile lp1
apic1(config-leaf-profile)# leaf-group lg1
apic1(config-leaf-group)# leaf 101
apic1(config-leaf-profile)# exit
apic1(config)# leaf-interface-profile lip1
apic1(config-leaf-if-profile)# exit
apic1(config)# leaf-interface-profile lip1
apic1(config-leaf-if-profile)# leaf-interface-group lig1
apic1(config-leaf-if-group)# interface ethernet 1/5
apic1(config-leaf-if-profile)# policy-group pg1
apic1(config-leaf-if-profile)# exit
apic1(config-leaf-profile)# exit

Step 8 Create a port channel policy group (AccBndlGrp).

Example:
apic1(config)# template port-channel po6
apic1(config-if)# ip flow monitor infraMonitor1
apic1(config-if)# ipv6 flow monitor infraMonitor1
apic1(config-if)# exit
apic1(config-leaf-profile)# leaf-profile lp2
apic1(config-leaf-group)# leaf-group lg2
apic1(config-leaf-profile)# leaf 101
apic1(config-leaf-profile)# exit
apic1(config)# leaf-interface-profile lip2
apic1(config-leaf-if-profile)# exit
apic1(config)# leaf-interface-profile lip2
apic1(config-leaf-if-profile)# leaf-interface-group lig2
apic1(config-leaf-if-group)# interface ethernet 1/6
apic1(config-leaf-if-profile)# channel-group po6
apic1(config-leaf-if-profile)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

455
 Configuring NetFlow
Configuring NetFlow Overrides Using the NX-OS-Style CLI

You can have one monitor policy per address family (IPv4 and IPv6). The interfaces can also be vPCs.

Configuring NetFlow Overrides Using the NX-OS-Style CLI

The following procudure configures NetFlow overrides using the NX-OS-Style CLI:

Procedure

Step 1 Enter the configuration mode.

Example:
apic1# config

Step 2 Create the override.

Example:
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant tn2 vrf vrf2
apic1(config-leaf)# exit
apic1(config)# interface ethernet 1/15
apic1(config-if)# ip flow monitor infraMonitor1
apic1(config-if)# ipv6 flow monitor infraMonitor2
apic1(config-if)# exit
apic1(config)# exit
apic1# exit

You can have one monitor policy per address family (IPv4 and IPv6). The interfaces can also be vPCs.

Configuring NetFlow Tenant Hierarchy Using the NX-OS-Style

CLI
The following example procedure uses the NX-OS-style CLI to configure the NetFlow tenant hierarchy:

Procedure

Step 1 Enter the configuration mode.

Example:
apic1# config

Step 2 Create a tenant and bridge domain, and add them to a VRF.
Example:
apic1(config)# tenant tn2
apic1(config-tenant)# vrf context vrf2

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

456
Configuring NetFlow
Configuring NetFlow Tenant Hierarchy Using the NX-OS-Style CLI

apic1(config-tenant-vrf)# exit
apic1(config-tenant)# bridge-domain bd2
apic1(config-tenant-bridge-domain)# vrf member vrf2
apic1(config-tenant-bridge-domain)# exit
apic1(config-tenant)# bridge-domain bd3
apic1(config-tenant-bridge-domain)# vrf member vrf2
apic1(config-tenant-bridge-domain)# exit

Step 3 Create an application endpoint group behind which the exporter resides.
Example:
apic1(config-tenant)# application ap2
apic1(config-tenant-app)# epg epg2
apic1(config-tenant-app)# bridge-domain member bd2
apic1(config-tenant-app-bridge-domain)# exit
apic1(config-tenant-app)# exit

Step 4 Create a second application endpoint group behind which the exporter resides.
Example:
apic1(config-tenant)# application ap3
apic1(config-tenant-app)# epg epg3
apic1(config-tenant-app)# bridge-domain member bd3
apic1(config-tenant-app-bridge-domain)# exit
apic1(config-tenant-app)# exit

Step 5 Attach a NetFlow monitor policy on the bridge domains.

Example:
apic1(config)# interface bridge-domain bd2
apic1(config-if)# ipv6 flow monitor tnMonitor1
apic1(config-if)# ip flow monitor tnMonitor1
apic1(config-if)# layer2-switched flow monitor tnMonitor1
apic1(config-if)# exit
apic1(config)# interface bridge-domain bd3
apic1(config-if)# ipv6 flow monitor tnMonitor1
apic1(config-if)# ip flow monitor tnMonitor1
apic1(config-if)# exit

You can have one monitor policy per address family (IPv4 and IPv6). The interfaces can also be vPCs.

Step 6 Create the Netflow exporter policy.

Example:
In the following commands, the destination endpoint group is the endpoint group that the exporter sits behind.
This endpoint group can also be an external Layer 3 endpoint group.
apic1(config)# flow exporter tnExporter1
apic1(config-flow-exporter)# transport udp 1234
apic1(config-flow-exporter)# destination address 2.2.2.2
apic1(config-flow-exporter)# destination epg tenant tn2 application ap2 epg epg2
apic1(config-flow-exporter)# vrf member tenant tn2 vrf vrf2
apic1(config-flow-exporter)# version v9
apic1(config-flow-exporter)# source address 1.1.1.1
apic1(config-flow-exporter)# exit

Step 7 Create a second Netflow exporter policy.

Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

457
 Configuring NetFlow
Configuring NetFlow Tenant Hierarchy Using the NX-OS-Style CLI

In the following commands, the destination endpoint group is the endpoint group that the exporter sits behind,
which in this case is an external Layer 3 endpoint group.
apic1(config)# flow exporter tnExporter2
apic1(config-flow-exporter)# transport udp 9990
apic1(config-flow-exporter)# destination address 2001:db5:a0c:1f0::2
apic1(config-flow-exporter)# destination external-l3 epg tenant tn2 vrf v2 epg accounting-inst
apic1(config-flow-exporter)# vrf member tenant tn2 vrf vrf2
apic1(config-flow-exporter)# version v5
apic1(config-flow-exporter)# source address 2001:db8:a0b:12f0::1
apic1(config-flow-exporter)# exit

Step 8 Create a NetFlow record policy.

Example:
apic1(config)# flow record tnRecord1
apic1(config-flow-record)# match dst-ip
apic1(config-flow-record)# match dst-ipv4
apic1(config-flow-record)# match dst-ipv6
apic1(config-flow-record)# match dst-mac
apic1(config-flow-record)# match dst-port
apic1(config-flow-record)# match ethertype
apic1(config-flow-record)# match proto
apic1(config-flow-record)# match src-ip
apic1(config-flow-record)# match src-ipv4
apic1(config-flow-record)# match src-ipv6
apic1(config-flow-record)# match src-mac
apic1(config-flow-record)# match src-port
apic1(config-flow-record)# match tos
apic1(config-flow-record)# match vlan
apic1(config-flow-record)# collect count-bytes
apic1(config-flow-record)# collect count-pkts
apic1(config-flow-record)# collect pkt-disp
apic1(config-flow-record)# collect sampler-id
apic1(config-flow-record)# collect src-intf
apic1(config-flow-record)# collect tcp-flags
apic1(config-flow-record)# collect ts-first
apic1(config-flow-record)# collect ts-recent
apic1(config-flow-record)# exit

Step 9 Create a NetFlow monitor policy.

Example:
apic1(config)# flow monitor tnMonitor1
apic1(config-flow-monitor)# record tnRecord1
apic1(config-flow-monitor)# exporter tnExporter1
apic1(config-flow-monitor)# exporter tnExporter2
apic1(config-flow-monitor)# exit

You can attach a maximum of two exporters.

Step 10 Add VLANs to the VLAN domain and configure a VRF for a leaf node.
Example:
apic1(config)# vlan-domain dom1
apic1(config-vlan)# vlan 5-100
apic1(config-vlan)# exit
apic1(config)# leaf 101
apic1(config-leaf)# vrf context tenant tn2 vrf vrf2
apic1(config-leaf-vrf)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

458
Configuring NetFlow
Configuring NetFlow Tenant Hierarchy Using the NX-OS-Style CLI

Step 11 Deploy an endpoint group on an interface to deploy the bridge domain.

Example:
apic1(config-leaf)# interface ethernet 1/10
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# switchport trunk allowed vlan 10 tenant tn2 application ap2 epg epg2
apic1(config-leaf-if)# exit

Step 12 Deploy another endpoint group on an interface.

Example:
apic1(config-leaf)# interface ethernet 1/11
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# switchport trunk allowed vlan 11 tenant tn2 application ap3 epg epg3
apic1(config-leaf-if)# exit

Step 13 Attach the monitor policy to the sub-interface.

Example:
apic1(config-leaf)# interface ethernet 1/20
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# exit
apic1(config-leaf)# interface ethernet 1/20.20
apic1(config-leaf-if)# vrf member tenant tn2 vrf vrf2
apic1(config-leaf-if)# ipv6 address 20::1/64 preferred
apic1(config-leaf-if)# ipv6 flow monitor tnMonitor1
apic1(config-leaf-if)# ip flow monitor tnMonitor2
apic1(config-leaf-if)# exit

Step 14 Attach the monitor policy to a switched virtual interface (SVI).

Example:
apic1(config-leaf)# interface vlan 30
apic1(config-leaf-if)# vrf member tenant tn2 vrf vrf2
apic1(config-leaf-if)# ipv6 address 64::1/64 preferred
apic1(config-leaf-if)# ip flow monitor tnMonitor1
apic1(config-leaf-if)# ip6 flow monitor tnMonitor1
apic1(config-leaf-if)# exit

Step 15 Associate the SVI to a Layer 2 interface.

Example:
apic1(config-leaf)# interface ethernet 1/30
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# switchport trunk allowed vlan 30 tenant tn2 external-svi
apic1(config-leaf-if)# exit
apic1(config-leaf)# exit
apic1(config)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

459
 Configuring NetFlow
Consuming a NetFlow Exporter Policy Under a VMM Domain Using the NX-OS-Style CLI for VMware VDS

Consuming a NetFlow Exporter Policy Under a VMM Domain

Using the NX-OS-Style CLI for VMware VDS
The following procedure uses the NX-OS-style CLI to consume a NetFlow exporter policy under a VMM
domain.

Procedure

Step 1 Enter the configuration mode.

Example:
apic1# config

Step 2 Consume the NetFlow exporter policy.

Example:
apic1(config)# vmware-domain mininet
apic1(config-vmware)# configure-dvs
apic1(config-vmware-dvs)# flow exporter vmExporter1
apic1(config-vmware-dvs-flow-exporter)# active-flow-timeout 62
apic1(config-vmware-dvs-flow-exporter)# idle-flow-timeout 16
apic1(config-vmware-dvs-flow-exporter)# sampling-rate 1
apic1(config-vmware-dvs-flow-exporter)# exit
apic1(config-vmware-dvs)# exit
apic1(config-vmware)# exit
apic1(config)# exit

Enabling or Disabling NetFlow on an Endpoint Group Using the

NX-OS-Style CLI for VMware VDS
The following procedure enables or disables NetFlow on an endpoint group using the NX-OS-style CLI.

Procedure

Step 1 Enable NetFlow:

Example:
apic1# config
apic1(config)# tenant tn1
apic1(config-tenant)# application app1
apic1(config-tenant-app)# epg epg1
apic1(config-tenant-app-epg)# vmware-domain member mininet
apic1(config-tenant-app-epg-domain)# flow monitor enable
apic1(config-tenant-app-epg-domain)# exit
apic1(config-tenant-app-epg)# exit
apic1(config-tenant-app)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

460
Configuring NetFlow
Enabling or Disabling NetFlow on an Endpoint Group Using the NX-OS-Style CLI for VMware VDS

apic1(config-tenant)# exit
apic1(config)# exit

Step 2 (Optional) If you no longer want to use NetFlow, disable the feature:
Example:
apic1(config-tenant-app-epg-domain)# no flow monitor enable

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

461
 Configuring NetFlow
Enabling or Disabling NetFlow on an Endpoint Group Using the NX-OS-Style CLI for VMware VDS

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

462
 CHAPTER 17
Managing Firmware
• Managing Firmware, on page 463
• Adding or Removing Repository Images, on page 463
• Changing Catalog Firmware, on page 464
• Upgrading Controller Firmware, on page 465
• Upgrading Switch Firmware, on page 467

Managing Firmware
Each firmware image includes a compatibility catalog that identifies supported types and switch models. APIC
maintains a catalog of the firmware images, switch types, and models that are allowed to use that firmware
image. The default setting is to reject a firmware update when it does not conform to the compatibility catalog.
APIC has an image repository for compatibility catalogs, controller firmware images, and switch images. The
administrator can download new firmware image to the APIC image repository from an external HTTP server
or SCP server.

Note Before you upgrade the switches, the APICs must have completed upgrading and have a health state of Fully
Fit.

Adding or Removing Repository Images

Procedure

Command or Action Purpose

Step 1 firmware repository add absolute-image-path Adds a firmware image to the repository.
Example:
apic1# firmware repository add
/home/admin/aci-catalog-dk9.1.2.1b.bin

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

463
 Managing Firmware
Changing Catalog Firmware

Command or Action Purpose

Step 2 firmware repository delete image Deletes a firmware image from the repository.
Example:
apic1# firmware repository delete
aci-catalog-dk9.1.2.1a.bin

Examples

apic1# firmware repository add /home/admin/aci-catalog-dk9.1.2.1b.bin

apic1# firmware repository delete aci-catalog-dk9.1.2.1a.bin

Changing Catalog Firmware

This procedures shows how to select a catalog firmware version from the repository.

Procedure

Command or Action Purpose

Step 1 show firmware repository [detail] Show firmware images present in repository.
The detail option displays additional
Example:
information such as MD5 checksum, release
apic1# show firmware repository date, and download date.

Step 2 configure Enters global configuration mode.

Example:
apic1# configure

Step 3 firmware Enters firmware upgrade configuration mode.

Example:
apic1(config)# firmware

Step 4 (Optional) show version Displays the currently-installed controller and

switch firmware versions.
Example:
apic1(config-firmware)# show version

Step 5 catalog-version firmware-name Changes the catalog version to an available

image in the repository.
Example:
apic1(config-firmware)# catalog-version
aci-catalog-dk9.1.2.1b.bin

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

464
 Managing Firmware
Upgrading Controller Firmware

Examples
This example shows how to select a catalog firmware version from the repository.

apic1# show firmware repository

Name Type Version Size(MB)
-------------------------- ------- ------- -------
aci-catalog-dk9.1.2.1a.bin catalog 1.2.1a 0.023
aci-catalog-dk9.1.2.1b.bin catalog 1.2.1b 0.025

apic1# configure
apic1(config)# firmware
apic1(config-firmware)# catalog-version aci-catalog-dk9.1.2.1b.bin

Upgrading Controller Firmware

The controllers upgrade in random order. Each APIC controller takes about 10 minutes to upgrade. Once a
controller image is upgraded, it drops from the cluster and reboots with the newer version while the other
APIC controllers in the cluster are still operational. Once the controller reboots, it joins the cluster again. Then
the cluster converges, and the next controller image starts to upgrade.
The catalog firmware image is upgraded when an APIC controller image is upgraded. You do not need to
upgrade the catalog firmware image separately.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 firmware Enters firmware upgrade configuration mode.

Example:
apic1(config)# firmware

Step 3 (Optional) show version Displays the currently-installed controller and

switch firmware versions.
Example:
apic1(config-firmware)# show version

Step 4 controller-group Enters controller upgrade configuration mode.

Example:
apic1(config-firmware)# controller-group

Step 5 firmware-version firmware-name Specifies the desired version for the upgrade.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

465
 Managing Firmware
Upgrading Controller Firmware

Command or Action Purpose

apic1(config-firmware-controller)#
firmware-version aci-apic-dk9.1.2.1b.iso

Step 6 [no] time start time Sets the starting time in the format
[[[yyyy:]mmm:]dd:]HH:MM. The date is
Example:
optional.
apic1(config-firmware-controller)# time
start 2016:jan:01:12:01 Note To upgrade the controllers
immediately, return to EXEC mode
and type the command firmware
upgrade controller-group .

Examples
This example shows how to upgrade the controllers.

apic1# show controller

Fabric Name : mininet
Operational Size : 3
Cluster Size : 3
Time Difference : 0
Fabric Security Mode : permissive

ID Address In-Band Address OOB Address Version Flags Serial Number Health

--- --------- --------------- ------------ ---------- ----- ------------- ---------

1* 10.0.0.1 192.168.11.1 192.168.10.1 1.2(1a) crva TEP-1-1 fully-fit

2 10.0.0.2 192.168.11.2 192.168.10.2 1.2(1a) crva TEP-1-2 fully-fit

3 10.0.0.3 192.168.11.3 192.168.10.3 1.2(1a) crva TEP-1-3 fully-fit

Flags - c:Commissioned | r:Registered | v:Valid Certificate | a:Approved

apic1# configure
apic1(config)# firmware
apic1(config-firmware)# show version
Role Id Name Version
---------- ---------- ----------------- -----------
controller 1 apic1 1.2(1a)
controller 2 apic2 1.2(1a)
controller 3 apic3 1.2(1a)
leaf 101 leaf1 n9000-11.2(1a)
leaf 102 leaf2 n9000-11.2(1a)
leaf 103 leaf2 n9000-11.2(1a)
spine 201 spine1 n9000-11.2(1a)
spine 202 spine2 n9000-11.2(1a)

apic1(config-firmware)# controller-group
apic1(config-firmware-controller)# firmware-version aci-apic-dk9.1.2.1b.iso
apic1(config-firmware-controller)# time start 2016:jan:01:12:01

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

466
 Managing Firmware
Upgrading Switch Firmware

Upgrading Switch Firmware

Before you begin
A scheduler must exist to specify when the upgrade will be executed.

Note Before you upgrade the switches, the APICs must have completed upgrading and have a health state of Fully
Fit.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 firmware Enters firmware upgrade configuration mode.

Example:
apic1(config)# firmware

Step 3 [no] switch-group group-name Creates (or deletes) switch group and enters
switch upgrade configuration mode.
Example:
apic1(config-firmware)# switch-group
mySwitchGroup5

Step 4 [no] switch Adds (or removes) a switch or a list of switches

node-id-or-name[,node-id-or-name,...] to the switch-group for upgrading. You can
specify the node ID (such as 101) or the name
Example:
(such as spine1). You can specify multiple
switches by using commas.
apic1(config-firmware-switch)# switch
leaf1-leaf3,leaf6
apic1(config-firmware-switch)# no switch
leaf4,leaf5

Step 5 firmware-version firmware-name Specifies the target firmware image.

Example:
apic1(config-firmware-switch)#
firmware-version aci-apic-dk9.11.2.1a.bin

Step 6 [no] run-mode {pause-never | Species whether to proceed to the next set of
pause-on-failure} nodes if the upgrade fails on the current set of
nodes.
Example:
apic1(config-firmware-switch)# run-mode
pause-on-failure

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

467
 Managing Firmware
Upgrading Switch Firmware

Command or Action Purpose

Step 7 schedule scheduler-name Assigns a scheduler for the upgrade. Enter the
name of a scheduler that has already been
Example:
defined.
apic1(config-firmware-switch)# schedule
myNextSunday Note To upgrade the switch group
immediately, return to EXEC mode
and type the command firmware
upgrade switch-group .

Step 8 [no] scheduler pause Pauses the maintenance policy scheduler. Use
the [no] prefix to resume.
Example:
apic1(config-firmware-switch)# scheduler
pause
apic1(config-firmware-switch)# no
scheduler pause

Step 9 show running-config Displays the configuration.

Example:
apic1(config-firmware-switch)# show run

Examples
This example shows how to upgrade the firmware for three leaf switches.

apic1# configure
apic1(config)# firmware
apic1(config-firmware)# switch-group mySwitchGroup5
apic1(config-firmware-switch)# switch leaf1,leaf3,leaf6
apic1(config-firmware-switch)# no switch leaf4,leaf5
apic1(config-firmware-switch)# firmware-version aci-apic-dk9.1.1.3f.bin
apic1(config-firmware-switch)# run-mode pause-on-failure
apic1(config-firmware-switch)# schedule myNextSunday
apic1(config-firmware-switch)# show run
# Command: show running-config firmware switch-group mySwitchGroup5
# Time: Fri Nov 6 23:55:35 2015
firmware
switch-group mySwitchGroup5
switch 101
switch 102
switch 103
switch 106
schedule myNextSunday
exit
exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

468
 CHAPTER 18
Managing the Configuration with Snapshots
• About Configuration Management and Snapshots, on page 469
• Exporting a Snapshot, on page 469
• Importing a Snapshot, on page 471
• Rollback Configuration Using Snapshots, on page 472
• Uploading or Downloading a Snapshot File to a Remote Path, on page 473
• Managing Snapshot Files and Jobs, on page 475

About Configuration Management and Snapshots

You can back up and restore your system configuration by exporting and importing configuration archives
(snapshots) to and from a local controller-managed folder. By exporting snapshots before and after making
configuration changes, you have the ability to roll back configuration changes that were applied between two
snapshots.
You can also upload and download the snapshot files to and from a remote server.
Each snapshot action (export, import, rollback, upload, and download) is performed by creating a policy for
the action and then triggering the action as a job. Export actions can also be scheduled to run at a future time
or periodically. Import, export, and rollback jobs cannot run in parallel. If a job is already running, triggering
a new job will fail.

Exporting a Snapshot
Before you begin
If you want to export snapshots according to a schedule, configure a scheduler before configuring the export
policy.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

469
 Managing the Configuration with Snapshots
Exporting a Snapshot

Command or Action Purpose

Step 2 [no] snapshot export policy-name Creates a policy for exporting snapshots.
Example:
apic1(config)# snapshot export
myExportPolicy

Step 3 format {xml | json} Specifies the data format for the exported
configuration file. The default is
Example:
apic1(config-export)# format json

Step 4 (Optional) [no] schedule schedule-name Specifies an existing scheduler for exporting
snapshots.
Example:
apic1(config-export)# schedule
EveryEightHours

Step 5 (Optional) [no] target [infra | fabric | Assigns the target of the export, which can be
tenant-name] fabric, infra, a specific tenant, or none. If no
target is specified, all configuration information
Example:
is exported. The default is no target.
apic1(config-export)# target
tenantExampleCorp

Step 6 (Optional) [no] remote path remote-path-name Specifies the name of a configured remote path
to which the file will be sent. If no remote path
Example:
is specified, the file is exported locally to a
apic1(config-export)# remote path folder in the controller. The default is no remote
myBackupServer
path.

Step 7 end Returns to EXEC mode.

Example:
apic1(config-export)# end

Step 8 Required: trigger snapshot export Executes the snapshot export task. If the export
policy-name policy is configured with a scheduler, this step
is unnecessary unless you want an immediate
Example:
export.
apic1# trigger snapshot export
myExportPolicy

Examples
This example shows how to configure the periodic export of a JSON-format snapshot file for a
specific tenant configuration.

apic1# configure
apic1(config)# snapshot export myExportPolicy
apic1(config-export)# format json
apic1(config-export)# target tenantExampleCorp
apic1(config-export)# schedule EveryEightHours

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

470
 Managing the Configuration with Snapshots
Importing a Snapshot

Importing a Snapshot
Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] snapshot import policy-name Creates a policy for importing snapshots.
Example:
apic1(config)# snapshot import
myImportPolicy

Step 3 file filename Specifies the name of the file to be imported.

Example:
apic1(config-import)# file
ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz

Step 4 action {merge | replace} Specifies whether the imported configuration

settings will be merged with the current settings
Example:
or whether the imported configuration will
apic1(config-import)# action replace completely replace the current configuration.

Step 5 [no] mode {atomic | best-effort} Specifies how the import process handles
configuration errors when applying the imported
Example:
settings. The best-effort import mode allows
apic1(config-import)# mode atomic skipping individual configuration errors in the
archive, while atomic mode cancels the import
upon any configuration error.

Step 6 (Optional) [no] remote path remote-path-name Specifies the name of a configured remote path
from which the file will be imported. If no
Example:
remote path is specified, the file is imported
apic1(config-import)# remote path locally from a folder in the controller. The
myBackupServer
default is no remote path.

Step 7 end Returns to EXEC mode.

Example:
apic1(config-import)# end

Step 8 Required: trigger snapshot import Executes the snapshot import task.
policy-name
Example:
apic1# trigger snapshot import
myImportPolicy

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

471
 Managing the Configuration with Snapshots
Rollback Configuration Using Snapshots

Examples
This example shows how to configure and execute the importing of a snapshot file to replace the
current configuration.

apic1# show snapshot files

File : ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz
Created : 2015-11-21T01:00:21.167+00:00
Root :
Size : 22926

apic1# configure
apic1(config)# snapshot import myImportPolicy
apic1(config-import)# file ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz
apic1(config-import)# action replace
apic1(config-import)# mode atomic
apic1(config-import)# end
apic1# trigger snapshot import myImportPolicy

Rollback Configuration Using Snapshots

The rollback feature provides an "undo" function that reverts changes made between one snapshot archive
and a later snapshot archive. Only locally stored snapshot files are supported for rollback. You can optionally
enable the preview mode to generate and view a rollback before implementing it.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] snapshot rollback policy-name Creates a policy for rollback using snapshots.
Example:
apic1(config)# snapshot rollback
myRollbackPolicy

Step 3 first-file filename Specifies the name of the earlier file.

Example:
apic1(config-rollback)# first-file
ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz

Step 4 second-file filename Specifies the name of the later file.

Example:
apic1(config-rollback)# second-file
ce2_DailyAutoBackup-2015-11-21T09-00-21.tar.gz

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

472
 Managing the Configuration with Snapshots
Uploading or Downloading a Snapshot File to a Remote Path

Command or Action Purpose

Step 5 [no] preview (Optional) Specifies that the rollback changes
are generated and previewed but not applied.
Example:
When preview mode is enabled, no changes to
apic1(config-rollback)# preview the configuration are made. After previewing
rollback changes, use the no preview command
to exit preview mode and enable the rollback
to be applied when you reenter the trigger
snapshot rollback commands.

Step 6 end Returns to EXEC mode.

Example:
apic1(config-rollback)# end

Step 7 Required: trigger snapshot rollback Executes the snapshot rollback task.
policy-name
Example:
apic1# trigger snapshot rollback
myRollbackPolicy

Examples
This example shows how to configure and execute a rollback without previewing it first.

apic1# show snapshot files

File : ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz
Created : 2015-11-21T01:00:21.167+00:00
Root :
Size : 22926

File : ce2_DailyAutoBackup-2015-11-21T09-00-21.tar.gz
Created : 2015-11-21T09:00:24.025+00:00
Root :
Size : 23588

apic1# configure
apic1(config)# snapshot rollback myRollbackPolicy
apic1(config-rollback)# first-file ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz
apic1(config-rollback)# second-file ce2_DailyAutoBackup-2015-11-21T09-00-21.tar.gz

apic1(config-rollback)# end
apic1# trigger snapshot rollback myRollbackPolicy

Uploading or Downloading a Snapshot File to a Remote Path

You can upload snapshot archive files from local storage to a remote path. You can also download snapshot
archive files from the remote path.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

473
 Managing the Configuration with Snapshots
Uploading or Downloading a Snapshot File to a Remote Path

Before you begin

You must configure a remote path to receive the file. See Configuring a Remote Path for File Export, on page
490.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] snapshot {upload | download} Creates a policy for uploading or downloading
policy-name remote-path-name snapshot files with a remote path.
Example:
apic1(config)# snapshot upload myUpPolicy

Step 3 remote path remote-path-name Specifies the name of a configured remote path
to which the snapshot file will be sent.
Example:
apic1(config-upload)# remote path
myBackupServer

Step 4 file filename Specifies the name of the snapshot file to be

sent.
Example:
apic1(config-upload)# file
ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz

Step 5 end Returns to EXEC mode.

Example:
apic1(config-upload)# end

Step 6 trigger snapshot {upload | download} Executes the snapshot upload or download task.
policy-name
Example:
apic1# trigger snapshot upload myUpPolicy

Examples
This example shows how to configure and execute the uploading of a snapshot file to a remote path.

apic1# show snapshot files

File : ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz
Created : 2015-11-21T01:00:21.167+00:00
Root :
Size : 22926

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

474
 Managing the Configuration with Snapshots
Managing Snapshot Files and Jobs

apic1# configure
apic1(config)# snapshot upload myUpPolicy
apic1(config-upload)# remote path myBackupServer
apic1(config-upload)# file ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz
apic1(config-upload)# end
apic1# trigger snapshot upload myUpPolicy

Managing Snapshot Files and Jobs

The following commands are available for managing snapshot files and jobs.

Command Description

clear snapshot file filename Removes a snapshot file from the local storage.

clear snapshot job job-name Removes a snapshot job from the history.

show snapshot files Displays the snapshot files in local storage.

show snapshot jobs Displays recent snapshot tasks.

show snapshot active jobs Displays currently-active snapshot tasks.

Examples
This example shows how to display snapshot files and the snapshot job history.

apic1# show snapshot files

File : ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz
Created : 2015-11-21T01:00:21.167+00:00
Root :
Size : 22926

File : ce2_DailyAutoBackup-2015-11-21T09-00-21.tar.gz
Created : 2015-11-21T09:00:24.025+00:00
Root :
Size : 23588

apic1# show snapshot jobs

Type : export
Run : 2015-11-21T01-00-17
State : success
Details : Success
File Name : ce2_DailyAutoBackup-2015-11-21T01-00-17.tar.gz

Type : export
Run : 2015-11-21T09-00-21
State : success
Details : Success
File Name : ce2_DailyAutoBackup-2015-11-21T09-00-21.tar.gz

Type : rollback
Run : 2015-11-22T00-25-06
State : running
Details :
File Name : not applicable

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

475
 Managing the Configuration with Snapshots
Managing Snapshot Files and Jobs

apic1# clear snapshot job 2015-11-22T00-25-06

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

476
 CHAPTER 19
Configuring Monitoring
• Configuring Syslog, on page 477
• Configuring Call Home, on page 480
• Configuring TACACS External Logging, on page 487
• Sending an On-Demand Tech Support File Using the NX-OS Style CLI, on page 489
• Configuring a Remote Path for File Export, on page 490
• Using Show Commands for Monitoring, on page 491
• Configuring SNMP, on page 498
• Configuring SNMP Policy Using CLI, on page 499
• Configuring Smart Callhome, on page 501

Configuring Syslog
Configuring a Logging Server Group
In the ACI fabric, one or more logging server-groups can be configured with one or more logging destination
servers.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 logging server-group server-group-name Configure a grouping of servers for monitoring.

Example:
apic1(config)# logging server-group
myLoggingGroup

Step 3 [no] description text Specifies

Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

477
 Configuring Monitoring
Configuring a Logging Server Group

Command or Action Purpose

apic1(config-logging)# logging
description "This is my logging server
group"

Step 4 [no] console [severity {alerts | critical | Enables logging to the console (only for
emergencies}] switches) and optionally sets the minimum
severity level for logging.
Example:
apic1(config-logging)# console severity
critical

Step 5 [no] logfile [severity {alerts | critical | Enables logging to the logfile and optionally
debugging | emergencies | errors | sets the minimum severity level for logging.
information | notifications | warnings}]
Example:
apic1(config-logging)# logfile severity
critical

Step 6 [no] server ip-address-or-hostname [facility Adds a destination logging server and optionally
local-level] [severity severity-level] [mgmtepg sets the minimum severity level for logging.
{inb | oob}] [port port-number]
• facility —Local facility in the form local
Example: n
apic1(config-logging)# server
reach.example.com level local4 mgmtepg
• severity —Minimum severity level for
inb port 514 logging. Can be one of the options shown
in the logfile command.
• mgmt —Management endpoint group,
either inb (inband) or oob (out of band).
• port —Service port number of the logging
server.

Examples
This example shows how to configure a syslog destination server group.

apic1# configure
apic1(config)# logging server-group myLoggingGroup
apic1(config-logging)# logging description "This is my logging server group"
apic1(config-logging)# console severity critical
apic1(config-logging)# logfile severity critical
apic1(config-logging)# server reach.example.com level local4 mgmtepg inb port 514

What to do next
Configure syslog with this logging server group as the logging destination.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

478
 Configuring Monitoring
Configuring Syslog

Configuring Syslog
In order to receive and monitor system log messages, you must specify a syslog destination, which can be the
console, a local file, or one or more remote hosts running a syslog server. In addition, you can specify the
minimum severity level of messages to be displayed on the console or captured by the file or host.

Before you begin

Configure a logging server-group containing the servers to which syslog messages will be sent.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 syslog common Enters syslog common policy configuration

mode.
Example:
apic1(config)# syslog common

Step 3 [no] logging description text Adds descriptive text about the policy.
Example:
apic1(config-syslog)# logging description
"This is the common logging policy"

Step 4 [no] logging severity {alerts | critical | Specifies the minimum severity level for
debugging | emergencies | errors | sending syslog messages.
information | notifications | warnings}
Example:
apic1(config-syslog)# logging severity
notifications

Step 5 [no] logging server-group server-group-name Specifies a destination logging server group.
Example:
apic1(config-syslog)# logging
server-group myLoggingGroup

Step 6 [no] logging audit Enables audit logs to the policy.

Example:
apic1(config-syslog)# logging audit

Step 7 [no] logging event Enables event logs to the policy.

Example:
apic1(config-syslog)# logging event

Step 8 [no] logging fault Enables fault logs to the policy.

Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

479
 Configuring Monitoring
Configuring Call Home

Command or Action Purpose

apic1(config-syslog)# logging fault

Step 9 [no] logging session Enables session logs to the policy.

Example:
apic1(config-syslog)# logging session

Examples
This example shows how to configure syslog for messages of 'notification' severity or higher. Syslog
messages from fault and event logs are sent to servers in server-group myLoggingGroup.

apic1# configure
apic1(config)# syslog common
apic1(config-syslog)# logging description "This is the common logging policy"
apic1(config-syslog)# logging severity notifications
apic1(config-syslog)# logging server-group myLoggingGroup
apic1(config-syslog)# logging audit
apic1(config-syslog)# logging event

Configuring Call Home

Configuring the Call Home Policy
In the ACI fabric, Cisco Call Home configuration can be added in the common monitoring policy.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 callhome common Enters Call Home common policy configuration

mode.
Example:
apic1(config)# callhome common

Step 3 [no] logging audit Enables audit logs to the policy.

Example:
apic1(config-callhome)# logging audit

Step 4 [no] logging event Enables event logs to the policy.

Example:
apic1(config-callhome)# logging event

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

480
Configuring Monitoring
Configuring the Call Home Policy

Command or Action Purpose

Step 5 [no] logging fault Enables fault logs to the policy.
Example:
apic1(config-callhome)# logging fault

Step 6 [no] logging severity {alert | critical | debug Specifies the minimum severity level for
| emergency | error | info | notice | warning} logging.
Example:
apic1(config-callhome)# logging severity
notice

Step 7 [no] periodic-inventory notification schedule Configures a periodic notification scheduler.

scheduler The scheduler must be previously configured.
Example:
apic1(config-callhome)#
periodic-inventory notification schedule
EveryEightHours

Step 8 show callhome common [destination-profile Shows Call Home configuration.

| query-profile | transport-email]
Example:
apic1(config-callhome)# show callhome
common

Examples
This example shows how to configure a basic Call Home policy.

apic1# configure
apic1(config)# callhome common
apic1(config-callhome)# logging event
apic1(config-callhome)# logging fault
apic1(config-callhome)# logging severity notice
apic1(config-callhome)# periodic-inventory notification schedule EveryEightHours
apic1(config-callhome)# end
apic1# show callhome common
Callhome : common

Logging Enabled : event,faults

Logging Severity : notice

Destination-Profile :

Admin State : Enabled

Contract-id : 12345678
Customer-id : ABCDEFG
Email-addr : [email protected]
From email-addr : [email protected]
Reply-To email-addr : [email protected]
Phone Number : +14085551212
SMTP Port num : 25

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

481
 Configuring Monitoring
Configuring a Call Home Destination Profile

SMTP Server : smtp.example.com

Destination Email-addr Format Message-Size Message-Level

----------- ----------------- ------ ------------ -------------
SanJose [email protected] xml 40000 alert

Query-Profile :

Query Name Query Type Dn/Class Target Respones Subtree Response Include
----------- ---------- -------- ------ ----------------
------------------------------
myUserQuery class User self children
ep-records,fault-records,stats

What to do next
Configure a destination profile and (optionally) a query profile.

Configuring a Call Home Destination Profile

You must configure at least one destination profile for Call Home. If the destination profile uses email message
delivery, you must specify a Simple Mail Transfer Protocol (SMTP) server.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 callhome common Enters Call Home common policy

configuration mode.
Example:
apic1(config)# callhome common

Step 3 [no] destination-profile Configures a destination profile.

Example:
apic1(config-callhome)#
destination-profile

Step 4 [no] destination dest-name Configures a destination where the Call Home
messages will be sent, including the format of
Example:
the messages and the severity level for sending.
apic1(config-callhome-destnprof)#
destination SanJose Note You can configure more than one
destination.

Step 5 [no] email-addr email Configures the e-mail address that will receive
the Call Home messages. Up to 255
Example:
alphanumeric characters are accepted in e-mail
apic1(config-callhome-destnprof-destn)# address format.
email-addr [email protected]

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

482
Configuring Monitoring
Configuring a Call Home Destination Profile

Command or Action Purpose

Step 6 [no] format {aml | xml | short-txt} Configures the format for Call Home
messages, which can be sent in the following
Example:
formats:
apic1(config-callhome-destnprof-destn)#
format xml • aml —Adaptive Messaging Language
(AML) XML schema definition (XSD)
• xml —The XML format enables
communication with the Cisco Systems
Technical Assistance Center (TAC).
• short-txt —Short text format provides a
one or two line description of the fault
that is suitable for pagers or printed
reports.

Step 7 [no] message-level {alert | critical | debug | Configures the minimum severity level for
emergency | error | info | notice | warning} sending messages.
Example:
apic1(config-callhome-destnprof-destn)#
message-level alert

Step 8 [no] message-size size Configures the size of the messages. The range
is 0 to 5000000 characters.
Example:
apic1(config-callhome-destnprof-destn)#
message-size 40000

Step 9 exit Returns to destination profile configuration

mode.
Example:
apic1(config-callhome-destnprof-destn)#
exit

Step 10 Configure the destination profile. Use the commands in Call Home Destination
Profile Configuration Commands, on page 484
Example:
apic1(config-callhome-destnprof)#
(various commands)

Step 11 show callhome common [destination-profile Shows Call Home configuration.

| query-profile | transport-email]
Example:
apic1(config-callhome-destnprof)# show
callhome common transport-email

Examples
This example shows how to configure Call Home to send email messages of severity 'alert' or higher
to [email protected].

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

483
 Configuring Monitoring
Call Home Destination Profile Configuration Commands

apic1# configure
apic1(config)# callhome common
apic1(config-callhome)# destination-profile
apic1(config-callhome-destnprof)# destination SanJose
apic1(config-callhome-destnprof-destn)# email-addr [email protected]
apic1(config-callhome-destnprof-destn)# format xml
apic1(config-callhome-destnprof-destn)# message-level alert
apic1(config-callhome-destnprof-destn)# message-size 40000
apic1(config-callhome-destnprof-destn)# exit
apic1(config-callhome-destnprof)# contract-id 12345678
apic1(config-callhome-destnprof)# customer-id ABCDEFG
apic1(config-callhome-destnprof)# description "Example Corporation"
apic1(config-callhome-destnprof)# site-id XYZ123
apic1(config-callhome-destnprof)# street-address "1 Cisco Way"
apic1(config-callhome-destnprof)# phone-contact +14085551212
apic1(config-callhome-destnprof)# email-contact [email protected]
apic1(config-callhome-destnprof)# transport email from [email protected]
apic1(config-callhome-destnprof)# transport email reply-to [email protected]
apic1(config-callhome-destnprof)# transport email mail-server smtp.example.com mgmtepg inb
port 25
apic1(config-callhome)# end
apic1# show callhome common transport-email
From email-addr : [email protected]
SMTP Port num : 25

SMTP Server : smtp.example.com

Call Home Destination Profile Configuration Commands

These commands are entered in the Call Home destination profile ( config-callhome-destnprof ) configuration
mode.

Command Purpose

contract-id contract-id The Call Home contract number for the customer.

customer-id customer-id The CCO ID that includes the contract numbers for the support
contract in its entitlements.

description text Descriptive text about this customer site.

email-contact email The email address for the main contact.

phone-contact phone-num The telephone number for the main contact.

site-id site-id The unique Call Home identification number for the customer
site.

street-address address The mailing address for the main contact.

transport email from email The email address that should appear in the From field on Call
Home alert messages sent by the system.

transport email reply-to email The return email address that should appear in the From field
on Call Home alert messages sent by the system.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

484
 Configuring Monitoring
Configuring a Call Home Query

transport email mail-server smtp-server The IP address or hostname of the SMTP server and the port
mgmtepg {inb | oob} port port-number number the system should use to talk to the SMTP server.

Configuring a Call Home Query

When an event triggers the sending of a Call Home report, information from your selected queries is included
in the report. You can configure a query based on a class name or a distinguished name, and you can further
qualify the query based on subtrees.

Before you begin

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 callhome common Enters Call Home common policy configuration

mode.
Example:
apic1(config)# callhome common

Step 3 [no] query-profile Enters Call Home query profile configuration

mode.
Example:
apic1(config-callhome)# query-profile

Step 4 [no] query query-name type {class class-name Configures a query profile.
| dn name}
Example:
apic1(config-callhome-queryprof)# query
myUserQuery type class User

Step 5 [no] response-subtree {full | children | no} Configures the response subtree. You can
choose to include the full subtree, only children,
Example:
or no subtree information.
apic1(config-callhome-queryprof-query)#
response-subtree children

Step 6 [no] response-incl {option[,option[,option...]]} Configures the specific subtree information

categories to be included in the response.
Example:
Multiple categories can be specified in a
apic1(config-callhome-queryprof-query)# comma-separated list. The available categories
response-incl
ep-records,fault-records,stats are listed in Query Subtree Categories, on page
486.

Step 7 [no] target {children | self | subtree} Configures the query target.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

485
 Configuring Monitoring
Query Subtree Categories

Command or Action Purpose

apic1(config-callhome-queryprof-query)#
target self

Step 8 show callhome common [destination-profile Shows Call Home configuration.

| query-profile | transport-email]
Example:
apic1(config-callhome-queryprof-query)#
show callhome common query-profile

Examples
This example shows how to configure a Call Home query.

apic1# configure
apic1(config)# callhome common
apic1(config-callhome)# query-profile
apic1(config-callhome-queryprof)# query myUserQuery type class User
apic1(config-callhome-queryprof-query)# response-subtree children
apic1(config-callhome-queryprof-query)# response-incl ep-records,fault-records,stats
apic1(config-callhome-queryprof-query)# target self
apic1(config-callhome)# end
apic1# show callhome common destination-profile
Query-Profile :

Query Name Query Type Dn/Class Target Respones Subtree Response Include
----------- ---------- -------- ------ ----------------
------------------------------
myUserQuery class User self children
ep-records,fault-records,stats

Query Subtree Categories

Query Category Description

add-mo-list

audit-logs

config-only

count

custom-path-hop

deployment

deployment-records

ep-records

event-logs

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

486
 Configuring Monitoring
Configuring TACACS External Logging

fault-count

fault-records

faults

full-deployment

health

health-records

local-prefix

no-scoped

none

port-deployment

record-subtree

relations

relations-with-parent

required

state

stats

tasks

Configuring TACACS External Logging

Creating a TACACS External Logging Destination Group Using the NX-OS-Style
CLI
You can use the NX-OS-style command line interface (CLI) to configure TACACS destination groups and
destinations. A TACACS destination group enables you to create a list of remote TACACS server destinations
to which AAA logging data is sent. You can create one or more destinations in a group. After the destination
group is created, you can associate it with a TACACS source, either for a fabric policy, an external access
policy, or a specific tenant policy configured on the Cisco Application Policy Infrastructure Controller (Cisco
APIC).

Note You must have administrator rights to access the TACACS External Logging commands in the NX-OS-style
CLI.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

487
 Configuring Monitoring
Creating a TACACS External Logging Source Using the NX-OS-Style CLI

The following example CLI commands show how to configure a TACACS destination group and destination
using the NX-OS-style CLI:

Procedure

Step 1 Enter the configuration mode.

Example:
apic1# config

Step 2 Create a TACACS destination group.

Example:
In the following command, a TACACS destination group named "tacacs-dest-grp-1" is created:
apic1(config)# tacacslog-group tacacs-dest-grp-1

Step 3 Create a TACACS destination in the new destination group.

Example:
In the following command, a remote TACSCS destination with an IP address of "1.1.1.1" is created and
includes the default port number 49:
apic1(config-tacacslog-group)# remote-dest 1.1.1.1 port 49

Note You can have logs sent to multiple ports on the same IP address by including additional port numbers
after the port keyword.

Step 4 Configure specific parameters for the new remote TACACS destination.
Example:
In the following command example, the following characteristics are configured for the new remote destination:
• Authentication key: 12345
• Authentication protocol: MS-CHAP
• Management EPG: Out-of-Band

apic1(config-remote-dest)# key
Enter Key: 12345
Enter Key again: 12345
apic1(config-remote-dest)# protocol mschap
apic1(config-remote-dest)# management-epg oob

The result of this configuration is the creation of a TACACS destination group containing a remote TACACS
server destination. If you want the same AAA logging data sent to multiple remote TACACS servers, you
can repeat steps 3 and 4 as many times as needed.

Creating a TACACS External Logging Source Using the NX-OS-Style CLI

You can use the NX-OS-style CLI to configure TACACS sources. In this configuration, the source is associated
with a TACACS destination group. Where a TACACS source is created determines which set of AAA logging

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

488
 Configuring Monitoring
Sending an On-Demand Tech Support File Using the NX-OS Style CLI

data is sent. For example, if you create the TACACS source in Fabric Policies, all AAA logging data for the
Cisco Application Centric Infrastructure (Cisco ACI) fabric supported by Cisco Application Policy Infrastructure
Controller (Cisco APIC) is sent to the associated TACACS destinations. You can create one or more sources
to support different destination groups.
The following example CLI commands show how to configure a TACACS source using the NX-OS-style
CLI:

Procedure

Step 1 Enter the configuration mode.

Example:
apic1# config

Step 2 Create a TACACS source.

Example:
In the following command, a TACSCS source named "tacacs-src-1" is created:
apic1(config)# tacacslog-monitoring common tacacslog-src tacacs-src-1

Step 3 Associate the TACACS source with a TACACS destination group.

Example:
In the following command, a TACSCS destination group named "tacacs-dest-grp-1" is associated with the
new TACACS source:
apic1(config-tacacslog-monitoring)# server-group tacacs-dest-grp-1

The result of this configuration is the creation of a TACACS source for the entire fabric and the association
of a destination group containing a remote TACACS server destination. All AAA logging data for the entire
fabric is then sent to the associated TACACS destination(s).

Sending an On-Demand Tech Support File Using the NX-OS

Style CLI

Note Do not trigger tech support file collection from more than five nodes simultaneously, especially if they are to
be exported into the APIC or to an external server with insufficient bandwidth and compute resources.
To avoid excessive storage usage in APIC, remove locally-stored tech support files promptly.

Before you begin

Configure a remote path for exporting the tech support file.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

489
 Configuring Monitoring
Configuring a Remote Path for File Export

Procedure

Command or Action Purpose

Step 1 trigger techsupport {all | controllers switch Triggers the export of a tech support file from
node-id} [remotename remote-path-name] the controllers, switches, or all to the remote
path. For switches, you can specify a range or
Example:
a comma-separated list. If no remote host is
apic1# trigger techsupport switch 101,103 specified, the file is collected in the controller
remotename remote5
itself.

Step 2 trigger techsupport host host-id Triggers the export of a tech support file from
the specified host to the remote host. If no
Example:
remote host is specified, the file is collected in
apic1# trigger techsupport host the controller itself.

Step 3 trigger techsupport local Triggers the export of a local tech support file
to the remote host. If no remote host is
Example:
specified, the file is collected in the controller
apic1# trigger techsupport local itself.

Step 4 show techsupport {all | controllers switch After a tech support file is triggered, this
node-id} status command shows the status of the tech support
report.
Example:
apic1# show techsupport switch 101 status

Examples
This example shows how to trigger a tech support file for switch 101, to be stored locally on the
apic1 controller.

apic1# trigger techsupport switch 101

Triggering techsupport for Switch 101 using policy supNode101, setting filters to default
value

Triggered on demand tech support successfully for Switch 101, will be available at:
/data/techsupport on
the controller. Use 'show techsupport' with your options to check techsupport status.

Configuring a Remote Path for File Export

In the ACI fabric, you can configure one or more remote destinations for exporting techsupport or configuration
files.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

490
 Configuring Monitoring
Using Show Commands for Monitoring

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] remote path remote-path-name Enters configuration mode for a remote path.
Example:
apic1(config)# remote path myFiles

Step 3 user username Sets the user name for logging in to the remote
server. You are prompted for a password.
Example:
apic1(config-remote)# user admin5

Step 4 path {ftp | scp | sftp} host [:port] Sets the path and protocol to the remote server.
[remote-directory ] You are prompted for a password.
Example:
apic1(config-remote)# path sftp
filehost.example.com:21 remote-directory
/reports/apic

Examples
This example shows how to configure a remote path for exporting files.

apic1# configure
apic1(config)# remote path myFiles
apic1(config-remote)# user admin5
You must reset the password when modifying the path:
Password:
Retype password:
apic1(config-remote)# path sftp filehost.example.com:21 remote-directory /reports/apic
You must reset the password when modifying the path:
Password:
Retype password:

Using Show Commands for Monitoring

About Using the Show Commands
The show commands for faults, events, health, statistics, and audit logs can be filtered to display specific types
of information or information from specific entities, such as controllers, leaf switches, spine switches, or
tenants.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

491
 Configuring Monitoring
Using the show faults Command

Broad queries are expensive in terms of system resources and storage. For example, using the show faults,
events, or audit commands without entity filters retrieves all logs or records from the entire system. We
recommend that you make use of the available data and entity filters to narrow your query as much as possible.
For example, the following command would result in a quicker and more filtered response by limiting the
query to the most recent 45 minute period:
show audits last-minutes 45

Tip At each point in the command, typing ‘ ?’ displays all possible keywords and options that can be used at that
point along with a brief explanation of each.

Using the show faults Command

The show faults command can combine several data filters and an entity filter to deliver a specific set of
faults. The command syntax is:
show faults [filter1 [filter2... ]] [entity-filter]
Entity filters restrict the output to faults of a controller, leaf, spine, or tenant. The available entity filters are
listed in Entity Filters for Show Commands, on page 497.
Data filters are provided to make the task of querying faults easier for the user. The available data filters are:

Filter Description

ack {yes | no} acknowledgment status

cause name cause

code fault-code fault code

controller controller information

detail detailed faults information

end-time YYYY-MM-DDTHR-MM:SS fault activity up to this time

history historical information

id fault-id fault ID

l4l7-cluster[cluster name | tenant name] L4 L7 device information

l4l7-graph[cluster name | tenant name] L4 L7 graph information

last-days days fault activity in the last N days

last-hours hours fault activity in the last N hours

last-minutes minutes fault activity in the last N minutes

lc lc-state lifecycle state

leaf [leaf-id] leaf switch information

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

492
 Configuring Monitoring
Using the show events Command

Filter Description

microsoft domain name Microsoft domain information

min-severity severity-value minimum severity

severity severity-value severity

spine [spine-id] spine switch information

start-time YYYY-MM-DDTHR-MM:SS fault activity starting from this time

tenant [name] tenant information

type fault-type fault type

vmware domain name VMware domain information

Examples
This example shows all faults that occurred in the past five days with code “F110473”, severity “warning”,
lifecycle “raised” and acknowledgment status “no” for the tenant TSW_Tenant0.

apic1# show faults code F110473 last-days 5 severity warning lc raised ack no tenant
TSW_Tenant0
Code : F110473
Severity : warning
Last Transition : 2015-11-03T01:19:04.913+00:00
Lifecycle : raised
DN : uni/tn-TSW_Tenant0/BD-tsw0ctx0BD1/fault-F110473
Description : TCA: ingress drop bytes rate(l2IngrBytesAg15min:dropRate)
value 160462 raised above threshold 100000

Using the show events Command

The show events command can combine several data filters and an entity filter to deliver a specific set of
events. The command syntax is:
show events [filter1 [filter2... ]] [entity-filter]
Entity filters restrict the output to events of a controller, leaf, spine, or tenant. The available entity filters are
listed in Entity Filters for Show Commands, on page 497.
Data filters are provided to make the task of querying events easier for the user. The available data filters are:

Filter Description

cause fault-value cause

code event-code event code

controller controller information

detail detailed events information

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

493
 Configuring Monitoring
Using the show health Command

Filter Description

end-time YYYY-MM-DDTHR-MM:SS event activity up to this time

id event-id event ID

last-days days event activity in the last N days

last-hours hours event activity in the last N hours

last-minutes minutes event activity in the last N minutes

leaf [leaf-id] leaf switch information

spine [spine-id] spine switch information

start-time YYYY-MM-DDTHR-MM:SS event activity starting from this time

tenant [name] tenant information

Examples
This example shows all events on leaf 101.

apic1# show events leaf 101

Severity : info
Affected Object : topology/pod-1/node-101/sys/phys-[eth1/28]
Code : E4208843
ID : 8589934758
Cause : transition
Description : PhysIf eth1/28 modified
Creation Time : 2015-11-03T01:11:16.763+00:00

Using the show health Command

The show health command can combine several data filters and an entity filter to deliver a specific health
report. The command syntax is:
show health [filter1 [filter2... ]] [entity-filter]
Entity filters restrict the output to health scores of a controller, leaf, spine, or tenant. The available entity filters
are listed in Entity Filters for Show Commands, on page 497.
Data filters are provided to make the task of querying health easier for the user. The available data filters are:

Filter Description

end-time YYYY-MM-DDTHR-MM:SS health activity up to this time

history historical information

max-change percentage minimum change in health score percentage

min-hs score maximum health score

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

494
 Configuring Monitoring
Using the show audits Command

Filter Description

start-time YYYY-MM-DDTHR-MM:SS health activity starting from this time

Examples
This example shows a brief health report for all tenants.

apic1# show health tenant

Tenant Score Change(%) Created
------------------------------------------------------------------------
infra 100 0 2015-05-12 18:45:47PDT
common 100 0 2015-05-12 18:45:47PDT
TSW_Tenant0 98 0 2015-05-12 18:20:58PDT
mgmt 100 0 2015-05-12 18:45:47PDT

This example shows all historical health records from the 4th of November that have a maximum health score
of 75 that have had a minimum change of 10% for the tenant TSW_Tenant0.

apic1# show health max-hs 75 min-change 10 start-time 2015-11-04T01:55:48 history tenant

TSW_Tenant0

Using the show audits Command

The show audits command can be used to view the audit-logs as well as the session logs for an entity. The
command can combine several data filters and an entity filter to deliver a specific set of audit logs. The
command syntax is:
show audits [filter1 [filter2... ]] [entity-filter]
Entity filters restrict the output to logs of a controller, leaf, spine, or tenant. The available entity filters are
listed in Entity Filters for Show Commands, on page 497.
Data filters are provided to make the task of querying audit logs easier for the user. The available data filters
are:

Filter Description

action {creation | deletion | failure | modification | object action indicator

special | state-transition}

controller controller information

detail detailed log information

end-time YYYY-MM-DDTHR-MM:SS log activity up to this time

id log-id log ID

last-days days log activity in the last N days

last-hours hours log activity in the last N hours

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

495
 Configuring Monitoring
Using the show stats Command

Filter Description

last-minutes minutes log activity in the last N minutes

leaf [leaf-id] leaf switch information

spine [spine-id] spine switch information

start-time YYYY-MM-DDTHR-MM:SS log activity starting from this time

tenant [name] tenant information

user user-name name of user

Examples
This example shows all audit logs in the last 45 minutes for the tenant TSW_Tenant0.

apic1# show audits last-minutes 45 tenant TSW_Tenant0

Creation Time : 2015-11-03T01:11:05.708+00:00
ID : 12884902085
User : admin
Action : creation
Affected Object : uni/tn-TSW_Tenant0/out-T0-sub-L3OUT-1/instP-
l3extInstP-1/extsubnet-[192.5.1.0/24]
Description : Subnet 192.5.1.0/24 created

Using the show stats Command

The show stats command can combine data filters and an entity filter to deliver a specific set of statistics.
The command syntax is:
show stats granularity granularity [cumulative] [history] [entity-filter]
Entity filters restrict the output to statistics of a leaf, spine, or tenant. The available entity filters are listed in
Entity Filters for Show Commands, on page 497.
Data filters are provided to make the task of querying statistics easier for the user. The available data filters
are:

Filter Description

cumulative cumulative statistics information

granularity {5min | 15min | 1h | 1d | 1w | 1mo | 1qtr the sampling interval size which can be 5 minutes, 15
| 1year} minutes, 1 hour, 1 day, 1 week, 1 month, 1 quarter,
or 1 year

history historical statistics information

Examples
This example shows 15 minute granularity statistics for the tenant TSW_Tenant0.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

496
 Configuring Monitoring
Entity Filters for Show Commands

apic1# show stats granularity 15min tenant TSW_Tenant0

This example shows 15 minute granularity statistics for a specific port.

apic1# show stats granularity 15min leaf 101 interface ethernet 1/1

Entity Filters for Show Commands

Entity filters can extend many show commands to restrict the output to faults of a controller, leaf, spine, or
tenant. The available entity filters are:

Filter

controller

leaf node-id [fex]

leaf node-id interface [ethernet slot/port | l3instance [instance-name] | mgmt [mgmt0] | portchannel |
tunnel [tunnel-name]]

leaf node-id inventory {chassis [number] | fans [number] | module [number] | powersupply [number] |
supervisor [number]}

leaf node-id protocol {arp | bgp | coop | ipv4 | ipv6 | isis | lldp | ospf | ospfv3}

leaf node-id vpc {

leaf node-id vrf [vrf-name]

spine node-id

spine node-id interface [ethernet slot/port | l3instance [instance-name] | mgmt [mgmt0] | tunnel
[tunnel-name]]

spine node-id inventory {chassis [number] | fabric [number] | fans [number] | module [number] |
powersupply [number] | supervisor [number] | system [number]}

spine node-id protocol {arp | bgp | coop | ipv4 | ipv6 | isis | lldp | ospf | ospfv3}

spine node-id vrf [vrf-name]

tenant tenant-name

tenant tenant-name application [app-name] [epg]

tenant tenant-name bridge-domain [bd-name]

tenant tenant-name interface bridge-domain [bd-name]

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

497
 Configuring Monitoring
Configuring SNMP

Configuring SNMP
Before you begin
To allow SNMP communications, you must configure an out-of-band contract allowing SNMP traffic, which
is normally on UDP:161.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 template snmp-fabric Enters template snmp-fabric mode.

snmp-fabric-template-name
Example:
apic1(config)# template snmp-fabric Pol1

Step 3 [no] snmp-server protocol enable Enables (or disables) SNMP protocol support.
Example:
apic1(config-template-snmp-fabric)#
snmp-server protocol enable

Step 4 [no] snmp-server community The community is required for SNMPv2 only.
community-name
Example:
apic1(config-template-snmp-fabric)#
snmp-server community mysecret

Step 5 snmp-server contact contact-name .

Example:
apic1(config-template-snmp-fabric)#
snmp-server contact admin80

Step 6 snmp-server location location-name Sets the location for the SNMP server.
Example:
apic1(config-template-snmp-fabric)#
snmp-server location SanJose

Step 7 exit Returns to global configuration mode

Example:
apic1(config-template-snmp-fabric)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

498
 Configuring Monitoring
Configuring SNMP Policy Using CLI

Command or Action Purpose

Step 8 template pod-group pod-group-template-name Configures a pod-group template (policy).
Example:
apic1(config)# template pod-group allPods

Step 9 inherit snmp-fabric Associates the pod-profile with the previously

snmp-fabric-template-name configured pod group.
Example:
apic1(config-pod-group)# inherit
snmp-fabric Pol1

Examples
The following example configures an out-of-band contract allowing SNMP traffic in the fabric.
apic1# configure
apic1(config)# template snmp-fabric Pol1
apic1(config-template-snmp-fabric)# snmp-server protocol enable
apic1(config-template-snmp-fabric)# snmp-server community mysecret
apic1(config-template-snmp-fabric)# snmp-server contact admin80
apic1(config-template-snmp-fabric)# snmp-server location SanJose
apic1(config-template-snmp-fabric)# exit
apic1(config)# template pod-group allPods
apic1(config-pod-group)# inherit snmp-fabric Pol1
apic1(config-pod-group)# exit
apic1(config)#

Configuring SNMP Policy Using CLI

Use this procedure to configure SNMP policy.

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:

apic1# configure

Step 2 template snmp-fabric default Creates a SNMP policy.

Example:

apic1(config)# template snmp-fabric

default

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

499
 Configuring Monitoring
Configuring SNMP Policy Using CLI

Command or Action Purpose

Step 3 snmp-server clientgroup Configures SNMP client group. A client group
is a group of client IP addresses that allows
SNMP access to routers or switches.

Step 4 snmp-server community Configures SNMP community. The SNMP

community profile enables access to the router
Example:
or switch statistics for monitoring.
apic1(config-template-snmp-fabric)#
snmp-server community abc

Step 5 snmp-server contact Configures SNMP contact information.

Step 6 snmp-server host Configures SNMP trap host.

Example:

apic1(config-template-snmp-fabric)#
snmp-server host 2001:420:28e:2020::10
traps-version 2c abc
apic1(config-template-snmp-fabric)#
snmp-server host 2001:420:28e:2020::2
traps-version 2c abc
apic1(config-template-snmp-fabric)#
snmp-server host 2001:420:28e:2020::11
traps-version 2c abc

Step 7 snmp-server location Configures SNMP location.

Step 8 snmp-server protocol Configures SNMP protocol.

Example:

apic1(config-template-snmp-fabric)#
snmp-server protocol enable

Step 9 snmp-server trap-fwd-server Configures SNMP trap forwarding server.

Example:

apic1(config-template-snmp-fabric)#
snmp-server trap-fwd-server
172.31.128.199

Step 10 snmp-server user Configures SNMP user. The SNMP user

profile is used to associate users with SNMP
Example:
policies for monitoring devices in a network.
apic1(config-template-snmp-fabric)#
snmp-server user test_user auth
hmac-md5-96 '' priv none
privacy-passphrase ''

Step 11 show running-config Verifies the configuration.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

500
 Configuring Monitoring
Configuring Smart Callhome

Configuring Smart Callhome

About Smart Callhome
Smart Callhome provides an email-based notification for critical system policies in a similar way as Callhome.
However, Smart Callhome collects a more specific selection of faults to deliver in email messages.

Note Smart Callhome only collects and delivers faults.

The fault triggers that are typical of the Smart Callhome feature correspond to the kind of events that threaten
to disrupt your network. Examples are:
• Temperature Faults: The temperature of a sensor exceeds a threshold.
• Fan/ Power Supply Faults: A fan or power supply unit goes offline.
• Disk Utilization Faults: The disk usage of a device exceeds a threshold.

Smart Callhome collects faults and emails them to a network support engineer, a Network Operations Center,
or to Cisco Smart Callhome services to generate a case with the Technical Assistance Center (TAC).

Creating a Smart Callhome Destination Group Using the NX-OS-Style CLI

Smart Callhome collects faults and emails them to a network support engineer, a Network Operations Center,
or to the Cisco Technical Assistance Center (TAC).
You can use the NX-OS-style CLI to configure Smart Callhome destination groups and destinations. A Smart
Callhome destination group enables you to create a list of email destinations to which fault data is sent. You
can create one or more destinations in a group. After the destination group is created, you can associate it with
a Smart Callhome source, either for the entire switch fabric supported by the Cisco Application Policy
Infrastructure Controller (Cisco APIC) or for a specific Tenant.
The following example CLI commands show how to configure a Smart Callhome destination group and
destination using the NX-OS-style CLI:

Procedure

Step 1 Enter the configuration mode.

Example:
apic1# config

Step 2 Enter the Smart Callhome common policy configuration mode.

Example:
apic1(config)# smartcallhome common

Note The default name for the common policy configuration mode is "common". It is the only name that
can be created.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

501
 Configuring Monitoring
Creating a Smart Callhome Destination Group Using the NX-OS-Style CLI

Step 3 Create a Smart Callhome destination group.

Example:
In the following command, a Smart Callhome destination group is created:
apic1(config-smartcallhome)# destination-profile

Step 4 Configure an SMTP server in the new destination group.

Example:
In the following command, an SMTP server with an IP address of "10.10.10.2" is added to the destination
group:
apic1(config-callhome-destnprof)# transport email mail-server 10.10.10.2

Step 5 Configure profile parameters about the new Smart Callhome destination group.
Example:
The following commands provide additional information about the destination group:
• contract-id: The service contract ID of the customer.
• customer-id: The customer ID.
• description: A description for the Smart Callhome destination profile.
• email-contact: The customer contact e-mail address.
• phone-contact: The customer contact phone number.
• site-id: The ID of the site where the network is deployed.
• street-address: The street address of the site.

Step 6 Create a Smart Callhome destination in the new destination group.

Example:
In the following command, a remote Smart Callhome destination named "sch-dest-1" is created:
apic1(config-callhome-destnprof)# destination sch-dest-1

Step 7 Configure specific parameters for the new remote Smart Callhome destination.
Example:
In the following command example, the following characteristics are configured for the new remote destination:
• Email address: [email protected]
• Message format: Short text
• RFC Compliant: True

apic1(config-callhome-destnprof-destn)# email-addr [email protected]

apic1(config-callhome-destnprof-destn)# format short-txt
apic1(config-callhome-destnprof-destn)# rfc-compliant true

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

502
Configuring Monitoring
Creating a Smart Callhome Destination Group Using the NX-OS-Style CLI

The result of this configuration is the creation of a Smart Callhome destination group containing a remote
email destination. If you want the same Smart Callhome fault data sent to multiple email destinations, you
can repeat steps 5 and 6 as many times as needed.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

503
 Configuring Monitoring
Creating a Smart Callhome Destination Group Using the NX-OS-Style CLI

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

504
 CHAPTER 20
Configuring SPAN
• Configuring SPAN and ERSPAN, on page 505

Configuring SPAN and ERSPAN

In the ACI Fabric, SPAN feature can be configured in three categories:
• Access – for monitoring traffic originating from access ports in leaf nodes
• Fabric – for monitoring traffic from fabric ports in leaf or spine nodes
• Tenant – for monitoring traffic from endpoint groups (EPGs) within a tenant

The following table shows the different configuration elements for each session.

Session Type Sources Filters Destination

Access Local Access Ports, Port-channels local to one EPG Port local to same leaf as
leaf sources

Access ERSPAN Access Ports, Port-channels, VPCs EPG EPG anywhere in the fabric
among one or more leaf nodes

Fabric ERSPAN Fabric ports in one or mode leaf or spine BD or VRF EPG anywhere in the fabric
nodes

Tenant ERSPAN EPG anywhere in the fabric - EPG anywhere in the fabric

SPAN Guidelines and Restrictions

• You cannot specify an l3extLIfP layer 3 subinterface as a SPAN source. You must use the entire port
for monitoring traffic from external sources.
• In local SPAN for FEX interfaces, the FEX interfaces can only be used as SPAN sources, not SPAN
destinations.
• On Generation 1 switches (Cisco Nexus 9000 Series switches without EX or FX on the switch
name), Tx SPAN does not work for any Layer 3 switched traffic.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

505
 Configuring SPAN
Configuring Local SPAN in Access Mode

• On Generation 2 switches (with EX or FX on the switch name), Tx SPAN does not work whether
traffic is Layer 2 or Layer 3 switched.

There are no limitations for Rx SPAN.

• For SPAN of FEX fabric port-channel (NIF), the member interfaces are supported as SPAN source
interfaces on Generation 1 leaf switches (Cisco Nexus 9000 Series switches without EX or FX on the
switch name).

Note While it is also possible to configure FEX fabric port-channel (NIF) member
interfaces as SPAN source interfaces on Generation 2 switches (Cisco Nexus
9000 Series switches with EX or FX on the switch name) for releases prior to
Cisco APIC Release 4.1, this is not supported.

• The type of SPAN supported varies:

• For Generation 1 switches, tenant and access SPAN use the encapsulated remote extension of SPAN
(ERSPAN) type I (Version 1 option in the APIC GUI). Generation 1 switches can be identified by
the lack of "EX", "FX", or "FX2" at the end of the switch name (for example, N9K-9312TX).
• For Generation 2 switches, tenant and access SPAN use the encapsulated remote extension of SPAN
(ERSPAN) type II (Version 2 option in the APIC GUI). Generation 2 switches can be identified
with "EX", "FX", or "FX2" at the end of the switch name.
• Fabric SPAN uses ERSPAN type II.

For information regarding ERSPAN headers, refer to the IETF Internet Draft at this URL:
https://tools.ietf.org/html/draft-foschiano-erspan-00.
• ERSPAN destination IPs must be learned in the fabric as an endpoint.
• SPAN supports IPv6 traffic but the destination IP for the ERSPAN cannot be an IPv6 address.
• See the Verified Scalability Guide for Cisco ACI document for SPAN-related limits, such as the maximum
number of active SPAN sessions.

Configuring Local SPAN in Access Mode

This is the traditional SPAN configuration local to an Access leaf node. Traffic originating from one or more
access ports or port-channels can be monitored and sent to a destination port local to the same leaf node.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

506
Configuring SPAN
Configuring Local SPAN in Access Mode

Command or Action Purpose

Step 2 [no] monitor access session session-name Creates an access monitoring session
configuration.
Example:
apic1(config)# monitor access session
mySession

Step 3 [no] description text Adds a description for this access monitoring
session. If the text includes spaces, it must be
Example:
enclosed in single quotes.
apic1(config-monitor-access)#
description "This is my SPAN session"

Step 4 [no] destination interface ethernet slot/port Specifies the destination interface. The
leaf node-id destination interface cannot be a FEX port or
port-channel.
Example:
apic1(config-monitor-access)#
destination interface eth 1/2 leaf 101

Step 5 [no] source interface ethernet {[fex/] Specifies the source interface port or port
slot/port | port-range} leaf node-id range.
Example:
apic1(config-monitor-access)# source
interface eth 1/2 leaf 101

Step 6 [no] direction {rx | tx | both} Specifies direction of traffic to be monitored.

The direction can be configured independently
Example:
for each source port range.
apic1(config-monitor-access-source)#
direction tx

Step 7 [no] filter tenant tenant-name application Filters traffic to be monitored. The filter can
application-name epg epg-name be configured independently for each source
port range.
Example:
apic1(config-monitor-access-source)#
filter tenant t1 application app1 epg
epg1

Step 8 exit Returns to access monitor session

configuration mode.
Example:
apic1(config-monitor-access-source)#
exit

Step 9 [no] source interface port-channel Specifies the source interface port channel.
port-channel-name-list leaf node-id [fex fex-id]
(Enters the traffic direction and filter
Example: configuration, not shown here.)
apic1(config-monitor-access)# source
interface port-channel pc5 leaf 101

Step 10 [no] shutdown Disables (or enables) the monitoring session.

Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

507
 Configuring SPAN
Configuring ERSPAN in Access Mode

Command or Action Purpose

apic1(config-monitor-access)# no shut

Examples
This example shows how to configure a local access monitoring session.

apic1# configure terminal

apic1(config)# monitor access session mySession
apic1(config-monitor-access)# description "This is my SPAN session"
apic1(config-monitor-access)# destination interface eth 1/2 leaf 101
apic1(config-monitor-access)# source interface eth 1/1 leaf 101
apic1(config-monitor-access-source)# direction tx
apic1(config-monitor-access-source)# filter tenant t1 application app1 epg epg1
apic1(config-monitor-access-source)# exit
apic1(config-monitor-access)# no shut
apic1(config-monitor-access)# show run
# Command: show running-config monitor access session mySession
# Time: Fri Nov 6 23:55:35 2015
monitor access session mySession
description "This is my SPAN session"
destination interface eth 1/2 leaf 101
source interface eth 1/1 leaf 101
direction tx
filter tenant t1 application app1 epg epg
exit
exit

Configuring ERSPAN in Access Mode

In the ACI fabric, an access mode ERSPAN configuration can be used for monitoring traffic originating from
access ports, port-channels, and vPCs in one or more leaf nodes.
For an ERSPAN session, the destination is always an endpoint group (EPG) which can be deployed anywhere
in the fabric. The monitored traffic is forwarded to the destination wherever the EPG is moved.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] monitor access session session-name Creates an access monitoring session
configuration.
Example:
apic1(config)# monitor access session
mySession

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

508
Configuring SPAN
Configuring ERSPAN in Access Mode

Command or Action Purpose

Step 3 [no] description text Adds a description for this monitoring session.
If the text includes spaces, it must be enclosed
Example:
in single quotes.
apic1(config-monitor-access)#
description "This is my access ERSPAN
session"

Step 4 [no] destination tenant tenant-name Specifies the destination interface as a tenant
application application-name epg epg-name and enters destination configuration mode.
destination-ip dest-ip-address
source-ip-prefix src-ip-address
Example:
apic1(config-monitor-access)#
destination tenant t1 application app1
epg epg1 destination-ip 192.0.20.123
source-ip-prefix 10.0.20.1

Step 5 [no] erspan-id flow-id Configures the ERSPAN ID for the ERSPAN
session. The ERSPAN range is from 1 to 1023.
Example:
apic1(config-monitor-access-dest)#
erspan-id 100

Step 6 [no] ip dscp dscp-code Configures the differentiated services code

point (DSCP) value of the packets in the
Example:
ERSPAN traffic. The range is from 0 to 64.
apic1(config-monitor-access-dest)# ip
dscp 42

Step 7 [no] ip ttl ttl-value Configures the IP time-to-live (TTL) value for
the ERSPAN traffic. The range is from 1 to
Example:
255.
apic1(config-monitor-access-dest)# ip
ttl 16

Step 8 [no] mtu mtu-value Configures the maximum transmit unit (MTU)
size for the ERSPAN session. The range is 64
Example:
to 9216 bytes.
apic1(config-monitor-access-dest)# mtu
9216

Step 9 exit Returns to monitor access configuration mode.

Example:
apic1(config-monitor-access-dest)#

Step 10 [no] source interface ethernet {[fex/] Specifies the source interface port or port
slot/port | port-range} leaf node-id range.
Example:
apic1(config-monitor-access)# source
interface eth 1/2 leaf 101

Step 11 [no] source interface port-channel Specifies the source interface port-channel.
port-channel-name-list leaf node-id [fex fex-id]

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

509
 Configuring SPAN
Configuring ERSPAN in Access Mode

Command or Action Purpose

Example:
apic1(config-monitor-access)# source
interface port-channel pc1 leaf 101

Step 12 [no] source interface vpc vpc-name-list leaf Specifies the source interface vPC.
node-id1 node-id2 [fex fex-id1 fex-id2]
Example:
apic1(config-monitor-access)# source
interface vpc pc1 leaf 101 102

Step 13 [no] direction {rx | tx | both} Specifies direction of traffic to be monitored.

The direction can be configured independently
Example:
for each source port range.
apic1(config-monitor-access-source)#
direction tx

Step 14 [no] filter tenant tenant-name application Filters traffic to be monitored. The filter can
application-name epg epg-name be configured independently for each source
port range.
Example:
apic1(config-monitor-access-source)#
filter tenant t1 application app1 epg
epg1

Step 15 exit Returns to access monitor session

configuration mode.
Example:
apic1(config-monitor-access-source)#
exit

Step 16 [no] shutdown Disables (or enables) the monitoring session.

Example:
apic1(config-monitor-access)# no shut

Examples
This example shows how to configure an ERSPAN access monitoring session.

apic1# configure terminal

apic1(config)# monitor access session mySession
apic1(config-monitor-access)# description "This is my access ERSPAN session"
apic1(config-monitor-access)# destination tenant t1 application app1 epg epg1 destination-ip
192.0.20.123 source-ip-prefix 10.0.20.1
apic1(config-monitor-access-dest)# erspan-id 100
apic1(config-monitor-access-dest)# ip dscp 42
apic1(config-monitor-access-dest)# ip ttl 16
apic1(config-monitor-access-dest)# mtu 9216
apic1(config-monitor-access-dest)# exit
apic1(config-monitor-access)# source interface eth 1/1 leaf 101
apic1(config-monitor-access-source)# direction tx
apic1(config-monitor-access-source)# filter tenant t1 application app1 epg epg1
apic1(config-monitor-access-source)# exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

510
 Configuring SPAN
Configuring ERSPAN in Fabric Mode

apic1(config-monitor-access)# no shut
apic1(config-monitor-access)# show run
# Command: show running-config monitor access session mySession
# Time: Fri Nov 6 23:55:35 2015
monitor access session mySession
description "This is my ERSPAN session"
source interface eth 1/1 leaf 101
direction tx
filter tenant t1 application app1 epg epg1
exit
destination tenant t1 application app1 epg epg1 destination-ip 192.0.20.123
source-ip-prefix 10.0.20.1
ip dscp 42
ip ttl 16
erspan-id 9216
mtu 9216
exit
exit

This example shows how to configure a port-channel as a monitoring source.

apic1(config-monitor-access)# source interface port-channel pc3 leaf 105

This example shows how to configure a one leg of a vPC as a monitoring source.

apic1(config-monitor-access)# source interface port-channel vpc3 leaf 105

This example shows how to configure a range of ports from FEX 101 as a monitoring source.

apic1(config-monitor-access)# source interface eth 101/1/1-2 leaf 105

Configuring ERSPAN in Fabric Mode

In the ACI fabric, a fabric mode ERSPAN configuration can be used for monitoring traffic originating from
one or more fabric ports in leaf or spine nodes. Local SPAN is not supported in fabric mode.
For an ERSPAN session, the destination is always an endpoint group (EPG) which can be deployed anywhere
in the fabric. The monitored traffic is forwarded to the destination wherever the EPG is moved. In the fabric
mode, only fabric ports are allowed as source, but both leaf and spine switches are allowed.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] monitor fabric session session-name Creates a fabric monitoring session
configuration.
Example:
apic1(config)# monitor fabric session
mySession

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

511
 Configuring SPAN
Configuring ERSPAN in Fabric Mode

Command or Action Purpose

Step 3 [no] description text Adds a description for this monitoring session.
If the text includes spaces, it must be enclosed
Example:
in single quotes.
apic1(config-monitor-fabric)#
description "This is my fabric ERSPAN
session"

Step 4 [no] destination tenant tenant-name Specifies the destination interface as a tenant
application application-name epg epg-name and enters destination configuration mode.
destination-ip dest-ip-address
source-ip-prefix src-ip-address
Example:
apic1(config-monitor-fabric)#
destination tenant t1 application app1
epg epg1 destination-ip 192.0.20.123
source-ip-prefix 10.0.20.1

Step 5 [no] erspan-id flow-id Configures the ERSPAN ID for the ERSPAN
session. The ERSPAN range is from 1 to 1023.
Example:
apic1(config-monitor-fabric-dest)#
erspan-id 100

Step 6 [no] ip dscp dscp-code Configures the differentiated services code

point (DSCP) value of the packets in the
Example:
ERSPAN traffic. The range is from 0 to 64.
apic1(config-monitor-fabric-dest)# ip
dscp 42

Step 7 [no] ip ttl ttl-value Configures the IP time-to-live (TTL) value for
the ERSPAN traffic. The range is from 1 to
Example:
255.
apic1(config-monitor-fabric-dest)# ip
ttl 16

Step 8 [no] mtu mtu-value Configures the maximum transmit unit (MTU)
size for the ERSPAN session. The range is 64
Example:
to 9216 bytes.
apic1(config-monitor-fabric-dest)# mtu
9216

Step 9 exit Returns to monitor access configuration mode.

Example:
apic1(config-monitor-fabric-dest)#

Step 10 [no] source interface ethernet {slot/port | Specifies the source interface port or port
port-range} switch node-id range.
Example:
apic1(config-monitor-fabric)# source
interface eth 1/2 switch 101

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

512
Configuring SPAN
Configuring ERSPAN in Fabric Mode

Command or Action Purpose

Step 11 [no] direction {rx | tx | both} Specifies direction of traffic to be monitored.
The direction can be configured independently
Example:
for each source port range.
apic1(config-monitor-fabric-source)#
direction tx

Step 12 [no] filter tenant tenant-name bd bd-name Filters traffic by bridge domain.
Example:
apic1(config-monitor-fabric-source)#
filter tenant t1 bd bd1

Step 13 [no] filter tenant tenant-name vrf vrf-name Filters traffic by VRF.
Example:
apic1(config-monitor-fabric-source)#
filter tenant t1 vrf vrf1

Step 14 exit Returns to access monitor session

configuration mode.
Example:
apic1(config-monitor-fabric-source)#
exit

Step 15 [no] shutdown Disables (or enables) the monitoring session.

Example:
apic1(config-monitor-fabric)# no shut

Examples
This example shows how to configure an ERSPAN fabric monitoring session.

apic1# configure terminal

apic1(config)# monitor fabric session mySession
apic1(config-monitor-fabric)# description "This is my fabric ERSPAN session"
apic1(config-monitor-fabric)# destination tenant t1 application app1 epg epg1 destination-ip
192.0.20.123 source-ip-prefix 10.0.20.1
apic1(config-monitor-fabric-dest)# erspan-id 100
apic1(config-monitor-fabric-dest)# ip dscp 42
apic1(config-monitor-fabric-dest)# ip ttl 16
apic1(config-monitor-fabric-dest)# mtu 9216
apic1(config-monitor-fabric-dest)# exit
apic1(config-monitor-fabric)# source interface eth 1/1 switch 101
apic1(config-monitor-fabric-source)# direction tx
apic1(config-monitor-fabric-source)# filter tenant t1 bd bd1
apic1(config-monitor-fabric-source)# filter tenant t1 vrf vrf1
apic1(config-monitor-fabric-source)# exit
apic1(config-monitor-fabric)# no shut

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

513
 Configuring SPAN
Configuring ERSPAN in Tenant Mode

Configuring ERSPAN in Tenant Mode

In the ACI fabric, a tenant mode ERSPAN configuration can be used for monitoring traffic originating from
endpoint groups within a tenant.
In the tenant mode, traffic originating from a source EPG is sent to a destination EPG within the same tenant.
The monitoring of traffic is not impacted if the source or destination EPG is moved within the fabric.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 [no] monitor tenant tenant-name session Creates a tenant monitoring session
session-name configuration.
Example:
apic1(config)# monitor tenant session
mySession

Step 3 [no] description text Adds a description for this access monitoring
session. If the text includes spaces, it must be
Example:
enclosed in single quotes.
apic1(config-monitor-tenant)#
description "This is my tenant ERSPAN
session"

Step 4 [no] destination tenant tenant-name Specifies the destination interface as a tenant
application application-name epg epg-name and enters destination configuration mode.
destination-ip dest-ip-address
source-ip-prefix src-ip-address
Example:
apic1(config-monitor-tenant)#
destination tenant t1 application app1
epg epg1 destination-ip 192.0.20.123
source-ip-prefix 10.0.20.1

Step 5 [no] erspan-id flow-id Configures the ERSPAN ID for the ERSPAN
session. The ERSPAN range is from 1 to 1023.
Example:
apic1(config-monitor-tenant-dest)#
erspan-id 100

Step 6 [no] ip dscp dscp-code Configures the differentiated services code

point (DSCP) value of the packets in the
Example:
ERSPAN traffic. The range is from 0 to 64.
apic1(config-monitor-tenant-dest)# ip
dscp 42

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

514
Configuring SPAN
Configuring ERSPAN in Tenant Mode

Command or Action Purpose

Step 7 [no] ip ttl ttl-value Configures the IP time-to-live (TTL) value for
the ERSPAN traffic. The range is from 1 to
Example:
255.
apic1(config-monitor-tenant-dest)# ip
ttl 16

Step 8 [no] mtu mtu-value Configures the maximum transmit unit (MTU)
size for the ERSPAN session. The range is 64
Example:
to 9216 bytes.
apic1(config-monitor-tenant-dest)# mtu
9216

Step 9 exit Returns to monitor access configuration mode.

Example:
apic1(config-monitor-tenant-dest)#

Step 10 [no] source application application-name epg Specifies the source interface port or port
epg-name range.
Example:
apic1(config-monitor-tenant)# source
application app2 epg epg5

Step 11 [no] direction {rx | tx | both} Specifies direction of traffic to be monitored.

The direction can be configured independently
Example:
for each source port range.
apic1(config-monitor-tenant-source)#
direction tx

Step 12 exit Returns to access monitor session

configuration mode.
Example:
apic1(config-monitor-tenant-source)#
exit

Step 13 [no] shutdown Disables (or enables) the monitoring session.

Example:
apic1(config-monitor-tenant)# no shut

Examples
This example shows how to configure an ERSPAN tenant monitoring session.

apic1# configure terminal

apic1(config)# monitor access session mySession
apic1(config-monitor-tenant)# description "This is my tenant ERSPAN session"
apic1(config-monitor-tenant)# destination tenant t1 application app1 epg epg1 destination-ip
192.0.20.123 source-ip-prefix 10.0.20.1
apic1(config-monitor-tenant-dest)# erspan-id 100
apic1(config-monitor-tenant-dest)# ip dscp 42
apic1(config-monitor-tenant-dest)# ip ttl 16
apic1(config-monitor-tenant-dest)# mtu 9216

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

515
 Configuring SPAN
Configuring ERSPAN in Tenant Mode

apic1(config-monitor-tenant-dest)# exit
apic1(config-monitor-tenant)# source application app2 epg epg5
apic1(config-monitor-tenant-source)# direction tx
apic1(config-monitor-tenant-source)# exit
apic1(config-monitor-tenant)# no shut

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

516
 CHAPTER 21
Applying the show running config Output to
Another Cisco APIC
This section explains how to use the export config and import config CLIs to use the show running
config output on another Cisco APIC.

• About Import and Export Configurations, on page 517

• Import and Export Configuration Guidelines and Limitations, on page 517
• Exporting a CLI Configuration, on page 517
• Importing a CLI Configuration, on page 518

About Import and Export Configurations

The import config and export config commands enable you to apply the show running config output to
another Cisco APIC. This section contains the guidelines for these commands and demonstrates how the
commands are executed.

Import and Export Configuration Guidelines and Limitations

This section explains the guidelines and limitations for the export config and import config commands.
• Passwords and other encrypted data are not included in the configuration file.
• Some REST API configurations may not be compatible with CLI configurations; this may cause errors
when applying a configuration file to a Cisco APIC.
• Some features require configurations to be in a specific order. These configurations are validated when
performed through the CLI. Configurations through the REST API, however, are not validated and may
cause errors when running the imported file due to missing configurations.
• Interactive commands are prefixed with a "#" and ignored when running the configuration file.

Exporting a CLI Configuration

This procedure shows how to export a configuration to a text file.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

517
 Applying the show running config Output to Another Cisco APIC
Importing a CLI Configuration

Procedure

Command or Action Purpose

Step 1 configure Enters configuration mode.
Example:
dev4-ifc1# configure

Step 2 leaf ID Identifies the leaf with the configuration to be

exported.
Example:
dev4-ifc1(config)# leaf 101

Step 3 interface ethernet slot/port Identifies the slot number and port number for
an existing Ethernet interface.
Example:
dev4-ifc1(config-leaf)# interface
ethernet 1/34

Step 4 export-config result-file-name Exports the configuration to a specified file

name.
Example:
dev4-ifc1(config-leaf-if)# export-config
/tmp/showRunnLeaf101.txt

Example
This example shows how to configure export-config.
dev4-ifc1# config
dev4-ifc1(config)# leaf 101
dev4-ifc1(config-leaf)# interface ethernet 1/34
dev4-ifc1(config-leaf-if)# export-config /tmp/showRunnLeaf101.txt
dev4-ifc1(config-leaf-if)# cat /tmp/showRunnLeaf101.txt
config
# Command: show running-config leaf 101 interface ethernet 1 / 34
# Time: Fri Sep 23 16:03:48 2016
leaf 101
interface ethernet 1/34
switchport trunk allowed vlan 602 tenant t1 external-svi l3out l3ext1sub1
exit
exit
dev4-ifc1(config-leaf-if)#

Importing a CLI Configuration

This procedure shows how to import a configuration from a text file.

Procedure

Command or Action Purpose

Step 1 import-config file-name

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

518
Applying the show running config Output to Another Cisco APIC
Importing a CLI Configuration

Command or Action Purpose

Example:
dev4-ifc1(config-tenant)# import-config
/tmp/showRunnLeaf101.txt
config
# Command: show running-config leaf 101
interface ethernet 1 / 34
# Time: Fri Sep 23 16:03:48 2016
leaf 101
interface ethernet 1/34
switchport trunk allowed vlan 602 tenant
t1 external-svi l3out l3ext1sub1
exit
exit
dev4-ifc1(config)#

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

519
 Applying the show running config Output to Another Cisco APIC
Importing a CLI Configuration

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

520
 CHAPTER 22
Configuring a Forwarding Scale Profile Policy
• Forwarding Scale Profile Policy Overview, on page 521
• Supported Platforms for Forwarding Scale Profile Policies, on page 523
• Configuring the Forwarding Scale Profile Policy Using the NX-OS-Style CLI, on page 523

Forwarding Scale Profile Policy Overview

The Forwarding Scale Profile policy provides different scalability options. For example:
• Dual Stack—provides scalability of up to 12,000 endpoints for IPv6 configurations and up to 24,000
endpoints for IPv4 configurations.
• IPv4 Scale—enables systems with no IPv6 configurations to increase scalability to 48,000 IPv4 endpoints.
• High Dual Stack—provides scalability of up to 64,000 MAC endpoints and 64,000 IPv4 endpoints.
IPv6 endpoint scale can be 24,000/48,000, depending on the switch hardware model.

Note With Cisco APIC Release 3.2(1), depending on your TOR switch hardware, a
Forwarding Scale Profile with the High Dual Stack option has different scales;
for example:
• For Cisco Nexus 9000 Series TOR switches with FX in the switch name,
the high dual-stack option has scalability of 48,000 IPv6 endpoints instead
of 24,000 and 128,000 policies instead of 8,000.
• For Cisco Nexus 9000 Series TOR switches with EX in the switch name,
the high dual-stack option has the same scale values as with earlier APIC
releases.

See the following table for more details.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

521
 Configuring a Forwarding Scale Profile Policy
Forwarding Scale Profile Policy Overview

Table 21: Forwarding Scale Profile Policy Scalability

Forwarding Scale Profile Policy TOR Switches with EX Names TOR Switches with FX Names
Options

Dual Stack • EP MAC: 24,000 Has the same scalability numbers

as Dual Stack scale on earlier
• EP IPv4: 24,000 switches.
• EP IPv6: 12,000
• LPM: 20,000
• Policy: 64,000
• Multicast: 8,000

High Dual Stack • EP MAC: 64,000 • EP MAC: 64,000

• EP IPv4: 64,000 • EP IPv4: 64,000
• EP IPv6: 24,000 • EP IPv6: 48,000
• LPM: 38,000 • LPM: 38,000
• Policy: 8,000 • Policy: 128,000
• Multicast: 0 • Multicast: 512

IPv4 Scale • EP MAC: 48,000 Has the same scalability numbers

as IPv4 scale on earlier switches.
• EP IPv4: 48,000
• EP IPv6: 0
• LPM: 38,000
• Policy: 60,000
• Multicast: 8,000

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

522
 Configuring a Forwarding Scale Profile Policy
Supported Platforms for Forwarding Scale Profile Policies

Note • Because the IPv4 forwarding scale profile policy does not support IPv6 configurations, all IPv6
configurations must be removed from switches configured with the IPv4 forwarding scale profile policy.
• Because the high dual stack profile has reduced-scale support for contract policies (8,000), the contracts
scale must be reduced accordingly prior to deploying that profile.
• Before migrating to minimal tenant multicast scale leaf profiles, such as high dual stack, we recommend
that you first disable Layer 2 IGMP snooping-, Layer 3 IGMP-, and PIM-related configurations to prevent
having a stale multicast state in your hardware.
• Applying a scale profile to a node requires a manual reload of that node. Any unsupported switches are
ignored. For a list of supported switches, see Supported Platforms for Forwarding Scale Profile Policies,
on page 523.
• VPCs associated with different scale profile settings are not supported. The VPC members must be
configured with the same scale profile settings.

Supported Platforms for Forwarding Scale Profile Policies

The forwarding scale profile policy is only supported on the following switches:
• Cisco Nexus 9300-EX Series switches
• N9K-C9348GC-FXP
• N9K-C93108TC-FX
• N9K-C93180YC-FX

Configuring the Forwarding Scale Profile Policy Using the

NX-OS-Style CLI
Before you begin
The Forwarding Scale Profile policy provides different scalability options. For more information on the
scalability options, see the Forwarding Scale Profile Policy Overview section in the chapter for your Cisco
APIC release.
The forwarding scale profile policy requires supported switches. For a list of supported switches, see the
Supported Platforms for Forwarding Scale Profile Policies section in the chapter for your Cisco APIC release.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

523
 Configuring a Forwarding Scale Profile Policy
Configuring the Forwarding Scale Profile Policy Using the NX-OS-Style CLI

Note • The switches that support the forwarding scale profile policy must be manually reloaded after the
forwarding scale profile policy is applied.
• Changing the scale profile for individual members of a VPC is not allowed. If members of the same VPC
are associated with different leaf profiles, then a new leaf profile should be created with both members
and the scale profile applied to it.

This section demonstrates how to configure the forwarding scale profile policy using the NX-OS-style CLI.

Procedure

Command or Action Purpose

Step 1 configure Enters global configuration mode.
Example:
apic1# configure

Step 2 no scale-profile name Defines the scale-profile policy.

Example:
apic1(config)# scale-profile
testFwdScaleProf

Step 3 profile-type {dual-stack | high-dual-stack | Sets the Forwarding Scale profile type.
high-lpm | high-policy | ipv4 }
Example:
apic1(config-scale-profile)#
profile-type ipv4

Step 4 exit Returns back to global configuration.

Example:
apic1(config-scale-profile)# exit

Step 5 template leaf-policy-group leaf_group_name Defines the leaf policy group.

Example:
apic1(config)# template
leaf-policy-group samplePolicyGrp

Step 6 scale-profile name Configures the relation between the

scale-profile policy and the leaf policy group.
Example:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

524
Configuring a Forwarding Scale Profile Policy
Configuring the Forwarding Scale Profile Policy Using the NX-OS-Style CLI

Command or Action Purpose

apic1(config-leaf-policy-group)# Note The switches that support the
scale-profile testFwdScaleProf
forwarding scale profile policy must
be manually reloaded after the
forwarding scale profile policy is
applied. For a list of supported
switches, see the Supported
Platforms for Forwarding Scale
Profile Policies section in the
chapter for your Cisco APIC
release.

Step 7 exit Returns back to global configuration.

Example:
apic1(config-leaf-policy-group)# exit

Step 8 leaf-profile leaf_profile_name Configures a leaf profile.

Example:
apic1(config)# leaf-profile
sampleLeafProf

Step 9 leaf-group leaf_group_name Specifies a group of leaf switches.

Example:
apic1(config-leaf-profile)# leaf-group
sampleLeafGrp

Step 10 leaf leaf_group_number Adds leaf switches to the leaf group.

Example:
apic1(config-leaf-profile)# leaf 201

Step 11 leaf-policy-group leaf_policy_group_name Specifies the leaf policy group to be associated

to the leaf switches.
Example:
apic1(config-leaf-group)#
leaf-policy-group samplePolicyGrp

Step 12 exit Exits command mode.

Example:
apic1(config-leaf-policy-group)# exit

Step 13 [show] running-config Displays the current running configuration.

Example:

apic1(config)# show running-config

# Command: show running-config
scale-profile testFwdScaleProf
# Time: Thu Jul 27 22:31:29 2017
scale-profile testFwdScaleProf
profile-type ipv4

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

525
 Configuring a Forwarding Scale Profile Policy
Configuring the Forwarding Scale Profile Policy Using the NX-OS-Style CLI

Command or Action Purpose

exit
apic1(config-scale-profile)#

Step 14 [show] template leaf-policy-group Displays the current running configuration.

Example:

Examples
This example shows how to configure the IPv4 scale profile policy.

apic1# configure
apic1(config)# scale-profile testFwdScaleProf
apic1(config-scale-profile)# profile-type ipv4
apic1(config-scale-profile)# exit
apic1(config)# template leaf-policy-group samplePolicyGrp
apic1(config-leaf-policy-group)# scale-profile testFwdScaleProf
apic1(config-leaf-policy-group)# exit
apic1(config)# leaf-profile sampleLeafProf
apic1(config-leaf-profile)# leaf-group sampleLeafGrp
apic1(config-leaf-profile)# leaf 201
apic1(config-leaf-group)# leaf-policy-group samplePolicyGrp
apic1(config-leaf-group)# show running-config scale-profile testFwdScaleProf
# Command: show running-config scale-profile testFwdScaleProf
# Time: Thu Jul 27 22:31:29 2017
scale-profile testFwdScaleProf
profile-type ipv4
exit
apic1(config-leaf-group)# show running-config template leaf-policy-group samplePolicyGrp
# Command: show running-config template leaf-policy-group samplePolicyGrp
# Time: Tue Aug 1 11:19:44 2017
template leaf-policy-group samplePolicyGrp
scale-profile testFwdScaleProf
exit
apic1(config-leaf-group)# show running-config leaf-profile sampleLeafProf
# Command: show running-config leaf-profile sampleLeafProf
# Time: Tue Aug 1 11:19:58 2017
leaf-profile sampleLeafProf
leaf-group sampleLeafGrp
leaf 201
leaf-policy-group samplePolicyGrp
exit

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

526
 APPENDIX A
Verified Scalability Using the CLI
• CLI Scalability Limits, on page 527

CLI Scalability Limits

Configurable Option Scale

Number of tenants 500

Number of Layer 3 (L3) contexts 300

Number of endpoint groups (EPGs) 3,500

Number of endpoints (EPs) 20,000

Number of bridge domains (BDs) 3,500

Number of BGP + number of OSPF sessions + EIGRP 300

(for external connection)

Maximum number of vPCs 48

Maximum number of PCs, access ports 48

Maximum number of encaps per access port 1,750

Number of multicast groups 8,000

Maximum number of vzAny provided contracts 16

Maximum number of vzAny consumed contracts 16

Maximum amount of encaps per endpoint group 2 static, 1 dynamic

Security TCAM size 4,000

Number of VRFs 500

Separate-Config-Set

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

527
 Verified Scalability Using the CLI
Verified Scalability Using the CLI

Configurable Option Scale

Tenants 100

Endpoint groups 1,000

Bridge domains 500

VRFs 100

SPAN destinations 3

NTP servers 2

Contracts 100

DNS servers 2

Syslog servers 1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

528
 APPENDIX B
Use Case: Three-Tier Application with Transit
Topology
• About Deploying a Three-Tier Application with Transit Topology, on page 529
• Deploying a Three-Tier Application, on page 531
• Transit Routing with OSPF and BGP, on page 533

About Deploying a Three-Tier Application with Transit Topology

Typically, the APIC fabric hosts a three-tier application within a tenant network. In this example, the application
is implemented by using three servers (a web server, an application server, and a database server). See the
following figure for an example of a three-tier application.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

529
 Use Case: Three-Tier Application with Transit Topology
Use Case: Three-Tier Application with Transit Topology

The web server has the HTTP filter, the application server has the Remote Method Invocation (RMI) filter,
and the database server has the Structured Query Language (SQL) filter. The application server consumes the
SQL contract to communicate with the database server. The web server consumes the RMI contract to
communicate with the application server. The traffic enters from the web server and communicates with the
application server. The application server then communicates with the database server, and the traffic can
also communicate externally.
To deploy the three-tier application, you must create the required EPGs, filters, and contracts.
A filter specifies the data protocols to be allowed or denied by a contract that contains the filter. A contract
can contain multiple subjects. A subject can be used to realize uni- or bidirectional filters. A unidirectional
filter is a filter that is used in one direction, either from consumer-to-provider (IN) or from provider-to-consumer
(OUT) filter. A bidirectional filter is the same filter that is used in both directions. It is not reflexive.
Contracts are policies that enable inter-End Point Group (inter-EPG) communication. These policies are the
rules that specify communication between application tiers. If no contract is attached to the EPG, inter-EPG
communication is disabled by default. No contract is required for intra-EPG communication because intra-EPG
communication is always allowed.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

530
 Use Case: Three-Tier Application with Transit Topology
Deploying a Three-Tier Application

About Transit Routing

Transit routing enables border routers to perform bidirectional redistribution with other routing domains.
Bidirectional redistribution passes routing information from one routing domain to another. Such redistribution
lets the ACI fabric provide full IP connectivity between different routing domains. Doing so can also provide
redundant connectivity by enabling backup paths between routing domains. For more information, see "ACI
Transit Routing" in the Cisco ACI Fundamentals Guide.

Deploying a Three-Tier Application

Configure the tenant VRF and bridge domain.

apic1(config)# tenant t1
apic1(config-tenant)# vrf context v1
apic1(config-tenant-vrf)# contract enforce
apic1(config-tenant)# bridge-domain b1
apic1(config-tenant-bd)# vrf member v1
apic1(config-tenant)# interface bridge-domain b1
apic1(config-tenant-interface)# ip address 159.10.10.1/24 scope public
apic1(config-tenant-interface)# exit

Configure three EPGs: web, app, and db.

apic1(config-tenant)# application retail

apic1(config-tenant-app)# epg web
apic1(config-tenant-app-epg)# bridge-domain member b1
apic1(config-tenant-app-epg)# contract provider web
apic1(config-tenant-app-epg)# contract consumer app
apic1(config-tenant-app)# epg app
apic1(config-tenant-app-epg)# bridge-domain member b1
apic1(config-tenant-app-epg)# contract provider app
apic1(config-tenant-app-epg)# contract consumer db
apic1(config-tenant-app)# epg db
apic1(config-tenant-app-epg)# bridge-domain member b1
apic1(config-tenant-app-epg)# contract provider db

Configure VLAN domain.

apic1(config)# vlan-domain dom100

apic1(config-vlan)# vlan 100-200

Create port-channel and deploy the web EPG.

apic1(config)# leaf 101

apic1(config-leaf)# interface ethernet 1/2-5
apic1(config-leaf-if)# channel-group po1

apic1(config-leaf)# interface port-channel po1

apic1(config-leaf-if)# vlan-domain member dom100
apic1(config-leaf-if)# switchport trunk allowed vlan 101 tenant t1 application retail epg
web

Create a vPC and deploy app and db EPGs.

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

531
 Use Case: Three-Tier Application with Transit Topology
Use Case: Three-Tier Application with Transit Topology

apic1(config)# leaf 101,102

apic1(config-leaf)# interface ethernet 1/6,1/7
apic1(config-leaf-if)# channel-group vpc1 vpc

apic1(config)# vpc domain explicit 100 leaf 101 102

apic1(config)# vpc context leaf 101 102
apic1(config-vpc)# interface vpc vpc1
apic1(config-vpc-if)# vlan-domain member dom100
apic1(config-vpc-if)# switchport trunk allowed vlan 102 tenant t1 application retail epg
app
apic1(config-vpc-if)# switchport trunk allowed vlan 103 tenant t1 application retail epg
db

Configure MP-BGP.

apic1(config)# bgp-fabric
apic1(config-bgp-fabric)# asn 100
apic1(config-bgp-fabric)# route-reflector spine 104,105

Configure External-l3 EPG.

apic1(config-tenant)# external-l3 epg l3epg1

apic1(config-tenant-l3ext-epg)# vrf member v1
apic1(config-tenant-l3ext-epg)# match ip 173.10.1.0/24
apic1(config-tenant-l3ext-epg)# contract consumer web

Configure VRF on Leaf , route-map and deploy external-l3 EPG.

apic1(config)# leaf 103

apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# external-l3 epg l3epg1
apic1(config-leaf-vrf)# route-map map1
apic1(config-leaf-vrf-route-map)# match bridge-domain b1

Configure OSPF area on a sub-Interface.

apic1(config-leaf)# router ospf default

apic1(config-leaf-ospf)# vrf member tenant t1 vrf v1
apic1(config-leaf-ospf-vrf)# area 0.0.0.1 route-map map1 out
apic1(config-leaf)# interface ethernet 1/2
apic1(config-leaf-if)# no switchport
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf)# interface ethernet 1/2.150
apic1(config-leaf-if)# vrf member tenant t1 vrf v1
apic1(config-leaf-if)# ip address 169.10.10.1/24
apic1(config-leaf-if)# ip router ospf default area 0.0.0.1

Configure filters.

apic1(config-tenant)# access-list http

apic1(config-tenant-acl)# match tcp dest 80
apic1(config-tenant-acl)# match tcp dest 443

apic1(config-tenant)# access-list rmi

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

532
 Use Case: Three-Tier Application with Transit Topology
Transit Routing with OSPF and BGP

apic1(config-tenant-acl)# match tcp dest 1099

apic1(config-tenant)# access-list sql

apic1(config-tenant-acl)# match tcp dest 1521

Configure contracts.

apic1(config-tenant)# contract rmi

apic1(config-tenant-contract)# subject rmi
apic1(config-tenant-contract-subj)# access-group rmi both

apic1(config-tenant)# contract web

apic1(config-tenant-contract)# subject web
apic1(config-tenant-contract-subj)# access-group http both

apic1(config-tenant)# contract db
apic1(config-tenant-contract)# subject sql
apic1(config-tenant-contract-subj)# access-group sql both

Transit Routing with OSPF and BGP

This procedure configures transit routing between Site1 and Site2 for the three-tier application described in
Deploying a Three-Tier Application in this chapter.
Configure External-l3 EPG (l3epg2) for Site2.

apic1(config-tenant)# external-l3 epg l3epg2

apic1(config-tenant-l3ext-epg)# vrf member v1
apic1(config-tenant-l3ext-epg)# match ip 174.10.1.0/24
apic1(config-tenant-l3ext-epg)# contract consumer transit
apic1(config)# leaf 102
apic1(config-leaf)# vrf context tenant t1 vrf v1
apic1(config-leaf-vrf)# external-l3 epg l3epg2

Configure BGP connectivity over External SVI and export route corresponding to Site1.

apic1(config)# leaf 102

apic1(config-leaf-vrf)# route-map map200
apic1(config-leaf-vrf-route-map)# ip prefix-list p1 match 173.10.1.0/24
apic1(config-leaf-vrf-route-map)# match prefix-list p1
apic1(config-leaf-vrf-route-map-match)# set community extended 200:1 replace

apic1(config-leaf)# interface vlan 160

apic1(config-leaf-if)# vrf member tenant t1 vrf v1
apic1(config-leaf-if)# ip address 208.1.1.2/24
apic1(config-leaf)# interface ethernet 1/11
apic1(config-leaf-if)# vlan-domain member dom1
apic1(config-leaf-if)# switchport trunk allowed vlan 160 tenant t1 external-svi

apic1(config-leaf)# router bgp 100

apic1(config-bgp)# vrf member tenant t1 vrf v1
apic1(config-leaf-bgp-vrf)# neighbor 208.1.1.1
apic1(config-leaf-bgp-vrf-neighbor)# remote-as 200
apic1(config-leaf-bgp-vrf-neighbor)# update-source vlan 160

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

533
 Use Case: Three-Tier Application with Transit Topology
Use Case: Three-Tier Application with Transit Topology

apic1(config-leaf-bgp-vrf-neighbor)# route-map map200 out

Configure contract provider on l3epg1 (Site1) to establish connection with l3epg2 (Site2)

apic1(config-tenant)# external-l3 epg l3epg1

apic1(config-tenant-l3ext-epg)# contract provider transit

Configure a route-map on Site1 to export the route corresponding to Site2.

apic1(config)# leaf 103

apic1(config-leaf-vrf)# route-map map1
apic1(config-leaf-vrf-route-map)# ip prefix-list p1 match 174.10.1.0/24
apic1(config-leaf-vrf-route-map)# match prefix-list p1
apic1(config-leaf-vrf-route-map-match)# set metric 100

Configure ACL and contract for transit routing.

apic1(config)# tenant t1
apic1(config-tenant)# access-list acl1
apic1(config-tenant-acl)# match ip
apic1(config-tenant)# contract transit
apic1(config-tenant-contract)# subject ip
apic1(config-tenant-contract-subj)# access-group acl1 both

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

534
 APPENDIX C
Examples: Show Commands
• Examples: Show Commands, on page 535

Examples: Show Commands

show running-config
show running-config “local” to the current mode.

apic1(config)# leaf 103

apic1(config-leaf)# interface ethernet 1/2.150
apic1(config-leaf-if)# show running-config
# Command: show running-config leaf 103 interface ethernet 1 / 2 . 150
# Time: Tue Dec 8 08:08:37 2015
leaf 103
interface ethernet 1/2.150
vrf member tenant t1 vrf v1
ip address 169.10.10.1/24
ip router ospf default area 0.0.0.1
exit
exit

show running-config with filters.

apic1(config-leaf)# interface ethernet 1/2.150

apic1(config-leaf-if)# show running-config leaf 103
# Command: show running-config leaf 103
# Time: Tue Dec 8 08:10:02 2015
leaf 103
vrf context tenant t1 vrf v1
external-l3 epg l3epg1
route-map map1
ip prefix-list p1 permit 181.1.1.0/24
match bridge-domain b1
match prefix-list p1
…

show vpc, port-channel

show vpc map

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

535
 Examples: Show Commands
Examples: Show Commands

apic1(config-leaf-if)# show vpc map

Legends:
N/D : Not Deployed

Virtual Port-Channel Name Domain VPC Leaf Id, Name Fex Id PC Id Ports

------------------------- ------ --- ------------- ------ ----- --------------------

vpc1 100 1 101,leaf1 po2 eth1/6-7, eth1/40-41

vpc1 100 1 102,leaf2 po1 eth1/6-7, eth1/40-41

show port-channel map

apic1(config-leaf-if)# show port-channel map

Legends:
N/D : Not Deployed
PC: Port Channel
VPC: Virtual Port Channel

Port-Channel Name Type Leaf ID, Name Fex Id Port Channel Ports
----------------- ---- ------------------ ------ ------------- --------------------
po1 PC 101,leaf1 po1 eth1/2-5, eth1/32-33
po1 PC 102,leaf2 po2 eth1/32-33

vpc1 VPC 101,leaf1 po2 eth1/6-7, eth1/40-41

vpc1 VPC 102,leaf2 po1 eth1/6-7, eth1/40-41

show vlan-domain
show vlan-domain name dom100

apic1# show vlan-domain name dom100

Legend:
vlanscope: L (Portlocal). Default is global

vlan-domain : dom100 Type : All

vlan : 100-200(static)

Leaf Interface Vlan Type Usage Operational State Operational Vlan

-------- ---------- ---- --------- ------------ ----------------- ----------------
101 PC: po1 101 App-Epg Tenant: t1 b1: down b1: vlan-18
App: retail web: down web: vlan-21
Epg: web

101,102 vPC: vpc1 102 App-Epg Tenant: t1 b1: down b1: vlan-18
App: retail app: down app: vlan-19
Epg: app

101,102 vPC: vpc1 103 App-Epg Tenant: t1 b1: down b1: vlan-18
App: retail db: down db: vlan-20
Epg: db

102 eth1/11 160 Ext-svi Tenant: t1 l2: down vlan-18

Vrf: v1 l3: down

103 eth1/2 150 Ext-subIf Tenant: t1 - eth1/2.14

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

536
Examples: Show Commands
Examples: Show Commands

Vrf: v1

show tenant
show tenant t1 detail

apic1# show tenant t1 detail

Detailed view for Tenant t1
Security Information:
Security Domain
----------------------------------------

VRF Information:
VRF Policy Enforcement
-------------------- --------------------
v1 enforced

Bridge-Domain Information:
BD VRF
-------------------- --------------------
b1 v1

Static VLAN Information:

Node VLANs VLAN Domains
-------- ---------------------------------------- ------------------------------
101 101 dom100
101 102 102,103 dom100

Static Application EPg Information:

Node Interface App:AEPg BD
Contract
-------- ------------------------------ -------------------- --------------------
--------
101 port-channel po1 retail:web b1 web,

app

101 102 vpc vpc1 retail:db,retail:app b1 app,

db

Application EPg Information:

App:AEPg BD
-------------------- --------------------
retail:app b1
retail:db b1
retail:web b1

External L2 EPg Information:

external-l2 BD
-------------------- --------------------

External L3 EPg Information:

external-l3 VRF
-------------------- --------------------
l3epg1 v1
l3epg2 v1

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

537
 Examples: Show Commands
Examples: Show Commands

show external-l3
show external-l3 interfaces

apic1# show external-l3 interfaces

Node Tenant VRF Interface Oper Interface IP Address
Oper IP
----- ------------ ------------ ---------------- ---------------- --------------
-------
102 t1 v1 vlan-160 eth1/11 vlan18 208.1.1.2/24 up

eth1/11

103 t1 v1 eth1/2.150 eth1/2.14 169.10.10.1/24 up

show external-l3 epg

apic1# show external-l3 epg

Name Flags Match Node Entry Oper
State
---------- ------------------------- -------------- ---------- ---------------
----------
t1: vxlan: 2457600 173.10.1.0/24
l3epg1 vrf: v1
Target dscp: unspecified
qosclass: unspecified
Contracts
---------
Provided: transit
Consumed: web
t1: vxlan: 2457600 173.10.1.0/24 node-103 173.10.1.0/24
disabled
l3epg2 vrf: v1 node-101 173.10.1.0/24
disabled
Target dscp: unspecified node-102 173.10.1.0/24
disabled
qosclass: unspecified
Contracts
---------
Provided:
Consumed: transit

show external-l3 ospf

apic1# show external-l3 ospf tenant t1 vrf v1

Area Id : 0.0.0.1
Tenant : t1
Vrf : v1
User Config :
Node ID Area Properties
---- ----------------------------------------------------------------------
103 Type: nssa, Cost: 1, Control: redistribute,summary

Configuration : Operational

Node ID Router ID Route Map Area Oper. Props

---- --------------- ---------------- -----------------------------------
103 10.1.0.103 map1 Type: nssa, Cost: 1, Control:
redistribute,summary, AreaId:

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

538
Examples: Show Commands
Examples: Show Commands

0.0.0.1
Interfaces :
Configuration : Operational

Node ID Interface IP Address Oper. Intf Oper. State

---- ------------ --------------- -------- --------
103 eth1/2.150 169.10.10.1/24 eth1/2.14 down

show external-l3 bgp

apic1# show external-l3 bgp

flags_match : Properties in logical and concrete MOs are symmetric
Tenant, vrf : t1, v1

Node Neighbor Flags RouteMap SourceIf Oper Peer Status

Session Status
---- ---------- ------------------------ ---------- ---------- -----------------
--------------
102 208.1.1.1 Allowed Self As Count: 3 no (in) Vlan vlan18
TTL: 1 map200 160 flags_match
(out)

show external-l3 route-map

apic1# show external-l3 route-map

Tenant : t1 VRF: v1
Table1: Route Map Configuration

Node Routemap Type Name Match Set Attributes

----- ---------------- ------- ------------------ --------------------

----------------------
102 map200 PfxList p1 100.100.100.0/24 Community
value:
173.10.1.0/24
extended:as4-nn2:200:1
103 map1 PfxList p1 181.1.1.0/24 Metric: 100
103 map1 BD b1 159.10.10.1/24

Table 2 : Route Map Usage

Node Routemap Protocol Neighbors Operational Attributes

----- ----------- ---------- ------------------ ----------------------
102 map200 bgp 208.1.1.1 Pfx List: p1
100.100.100.0/24
173.10.1.0/24
::/0
103 map1 ospf 0.0.0.1 Pfx List: p1
Metric: 100
181.1.1.0/24
::/0

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

539
 Examples: Show Commands
Examples: Show Commands

Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

540