Cisco OperatingApplicationCentricInfrastructure

Operating Cisco
Application Centric
Infrastructure
Alejandra Sanchez, Andres Vega, Arvind Chari, Carly Stoughton,

Christopher Stokes, Gabriel Fontenot, Jonathan Cornell, Ken Fee,
Kevin Corbin, Lauren Malhoit, Loy Evans, Lucien Avramov,
Paul Lesiak, Steven Lym, Rafael Muller, Robert Burns
Operating Cisco
Application Centric
Infrastructure
Copyright
Op
er
at
ing Cisco Ap
pli
ca
tion Cen
tric In
fra
struc
ture
Ale
jan
dra Sanchez, An
dres Vega, Arvind Chari, Carly Stoughton, Christo
pher Stokes,
Gabriel Fontenot, Jonathan Cor
nell, Ken Fee, Kevin Corbin, Lau
ren Mal
hoit, Loy Evans,
Lu
cien Avramov, Paul Lesiak, Rafael Muller, Robert Burns. Steven Lym
Copy
right 2015 Cisco Sys
tems, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this
file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or
agreed to in writing, software distributed under the License is distributed on an "AS IS"
BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied. See the License for the specific language governing permissions and
limitations under the License.
Cisco Press logo is a trade
mark of Cisco Sys
tems, Inc.
Pri
vately pub
lished by Cisco Sys
tems, Inc.
Warning and Disclaimer

This book is de
signed to pro
vide in
for
ma
tion about Cisco ACI. Every ef
fort has been
made to make this book as com
plete and as ac
cu
rate as pos
si
ble, but no war
ranty or
fit
ness is im
plied.
The in
for
ma
tion is pro
vided on an as is basis. The au
thors, and Cisco Sys
tems, Inc.
shall have nei
ther li
ab
il
ity nor re
spon
si
bil
ity to any per
son or en
tity with re
spect to any
loss or dam
ages aris
ing from the in
for
ma
tion con
tained in this doc
um
ent.
The opin
ions ex
pressed in this book be
long to the au
thor and are not nec
es
sar
ily those
of Cisco Sys
tems, Inc.
Feed
back In
for
ma
tion
Read
ers feed
back is a nat
ural con
tin
ua
t
ion of this process. If you have any com
ments
re
gard
ing how we could im
prove the qual
ity of this book, or oth
er
wise alter it to bet
ter
suit your needs, you can contact
us through email at ops-booksprint@
cisco.
com .
Contents
. . . . . Prologue
........................................................................
. . . . . Abstract
...................................................................
. . . . . Authors
...................................................................
. . . . . Book
. . . . . Writing
. . . . . . . .Methodology
......................................................
. . . . . Hardware
. . . . . . . . . .and
. . . .Software
. . . . . . . . .Included
. . . . . . . . in
. . the
. . . .Book
..............................
. . . . . Introduction
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
. . . . . The
. . . . Story
. . . . . .of
. . ACME
. . . . . . Inc.
.................................................
13
. . . . . The
. . . . Why,
. . . . . .Who,
. . . . .What,
. . . . . .When
. . . . . .and
. . . .How
....................................
15
. . . . . Management
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
. . . . . Section
. . . . . . . .Content
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
. . . . . APIC
. . . . . Overview
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
. . . . . Configuring
. . . . . . . . . . . .Management
. . . . . . . . . . . . Protocols
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
. . . . . Role-Based
. . . . . . . . . . . Access
. . . . . . .Control
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
. . . . . Import
. . . . . . . and
. . . . Export
. . . . . . .Policies
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
. . . . . Upgrading
. . . . . . . . . . .and
. . . Downgrading
. . . . . . . . . . . . . Firmware
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
. . . . . Section
. . . . . . . .Content
...........................................................
51
. . . . . Firmware
. . . . . . . . . .Management
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
. . . . . Upgrading
. . . . . . . . . . .and
. . . Downgrading
. . . . . . . . . . . . . Considerations
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
. . . . . Upgrading
. . . . . . . . . . .the
. . . Fabric
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
. . . . . Fabric
. . . . . . .Connectivity
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
. . . . . Section
. . . . . . . .Content
...........................................................
71
. . . . . Understanding
. . . . . . . . . . . . . . .Fabric
. . . . . .Policies
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
. . . . . Adding
. . . . . . . New
. . . . .Devices
. . . . . . . .to
. .the
. . . .Fabric
.........................................
81
. . . . . Server
. . . . . . .Connectivity
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
. . . . . Virtual
. . . . . . . Machine
. . . . . . . . Networking
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
. . . . . Deploying
. . . . . . . . . . the
. . . .Application
. . . . . . . . . . .Virtual
. . . . . . Switch
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
. . . . . External
. . . . . . . . .Connectivity
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
. . . . . Application
. . . . . . . . . . . Migration
. . . . . . . . . .Use
. . . .Case
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
. . . . . Tenants
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .159
. . . . . Section
. . . . . . . .Content
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
. . . . . ACI
. . . . Tenancy
. . . . . . . . .Models
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
. . . . . Application
. . . . . . . . . . . Profile
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
. . . . . Endpoint
. . . . . . . . . Group
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
. . . . . Endpoint
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
. . . . . Private
. . . . . . . Network
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
. . . . . Bridge
. . . . . . .Domain
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
. . . . . Tenant
. . . . . . . Networking
. . . . . . . . . . . .Use
. . . .Cases
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
. . . . . Working
. . . . . . . . .with
. . . . Contracts
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .215
. . . . . Section
. . . . . . . .Content
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
. . . . . Contracts
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
. . . . . Filters
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
. . . . . Taboo
. . . . . . .Contracts
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
. . . . . Inter-Tenant
. . . . . . . . . . . . .Contracts
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
. . . . . Contracts
. . . . . . . . . .Use
. . . .Cases
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
. . . . . Layer
. . . . . .4. .to
. .Layer
. . . . . .7. Services
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .249
. . . . . Section
. . . . . . . .Content
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
. . . . . . . . . . . . . . .Layer
. . . . . 4. .to
. . Layer
. . . . . .7. Integration
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
. . . . . Services
. . . . . . . . .Deployment
. . . . . . . . . . . Guide
. . . . . . Reference
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
. . . . . Service
. . . . . . . .Graph
. . . . . .Monitoring
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
. . . . . ASAv
. . . . . Sample
. . . . . . . .Configuration
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
. . . . . Health
. . . . . . .Scores
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .269
. . . . . Section
. . . . . . . .Content
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
. . . . . . . . . . . . . . .Health
. . . . . . Scores
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
. . . . . . . . . . . . . . .Faults
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
. . . . . How
. . . . . Health
. . . . . . .Scores
. . . . . . Are
. . . . Calculated
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
. . . . . Health
. . . . . . .Score
. . . . . .Use
. . . .Cases
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
. . . . . Monitoring
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .285
. . . . . Section
. . . . . . . .Content
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
. . . . . Proactive
. . . . . . . . . .Monitoring
. . . . . . . . . . -. .Tenant
. . . . . . .and
. . . .Fabric
. . . . . .Policies
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
. . . . . Proactive
. . . . . . . . . .Monitoring
. . . . . . . . . . -. .Infrastructure
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
. . . . . Proactive
. . . . . . . . . .Monitoring
. . . . . . . . . . Use
. . . . Cases
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
. . . . . Reactive
. . . . . . . . .Monitoring
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
. . . . . Reactive
. . . . . . . . .Monitoring
. . . . . . . . . . Tools
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335
. . . . . Reactive
. . . . . . . . .Monitoring
. . . . . . . . . . Use
. . . . Cases
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
. . . . . Scripting
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .349
. . . . . Section
. . . . . . . .Content
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
. . . . . Leveraging
. . . . . . . . . . .Network
. . . . . . . . Programmability
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
. . . . . ACI
. . . . and
. . . . Scripting
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
. . . . . API
. . . .Inspector
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
. . . . . Development
. . . . . . . . . . . . . Techniques
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
. . . . . STman
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
. . . . . Cobra
. . . . . . SDK
. . . . .and
. . . .Arya
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
. . . . . ACI
. . . . Toolkit
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
. . . . . GitHub
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
. . . . . Hardware
. . . . . . . . . .Expansion
. . . . . . . . . .and
. . . .Replacement
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .389
. . . . . Section
. . . . . . . .Content
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
. . . . . Expanding
. . . . . . . . . . .and
. . . Shrinking
. . . . . . . . . .the
. . . Fabric
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393
. . . . . Hardware
. . . . . . . . . .Diagnostics
. . . . . . . . . . . and
. . . .Replacement
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399
. . . . . Appendix
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .413
. . . . . Classes
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
. . . . . Package
. . . . . . . . Decoder
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435
. . . . . Acronyms
. . . . . . . . . . and
. . . .Definitions
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445
. . . . . Reference
. . . . . . . . . . Material
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453
....
Prologue
Prologue
Abstract
Cisco's Ap
pli
ca
tion Cen
tric In
fra
struc
ture (ACI) pro
vides pow
er
ful new ways to dy
nam
i
cally man
age in
fra
struc
ture in the mod
ern world of IT au
toma
tion and De
vOps. Hav
ing
the tools to change how in
fra
struc
ture is built is one thing, but being able to ef
fec
tively
op
er
ate the in
fra
struc
ture be
yond the Day Zero build ac
tiv
i
ties is cru
cial to long term ef
fec
tive
ness and ef
fi
ciency. To ef
fec
tively har
ness the power of ACI, or
ga
ni
za
tions will
need to un
der
stand how to in
cor
po
rate ACI into their daily op
er
a
tions. This book ex
am
ines some of the com
mon op
er
a
tional ac
tiv
i
ties that IT teams use to pro
vide con
tin
ued
in
fra
struc
ture op
er
a
tions and gives the reader ex
po
sure to the tools, method
olo
gies, and
processes that can be em
ployed to sup
port day 1+ op
er
a
tions within an ACI-based fab
ric.
Prologue
Authors
This book rep
re
sents a joint in
tense col
lab
or
a
tive ef
fort over the course of a week, with
par
tic
ip
a
tion of a num
ber of Cisco func
tional or
ga
ni
za
tions in
clud
ing Cisco Ad
vanced
Ser
vices, Tech
ni
cal Ser
vices, Prod
uct Mar
ket
ing, So
lu
tion Val
id
a
tion Test
ing, IT, and
Sales.
Authors
Ale
jan
dra Sanchez - Cus
tomer Sup
port En
gi
neer, Tech
ni
cal Ser
vices
An
dres Vega - Cus
tomer Sup
port En
gi
neer, Tech
ni
cal Ser
vices
Arvind Chari - So
lu
tions Ar
chi
tect, Ad
vanced Ser
vices
Carly Stoughton - Tech
ni
cal Mar
ket
ing En
gi
neer, INSBU
Christo
pher Stokes - Cisco IT Net
work Con
sult
ing En
gi
neer
Gabriel Fontenot - Net
work Con
sult
ing En
gi
neer, Ad
vanced Ser
vices
Jonathan Cor
nell - Sys
tems En
gi
neer, Sales
Ken Fee - Con
sult
ing Ar
chi
tect, Busi
ness Tech
nol
ogy Ar
chi
tects
Kevin Corbin - Tech
ni
cal So
lu
tions Ar
chi
tect, Sys
tems En
gi
neer
ing
Lau
ren Mal
hoit - Mar
ket
ing Con
sul
tant, INSBU
Loy Evans - Con
sult
ing Sys
tem En
gi
neer, Sales
Lu
cien Avramov - Tech
ni
cal Mar
ket
ing En
gi
neer, INSBU
Paul Lesiak - So
lu
tions Ar
chi
tect, Ad
vanced Ser
vices
Rafael Muller - Tech
ni
cal Leader, So
lu
tion Val
id
a
tion Ser
vices
Robert Burns - Tech
ni
cal Leader, Tech
ni
cal Ser
vices
Steven Lym - Tech
ni
cal Writer, INSBU
Prologue
Book Sprint Facilitation

Bar
bara Rhling
Faith Bosworth
Illustrations
Hen
rik van Leuwen
Book Production
Julien Taquet
Clean Up
Raewyn Whyte
Prologue
Book Writing Methodology

The Book Sprint (booksprints.
net) method
ol
ogy was used for writ
ing this book. The
Book Sprint method
ol
ogy is an in
no
v
at
ive new style of co
op
er
at
ive and col
lab
o
ra
tive
book pro
duc
tion. Book Sprints are strongly fa
cil
it
ated and lever
age team-ori
ented in
spi
ra
tion and mo
ti
va
tion to rapidly de
liver large amounts of well-au
thored and re
viewed con
tent, and in
cor
po
rate it into a com
plete nar
ra
tive in a short amount of time.
By lever
ag
ing the input of many ex
perts, the com
plete book was writ
ten in only five
days, but in
volved hun
dreds of au
thor
ing per
son-hours, and in
cluded thou
sands of ex
pe
ri
enced en
gi
neer
ing hours, al
low
ing for ex
tremely high qual
ity in a very short pro
duc
tion time pe
riod.
Prologue
Hardware and Software Included in the Book

The Cisco Ap
pli
ca
tion Cen
tric In
fra
struc
ture (ACI) com
bines hard
ware, soft
ware, and
ASIC in
no
va
tions into an in
te
grated sys
tems ap
proach and pro
vides a com
mon man
age
ment frame
work for net
work, ap
pli
ca
tion, se
cu
rity, and vir
tu
al
iza
tion.
For the pur
pose of writ
ing this book, the fol
low
ing hard
ware de
vices were used:
Cisco Application Policy Infrastructure Controller (APIC)
ACI Spine switches, including Cisco Nexus 9508 and 9336PQ
ACI Leaf switches, including Cisco Nexus 9396PX, 9396TX, and 93128TX
Cisco Application Virtual Switch (AVS)
Cisco UCS B and C series servers
Several models of switches and routers, including Cisco Nexus 5000 switches
and Cisco Integrated Services Routers (ISR)
A variety of hypervisors, including KVM, Microsoft Hyper-V, and VMware
IXIA IxVM product family traffic generators
vSphere
This book was writ

ten based on the Cisco ACI 1.1 soft
ware re
lease.
11
Introduction
Introduction
13
The Story of ACME Inc.

ACME Inc. is a multi
na
tional cor
po
ra
tion that spe
cial
izes in man
uf
ac
tur
ing, sales, and
dis
tri
bu
t
ion of a di
verse prod
uct port
fo
lio, in
clud
ing rocket-pow
ered roller skates, jetpro
pelled uni
cy
cles, and var
io
us ex
plo
sive ma
te
ri
als. These prod
uct groups op
er
ate as
sep
ar
ate busi
ness groups within the com
pany, and have pre
vi
ously main
tained sep
ar
ate
in
fra
struc
ture and ap
pli
ca
tions. They have largely fo
cused on re
tail routes to mar
ket,
but have re
cently de
cided to pur
sue a more di
rect-to-con
sumer busi
ness model due
to in
tense pres
sure from new com
peti
tors who have dom
in
ated the on
line sales chan
nels. In an ef
fort to be more com
pet
it
ive, ACME has un
der
taken a pro
ject to build a mo
bile ap
pli
ca
tion plat
form to sup
port or
der
ing and lo
gis
tics for prod
uct de
liv
ery to their
cus
tomers for their en
tire port
fo
lio.
Tra
di
tion
ally, ACME busi
ness units have lever
aged third party soft
ware com
pa
nies and
com
mer
cially avail
able soft
ware to meet their IT de
mands, but would like to cre
ate a
more in
ti
mate re
la
tion
ship with their con
sumers and be able to take feed
back on the
plat
form di
rectly from those users, while in
cor
po
rat
ing an on
go
ing im
prove
ment cycle
so they can react to chang
ing mar
ket dy
nam
ics in a more nim
ble fash
ion. Where they
have used cus
tom soft
ware in the past, they have lever
aged a tra
di
tional in
fra
struc
ture
and soft
ware model that does not allow them to keep up with the chang
ing re
quire
ments, and there
fore ACME is look
ing for a new ap
proach to both ap
pli
ca
tion and in
fra
struc
ture life
cy
cle man
age
ment. The ap
pli
ca
tion de
vel
op
ers have been look
ing at
new ap
pli
ca
tion de
vel
op
ment trends such as Con
tin
uo
us De
liv
ery and Con
tin
uo
us In
te
gra
tion, and the new ap
pli
ca
tion plat
form is to be de
vel
oped in this man
ner. To sup
port
this, the in
fra
struc
ture com
po
nents need to be ca
pa
ble of map
ping to these new par
a-
digms in a way that is not pos
si
ble using tra
di
tional con
cepts.
One of the largest chal
lenges ACME has his
tor
ic
ally faced is that op
er
at
ions and in
fra
struc
ture has been an af
ter
thought to prod
uct de
vel
op
ment. This has led to sev
eral sit
u
at
ions where ap
pli
ca
tion de
ploy
ments have meant long week
end hours for all of the
teams, caused cus
tomer-im
pact
ing out
ages, and taken longer to ac
com
plish than the
busi
ness lead
ers would have liked. For this rea
son, ACME Inc. has de
cided to change by
cre
at
ing an en
vi
ron
ment where in
fra
struc
ture ar
ti
facts are treated as part of the ap
pli
ca
tion, can be checked into ver
sion con
trol, can be tested along
side the ac
tual ap
pli
ca
tion, and can con
tin
ua
lly im
prove.
14
Introduction
While ACME is in
tensely fo
cused on de
liv
er
ing the new ap
pli
ca
tion plat
form in a timely
man
ner, ACME is also in
ter
ested in cre
at
ing a foun
da
tion on which it can grow to de
liver a com
mon pool of in
fra
struc
ture that is shared across all busi
ness groups and op
er
ated in a multi-ten
ant fash
ion to in
crease ef
fi
ciency.
At an ex
ec
ut
ive brief
ing, John Cham
bers, the CEO of Cisco Sys
tems, told ACME:
"The world is chang
ing. Every com
pany is a tech
nol
ogy com
pany, and if you don't
adapt, you'll get left be
hind."
As ev
id
enced by the suc
cess of cloud plat
forms, such as Ama
zon Web Ser
vices
(AWS) and Open
stack, con
sump
tion mod
els of tech
nol
ogy de
liv
ery have the abil
ity to
adapt tech
nol
ogy more quickly to rapid busi
ness re
quire
ments changes. This is the type
of con
sump
tion that ACME Inc.'s busi
ness own
ers need. Con
trol of op
er
at
ions is what
op
er
at
ions groups are fo
cused on, but con
trol can be a bar
rier to a pure con
sump
tion
model. Un
less com
pa
nies make in
vest
ments in tech
nolo
gies that allow for con
sump
tion
of au
to
mated com
po
nents, the only other way to scale is by break
ing the human level
com
po
nent, and few peo
ple would re
ally choose to work for that type of com
pany.
After an
al
yz
ing cur
rent of
fers from var
io
us tech
nol
ogy ven
dors, ACME Inc. se
lected
Cisco Ap
pli
ca
tion Cen
tric In
fra
struc
ture (ACI). The abil
ity to ab
stract all phys
ic
al and
vir
tual in
fra
struc
ture con
fig
ur
a
tion into a sin
gle con
fig
ur
a
tion that is con
sis
tent across
dev, test, and prod en
vi
ron
ments, as well as portable across the var
io
us data cen
ter lo
ca
tions cur
rently main
tained by ACME, is highly de
sir
able. ACI has been built from the
ground up to change the sub
struc
ture used to build net
work de
vices and pro
to
cols. In
no
va
tion at this level will pro
vide more op
por
tu
ni
ties for ex
pand
ing the tools with
which users in
ter
act. This is where the ful
crum will tilt in the favor of IT and in
fra
struc
ture being more dy
namic, thus al
low
ing IT to op
er
ate and man
age at the speed of busi
ness. How
ever, with a change of this na
ture comes fear, un
cer
tainty, and doubt. This
book will at
tempt to bring some level of com
fort and fa
mil
iar
ity with op
er
at
ions ac
tiv
i-
ties within an ACI fab
ric.
While ACME Inc. is a fic
ti
tious com
pany, this is the true story of every com
pany, and
just im
por
tant this is the story of the em
ploy
ees of those com
pa
nies. Work
ers in the IT
in
dus
try need to adapt to keep up with the rapid change of the busi
ness. How
ever,
this runs con
trary to how most op
er
at
ions groups exist in the re
la
tion
ship be
tween
busi
ness and tech
nol
ogy. Most IT op
er
at
ions groups in
vest a lot of time in the tools
needed to de
liver ser
vices today and there is an or
ganic re
sis
tance to re-in
vest
ing.
The thought is, "Why fix what is al
ready work
ing?"
Introduction
15
The Why, Who, What, When and How

Within ACI, ACME is look
ing to sim
plify how it op
er
ates in
fra
struc
ture, but rec
og
nizes
that this ini
tia
tive, this ap
pli
ca
tion, and this in
fra
struc
ture are new to ACME Inc. ACME
must ad
dress fun
da
men
tal ques
tions in
clud
ing Who man
ages What, and How they go
about their tasks. When dif
fer
ent groups per
form reg
ul
ar op
er
at
ions, and Where they
go to per
form these op
er
at
ions, are also con
sid
er
at
ions, but more tac
ti
cal and pointin-time-rel
e
vant. This sec
tion dis
cusses the rel
e
vant as
pects of these monikers as it re
lates to ACI fab
ric op
er
at
ions and how a com
pany such as ACME Inc. can di
vide the
work
load.
Why
"Why" is the most im
por
tant as
pect of what should be con
sid
ered in op
er
at
ional
iz
ing an
ACI fab
ric. In the case of ACME Inc. the key suc
cess cri
te
ria is to stream
line processes
and pro
ce
dures re
lated to the de
ploy
ment of in
fra
struc
ture re
quired to sup
port the ap
pli
ca
tion ini
tia
tives. To achieve the de
sired re
sult, a high de
gree of au
toma
tion is re
quired. Au
toma
tion adds speed to repet
it
ive tasks and elim
in
ates er
rors or missed
steps. Ini
tially, au
toma
tion can be a scary propo
si
tion for some of the key stake
hold
ers,
as it could be looked at as a threat to their own job. Quite the op
po
site, au
toma
tion is
about mak
ing work more en
joy
able for all team mem
bers, al
low
ing them the free
dom to
in
no
vate and add value, while re
mov
ing mun
dane, repet
it
ive tasks. Look
ing at why an
au
to
mated fab
ric is ben
e
fi
cial to an or
ga
ni
za
tion is im
por
tant for set
ting ex
pec
ta
tions
of re
turn on in
vest
ment. Also, look
ing at why an op
er
at
ional prac
tice is done a spe
cific
way can help with fram
ing the tools and processes that are em
ployed.
Who
As with most or
ga
ni
za
tions, ACME Inc. tra
di
tion
ally had dif
fer
ent types of stake
hold
ers
in
volved in mak
ing any IT ini
tia
tive suc
cess
ful, and each has a spe
cific el
e
ment of the
in
fra
struc
ture in which they have spe
cific ex
per
tise and about which they care most. In
any IT or
ga
ni
za
tion, these groups can have dis
tinct or
ga
ni
za
tional bound
aries, but
more likely the bound
aries are blurred to some de
gree. Listed below are some char
ac
ter
is
tics of these groups, but keep in mind that some of these char
ac
ter
is
tics might be
com
bined. At the macro level, the fact that these dif
fer
ent or
ga
ni
za
tions exist should
16
Introduction
not be ev
id
ent to the end-user. In
stead, the en
tire or
ga
ni
za
tion should be seen as one
team with a com
mon goal of de
liv
er
ing value to their or
ga
ni
za
tion.
ACME's De
vel
op
ment and Ap
pli
ca
tion Team is fo
cused on the soft
ware and ap
pli
ca
tions
the com
pany uses in
ter
nally and the soft
ware that it de
liv
ers to its cus
tomers. The Ap
pli
ca
tion part of the team con
tains ap
pli
ca
tion own
ers and sub
ject mat
ter ex
perts that
en
sure other busi
ness units are able to do their jobs by uti
liz
ing the busi
ness ap
pli
ca
tions avail
able. The De
vel
op
ment part of the team will be writ
ing the mo
bile ap
pli
ca
tion soft
ware plat
form. Both parts of this team will need to work closely with the other
teams in this sec
tion to en
able the best de
sign, per
for
mance, and avail
abil
ity of ap
pli
ca
tions for the end users.
ACME's Net
work Team is pri
mar
ily fo
cused on cre
at
ing and man
ag
ing net
work
ing con
structs to for
ward pack
ets at Layer 2 (MAC/switch
ing) and Layer 3 (IP rout
ing). The
team is chal
lenged with jug
gling ap
pli
ca
tion re
quire
ments, man
ag
ing SLA, and as
sist
ing
in the en
force
ment of in
for
ma
tion se
cu
rity, all while main
tain
ing high lev
els of avail
abil
ity. What the team needs to know is how to con
fig
ure the net
work
ing con
structs,
how to tie Layer 2 to Layer 3, how to ver
ify for
ward
ing, and how to trou
bleshoot net
work for
ward
ing as
pects in the fab
ric. With ACI, the team is the most in
ter
ested in de
cou
pling over
loaded net
work con
structs and re
turn
ing to the spe
cific net
work prob
lems that the team was in
tended to solve, while al
low
ing other groups to lever
age their
spe
cific ex
per
tise to ma
nip
ul
ate se
cu
rity and ap
pli
ca
tion level poli
cies. The team is also
in
ter
ested in al
low
ing more trans
parency in the per
for
mance of the net
work for
ward
ing, and the team is mak
ing key met
rics avail
able on de
mand in a self-ser
vice ca
pac
ity.
ACME's Stor
age Team is pri
mar
ily fo
cused on de
liv
ery of data stor
age re
sources to the
or
ga
ni
za
tion. The stor
age team is con
cerned with pro
tect
ing the data in terms of avail
abil
ity, as well as mak
ing sure that sen
si
tive data is se
cure. The stor
age team has been
very suc
cess
ful in main
tain
ing very tight SLAs and has tra
di
tion
ally man
aged sep
ar
ate
in
fra
struc
ture for stor
age ac
cess. The ca
pa
bil
it
ies pro
vided by the ACI fab
ric allow
them to con
fi
dently de
ploy newer IP-based stor
age and clus
ter
ing tech
nolo
gies. The
team is also very in
ter
ested in being able to see how the stor
age ac
cess is per
form
ing and would like to be no
ti
fied in the event of con
tention. The team typ
ic
ally
has some spe
cific re
quire
ments around QoS, multi-pathing, and so on. His
tor
ic
ally, the
team had to worry about de
liv
er
ing a stor
age fab
ric in ad
di
tion to man
ag
ing stor
age de
vices them
selves. ACI will pro
vide the stor
age team with the vis
ib
il
ity they will re
quire.
These ca
pa
bil
it
ies are pri
mar
ily dis
cussed in the mon
it
or
ing sec
tions.
Introduction
17
The Com
pute and Vir
tu
al
iza
tion Team at ACME Inc. is wrap
ping up a major ini
tia
tive to
vir
tu
al
ize the server farms that it is re
spon
si
ble for main
tain
ing. The team also re
cently
em
ployed new con
fig
ur
a
tion man
age
ment tools to ac
count for new work
loads that fell
out
side of the vir
tu
al
iza
tion ef
fort to get sim
il
ar agility for bare metal servers that the
team gained from its vir
tu
al
iza
tion ef
forts. This is timely as the ap
pli
ca
tion roll
out will
have both vir
tu
al
ized and non-vir
tu
al
ized work
loads. Ad
di
tion
ally, the ap
pli
ca
tion de
vel
op
ers are in
creas
ingly in
ter
ested in lever
ag
ing Linux con
tainer tech
nolo
gies to allow
for even greater ap
pli
ca
tion porta
bil
ity. The Com
pute and Vir
tu
al
iza
tion teams are in
ter
ested in ACI for its abil
ity to pro
vide com
mon ac
cess to phys
ic
al and vir
tual servers,
al
low
ing the team to pub
lish end
point groups to vir
tu
al
iza
tion clus
ters from a cen
tral
ized place across mul
ti
ple hy
per
vi
sors. These ca
pa
bil
it
ies are dis
cussed fur
ther in the
Fab
ric Con
nec
tiv
ity chap
ter.
The In
for
ma
tion Se
cu
rity Team at ACME Inc. has tra
di
tion
ally been en
gaged late in an
ap
pli
ca
tion de
ploy
ment process, and has been re
spon
si
ble for per
form
ing vul
ner
ab
il
ity
as
sess
ment and data clas
si
fi
ca
tion ef
forts. With the cur
rent pro
ject, the new ap
pli
ca
tion will be stor
ing sen
si
tive cus
tomer in
for
ma
tion, in
clud
ing credit card num
bers. Due
to the sen
si
tiv
ity of this in
for
ma
tion and the se
cu
rity as
pects of the ACI fab
ric, the In
for
ma
tion Se
cu
rity Team is able to pro
vide input ear
lier in the process and avoid re-do
ing work be
cause of se
cu
rity or com
pli
ance is
sues. The In
for
ma
tion Se
cu
rity Team is
in
ter
ested in the op
er
at
ional as
pects of the ACI se
cu
rity model as it re
lates to the fol
low
ing ca
pa
bil
it
ies: ten
ancy, Role Based Ac
cess Con
trol (RBAC), mon
it
or
ing, and Layer 4
to Layer 7 ser
vices.
What
The as
pect of "what" can be looked at in many dif
fer
ent ways, but the main con
cept in
the con
text of this book is what tools are used to man
age op
er
at
ions of an ACI fab
ric. In
a tra
di
tional net
work, you have some tra
di
tional tools, such as CLI and SNMP, to man
age net
work op
er
at
ions, and these tools in
te
grate into man
age
ment plat
forms and con
fig
ur
a
tion and man
age
ment processes.
In ACI there are some el
e
ments of the tra
di
tional tools, but the fab
ric man
age
ment is
rooted in an ab
stracted ob
ject model that pro
vides a more flex
ib
le base. With this base,
the op
er
at
or of the fab
ric can choose from mul
ti
ple modes of man
age
ment, such as
GUI, CLI, API in
te
gra
tion, pro
gram
ming, script
ing, or some com
bi
na
tion of these. How
a tool is se
lected in ACI will often be a prod
uct of what is being done and the as
pects of
how the tool is used. For ex
am
ple, if an op
er
at
ions staff is try
ing to gather a bunch of
in
for
ma
tion across a num
ber of in
ter
faces and switches or is man
ag
ing the con
fig
ur
a
-
Introduction
18
tion of many dif

fer
ent ob
jects at once, script
ing might be more ef
fi
cient, whereas sim
ple dash
board mon
it
or
ing might be more suited to a GUI.
When
"When" refers to when the teams listed above are in
volved in the plan
ning. It is a good
idea to in
volve the dif
fer
ent teams early when build
ing poli
cies and processes for how
the fab
ric is im
ple
mented and then man
aged. The col
lab
o
ra
tive na
ture of ACI al
lows for
a high de
gree of par
al
leliza
tion of work flow. This is a key dif
fer
ence be
tween ACI and
tra
di
tional processes that were very se
ri
al in na
ture, re
sult
ing in a longer de
ploy
ment
time for ap
pli
ca
tions and a higher mean-time to res
ol
u
tion when is
sues arise.
How
"How" an
swers the fol
low
ing basic ques
tions:
How does a networking person go about configuring the network forwarding?
How does the compute team get information from the infrastructure to make
How do the application team track performance and usage metrics?
How does a storage team track the access to storage subsystems and ensure
optimal workload placement decisions?
that it is performant?
When "how" in
volves mak
ing a change to the con
fig
ur
a
tion of an en
vi
ron
ment, an im
por
tant con
sid
er
at
ion is change con
trol. Change con
trol is a fact of life in the mis
sioncrit
ic
al en
vi
ron
ments that ACI has been de
signed to sup
port. The ACI pol
icy model has
been de
signed to re
duce the over
all size of a fault do
main and pro
vide a mech
an
ism for
in
cre
men
tal change. There are mech
an
isms for backup and re
store that will be dis
cussed in fol
low-on chap
ters. We will also dis
cuss the model and which ob
jects af
fect
the ten
ants and the fab
ric as a whole.
An eval
ua
t
ion of cur
rent change con
trol and con
tin
uo
us in
te
gra
tion/de
liv
ery strate
gies
is war
ranted as op
er
at
ional pro
ce
dures evolve. Through
out this book we will high
light
the meth
ods and pro
ce
dures to proac
tively and re
ac
tively man
age the fab
ric.
As a base
line, most or
ga
ni
za
tions are im
ple
ment
ing some kind of struc
tured changecon
trol method
ol
ogy to mit
ig
ate busi
ness risk and en
hance sys
tem avail
abil
ity. There
are a num
ber of change/IT man
age
ment prin
ci
ples (Cisco Life
cy
cle ser
vices, FCAPS,
Introduction
19
and ITIL) that are good guides from which to start. A com
mon sense ap
proach to
change man
age
ment and con
tin
uo
us in
te
gra
tion should be a premise that is dis
cussed
early in the de
sign and im
ple
men
ta
tion cycle be
fore hand
ing the fab
ric to the op
er
a-
tions teams for day-to-day main
te
nance, mon
it
or
ing, and pro
vi
sion
ing. Train
ing op
er
a
tions teams on norms (a stated goal of this book) is also key. Ap
ply
ing change man
age
ment prin
ci
ples based on tech
nol
ogy from five years ago would not en
able the rapid
de
ploy
ment of tech
nol
ogy that ACI can de
liver.
The multi-ten
ant and role-based ac
cess con
trol fea
tures in
her
ent to the ACI so
lu
tion
allow the iso
la
tion or draw
ing of a very clean box around the scope and im
pact of the
changes that can be made. For more de
tails, see the RBAC sec
tion of this book.
Ul
ti
mately each change must be eval
ua
ted pri
mar
ily in terms of both its risk and value
to the busi
ness. A way to en
able a low-over
head change man
age
ment process is to re
duce the risk of each change and in
crease its value. Con
tin
uo
us de
liv
ery does ex
actly
this by en
sur
ing that re
leases are per
formed reg
ul
arly from early on in the de
liv
ery
process, and en
sur
ing that de
liv
ery teams are work
ing on the most valu
able thing they
could be at any given time, based on feed
back from users.
In the In
for
ma
tion Man
age
ment Sys
tems world, there are three fun
da
men
tal kinds of
changes:
Emergency changes
Normal
Standard
Emer
gency changes are by de
f
i
n
i
tion a re
sponse to some kind of tech
ni
cal out
age (hard
ware, soft
ware, in
fra
struc
ture) and are per
formed to re
store ser
vice to af
fected sys
tems.
Nor
mal changes are those that go through the reg
ul
ar change man
age
ment process,
which starts with the cre
ation of a re
quest for change which is then re
viewed, as
sessed,
and then ei
ther au
tho
rized or re
jected, and then (as
sum
ing it is au
tho
rized) planned
and im
ple
mented. In an ACI en
vi
ron
ment a nor
mal change could apply to any
thing
within the fol
low
ing com
po
nents:
Fabric Policies (fabric internal and access will be discussed in detail later)
Configuration objects in the Common tenant that are shared with all other
tenants (things that affect the entire fabric)
Private Networks
Introduction
20
Bridge Domains
Subnets
Virtual Machine Manager (VMM) integrations
Layer 4 to Layer 7 devices

Device packages
Creation of logical devices
Creation of concrete devices
Layer 2 or Layer 3 external configuration
Attachable Entity Profile (AEP) creation
Server or external network attachment
Changes to currently deployed contracts and filters that would materially

change the way traffic flows
Stan
dard changes are low-risk changes that are pre-au
tho
rized. Each or
ga
ni
za
tion will
de
cide the kind of stan
dard changes that they allow, who is al
lowed to ap
prove
them, the cri
te
ria for a change to be con
sid
ered "stan
dard", and the process for man
ag
ing them. As with nor
mal changes, they must still be recorded and ap
proved. In the ACI
en
vi
ron
ment some ex
am
ples of "stan
dard" changes could be:
Tenant creation
Application profile creation
Endpoint group (EPG) creation
Contracts scoped at a tenant level
Layer 4 to Layer 7 service graphs
Domain associations for EPGs
The items men

tioned above are not in
tended to be all-in
clu
sive, but are rep
re
sen
ta
tive
of com
mon tasks per
formed day-to-day and week-to-week.
The abil
ity to audit changes that are hap
pen
ing to the en
vi
ron
ment is a re
quire
ment for
ACME Inc. APIC main
tains an audit log for all con
fig
ur
a
tion changes to the sys
tem. This
is a key trou
bleshoot
ing tool for "when some
thing mag
ic
ally stops work
ing". Im
me
di
ate
ac
tion should be to check the audit log as it will tell who made what change and when,
cor
re
lat
ing this to any faults that re
sult from said change. This en
ables the change to be
re
verted quickly.
Introduction
21
A more in-depth dis

cus
sion of con
tin
u
ous de
liv
ery in the con
text of in
fra
struc
ture
man
age
ment is out
side of the scope of this book.
The re
main
der of this book an
swers these ques
tions, pro
vid
ing you with a frame
work
of how to take the con
cepts and pro
ce
dures and apply them to sim
i
lar ini
tia
tives within
your or
ga
ni
za
tions. The book is laid out in a spe
cific order. How
ever, ACI en
ables
ACME Inc. to com
plete these tasks in par
al
lel with the var
i
ous stake
hold
ers who are
high
lighted through
out, and this book il
lus
trates how the stake
hold
ers can work to
gether in a more col
lab
o
ra
tive man
ner than they have in the past. While some script
ing
op
por
tu
ni
ties are called out through
out the book, there is a more in-depth sec
tion at
the end that ex
plains how to use the ACI API to au
to
mate most op
er
a
tional tasks. While
or
ga
ni
za
tional struc
tures might be siloed into these teams, to pro
vide the great
est
value to the cus
tomer, user, and ul
ti
mately the busi
ness, the most im
por
tant thing is
for these groups to work in
sieme (to
gether).
ACI addresses IT requirements from across the organization
23
Management
Management
Section Content
APIC Overview
Configuring Management Protocols

Cisco Discovery Protocol
Link Layer Discovery Protocol
Time Synchronization and NTP
Out-of-Band Management NTP
In-Band Management NTP
Verify NTP Operation
Verifying that the NTP Policy Deployed to Each Node
Domain Name Services (DNS)
Verifying DNS Operation
Role-Based Access Control

Multiple Tenant Support
User Roles
Security Domains
Creation of a Security Domain
Adding Users
Remote Authentication
Import and Export Policies

Configuration Export (Backup)
Add a Remote Location (SCP)
Create a One Time Export Policy
Verify Export Policy was Successful
Extract and View Configuration Files
Configuration Import (Restore/Merge)
25
Management
27
APIC Overview
There are a num
ber of fun
da
men
tal dif
fer
ences be
tween the op
er
at
ions of tra
di
tional
net
work
ing hard
ware and an ACI fab
ric. These dif
fer
ences serve to sim
plify the man
age
ment greatly, re
duce the num
ber of touch
points, and de
cou
ple the switch
ing hard
ware from the de
sired con
fig
ur
a
tion in
tent. These changes in
clude:
Single point of management controller-based architecture
Stateless hardware
Desired state-driven eventual consistency model
The sin
gle point of man
age
ment within the ACI ar
chi
tec
ture is known as the Ap
pli
ca
tion Pol
icy In
fra
struc
ture Con
troller (APIC). This con
troller pro
vides ac
cess to all con
fig
ur
a
tion, man
age
ment, mon
it
or
ing, and health func
tions. Hav
ing a cen
tral
ized con
troller with an ap
pli
ca
tion pro
gram
ming in
ter
face (API) means that all func
tions con
fig
ured or ac
cessed through the fab
ric can be uni
formly ap
proached through a graph
ic
al
user in
ter
face (GUI), com
mand line in
ter
face (CLI), and ap
pli
ca
tion pro
gram
ming in
ter
face (API), with no risk of in
con
sis
tency be
tween the var
io
us data in
ter
faces. This re
sults in a clean and pre
dictable tran
si
tion be
tween the in
ter
faces. The un
der
ly
ing in
ter
face for all ac
cess meth
ods is pro
vided through a REST-based API, which mod
if
ies the
con
tents of a syn
chro
nized data
base that is repli
cated across APICs in a clus
ter and
pro
vides an ab
strac
tion layer be
tween all of the in
ter
faces.
This con
troller-based ar
chi
tec
ture also makes pos
si
ble a state
less con
fig
ur
a
tion
model that de
cou
ples the hard
ware from the con
fig
ur
a
tion run
ning on it. This trans
lates to an APIC clus
ter that man
ages in
di
vid
ual fab
ric nodes of leaf and spine switches
that de
rive their iden
tity from what the con
troller de
fines as being the de
sired in
tent,
and not from the se
ri
al num
ber of the chas
sis, nor from a con
fig
ur
a
tion file re
sid
ing on
the de
vices. Each node re
ceives a unique node iden
ti
fier, which al
lows for the de
vice to
down
load the cor
rect con
fig
ur
a
tion at
trib
utes from the con
troller. The de
vice can
also be sub
sti
tuted in a state
less fash
ion, mean
ing that hard
ware swaps can be faster,
topol
ogy changes are less im
pact
ful, and net
work man
age
ment is sim
pli
fied.
The de
sired state model for con
fig
ur
a
tion fur
ther com
ple
ments these con
cepts of con
troller-based man
age
ment and state
less
ness by tak
ing ad
van
tage of a con
cept known
as de
clar
at
ive con
trol-based man
age
ment, based on a con
cept known as the promise
28
Management
the
ory. De
clar
at
ive con
trol dic
tates that each ob
ject is asked to achieve a de
sired state
and makes a "promise" to reach this state with
out being told pre
cisely how to do so.
This stands in con
trast with the tra
di
tional model of im
per
at
ive con
trol, where each
man
aged el
e
ment must be told pre
cisely what to do, be told how to do it, and take into
ac
count the spe
cific sit
ua
t
ional as
pects that will im
pact its abil
ity to get from its cur
rent state to the con
fig
ured state. A sys
tem based on de
clar
at
ive con
trol is able to scale
much more ef
fi
ciently than an im
per
at
ive-based sys
tem, since each en
tity within the
do
main is re
spon
si
ble for know
ing its cur
rent state and the steps re
quired to get to the
de
sired state, dic
tated by the man
ag
ing con
troller.
Management
29
Configuring Management Protocols

For the op
ti
miza
tion of ACME's Cisco Ap
pli
ca
tion Cen
tric In
fra
struc
ture (ACI) fab
ric,
the net
work team cre
ated the fol
low
ing man
age
ment pro
to
col poli
cies:
Cisco Discovery Protocol (CDP) - These policies are primarily consumed by the
network team, as CDP is the standard for their existing network equipment.
Link Layer Discovery Protocol (LLDP) - Discovery protocols are no longer

limited to just network devices. LLDP is a standards-based protocol for
discovering topology relationships. ACME is deploying LLDP on servers, Layer 4
to Layer 7 services devices, and storage arrays. Therefore, these policies will be
available for consumption by all of the teams.
Network Time Protocol (NTP) - Maintaining accurate time across the

application logs and infrastructure components is extremely important for
being able to correlate events. ACME has existing NTP servers and will
leverage them for maintaining time in their ACI fabric deployment. With ACI,
maintaining accurate time is of increased importance as accurate time is a
prerequisite for features such as atomic counters.
Domain Name Services (DNS) - providing name resolution to fabric components

consistent with the application and server teams. In addition to be able to
resolve names within the fabric, the important ACI components are also
registered with ACME's DNS server so teams can easily access them in their
management tasks.
Cisco Discovery Protocol

Cisco Dis
cov
ery Pro
to
col (CDP) is a valu
able source of in
for
ma
tion in trou
bleshoot
ing
phys
ic
al and data-link layer is
sues. In the course of ACI op
er
at
ions, there may be times
when you must pro
vi
sion a par
tic
ul
ar host-fac
ing port man
ua
lly with a CDP on or off
pol
icy. By de
fault, the ACI fab
ric pro
vides a de
fault pol
icy where CDP is dis
abled. As
a rec
om
mended prac
tice, man
ua
lly cre
ate a CDP-EN
ABLED" pol
icy with CDP en
abled,
as well as a CDP-DIS
ABLED pol
icy with CDP dis
abled that can be ref
er
enced through
out all in
ter
face pol
icy con
fig
ur
a
tions. It is also im
por
tant to note that CDP might be
re
quired for cer
tain con
fig
ur
a
tions, such as when to cre
ate switch pro
files to con
nect
to Cisco UCS B Se
ries servers.
Management
30
To cre
ate CDP poli
cies:
1
On the menu bar, choose Fabric > Access Policies.
In the Navigation pane, choose Interface Policies > Policies > CDP Interface.
In the Work pane, choose Actions > Create CDP Interface Policy.
In the Create CDP Interface Policy dialog box, perform the following actions:
a. In the Name field enter the name of the policy, such as "CDP-ENABLED".
b. For the Admin State radio buttons, click Enabled.
c. Click Submit.
5
6
a. In the Name field enter the name of the policy, such as "CDP-DISABLED".
b. For the Admin State radio buttons, click Disabled.
c. Click Submit.
Your CDP pol

icy is now ready for de
ploy
ment to the ACI fab
ric in
ter
faces.
Link Layer Discovery Protocol

Link Layer Dis
cov
ery Pro
to
col (LLDP) is a valu
able source of in
for
ma
tion for trou
bleshoot
ing phys
ic
al and data-link layer is
sues. In the course of ACI op
er
at
ions, by de
fault, the LLDP fea
ture is en
abled on the fab
ric. There might be times in the op
er
at
ion
of the ACI fab
ric in which you must man
ua
lly ad
just the LLDP pro
to
col to con
form with
in
ter
op
er
abil
ity re
quire
ments of end-host de
vices. For ex
am
ple, when con
nect
ing to
Cisco Uni
fied Com
put
ing Sys
tem (UCS) Blade servers, you must dis
able LLDP.
Cisco rec
om
mends that you pre-pro
vi
sion LLDP en
able and dis
able pro
to
col poli
cies to
make fu
ture in
ter
face pol
icy de
ploy
ment de
ci
sions stream
lined and eas
ily con
fig
ured.
To cre
ate an LLDP pol
icy:
1
In the Navigation pane, choose Interface Policies > Policies > LLDP Interface.
In the Work pane, choose Actions > Create LLDP Interface Policy.
In the Create LLDP Interface Policy dialog box, perform the following actions:
Management
31
a. In the Name field, enter "LLDP-TX-ON-RX-ON".

b. For the Receive State radio buttons, click Enabled.
c. For the Transmit State radio buttons, click Enabled.
d. Click Submit.
5
6
a. In the Name field, enter "LLDP-TX-ON-RX-OFF".
b. For the Receive State radio buttons, click Disabled.
c. For the Transmit State radio buttons, click Enabled.
d. Click Submit.
7
8
a. In the Name field, enter "LLDP-TX-OFF-RX-ON".
b. For the Receive State radio buttons, click Enabled.
c. For the Transmit State radio buttons, click Disabled.
d. Click Submit.
a. In the Name field, enter "LLDP-TX-OFF-RX-OFF".
b. For the Receive State radio buttons, click Disabled.
c. For the Transmit State radio buttons, click Disabled.
d. Click Submit.
Your LLDP pol

ploy
ment to the ACI fab
ric in
ter
faces.
Time Synchronization and NTP

Within the Cisco Ap
pli
ca
tion Cen
tric In
fra
struc
ture (ACI) fab
ric, time syn
chro
niza
tion is
a cru
cial ca
pa
bil
ity upon which many of the mon
it
or
ing, op
er
at
ional, and trou
bleshoot
ing tasks de
pend. Clock syn
chro
niza
tion is im
por
tant for proper analy
sis of traf
fic flows
as well as for cor
re
lat
ing debug and fault time
stamps across mul
ti
ple fab
ric nodes.
An off
set pre
sent on one or more de
vices can ham
per the abil
ity to prop
erly di
ag
nose
and re
solve many com
mon op
er
at
ional is
sues. In ad
di
tion, clock syn
chro
niza
tion al
lows
for the full uti
liza
tion of the atomic counter ca
pa
bil
ity that is built into the ACI upon
which the ap
pli
ca
tion health scores de
pend. Nonex
is
tent or im
proper con
fig
ur
a
tion of
time syn
chro
niza
tion does not nec
es
sar
ily trig
ger a fault or a low health score. You
Management
32
should con
fig
ure time syn
chro
niza
tion be
fore de
ploy
ing a full fab
ric or ap
pli
ca
tions so
as to en
able proper usage of these fea
tures. The most widely adapted method for syn
chro
niz
ing a de
vice clock is to use Net
work Time Pro
to
col (NTP). For more in
for
ma
tion
on atomic coun
ters, see the Re
ac
tive Mon
it
or
ing Tools chap
ter.
Prior to con
fig
ur
ing NTP, con
sider what man
age
ment IP ad
dress scheme is in place
within the ACI fab
ric. There are two op
tions for con
fig
ur
ing man
age
ment of all ACI
nodes and Ap
pli
ca
tion Pol
icy In
fra
struc
ture Con
trollers (APICs), in-band man
age
ment
and/or out-of-band man
age
ment. De
pend
ing on which man
age
ment op
tion was cho
sen for the fab
ric, con
fig
ur
a
tion of NTP will vary.
An
other con
sid
er
at
ion in de
ploy
ing time syn
chro
niza
tion where the time source is lo
cated. The re
li
ab
il
ity of the source must be care
fully con
sid
ered when de
ter
min
ing if
you will use a pri
vate in
ter
nal clock or an ex
ter
nal pub
lic clock.
Out-of-Band Management NTP

When an ACI fab
ric is de
ployed with out-of-band man
age
ment, each node of the fab
ric,
in
clu
sive of spines, leaves and all mem
bers of the APIC clus
ter, is man
aged from out
side the ACI fab
ric. This IP reach
ab
il
ity will be lever
aged so that each node can in
di
vid
u
ally query the same NTP server as a con
sis
tent clock source.
To con
fig
ure NTP, a Date and Time pol
icy must be cre
ated that ref
er
ences an out-ofband man
age
ment end
point group. Date and Time poli
cies are con
fined to a sin
gle
pod and must be de
ployed across all pods pro
vi
sioned in the ACI fab
ric. As of the pub
lish
ing of this doc
um
ent, only one pod per ACI fab
ric is al
lowed.
1
On the menu bar, choose Fabric > Fabric Policies.
In the Navigation pane, choose Pod Policies > Policies.
In the Work pane, choose Actions > Create Date and Time Policy.
In the Create Date and Time Policy dialog box, perform the following actions:
a. Provide a name for the policy to easily distinguish between your
environment's different NTP configurations, such as "Production-NTP".
b. Click Next.
c. Click the + sign to specify the NTP server information (provider) to be used.
d. In the Create Providers dialog box, enter all relevant information, including
the following fields: Name, Description, and Minimum Polling Intervals, and
Maximum Polling Intervals.
Management
33
i. If you are creating multiple providers, click the Preferred check box for
the most reliable NTP source.
ii. In the Management EPG drop-down list, if the NTP server is reachable by
all nodes on the fabric through out-of-band management, choose Out-ofBand. If you have deployed in-band management, see the "In-Band
Management NTP" section.
iii. Click OK.
e. Repeat steps c and d for each provider that you want to create.
Your NTP pol
ploy
ment to the ACI fab
ric nodes.
In-Band Management NTP

When an ACI Fab
ric is de
ployed with in-band man
age
ment, con
sider the reach
ab
il
ity of
the NTP server from within the ACI in-band man
age
ment net
work. In-band IP ad
dress
ing used within the ACI fab
ric is, by de
f
in
i
t
ion, not reach
able from any
where out
side
the fab
ric. To lever
age an NTP server ex
ter
nal to the fab
ric with in-band man
age
ment,
you must con
struct a pol
icy to en
able this com
mu
ni
ca
tion. The steps used to con
fig
ure
in-band man
age
ment poli
cies are iden
ti
cal to those used to es
tab
lish an out-of-band
man
age
ment pol
icy: the dis
tinc
tion is around how to allow the fab
ric to con
nect to the
NTP server.
Once you have es
tab
lished a pol
icy to allow com
mu
ni
ca
tion, you can cre
ate the NTP
pol
icy as es
tab
lished in the "Out-of-Band Man
age
ment NTP" sec
tion.
Verifying NTP Operation

Full ver
if
i
ca
tion of NTP func
tion
al
ity is best ac
com
plished by lever
ag
ing both the ACI
CLI and the APIC GUI. Start ver
if
y
ing time syn
chro
niza
tion con
fig
ur
a
tion by en
sur
ing
that all poli
cies are con
fig
ured prop
erly in
side of the APIC GUI. Pol
icy con
fig
ur
a
tion can
be ver
if
ied by these steps:
1
In the Navigation pane, choose Pod Policies > Policies > Date and Time >
ntp_policy > server_name. The ntp_policy is the previously created policy.
In the Work pane, verify the details of the server.
Management
34
Verifying that the NTP Policy Deployed to Each Node
To ver
ify that the pol
icy has been suc
cess
fully de
ployed to each node, you should use
the NX-OS Vir
tual Shell that is pre
sent in the APIC. To ac
cess the NX-OS Vir
tual Shell,
open an SSH ses
sion to the out-of-band man
age
ment IP in
ter
face of any of the APIC
nodes. From this prompt, ex
e
cute the "ver
sion" com
mand to ob
tain a con
sol
id
ated list
of node host
names.
1
SSH to an APIC in the fabric.
Press the Tab key twice after the entering the attach command to list all of the
available node names:
admin@apic1:~> attach <Tab> <Tab>
Log in to one of the nodes using the same password that you used to access the
APIC.
admin@apic1:~> attach node_name
View the NTP peer status:
leaf-1# show ntp peer-status
A reach
able NTP server has its IP ad
dress pre
fixed by an as
ter
isk (*), and the
delay is a non-zero value.
5
Repeat steps 3 and 4 for each node on the fabric.
Domain Name Services (DNS)

Set
ting up a DNS server al
lows the APIC to re
solve var
io
us host
names to IP ad
dresses.
This is use
ful when in
te
grat
ing VMM do
mains or other Layer 4 to Layer 7 de
vices and
the host
name is ref
er
enced.
1
In the Navigation pane, choose Global Policies > DNS Profiles > default.
In the Work pane, in the Management EPG drop-down list, choose the
appropriate management EPG.
Management
35
Note: The default is default (Out-of-Band).

4
Click + next to DNS Providers to add a DNS provider.

a. In the Address field, enter the provider address.
b. In the Preferred field, click the check box if you want to have this address as
the preferred provider.
Note: You can have only one preferred provider.
c. Click Update.
Repeat step 4 for each additional DNS provider.
Click + next to DNS Domains to add a DNS domain.

a. In the Name field, enter the domain name, such as "cisco.com".
b. In the Default field , click the check box to make this domain the default
domain.
Note: You can have only one domain name as the default.
c. Click Update.
Repeat step 6 for each additional DNS Domain suffix.
Click Submit.
Verifying DNS Operation
Check the resolv.conf file from the APIC CLI:
admin@apic1:~> cat /etc/resolv.conf

# Generated by IFC
search cisco.com
nameserver 171.70.168.183
nameserver 173.36.131.10
Management
36
Ping a host by DNS name that will be reachable from the APIC out-of-band
management.
admin@apic1:~> ping cisco.com

PING cisco.com (72.163.4.161) 56(84) bytes of data.
64 bytes from www1.cisco.com (72.163.4.161): icmp_seq=1 ttl=241 time=42.7 ms
^C
--- cisco.com ping statistics --3 packets transmitted, 3 received, 0% packet loss, time 2102ms
rtt min/avg/max/mdev = 42.485/43.038/43.903/0.619 ms
admin@apic1:~>
Management
37
Role-Based Access Control

Role-based ac
cess con
trol (RBAC) and ten
ancy are im
por
tant con
cepts to mas
ter when
op
er
at
ing a Cisco Ap
pli
ca
tion Cen
tric In
fra
struc
ture (ACI) fab
ric. Role-based ac
cess and
the con
cept of ten
ancy are two core foun
da
tions upon which the ACI man
age
ment and
pol
icy mod
els are built.
In an ACI fab
ric, the Cisco Ap
pli
ca
tion Pol
icy In
fra
struc
ture Con
troller (APIC) grants ac
cess to all ob
jects based on a user's es
tab
lished role in the con
fig
ur
a
tion. Each user can
be as
signed to a set of roles, a priv
il
ege type, and a se
cu
rity do
main. Think of this in
terms of Who, What, and Where. The role is the Who (log
ic
al col
lec
tion of priv
il
eges),
priv
il
eges de
fine What func
tions a user can per
form, and the se
cu
rity do
main de
fines Where the user can per
form these ac
tions. When com
bin
ing these ob
jects, you
can im
ple
ment very gran
ul
ar ac
cess con
trol in the sys
tem.
ricThe role clas
si
fi
ca
tion is an es
tab
lished group
ing of per
mis
sions. For ex
am
ple, a fab
ad
min can have ac
cess to the en
tire fab
ric as well as to as
sign
ing other user roles and
as
so
ci
ate them to se
cu
rity do
mains, whereas a ten
ant-ad
min can only have ac
cess to
the com
po
nents within the ten
ant to which they're as
so
ci
ated, and is un
able to man
age other user roles at a fab
ric wide level. ACME's IT or
ga
ni
za
tion fits per
fectly into this
ac
cess con
trol model.
The APIC pro
vides ac
cess ac
cord
ing to a users role through RBAC. An ACI fab
ric user is
as
so
ci
ated with the fol
low
ing:
A set of roles
For each role, privileges and privilege types (which can be "no access", "read-
One or more security domain tags that identify the portions of the management
only", or "read-write")
information tree (MIT) that a user can access
38
Management
Authenticated user creation lifecycle

The ACI fab
ric man
ages ac
cess priv
i
leges at the man
aged ob
ject (MO) level. A priv
i
lege
is an MO that en
ables or re
stricts ac
cess to a par
tic
u
lar func
tion within the sys
tem. For
ex
am
ple, fab
ric-equip
ment is a priv
i
lege flag. This flag is set by the APIC on all ob
jects
that cor
re
spond to equip
ment in the phys
i
cal fab
ric.
APIC poli
cies man
age the ac
cess, au
then
ti
ca
tion, and ac
count
ing (AAA) func
tions of the
Cisco ACI fab
ric. The com
bi
na
tion of user priv
i
leges, roles, and se
cu
rity do
mains with
ac
cess rights in
her
i
tance en
ables ad
min
is
tra
tors to con
fig
ure AAA func
tions at the
man
aged ob
ject level in a very gran
u
lar fash
ion. These con
fig
u
ra
tions can be im
ple
mented using the REST API, CLI, or GUI.
Multiple Tenant Support

A core APIC in
ter
nal data ac
cess con
trol sys
tem pro
vides multi-ten
ant iso
la
tion and
pre
vents in
for
ma
tion pri
vacy from being com
pro
mised across ten
ants. Read/write re
stric
tions pre
vent any ten
ant from see
ing any other ten
ants con
fig
u
ra
tion, sta
tis
tics,
Management
39
faults, or event data. Un

less the ad
min
is
tra
tor as
signs per
mis
sions to do so, ten
ants are
re
stricted from read
ing fab
ric con
fig
ur
a
tion, poli
cies, sta
tis
tics, faults, or events.
If a vir
tual ma
chine man
age
ment (VMM) do
main is tagged as a se
cu
rity do
main, the
users con
tained in the se
cu
rity do
main can ac
cess the cor
re
spond
ing VMM do
main. For
ex
am
ple, if a ten
ant named solar is tagged with the se
cu
rity do
main called sun and a
VMM do
main is also tagged with the se
cu
rity do
main called sun, then users in the solar
ten
ant can ac
cess the VMM do
main ac
cord
ing to their ac
cess rights.
User Roles
You can view the built-in user roles as well as cre
ate cus
tom roles to meet spe
cific re
quire
ments.
1
On the menu bar, choose Admin > AAA.
In the Navigation pane, choose Security Management > Roles.

Note: From here, you can see each built-in role and the associated privileges.
Additionally, you can create a custom role from the Actions menu.
Security Domains
The se
cu
rity do
main con
cept is cru
cial to proper op
er
at
ion of the ACI fab
ric. By using
se
cu
rity do
mains, users can be or
ga
nized into var
io
us per
mis
sion struc
tures, most
com
monly ap
plied to ten
ants. Using the ten
ancy ca
pa
bil
it
ies of the ACI fab
ric in con
junc
tion with prop
erly con
fig
ured se
cu
rity do
mains, it will be pos
si
ble to com
pletely
sep
ar
ate con
fig
ur
a
tion of ap
pli
ca
tion work
loads while only pro
vid
ing ac
cess to those
who need it.
A key con
cept to keep in mind when con
fig
ur
ing se
cu
rity do
mains is that the con
fig
u-
ra
tion only ap
plies at a ten
ant level. The poli
cies can
not cur
rently be ap
plied to in
di
vid
ual ob
jects de
spite ref
er
ences at the ob
ject level in
side of the RBAC screen in the APIC.
Cisco rec
om
mends that you con
fig
ure se
cu
rity do
mains and ac
cess lev
els prior to de
ploy
ment of ten
ants. Cisco does not rec
om
mend that you pro
vide user ac
cess con
fig
u-
ra
tion by mod
if
ing per
mis
sions of the "all" do
main. Changes in the "all" do
main will af
fect ac
cess per
mis
sions for all users. If you need to make se
lec
tive changes to allow ac
cess out
side of a user's se
cu
rity do
main, be sure to set up a dis
crete user ac
cess pol
icy
just for that com
mu
ni
ca
tion.
Management
40
Creation of a Security Domain

There are three se
cu
rity do
mains that are pro
vi
sioned by de
fault on the ACI fab
ric: "all",
"com
mon", and "mgmt". The "all" se
cu
rity do
main usu
ally in
cludes ac
cess to every
thing
within the man
age
ment in
for
ma
tion tree (MIT). "Com
mon" is usu
ally used when there is
a need for shared re
sources be
tween ten
ants, such as DNS or di
rec
tory ser
vices. The
"mgmt" se
cu
rity do
main is for man
age
ment traf
fic poli
cies. You can add se
cu
rity do
mains as nec
es
sary. For ex
am
ple, for a multi-ten
ant en
vi
ron
ment, a se
cu
rity do
main
would be cre
ated for each ten
ant. Users are then cre
ated and as
signed to cer
tain se
cu
rity do
mains, or ten
ants. RBAC poli
cies are con
fig
ured under the "Admin" tab of the
APIC GUI.
For more in
for
ma
tion about the MIT, see the doc
um
ent at the fol
low
ing URL:
http://
www.
cisco.
com/
c/
en/
us/
products/
collateral/
cloud-systems-management/
aci-fabric-controller/
white-paper-c11-729586.
html
To cre
ate a se
cu
rity do
main:
1
In the Navigation pane, choose Security Management > Security Management.
In the Work pane, choose Actions > Create a Security Domain.
In the Create a Security Domain dialog box, perform the following actions:
a. Give the security domain a name and an optional description.
Adding Users
You might need to add new users within the ACI fab
ric. ACI can have local users man
u-
ally en
tered by an admin, or use some
thing such as LDAP, RA
DIUS, Ac
tive Di
rec
tory, or
TACACS+ to spec
ify users that will be al
lowed to man
age cer
tain parts of the ACI net
work.
Con
fig
ure Local Users:
1
In the Navigation pane, choose AAA Authentication.
In the Work pane, in the Realm drop-down list, choose Local and click
Submit if Local is not already chosen.
Management
In the Navigation pane, choose Security Management.
In the Work pane, choose Actions > Create Local User.
41
In the Create Local User dialog box, perform the following actions:
a. Specify any security information that is necessary and click Next.
b. Select the roles to be given to this user, such as Read/Write for admin or
tenant admin, and click Next.
c. Specify login information for the user and
Click Finish.
Remote Authentication
Cre
at
ing re
mote user ac
counts in the ACI is sim
il
ar to most other data cen
ter sys
tems.
If cre
at
ing an LDAP ac
count, then the LDAP provider must be con
fig
ured first. The IP
ad
dress or host name of the LDAP server is needed, along with the port it uses to com
mu
ni
cate as well as any other rel
e
vant in
for
ma
tion that will allow con
nec
tion to that
server. The same is true for RA
DIUS and TACACS+. The next op
tion is to con
fig
ure
groups from which it is al
lowed to read data and grab it for the pur
poses of se
lect
ing
re
mote users granted ac
cess to the ACI net
work. Re
mote au
then
ti
ca
tion is cov
ered in
da
men
tals Guide.
de
tail in the Cisco ACI Fun
Management
43
Import and Export Policies

All of the stake
hold
ers at ACME in
volved in the de
ploy
ment and ad
min
is
tra
tion of the
new mo
bile ap
pli
ca
tion plat
form need to know that they will be able to eas
ily re
cover
from any loss of con
fig
ur
a
tion quickly and eas
ily. Since all APIC poli
cies and con
fig
ur
a
tion data can be ex
ported to cre
ate back
ups and tech sup
port files for Dis
as
ter Re
cov
ery, trou
bleshoot
ing, and of
fline analy
sis, ACI is a good choice on this front as well. As
with all things APIC, a pol
icy needs to be con
fig
ured to ex
port or backup the con
fig
ur
a
tion poli
cies. This can be done from any ac
tive and fully fit APIC within the ACI fab
ric.
The backup and re
store process does not re
quire backup of in
di
vid
ual com
po
nents.
Back
ups are con
fig
urable through an ex
port pol
icy that al
lows ei
ther sched
uled or im
me
di
ate back
ups to a re
mote server (pre
ferred) or, in the case where an ex
ter
nal
SCP/FTP server is not avail
able, back
ups to be writ
ten to the local APIC file sys
tem.
Backup/ex
port poli
cies can be con
fig
ured to be run on-de
mand or based on a re
cur
ring sched
ule. Cisco rec
om
mends that a cur
rent Backup be per
formed be
fore mak
ing
any major sys
tem con
fig
ur
a
tion changes or ap
ply
ing soft
ware up
dates. Con
fig
ur
a
tion
Im
port poli
cies (pol
icy re
store) will be dis
cussed later in this sec
tion.
By de
fault, all poli
cies and ten
ants are backed up, but the ad
min
is
tra
tor can op
tion
ally
spec
ify only a spe
cific sub
tree of the man
age
ment in
for
ma
tion tree. This is use
ful in
multi-ten
ant de
ploy
ments where in
di
vid
ual ten
ants wish to backup or re
store their re
spec
tive poli
cies.
An
other Ex
port pol
icy re
quired on oc
ca
sion is for gath
er
ing tech
ni
cal sup
port in
for
ma
tion. If is
sues are en
coun
tered within the sys
tem, you might need to con
tact the Cisco
Tech
ni
cal As
sis
tance Cen
ter (TAC). Pro
vid
ing a tech
ni
cal sup
port bun
dle will give Cisco
sup
port en
gi
neers the in
for
ma
tion needed to help iden
tify and sug
gest re
me
di
at
ion to
is
sues. Tech
ni
cal sup
port ex
port poli
cies can be con
fig
ured to run on-de
mand or
sched
uled for re
cur
ring pur
poses. Tech
ni
cal sup
port bun
dles are con
fig
ured very sim
i-
lar to con
fig
ur
a
tion ex
port poli
cies, and so the bun
dles will not be cov
ered in de
tail
bleshoot
here. For de
tails on tech
ni
cal sup
port ex
port poli
cies, see the Cisco APIC Trou
ing Guide that is ref
er
enced in the Ap
pen
dix of this book.
Management
44

Since ACI is pol
icy dri
ven, more than a backup job is re
quired. First, a re
mote lo
ca
tion
is cre
ated (the ex
ter
nal server to which you want to ex
port in
for
ma
tion), then an Ex
port pol
icy task, and op
tion
ally a Sched
uler (for re
cur
ring tasks). The pro
ce
dure below
de
tails how to cre
ate a re
mote lo
ca
tion and ex
port pol
icy in ACI to trans
fer the con
fig
u
ra
tion file bun
dle to an SCP server. The in
di
vid
ual XML files can be ex
tracted and
viewed after the con
fig
u
ra
tion bun
dle is trans
ferred to the desk
top. An ex
trac
tion util
ity is needed to de
com
press the tar.
gz file that is cre
ated.
Configuration Export Policy
Add a Remote Location (SCP)

1
On the menu bar, choose Admin > Import/Export.
In the Navigation pane, choose Remote Locations.
In the Work pane, choose Actions > Create Remote Location
In the Create Remote Location dialog box, perform the following actions:
a. Enter a Remote location name
b. Enter a Hostname/IP address
c. Choose a Protocol
d. Enter a Remote path
e. Enter a Remote port
f. Enter a Username
g. Enter a Password
h. Choose a Management EPG
Note: default (Out-of-Band)
Click Submit.
Management
45
Create a One Time Export Policy

The pro
ce
dure below de
tails a con
fig
ur
a
tion ex
port pol
icy, but the pro
ce
dure for a
tech
ni
cal sup
port ex
port pol
icy is very sim
il
ar.
1
In the Navigation pane, choose Export Policies > Configuration.
In the Work pane, choose Actions > Create Configuration Export Policy
In the Create Configuration Export Policy dialog box, perform the following
actions:
a. Name = Export_Policy_Name
b. Format = XML
c. Start Now = Yes
d. Export Destination = Choose_the_Remote_location_created_above
Click Submit.
Two op
tional con
fig
ur
a
tions are ap
ply
ing a Sched
uler pol
icy if you want to setup a re
cur
ring op
er
at
ion, and spec
if
y
ing a spe
cific Dis
tin
guished Name (DN) if you want to
backup only a sub
set of the Man
age
ment In
for
ma
tion Tree (MIT).
Verify Export Policy was Successful

1
In the Navigation pane, choose Export Policies > Configuration

> Export_Name.
In the Work pane, choose the Operational tab.

a. The State should change from "pending" to "success" when the export
completes correctly.
b. (Optional) Confirm on the SCP server that the backup filename exists.
Management
46
Extract and View Configuration Files

A con
fig
ur
a
tion ex
port task ex
ports a com
pressed bun
dle of in
di
vid
ual XML files. These
XML con
fig
ur
a
tion files can be ex
tracted and viewed on an
other work
sta
tion.
1
From the workstation where the exported bundle has been copied, select the
archive utility of choice.
Navigate to the folder where the config export is located (tar.gz), highlight the
file, and then select Extract.
Double-click one of the XML files to view the contents in a browser.
Examine the various XML configuration files for parameters that have been
configured.
Configuration Import (Restore/Merge)

To re
store a con
fig
ur
a
tion from a pre
vi
ous backup:
If the re
mote lo
ca
tion does not exist, cre
ate a re
mote lo
ca
tion per the "Adding a Re
mote Lo
ca
tion (SCP)" sec
tion.
1
In the Navigation pane, choose Import Policies > Configuration.
In the Work pane, choose Actions > Create Import Configuration Policy.
In the Create Import Configuration Policy dialog box, perform the following
actions:
a. Enter a name and the filename for the import policy, such as "backupfile.tar.gz". Do not include the file path.
b. Choose the import type:
Merge - Will merge backup configuration with existing running
configurations. Will not overrite any existing policies.
Replace - Will replace all configurations with those only from the backup file.
Management
47
c. The import mode must be specified when you attempt to perform

a Merge import. The configuration data is imported per shard with each
shard holding a certain part of the system configuration objects. The default
is best-effort. The Replace Import Mode can be either:
best-effort - Each shard is imported, skipping any invalid objects
atomic - Attempts to import each shard, skipping those which contain an
invalid configuration. A shard's entire configuration must be valid to be

imported in this mode.
d. Choose Start Now.
e. Choose the Remote Location that was previously created in the "Adding a
Remote Location (SCP)" section.
49
Upgrading and
Downgrading Firmware
Upgrading and Downgrading Firmware
Section Content
Firmware Management
Firmware Versions
Firmware Components
Firmware Policies
Firmware Groups
Maintenance Groups
Controller Firmware
Catalogue Firmware
Upgrading and Downgrading Considerations
Upgrading the Fabric

Downloading the Firmware Images
Upgrading the APIC Controller Software
Upgrading the Switch Software Using the GUI
Upgrading the APIC Controller Software Using the CLI
Upgrading the Switch Software Using the CLI
Verifying Cluster Convergence
Troubleshooting Failures During the Upgrade Process
51
53
Firmware Management
ACME Inc., in part
ner
ship with Cisco, has eval
ua
ted the re
quire
ments for their de
ploy
ment based on the soft
ware fea
tures re
quired, the sup
port for the hard
ware plat
forms
they have se
lected, and the ma
tu
rity of the soft
ware re
leases. They have se
lected a tar
get ver
sion of soft
ware for their de
ploy
ment. Ad
di
tion
ally, they have put a proac
tive
plan in place to re
visit this de
ci
sion pe
ri
od
ic
ally to de
ter
mine if fu
ture up
grades are re
quired.
Firmware Versions
The soft
ware ver
sions for ACI are listed in the fol
low
ing for
mat:
major.minor(mntnc)
major - Represents major changes in the product architecture, platform, or

features content.
minor - Represents a minor release with new software features.
mntnc - Represents bug fixes to a feature release of APIC. This changes when
there are fixes for product defects in the software, but no additional new
features.
Ex
am
ple:
APIC version: 1.1(1d)
Both the soft

ware for the APIC and the fab
ric nodes are de
noted by the same ver
sion
ing scheme, For ex
am
ple, APIC re
lease 1.0(2j) cor
re
sponds to switch soft
ware
11.0(2j). Re
lease notes for the APIC ver
sions ref
er
ence the cor
re
spond
ing switch ver
sions and vice-versa.
All com
po
nents of the ACI in
fra
struc
ture in
clud
ing the Cisco Ap
pli
ca
tion Pol
icy In
fra
struc
ture Con
troller (APIC), leaf switches, and spine switches, should be on the same
54
ver
sion. While at the time of up
grad
ing, dis
parate ver
sions may exist be
tween APIC and
the switches, do not op
er
ate the fab
ric for ex
tended pe
ri
ods of time in this state.
When con
sid
er
ing the im
pact and risk of up
grad
ing, you can as
sume that a main
te
nance ver
sion up
grade, such as 1.1.(1d) => 1.1(1d), will have less im
pact than a
major/minor ver
sion up
grade, as there will be only bug fixes and no new fea
tures
added.
Firmware Components
There are three main com
po
nents that can be up
graded:
Switches (leaf and spine)
Application Policy Infrastructure Controller (APIC)
Catalog firmware
Firmware Policies
Firmware Groups
Firmware Group poli
cies on the APIC de
fine which group of nodes on which firmware
will be up
graded. For most de
ploy
ments, a sin
gle firmware group is ad
e
quate. Con
trol
over which switches up
grade to the new ver
sion can be de
ter
mined through main
te
nance groups.
Maintenance Groups
Main
te
nance Group poli
cies de
fine a group of switches that will be jointly up
graded to
the as
so
ci
ated firmware set. Main
te
nance groups can be up
graded on de
mand or ac
cord
ing to a sched
ule, mak
ing it pos
si
ble to defer an up
grade task to a busi
ness main
te
nance win
dow.
Controller Firmware
The APIC con
troller firmware pol
icy al
ways ap
plies to all con
trollers in the clus
ter, but
the up
grade is al
ways done se
quen
tially. The APIC GUI pro
vides real-time sta
tus in
for
-
55
ma
tion about firmware up
grades. Con
troller Firmware poli
cies can be up
graded on de
mand or also ac
cord
ing to a sched
ule.
Catalogue Firmware
Each firmware image in
cludes a com
pat
i
bil
ity cat
a
log that iden
ti
fies sup
ported switch
mod
els. The APIC main
tains a cat
a
log of the firmware im
ages, switch types, and mod
els
that are al
lowed to use that firmware image. The APIC, which per
forms image man
age
ment, has an image repos
i
tory for com
pat
i
bil
ity cat
a
logs, APIC con
troller firmware im
ages, and switch im
ages.
Firmware upgrade policy relationships
57
Upgrading and Downgrading Considerations

Be
fore start
ing the up
grade or down
grade process, con
sider the fol
low
ing things:
Before starting the upgrade process, your controllers should be in good health.
On the menu bar, choose System > Controllers. In the navigation pane,
choose Controllers > apic_controller_name > Cluster. Verify that the health
state of all of the controllers in the cluster are Fully Fit before you proceed. To
resolve issues for controllers that are not fully fit see the Troubleshooting Cisco
Application Centric Infrastructure document:https://datacenter.github.io/acitroubleshooting-book/
Configuration backup.
Before starting any upgrade, always export your
configuration to an external source.

For information about exporting
configurations, see the Import and Export Policies chapter.
Permissions.
A user must have the fabric administrator role to perform
firmware upgrade tasks.
Verify free space. On the menu bar, choose System > Controllers. In the
navigation pane, choose Controllers > apic_controller_name > Storage.
Confirm that the /firmware partition is not filled beyond 75%. If the partition
is filled beyond 75%, you might be required to remove some unused firmware
files from the repository to accommodate the compressed image as well as
provide adequate space to extract the image. The APIC automatically extracts
the image.
Upgrade order. Typically, the controllers should be upgraded first, followed by

the switch nodes. Always refer to the relevant release notes of the destination
firmware version for any changes to this order.
Maintenance windows. Although it is possible to upgrade the fabric without

impacting the dataplane, you should perform an upgrade during a scheduled
maintenance window according to your change control policy. This window
should account for any unforseen issues that might arise during the
upgrade, and allocate enough time to troubleshoot or perform a rollback.
Maintenance groups. To help minimize the impact to hosts during an upgrade,

you should set up at least two separate maintenance groups. A common
separation is by odd and even node IDs. Assuming that your hosts are dualconnected to at least one odd and one even leaf node, there should not be any
58
impact to your hosts. Maintenance group creation is covered in detail later in

the chapter. Another consideration is that your leaf VPC pairs should contain
one odd and one even node.
Upgrading a fabric with the Application Virtual Switch (AVS) deployed. The AVS
Device packages. Device packages are not always tied to the APIC software.
software is not specifically tied to the APIC or switch software version.

You can confirm the device compatability for Layer 4 to Layer 7 devices using
the online Application Centric Infrastructure (ACI) Compatability
tool: http://www.cisco.com/web/techdoc/aci/acimatrix/matrix.html
59
Upgrading the Fabric

Downloading the Firmware Images
You must down
load both the con
troller soft
ware pack
age and switch soft
ware pack
age for the Ap
pli
ca
tion Pol
icy In
fra
struc
ture Con
troller (APIC) from Cisco.
com.
To down
load the firmware im
ages using the APIC GUI:
1
On the menu bar, choose Admin > Firmware.
In the Navigation pane, choose Fabric Node Firmware.

Note: In the Work pane, the list of all switches in the fabric and the status of
when the firmware was last upgraded are displayed.
In the Navigation pane, choose Download Tasks.
In the Work pane, choose Actions > Create Outside Firmware Source.
In the Create Outside Firmware Source dialog box, perform the following
actions:
a. In the Source Name field, enter a name for the switch image, such as
"apic_1.1.3d".
b. For the Protocol radio buttons, click the Secure copy or HTTP radio button.
c. In the URL field, enter the URL from where the image must be downloaded.
HTTP Example: http://192.168.0.50/aci-apic-dk9.1.1.3d.iso
SCP Example: 192.168.0.50:/tmp/aci-firmware/aci-apicdk9.1.1.3d.iso

For SCP, enter your username and password.
d. Click Submit.
6
In the Navigation pane, choose Download Tasks.
In the Work pane, choose the Operational tab to view the download status of
the images.
Repeat the steps for the switch image.
Once the download reaches 100%, in the Navigation pane, choose Firmware
Repository.
60
10
In the Work pane, the downloaded version numbers and image sizes are
displayed.
To down
load the firmware im
ages using the APIC CLI:
1
SSH to an APIC and log in as "admin".
Pull the software using SCP:
admin@apic1:~> scp
username@IP_address_with_the_image:/absolute_path_to_image_plus_image_filename &#10!
Place the image into the image repository:
admin@apic1:~> firmware add ver_no.iso &#10!
The firmware image ver_no.iso is added to the repos

it
ory.
4
Verify the software has been added to the respository:
admin@ifc1:~> firmware list

Name : aci-apic-dk9.1.1.1d.bin
Type : controller
Version : 1.1(1d)
Upgrading the APIC Controller Software

The cat
al
og firmware image is up
graded when an APIC con
troller image is up
graded.
You do not need to up
grade the cat
al
og firmware image sep
ar
ately.
To up
grade the APIC con
troller soft
ware using the GUI:
1
In the Navigation pane, click Controller Firmware.
In the Work pane, choose Actions > Upgrade Controller Firmware Policy.
In the Upgrade Controller Firmware Policy dialog box, perform the following
actions:
61
a. In the Target Firmware Version field, from the drop-down list, choose the
image version to which you want to upgrade.
b. In the Apply Policy field, click the Apply now radio button. Alternately, you
can apply a schedule policy if you wish to defer the task to a specific
date/time.
c. Click Submit to complete the task.
The Sta
tus di
al
og box dis
plays the Changes Saved Suc
cess
fully mes
sage, and the
up
grade process be
gins. The APIC con
trollers are up
graded se
ri
ally so that the
con
troller clus
ter is avail
able dur
ing the up
grade.
5
Verify the status of the upgrade in the Work pane by clicking Controller
Firmware in the Navigation pane.
The con
trollers up
grade in ran
dom order. Each APIC con
troller takes about 10
min
utes to up
grade. Once a con
troller image is up
graded, it drops from the
clus
ter, and it re
boots with the newer ver
sion while the other APIC con
trollers
in the clus
ter are still op
er
at
ional. Once the con
troller re
boots, it joins the clus
ter again. Then the clus
ter con
verges, and the next con
troller image starts to
up
grade. If the clus
ter does not im
me
di
ately con
verge, and is not fully fit, the
up
grade will wait until the clus
ter con
verges and is fully fit. Dur
ing this pe
riod, a
Wait
ing for Clus
ter Con
ver
gence mes
sage is dis
played in the Sta
tus col
umn for
each APIC as it up
grades.
When the APIC con
troller that the browser is con
nected to is up
graded and it
re
boots, the browser dis
plays an error mes
sage.
In the browser URL field, enter the URL for the APIC controller that has already
been upgraded, and sign in to the APIC controller as prompted.
Upgrading the Switch Software Using the GUI

Be
fore you up
grade the switches, the APICs must have com
pleted up
grad
ing and have
a health state of Fully Fit.
To up
grade the switch soft
ware using the GUI:
1
62
In the Navigation pane, choose Fabric Node Firmware.

Note: In the Work pane, the switches that are operating in the fabric are
displayed.
If you have not created a firmware group, perform the following substeps:
a. In the Work pane, choose the Policy tab.
b. Choose Actions > Create Firmware Group.
c. In the Create Firmware Group dialog box, perform the following actions:
i. In the Group Name field, enter the name of the firmware group.
ii. In the Target Firmware Version drop-down list, choose the
firmware version to which you will upgrade.
iii. In the Group Node IDs field, enter a comma-separated list or a range of
node IDs to include in the group. For example, "101, 103-105, 108".
iv. Click Submit.
d. To verify that the firmware group was created, in the Navigation pane,
choose Fabric Node Firmware > Firmware Groups >
new_firmware_group. The Work pane displays details about the firmware
policy that was created earlier.
If you have not created maintenance groups, perform the following substeps:
a. In the Navigation pane, click Maintenance Groups.
Note: Cisco recommends that you create two maintenance groups for all of
the switches. For example, create one group with the even-numbered
devices and the other group with the odd-numbered devices.
b. In the Work pane, choose Action > Create Maintenance Group.
c. In the Create Maintenance Group dialog box, perform the following actions:
i. In the Group Name field, enter the name of the maintenance group. For
example, "Even-Nodes".
ii. For the Run Mode radio buttons, click the Pause Upon Upgrade
Failure radio button, which is the default mode.
iii. In the Group Node IDs field, enter a comma-separated list or a range of
node IDs to include in the group. For example, "102, 104, 106, 108, 110".
iv. In the Scheduler drop-down list, you can choose to create a schedule
for upgrading or leave the drop-down list blank so that you can upgrade on
demand.
v. Click Submit.
63
vi. To verify that the maintenance group was created, in the Navigation pane,
choose Fabric Node Firmware > Maintenance Groups
> new_maintenance_group, and click the name of the maintenance
group that you created.
Note: The Work pane displays details about the maintenance policy.
vii. Repeat this step for the second maintenance group. For example, a group
named "Odd-Nodes".
5
Right-click one of the maintenance groups that you created and

choose Upgrade Now.
In the Upgrade Now dialog box, for Do you want to upgrade the maintenance
group policy now?, click Yes.
Click OK.
Note: In the Work pane, the Status displays that all the switches in the group
are being upgraded simultaneously. The default concurrency in a group is set at
20. Therefore, up to 20 switches at a time will get upgraded, and then the next
set of 20 switches are upgraded. In case of any failures, the scheduler pauses
and manual intervention is required by the APIC administrator. The switch
upgrade takes up to 12 minutes for each group. The switches will reboot when
they upgrade, connectivity drops, and the controllers in the cluster will not
communicate for some time with the switches in the group. Once the switches
rejoin the cluster after rebooting, you will see all the switches listed under the
controller node. If there are any VPC configurations in the cluster, the upgrade
process will upgrade only one switch at a time out of the two switches in a VPC
domain.
In the Navigation pane, click Fabric Node Firmware.

Note: In the Work pane, view all of the switches that are listed. In the Current
Firmware column, view the upgrade image details listed against each switch.
Verify that the switches in the fabric are upgraded to the new image.
Upgrading the APIC Controller Software Using the CLI

The cat
al
og firmware image is up
graded when an APIC con
troller image is up
graded.
You do not need to up
grade the cat
al
og firmware image sep
ar
ately. Cisco rec
om
mends
that you to per
form the firmware up
grade from the GUI. When you use the GUI, the
APIC per
forms ad
di
tional ver
if
i
ca
tion and in
tegrity checks on the soft
ware image.
64
To up
grade the APIC con
troller soft
ware using the CLI:
1
List the current software in the respository that was previously downloaded.
Ex
am
ple:
admin@apic1:~> firmware list

Name : aci-apic-dk9.1.1.1d.bin
Type : controller
Version : 1.1(1d)
Upgrade the firmware on the controllers.

Ex
am
ple:
admin@apic1:~> firmware upgrade controllers ver_no.bin
The APIC con

trollers are up
graded se
ri
ally so that the con
troller clus
ter is avail
able dur
ing the up
grade. The up
grade oc
curs in the back
ground.
3
Check the status of the upgrade.

Ex
am
ple:
admin@apic1:~> firmware upgrade status
Node-Id
Role
Current-
Target-
Upgrade-
Progress-Percent
Firmware
Firmware
Status
(if inprogress)
---------
-----------
------------
------------------ ----------
controller
1.0(1.200)
apic-1.0(1.202a)
inqueue
-----------------0
controller
1.0(1.200)
apic-1.0(1.202a)
inqueue
controller
1.0(1.200)
apic-1.0(1.202a)
inprogress
The Up
grade-Sta
tus field will show "in
queue", "in
progress", or "com
ple
teok". If you
see "un
known" in this field, that con
troller has prob
a
bly up
graded and is re
boot
ing.
When the APIC con
troller to which you have con
nected com
pletes up
grad
ing
and re
boots, you can close the SSH win
dow.
65

Be
fore you up
grade the switches, the APICs must have com
pleted up
grad
ing and have a
health state of Fully Fit.
To up
grade the switch soft
ware using the CLI
1
Check that the output of the following command appears like the output shown
below, with the correct version number:
Ex
am
ple:
admin@apic1:~> firmware list

Name
: aci-n9000-dk9.11.1.1d.bin
Type
: switch
Version : 11.1(1d)
The name changes from ".iso" to ".bin".

2
Upgrade the switches.

Ex
am
ple:
admin@apic1:~> firmware upgrade switch node 101 ver_no.bin

Firmware Installation on Switch Scheduled
You must up
grade each switch sep
ar
ately.
3
Check the upgrade status for the switch. The output that appears from the
following command will appear like the following sample:
Ex
am
ple:
admin@apic1:~> firmware upgrade status node node_id

Node-Id
Role
Current-
Target-
Upgrade-
Progress-Percent
Firmware
Firmware
Status
(if inprogress)
---------
-----------
-------------------
------------------ ----------
1017
leaf
n9000-11.0(1.869S1)
n9000-11.1(1d)
completeok
-----------------100
66
You can check the sta

tus of all nodes at once, by en
ter
ing the firmware up
grade
sta
tus com
mand.
4
Repeat Steps 2 and 3 for each additional switch.
Verifying Cluster Convergence

You can mon
it
or the progress of the clus
ter con
ver
gence after a sched
uled main
te
nance. You view the Con
troller Firmware screen on the GUI, which pre
sents you with
a se
ries of mes
sages dur
ing the process of one clus
ter con
verg
ing and then the next
clus
ter. These mes
sages are dis
played in the Sta
tus field.
As the con
troller and switches move through the up
grade, you will see mes
sages about
the num
ber of nodes queued and the num
ber in the process of up
grad
ing, as well as
how many have up
graded suc
cess
fully.
The fol
low
ing are the pos
si
ble up
grade states for a node:
NotScheduled: No upgrade is currently scheduled for this node.
Scheduled: Upgrade is scheduled for this node.
Queued: There is a currently active window (schedule) and the node is
Inprogress: Upgrade is currently in progress on this node.
CompleteOK: Upgrade completed successfully.
CompleteNOK: Upgrade failed on this node.
Inretryqueue: Node is queued again for upgrade retry (5 attempts are made
requesting permission to upgrade.
before declaring failure).

This may take a while. When all the clus
ters have con
verged suc
cess
fully, you will see
"No" in the Wait
ing for Clus
ter Con
ver
gence field of the Con
troller Firmware screen.
Troubleshooting Failures During the Upgrade Process

There is one sched
uler per main
te
nance pol
icy. By de
fault, when an up
grade fail
ure is
de
tected, the sched
uler pauses, and no more nodes in that group begin to up
grade. The
sched
uler ex
pects man
ual in
ter
ven
tion to debug any up
grade fail
ures. Once man
ual in
ter
ven
tion is com
plete, you must re
sume the paused sched
uler.
67
If you no
tice that switches are in the queued state, then check the fol
low
ing:
Is the controller cluster healthy? The controller cluster must be healthy. If you
see waitingForClusterHealth = yes in the API or "Waiting for Cluster
Convergence" showing "Yes" in the GUI, that means the controller cluster is not
healthy. Until the controller cluster is healthy, switches which have not already
started their upgrade will be in queued state.
Is the switch maintenance group paused? The group will be paused if any
switch fails its upgrade.
If the sys
tem takes longer than about 60 min
utes for a switch to dis
play wait
ing
For
Clus
ter
Health = no in the API or "Wait
ing for Clus
ter Con
ver
gence" show
ing "No" in
the GUI, you should work through the steps for ver
if
y
ing a pause in the sched
uler.
For ad
di
tional trou
bleshoot
ing pro
ce
dures, see the Trou
bleshoot
ing Cisco Ap
pli
ca
tion
Cen
tric In
fra
struc
ture doc
um
ent at the fol
low
ing URL:
https://
datacenter.
github.
io/
aci-troubleshooting-book/
69
Fabric Connectivity
Fabric Connectivity
Section Content
Understanding Fabric Policies
Fabric - Access Policies

Domains
VLAN Pools
AEPs
Policy Types
Interface Policies
Switch Policies
Interface Policy Groups
Switch Policy Groups
Interface Profiles
Switch Profiles
Best Practices
Adding New Devices to the Fabric

Sample Configuration
Creating VLAN Pools
Create VLAN Pool
Create Physical Domain
Create an Attachable Access Entity Profile (AEP)
Interface Policies
Interface Profile
Switch Profiles
Reusability
Sample vPC Creation
Create VLAN Pool
Create a Physical Domain
71
Fabric Connectivity
72
Create Access Entity Profile

Interface Policies
Switch Profile
Create vPC domain
Validate Operation of Configured vPC
Server Connectivity
Cisco UCS B-Series Servers
Standalone Rack Mount Servers or Non-Cisco Servers
Virtual Machine Networking

Understanding VM Networking in ACI
ACI VM Integration Workflow
VMware Integration
VMM Policy Model Interaction
Publishing EPGS to a VMM Domain
Connecting VMs to the EPG Port Groups on vCenter
Verifying Virtual Endpoint Learning
Verifying VM Endpoint Learning on the APIC from the CLI
VMware Integration Use Case
Deploying the Application Virtual Switch

Prerequisites
Getting Started
Install the AVS VIB
Manual Installation
DHCP Relay
Attachable Access Entity Profile (AEP) and AVS
VMM Domains for vCenter
AVS Switching Modes
Create the VMM Domain for AVS
Verify AVS Deployment on vCenter
Add vSphere Hosts to the AVS
Fabric Connectivity
Verify AVS on ESX

VXLAN Load Balancing
IGMP Snooping Policy for AVS
External Connectivity
Extending ACI to External Layer 2
Extending an ACI Bridge Domain Outside of the Fabric
Extending Endpoint Groups Outside the ACI Fabric
Supported Routing Protocols
Configure MP-BGP Spine Route Reflectors
Layer 3 Integration Through Tenant Network with OSPF NSSA
External Layer 3 for Multiple Tenants
Application Migration Use Case

Extending the Network to ACI
73
Fabric Connectivity
75
Understanding Fabric Policies

Now that ACME has been pro
vi
sioned with ACI fab
ric and in
fra
struc
ture space has
been con
fig
ured be
tween the leaf and spine switches, ac
cess priv
il
eges, and basic man
age
ment poli
cies, it is time to start cre
at
ing con
nec
tiv
ity poli
cies within the ACI fab
ric. The fab
ric tab in the APIC GUI is used to con
fig
ure sys
tem-level fea
tures in
clud
ing,
but not lim
ited to, de
vice dis
cov
ery and in
ven
tory man
age
ment, di
ag
nos
tic tools, con
fig
ur
ing do
mains, and switch and port be
hav
ior. The fab
ric pane is split into three sec
tions: in
ven
tory, fab
ric poli
cies, and ac
cess poli
cies. Un
der
stand
ing how fab
ric and ac
cess poli
cies con
fig
ure the fab
ric is key for the ACME net
work teams, as they will be
main
tain
ing these poli
cies for the pur
poses of in
ter
nal con
nec
tions be
tween fab
ric leaf
nodes, con
nec
tions to ex
ter
nal en
ti
ties such as servers, net
work
ing equip
ment, and
stor
age ar
rays. It is im
por
tant that other teams such as server teams un
der
stand these
con
cepts as well, as they will be in
ter
act
ing with them, par
tic
ul
arly in the case of their
build processes for adding ad
di
tional ca
pac
ity.
This chap
ter will re
view the key ob
jects in the ac
cess poli
cies sub
sec
tion of the fab
ric
tab many of which are reusable; demon
strate how to add and pre-pro
vi
sion switches,
and walk through the steps and ob
jects re
quired when new de
vices are added to the
fab
ric to ef
fec
tively op
er
ate an ACI fab
ric. While many poli
cies are reusable, it is also
key to un
der
stand the im
pli
ca
tions of delet
ing poli
cies on the ACI fab
ric.
The ac
cess poli
cies sub
sec
tion is split into fold
ers sep
ar
at
ing out dif
fer
ent types of
poli
cies and ob
jects that af
fect fab
ric be
hav
ior. For ex
am
ple, the in
ter
face poli
cies
folder is where port be
hav
ior is con
fig
ured, like port speed, or whether or not to run
pro
to
cols like LACP on leaf switch in
ter
faces is set. Do
mains and AEPs are also cre
ated
in the ac
cess poli
cies view. The fab
ric ac
cess poli
cies pro
vide the fab
ric with the base
con
fig
ur
a
tion of the ac
cess ports on the leaf switches.
Fabric - Access Policies

Domains
EPGs are con
sid
ered the who in ACI; con
tracts are con
sid
ered the
what/when/why; AEPs can be con
sid
ered the "where" and do
mains can be thought
76
Fabric Connectivity
of as the how of the fab

ric. Dif
fer
ent do
main types are cre
ated de
pend
ing on how a
de
vice is con
nected to the leaf switch. There are four dif
fer
ent do
main types: phys
ic
al
do
mains, ex
ter
nal bridged do
mains, ex
ter
nal routed do
mains, and VMM do
mains.
Phys
ic
al do
mains are gen
er
ally used for bare metal servers or servers where hy
per
vi
sor
in
te
gra
tion is not an op
tion.
Ex
ter
nal bridged do
mains are used for Layer 2 con
nec
tions. For ex
am
ple, an ex
ter
nal
bridged do
main could be used to con
nect an ex
ist
ing switch trunked-up to a leaf
switch.
Ex
ter
nal routed do
mains are used for Layer 3 con
nec
tions. For ex
am
ple, an ex
ter
nal
routed do
main could be used to con
nect a WAN router to the leaf switch.
Do
mains act as the glue be
tween the con
fig
ur
a
tion done in the fab
ric tab to the pol
icy
model and EPG con
fig
ur
a
tion found in the ten
ant pane. The fab
ric op
er
at
or cre
ates the
do
mains, and the ten
ant ad
min
is
tra
tors as
so
ci
ate do
mains to EPGs.
For an in-depth white
board ex
pla
na
tion on do
mains, check out the fol
low
ing video ti
tled "How De
vices Con
nect to the Fab
ric: Un
der
stand
ing Cisco ACI Do
mains": https://
www.
youtube.
com/
watch?
v=_
iQvoC9zQ_
A
VLAN Pools
VLAN pools con
tain the VLANs used by the EPGs the do
main will be tied to. A do
main is
as
so
ci
ated to a sin
gle VLAN pool. VXLAN and mul
ti
cast ad
dress pools are also con
fig
urable. VLANs are in
stan
ti
ated on leaf switches based on AEP con
fig
ur
a
tion. For
ward
ing
de
ci
sions are still based on con
tracts and the pol
icy model, not sub
nets and VLANs.
AEPs
At
tach
able Ac
cess En
tity Pro
files (AEPs) can be con
sid
ered the "where" of the fab
ric
con
fig
ur
a
tion, and are used to group do
mains with sim
il
ar re
quire
ments. AEPs are tied
to in
ter
face pol
icy groups. One or more do
mains are added to an AEP. By group
ing do
mains into AEPs and as
so
ci
at
ing them, the fab
ric knows where the var
io
us de
vices in
the do
main live and the APIC can push the VLANs and pol
icy where it needs to be. AEPs
are con
fig
ured under global poli
cies.
Fabric Connectivity
77
Policy Types
Most of the poli
cies fold
ers have sub
fold
ers. For ex
am
ple, under the in
ter
face poli
cies
folder there are fold
ers for con
fig
ur
a
tion called poli
cies, pol
icy groups, and pro
files.
Interface Policies
First, in
ter
face poli
cies are cre
ated to dic
tate in
ter
face be
hav
ior, are are later tied to
in
ter
face pol
icy groups. For ex
am
ple, there should be a pol
icy that dic
tates CDP is dis
abled, and a pol
icy that dic
tates CDP is en
abled - these can be reused as new de
vices
are con
nected to the leaf switches.
Switch Policies
There are also poli

cies for switches - for ex
am
ple, con
fig
ur
ing vPC do
mains, which are
called ex
plicit vPC pro
tec
tion groups in the APIC GUI. Ide
ally, poli
cies should be cre
ated once and reused when con
nect
ing new de
vices to the fab
ric. Max
im
iz
ing reusabil
ity of pol
icy and ob
jects makes day-to-day op
er
at
ions ex
po
nen
tially faster and eas
ier to
make large-scale changes.
In
ter
face pol
icy groups are tem
plates to dic
tate port be
hav
ior and are as
so
ci
ated to an
AEP. In
ter
face pol
icy groups use the poli
cies de
scribed in the pre
vi
ous para
graph to
spec
ify how links should be
have. These are also reusable ob
jects as many de
vices are
likely to be con
nected to ports that will re
quire the same port con
fig
ur
a
tion. There are
three types of in
ter
face pol
icy groups de
pend
ing on link type: in
di
vid
ual, Port Chan
nel,
and vPC. Note that the ports on the leaf switches de
fault to 10GE, and a 1GE link level
pol
icy must be cre
ated for de
vices con
nected at that speed. Just like poli
cies, there
should ide
ally be a set of pol
icy groups cre
ated once and reused as new de
vices are
con
nected to the fab
ric. For ex
am
ple, there may be a pol
icy that dic
tates 10GE, CDPen
abled. Note the in
ter
face pol
icy groups sim
ply dic
tate pol
icy. Pol
icy groups do not
ac
tu
ally spec
ify where the pro
to
cols and port be
hav
ior should be im
ple
mented. The
"where" hap
pens by as
so
ci
at
ing one or more in
ter
face pro
files to a switch pro
file, cov
ered in the fol
low
ing para
graphs.
Switch Policy Groups
Switch pol
icy groups allow lever
ag
ing of ex
ist
ing switch po
lices like Span
ning Tree and
mon
it
or
ing poli
cies.
Interface Profiles
In
ter
face pro
files help tie the pieces to
gether. In
ter
face pro
files con
tain blocks of ports
- in
ter
face se
lec
tors - and are also tied to the in
ter
face pol
icy groups de
scribed in the
78
Fabric Connectivity
pre
vi
ous para
graphs. Again, this is just an ar
bi
trary port, such as e1/1, the pro
file must
be as
so
ci
ated to a spe
cific switch pro
file (dis
cussed in the next para
graph) to con
fig
ure
the ports.
Switch Profiles
Lastly, switch pro

files allow the se
lec
tion of one or more leaf switches and as
so
ci
ate in
ter
face pro
files to con
fig
ure the ports on that spe
cific node. This as
so
ci
a
tion pushes
the con
fig
u
ra
tion to the in
ter
face, and cre
ates a Port Chan
nel or vPC if one has been
con
fig
ured in the in
ter
face pol
icy.
The fol
low
ing fig
ure high
lights the re
la
tion
ship be
tween the var
i
ous global, switch, and
in
ter
face poli
cies:
Relationships to allow a physical interface or interfaces to be attached to an EPG

Layer 2 In
ter
face Pol
icy
In Cisco ACI ver
sion 1.1, a new con
fig
urable In
ter
face Pol
icy was added to allow a per
port-VLAN sig
nif
i
cance.
To con
nect de
vices to the ACI fab
ric we can use un
tagged traf
fic,VLAN en
cap
su
la
tion or VXLAN en
cap
su
la
tion.
Fabric Connectivity
79
In tra
di
tional net
work
ing one of the lim
it
a
tions re
lated to VLAN en
cap
su
la
tion is scal
a-
bil
ity and re-us
abil
ity due to the limit of 4096 VLANs in net
work
ing de
vices.
In ACI, with the de
fault con
fig
ur
a
tion (global), EPGs can use the same VLAN en
cap
su
la
tion as long as EPGs are bound to sep
ar
ate switches. This al
lows ten
ants to re-use
VLAN en
cap
su
la
tion IDs thor
ough the fab
ric with
out al
low
ing com
mu
ni
ca
tion be
tween
ten
ants. How
ever, global con
fig
ur
a
tion as
sumes that ten
ants do not share leaf switches
and there
fore there is no VLAN over
lap
ping within the same leaf.
Per Port-VLAN limitations and considerations

When per port-VLAN is used, a port and VLAN pair (P,V) is registered
internally instead of just a VLAN encapsulation ID. This increases the
consumption of hardware resources at a per switch level.
Two EPGs belonging to a single Bridge Domain cannot share the same
encapsulation ID on a given leaf switch.
It is expected that the port will flap when the Layer 2 interface policy
changes between global and local. That is, traffic will get affected.
Best Practices
Cisco has es
tab
lished sev
eral best prac
tices for fab
ric con
fig
ur
a
tion. These are not re
quire
ments and might not work for all en
vi
ron
ments or ap
pli
ca
tions, but might help
sim
plify day-to-day op
er
at
ion of the ACI fab
ric.
Policies
Reuse policies whenever possible. For example, there should be policies for
LACP active/passive/off, 1GE port speed, and 10GE port speed.
When naming policies, use names that clearly describe the setting. For
example, a policy that enables LACP in mode active could be called "LACPActive". There are many "default" policies out of the box. However, it can be
hard to remember what all the defaults are, which is why policies should be
clearly named to avoid making a mistake when adding new devices to the
fabric.
A set of interface policy groups should be created for each type of similar
devices connected. For example, there can be a set of interface policy
groups for all VMware ESXi servers connected via 10GE vPCs, and a
different set of interface policy groups for all bare metal servers running
Fabric Connectivity
80
1GE with CDP disabled. Since interface policy groups are tied to a single
AEP, each AEP will have its own set of interface policy groups.
Create a switch profile for each leaf switch individually, and additionally,
create a switch profile for each vPC pair (if using vPC).
Domains
Build one physical domain per tenant for bare metal servers or servers
without hypervisor integration requiring similar treatment.
Build one physical domain per tenant for external connectivity.
If a VMM domain needs to be leveraged across multiple tenants, a single
VMM domain can be created and associated with all leaf ports where
VMware ESXi servers are connected.
AEPs
Multiple domains can be associated to a single AEP for simplicity's sake.
There are some cases where multiple AEPs may need to be configured to
enable the infrastructure VLAN, such as overlapping VLAN pools, or to limit
the scope of the presence of VLANs across the fabric.
Fabric Connectivity
81
Adding New Devices to the Fabric

This sec
tion will demon
strate how to con
fig
ure ACI to re-use the fab
ric ac
cess poli
cies,
sim
pli
fy
ing day-to-day op
er
a
tion of the fab
ric. This sec
tion will walk through set
ting up
pro
files from scratch, with a focus on how to re-use these pro
files across the fab
ric. As
out
lined in the pre
vi
ous sec
tion, these var
i
ous pro
files are linked to
gether and have de
pen
den
cies. The fol
low
ing di
a
gram re
it
er
ates the ob
ject re
la
tion
ships:
Object relationships
Whereas a tra
di
tional com
mand line in
ter
face on a switch gen
er
ally re
quires a port-byport confugu
ra
tion, ACI al
lows de
f
i
n
i
tion of ob
jects and poli
cies that can be re-used.
The re-us
abil
ity of these poli
cies makes it pos
si
ble to repli
cate the con
fig
u
ra
tion of a
switch very eas
ily. The fol
low
ing di
a
gram de
picts how this re-us
abil
ity sim
pli
fies the
op
er
a
tion of the fab
ric over time.
82
Fabric Connectivity
Policy Re-use
In any data cen
ter, the con
fig
u
ra
tion of a cou
ple of switches does not re
quire many
processes or au
toma
tion. As the data cen
ter size in
creases, au
toma
tion be
comes more
and more crit
i
cal as it has a di
rect im
pact on the cost of busi
ness op
er
a
tions. In tra
di
tional net
works, when changes that im
pact a large set of de
vices need to be made, the
op
er
a
tor is faced with the cost of de
sign
ing processes to man
age these de
vices. These
can be net
work man
age
ment tools, scripts, or spe
cial
ized ap
pli
ca
tions. Lever
ag
ing the
Cisco ACI pol
icy model, an op
er
a
tor can lever
age pro
files to stream
line the op
er
a
tion of
adding de
vices and man
ag
ing those de
vices. This is what is de
picted as the pol
icy reuse in
flec
tion point in the pre
vi
ous di
a
gram.
Sample Configuration
The fol
low
ing sec
tions will walk through sam
ple con
fig
u
ra
tion of set
ting up in
di
vid
u
ally
con
nected de
vices, Port Chan
nel-con
nected de
vices, and vPC-con
nected de
vices from
scratch, and will in
clude a re
view of the ob
jects as they are con
fig
ured. These are the
steps to be taken in the APIC GUI when new de
vices are con
nected to the leaf switches
to en
sure the ac
cess ports on the leaf switches have the right switch
port con
fig
u
ra
tion,
and the ver
i
fi
ca
tion steps to en
sure proper con
fig
u
ra
tion. The fol
low
ing steps rep
re
sent the use case of adding a new bare metal server con
nected to a leaf switch.
Fabric Connectivity
83
Be
fore get
ting into the con
fig
u
ra
tion of vPC's, which are a pop
u
lar server con
nec
tiv
ity
method
ol
ogy, it is im
por
tant to un
der
stand what vPC's are and how they are dif
fer
ent from
tra
di
tional meth
ods of server con
nec
tiv
ity. This sec
tion of the chap
ter at
tempts to clar
ify
at a high level what vPC's are, the ben
e
fits they pro
vide and how vPC's in the ACI fab
ric dif
fer from how they are de
ployed on Cisco Nexus switches run
ning NX-OS soft
ware.
At a high level, vPC ex
tends link ag
gre
ga
tion to two sep
a
rate phys
i
cal switches.
vPC Topology
In the fig
ure above, a sin
gle server is dual homed to two dif
fer
ent switches for re
dun
dancy. With
out vPC's, the server will likely use an ac
tive-standby con
fig
u
ra
tion, or a
spe
cial con
fig
u
ra
tion on the NIC dri
ver or the ker
nel that al
lows it to in
tel
li
gently loadbal
ance traf
fic using an al
go
rithm.
By con
fig
ur
ing ports on two dif
fer
ent switches as the same port-chan
nel and using an
in
ter-switch mes
sag
ing chan
nel (such as the in
ter-switch port-chan
nel in the green
box on the left hand side) to cover re
dun
dancy sce
nar
ios, we pro
vide a log
i
cal topol
ogy
that greatly sim
pli
fies server pro
vi
sion
ing and man
age
ment.
This al
lows for sev
eral key ad
van
tages from a server de
ploy
ment per
spec
tive:
You can create resilient Layer 2 topologies based on link aggregation
You do not need STP
You have increased bandwidth, as all links are actively forwarding
Your server configurations are simplified since the configurations simply

appears as port-channels without the need for special software, from a driver
or kernel-tuning standpoint
84
Fabric Connectivity
vPCs can also be used to con

nect other down
stream de
vices, such as Cisco UCS fab
ricin
ter
con
nects, to pro
vide sim
i
lar ben
e
fits.
The fig
ure below shows a sin
gle tra
di
tional Layer 2 switch con
nected to a VPC en
abled Cisco switch pair.
Legacy connectivity compared to vPC

The com
po
nents of a tra
di
tional vPC do
main are il
lus
trated below:
Traditional vPC topology
Fabric Connectivity
85
As il
lus
trated above, in Cisco switch
ing prod
ucts run
ning NX-OS soft
ware, vPC con
fig
u
ra
tions need to be done man
u
ally by the op
er
a
tor and re
quire a pair of ded
i
cated "in
ter-switch" links also called a peer-link. There is also a peer-keepalive link, typ
i
cally on
the out-of-band man
age
ment port, that is used to de
ter
mine peer live
li
ness to de
tect a
vPC peer-switch fail
ure. Mak
ing con
fig
u
ra
tion changes in such sce
nar
ios with
out the
con
fig-sync fea
ture en
abled may lead to sce
nar
ios where there are mis
matched vPC
pa
ra
me
ters be
tween the vPC pri
mary and the vPC sec
ondary switches that may cause
par
tial con
nec
tiv
ity loss dur
ing the change it
self if a type-1 in
con
sis
tency is de
tected.
The ACI fab
ric greatly sim
pli
fies VPC con
fig
u
ra
tions.
vPC topology in ACI

The key dif
fer
ences to note here are that rel
a
tive to tra
di
tional vPC de
sign, there is no
re
quire
ment for set
ting up vPC peer-links. There are also no keepalives being sent on
the man
age
ment ports. The fab
ric it
self serves as the peer-link. The rich in
ter
con
nec
tiv
ity be
tween fab
ric nodes makes it un
likely that peers will have an ac
tive path be
tween them.
Note that at
tempt
ing to cable a leaf switch to an
other leaf switch will lead to a "wiring
mis
match" fault in the GUI and re
sult in a black
listed port that will have to be man
u
ally
re
cov
ered.
Fabric Connectivity
86
The fol
low
ing are some other key be
hav
ioral changes to vPC as it ap
plies to the ACI
fab
ric rel
at
ive to clas
sic vPC that are im
por
tant for op
er
at
ors to un
der
stand:
Configurations are automatically synchronized to avoid an error-free

configuration by the APIC which is the central point of control for all
configurations in the ACI fabric.
In traditional vPC solution, the slave switch brings down all its vPC links if the
MCT goes down.
In the ACI fabric, it is very unlikely that all the redundant paths between vPC
peers fail at the same time. Hence if the peer switch becomes unreachable, it is
assumed to have crashed. The slave switch does not bring down vPC links.
Role election still happens, peers assume master-slave roles.
Role is used in case of vpc type-1 consistency failure. Slave switch brings down
all its vPC ports. A list of type-1 parameters used for consistency checking for a
given vPC domain specific to the ACI fabric are listed below.
Global type-1 parameters:

STP
Interface type-1 parameters:

STP: Only BPDU Guard is configurable
EthPM
Port speed
Duplex mode
Port mode
MTU
Native VLAN
PCM: Channel mode, static vs lacp
LACP: Lag ID
The fol
low
ing di
ag
rams il
lus
trate how the ACI fab
ric for
wards traf
fic from a vPC do
main
to a non-vPC con
nected host in the fab
ric, and vice-versa.
Fabric Connectivity
87
vPC forwarding
Uni
cast packet flow H2 -> H1
1
H2 sends a pkt towards H1 on its link to S3.
S3 does a table lookup and routes with vPC Virtual IP (VIP).
Spine switch sees multiple routes for VIP and picks one of them (S2 in this case).
S2 delivers the pkt to locally attached host H1.
H1 -> H2
1
H1 sends a pkt towards H2 on one of its PC link (S1 in this case).
S1 does a table lookup and routes with IP of S3.
Spine switch routes to S3.
S3 delivers the pkt to locally attached host H2.
Creating VLAN Pools

In this ex
am
ple, con
fig
ur
ing newly-con
nected bare metal servers first re
quires cre
ation
of a phys
i
cal do
main and then as
so
ci
a
tion of the do
main to a VLAN pool. As men
tioned
in the pre
vi
ous sec
tion, VLAN pools de
fine a range of VLAN IDs that will be used by the
EPGs.
Fabric Connectivity
88
The servers are con

nected to two dif
fer
ent leaf nodes in the fab
ric. Each server will be
tag
ging using 802.1Q or VXLAN en
cap
su
la
tion. The range of VLANs used in the con
fig
u
ra
tion ex
am
ple is 100-199. As de
picted in the fol
low
ing fig
ure, the ACI fab
ric can also act as
a gate
way be
tween dis
parate en
cap
su
la
tion types such as un
tagged traf
fic, 802.1Q VLAN
tags, VXLAN VNIDs, and NVGRE tags. The leaf switches nor
mal
ize the traf
fic by strip
ping
off tags and reap
ply
ing the re
quired tags on fab
ric egress. In ACI, it is im
por
tant to un
der
stand that the de
f
i
n
i
tion of VLANs as they per
tain to the leaf switch ports is uti
lized only
for iden
ti
fi
ca
tion pur
poses. When a packet ar
rives ingress to a leaf switch in the fab
ric,
ACI has to know be
fore
hand how to clas
sify pack
ets into the dif
fer
ent EPGs, using iden
ti
fiers like VLANs, VXLAN, NVGRE, phys
i
cal port IDs, vir
tual port IDs.
Encapsulation normalization
Create VLAN Pool

1
In the Navigation pane, choose Pools > VLAN.
In the Work pane, choose Actions > Create VLAN Pool.
In the Create VLAN Pool dialog box, perform the following actions:
a. Define a meaningful name for the VLAN pool.
b. Optionally, provide a description for the VLAN pool.
c. There are two allocation modes: dynamic or static.
Note: When dynamic allocation is selected, the APIC selects VLANs from the
pool dynamically. This is common in VMM integration mode where the actual
Fabric Connectivity
89
VLAN ID is not important. Remember it's the EPG itself which policies are
applied to. Static allocation is typically used when the pool will be referenced
from a static source like a static path binding for an EPG for use with bare
metal servers.
d. The encap blocks are used to define the range of VLANs in the VLAN pool.
Note multiple ranges can be added to a single pool.
XML Ob
ject
<fvnsVlanInstP allocMode="static" childAction="" configIssues="" descr=""
dn="uni/infra/vlanns-[bsprint-vlan-pool]-static" lcOwn="local" modTs="2015-02-
23T15:58:33.538-08:00" monPolDn="uni/fabric/monfab-default" name="bsprint-vlan-pool"

ownerKey="" ownerTag="" status="" uid="8131">
<fvnsRtVlanNs childAction="" lcOwn="local" modTs="2015-02-25T11:35:33.365-08:00"
rn="rtinfraVlanNs-[uni/l2dom-JC-L2-Domain]" status="" tCl="l2extDomP" tDn="uni/l2domJC-L2-Domain"/>
<fvnsRtVlanNs childAction="" lcOwn="local" modTs="2015-02-23T16:13:22.007-08:00"
rn="rtinfraVlanNs-[uni/phys-bsprint-PHY]" status="" tCl="physDomP" tDn="uni/physbsprint-PHY"/>
<fvnsEncapBlk childAction="" descr="" from="vlan-100" lcOwn="local" modTs="2015-02-
23T15:58:33.538-08:00" name="" rn="from-[vlan-100]-to-[vlan-199]" status="" to="vlan199" uid="8131"/>

</fvnsVlanInstP>
Create Physical Domain

A phys
ic
al do
main acts as the link be
tween the VLAN pool and the Ac
cess En
tity Pro
file
(AEP). The do
main also ties the fab
ric con
fig
ur
a
tion to the ten
ant con
fig
ur
a
tion, as the
ten
ant ad
min
is
tra
tor is the one who as
so
ci
ates do
mains to EPGs, while the do
mains are
cre
ated under the fab
ric tab. When con
fig
ur
ing in this order, only the pro
file name and
the VLAN pool are con
fig
ured. The cre
ation of the AEP and its as
so
ci
at
ion will be cov
ered later in this sec
tion.
1
In the Navigation pane, choose Physical and External Domains > Physical
Domains.
3
4
In the Work pane, choose Actions > Create Physical Domain.

In the Create Physical Domain dialog box, perform the following actions:
a. Define a meaningful name for the profile.
b. Select the VLAN pool you just created.
Fabric Connectivity
90
XML Ob
ject
<physDomP childAction="" configIssues="" dn="uni/phys-bsprint-PHY" lcOwn="local"
modTs="2015-02-23T16:13:21.906-08:00" monPolDn="uni/fabric/monfab-default"
name="bsprint-PHY" ownerKey="" ownerTag="" status="" uid="8131">
<infraRsVlanNs childAction="" forceResolve="no" lcOwn="local" modTs="2015-02-
23T16:13:22.065-08:00" monPolDn="uni/fabric/monfab-default" rType="mo" rn="rsvlanNs"
state="formed" stateQual="none" status="" tCl="fvnsVlanInstP" tDn="uni/infra/vlanns[bsprint-vlan-pool]-static" tType="mo" uid="8131"/>
<infraRsVlanNsDef childAction="" forceResolve="no" lcOwn="local" modTs="2015-02-
23T16:13:22.065-08:00" rType="mo" rn="rsvlanNsDef" state="formed" stateQual="none"

status="" tCl="fvnsAInstP" tDn="uni/infra/vlanns-[bsprint-vlan-pool]-static"
tType="mo"/>
<infraRtDomP childAction="" lcOwn="local" modTs="2015-02-23T16:13:52.945-08:00"
rn="rtdomP-[uni/infra/attentp-bsprint-AEP]" status="" tCl="infraAttEntityP"

tDn="uni/infra/attentp-bsprint-AEP"/>
</physDomP>
Create an Attachable Access Entity Profile (AEP)

The AEP links the phys
ic
al do
main and its VLAN Pool to the in
ter
face poli
cies. The con
fig
ur
a
tion for an AEP is straight
for
ward.
1
In the Navigation pane, choose Global Policies > Attached Acess Entity Profle.
In the Work pane, choose Actions > Create Attached Entity Profile.
In the Create Attached Entity Profile dialog box, perform the following actions:
a. Define the a meaningful name for the profile.
b. Optionally, enter a description.
c. Click + to associate the domain to the AEP.
d. Select the physical domain that was previously configured.
Click Next.
Click Submit.
Fabric Connectivity
91
XML Ob
ject
<infraAttEntityP childAction="" configIssues="" descr="" dn="uni/infra/attentp-
bsprint-AEP" lcOwn="local" modTs="2015-02-23T16:13:52.874-08:00"
monPolDn="uni/fabric/monfab-default" name="bsprint-AEP" ownerKey="" ownerTag=""
status="" uid="8131">
<infraContDomP childAction="" lcOwn="local" modTs="2015-02-23T16:13:52.874-08:00"
rn="dompcont" status="">
<infraAssocDomP childAction="" dompDn="uni/phys-bsprint-PHY" lcOwn="local"
modTs="2015-02-23T16:13:52.961-08:00" rn="assocdomp-[uni/phys-bsprint-PHY]" status=""/>

<infraAssocDomP childAction="" dompDn="uni/l2dom-JC-L2-Domain" lcOwn="local"
modTs="2015-02-25T11:35:33.570-08:00" rn="assocdomp-[uni/l2dom-JC-L2-Domain]"
status=""/>
</infraContDomP>
<infraContNS childAction="" lcOwn="local" modTs="2015-02-23T16:13:52.874-08:00"
monPolDn="uni/fabric/monfab-default" rn="nscont" status="">
<infraRsToEncapInstDef childAction="" deplSt="" forceResolve="no"
lcOwn="local" modTs="2015-02-23T16:13:52.961-08:00" monPolDn="uni/fabric/monfabdefault" rType="mo" rn="rstoEncapInstDef-[allocencap-[uni/infra]/encapnsdef-
[uni/infra/vlanns-[bsprint-vlan-pool]-static]]" state="formed" stateQual="none"
status="" tCl="stpEncapInstDef" tDn="allocencap-[uni/infra]/encapnsdef[uni/infra/vlanns-[bsprint-vlan-pool]-static]" tType="mo">
<fabricCreatedBy childAction="" creatorDn="uni/l2dom-JC-L2-Domain"
deplSt="" domainDn="uni/l2dom-JC-L2-Domain" lcOwn="local" modTs="2015-02-
25T11:35:33.570-08:00" monPolDn="uni/fabric/monfab-default" profileDn="" rn="source[uni/l2dom-JC-L2-Domain]" status=""/>
<fabricCreatedBy childAction="" creatorDn="uni/phys-bsprint-PHY" deplSt=""
domainDn="uni/phys-bsprint-PHY" lcOwn="local" modTs="2015-02-23T16:13:52.961-08:00"
monPolDn="uni/fabric/monfab-default" profileDn="" rn="source-[uni/phys-bsprint-PHY]"
status=""/>
</infraRsToEncapInstDef>
</infraContNS>
<infraRtAttEntP childAction="" lcOwn="local" modTs="2015-02-24T11:59:37.980-08:00"
rn="rtattEntP-[uni/infra/funcprof/accportgrp-bsprint-AccessPort]" status=""
tCl="infraAccPortGrp" tDn="uni/infra/funcprof/accportgrp-bsprint-AccessPort"/>
<infraRsDomP childAction="" forceResolve="no" lcOwn="local" modTs="2015-02-
25T11:35:33.570-08:00" monPolDn="uni/fabric/monfab-default" rType="mo" rn="rsdomP-
[uni/l2dom-JC-L2-Domain]" state="formed" stateQual="none" status="" tCl="l2extDomP"

tDn="uni/l2dom-JC-L2-Domain" tType="mo" uid="8754"/>
<infraRsDomP childAction="" forceResolve="no" lcOwn="local" modTs="2015-02-
23T16:13:52.961-08:00" monPolDn="uni/fabric/monfab-default" rType="mo" rn="rsdomP[uni/phys-bsprint-PHY]" state="formed" stateQual="none" status="" tCl="physDomP"
tDn="uni/phys-bsprint-PHY" tType="mo" uid="8131"/>

</infraAttEntityP>
Create Interface Policies

Next, de
fine the in
ter
face pro
files and show
case the re-us
abil
ity of the fab
ric poli
-
Fabric Connectivity
92
cies. In
ter
face poli
cies can be re-used as needed by dif
fer
ent in
ter
face pro
file de
f
in
i
t
ion
re
quire
ments. This sec
tion will il
lus
trate cre
ation of new pro
files, but ide
ally there are
al
ready poli
cies in place that can sim
ply be se
lected.
Create Link Level Policies
Link level poli

cies dic
tate con
fig
ur
a
tion like the speed of ports.
1
In the Navigation pane, choose Interface Policies > Policies > Link Level.
In the Work pane, choose Actions > Create Link Level Policy.
In the Create Link Level Policy dialog box, perform the following actions:
a. Define the meaningful name for the policy.
b. Optionally, provide a description for the policy.
c. Select the auto negotiation mode for the interface.
d. Select the interface speed. Leaf switch ports default to 10GE.
e. Change the de-bounce interval if required.
Click Submit.
XML Ob
ject
<fabricHIfPol autoNeg="on" childAction="" descr="" dn="uni/infra/hintfpol-1G-Auto"
lcOwn="local" linkDebounce="100" modTs="2015-01-14T06:47:15.693-08:00" name="1G-Auto"

ownerKey="" ownerTag="" speed="1G" status="" uid="15374">
<fabricRtHIfPol childAction="" lcOwn="local" modTs="2015-01-14T06:48:48.081-
08:00" rn="rtinfraHIfPol-[uni/infra/funcprof/accportgrp-UCS-1G-PG]" status=""

tCl="infraAccPortGrp" tDn="uni/infra/funcprof/accportgrp-UCS-1G-PG"/>
<fabricRtHIfPol childAction="" lcOwn="local" modTs="2015-02-25T11:48:11.331-
08:00" rn="rtinfraHIfPol-[uni/infra/funcprof/accportgrp-L3-Example]" status=""

tCl="infraAccPortGrp" tDn="uni/infra/funcprof/accportgrp-L3-Example"/>
</fabricHIfPol>
Create a CDP Interface Policy
In the Navigation pane, choose Interface Policies > Policies > CDP Interface.
a. Define a meaningful name for the policy such as 'CDP-Enable'.
Fabric Connectivity
93

c. Choose either admin state enabled or disabled.
5
Click Submit.
XML Ob
ject
<cdpIfPol adminSt="enabled" childAction="" descr="" dn="uni/infra/cdpIfP-CDP-Enable"
lcOwn="local" modTs="2015-01-14T06:47:25.470-08:00" name="CDP-Enable" ownerKey=""
ownerTag="" status="" uid="15374">
<cdpRtCdpIfPol childAction="" lcOwn="local" modTs="2015-01-14T07:23:54.957-
08:00" rn="rtinfraCdpIfPol-[uni/infra/funcprof/accportgrp-UCS-10G-PG]" status=""
08:00" rn="rtinfraCdpIfPol-[uni/infra/funcprof/accbundle-ACI-VPC-IPG]" status=""
tCl="infraAccBndlGrp" tDn="uni/infra/funcprof/accbundle-ACI-VPC-IPG"/>
08:00" rn="rtinfraCdpIfPol-[uni/infra/funcprof/accportgrp-UCS-1G-PG]" status=""

08:00" rn="rtinfraCdpIfPol-[uni/infra/funcprof/accportgrp-bsprint-AccessPort]"
status="" tCl="infraAccPortGrp" tDn="uni/infra/funcprof/accportgrp-bsprintAccessPort"/>
08:00" rn="rtinfraCdpIfPol-[uni/infra/funcprof/accportgrp-L3-Example]" status=""
tCl="infraAccPortGrp" tDn="uni/infra/funcprof/accportgrp-L3-Example"/>
</cdpIfPol>
Create an LLDP Interface Policy
In the Navigation pane, choose Interface Policies > Policies > LLDP Interface.
a. Define a meaningful name for the policy.
c. Choose the receive state.
d. Choose the transmit state.
Click Submit.
Fabric Connectivity
94
XML Object
descr=""
<lldpIfPol adminRxSt="enabled" adminTxSt="enabled" childAction=""
dn="uni/infra/lldpIfP-LLDP-Enable" lcOwn="local" modTs="2015-02-11T07:40:35.664-08:00"

status=""
name="LLDP-Enable" ownerKey="" ownerTag="" status="" uid="15374">
/>
<lldpRtLldpIfPol childAction="" lcOwn="local" modTs="2015-02-24T14:59:11.154-
08:00" rn="rtinfraLldpIfPol-[uni/infra/funcprof/accbundle-ACI-VPC-IPG]"
tCl="infraAccBndlGrp" tDn="uni/infra/funcprof/accbundle-ACI-VPC-IPG"
status=""
/>
<lldpRtLldpIfPol childAction="" lcOwn="local" modTs="2015-02-25T11:48:11.331-
08:00" rn="rtinfraLldpIfPol-[uni/infra/funcprof/accportgrp-L3-Example]"
tCl="infraAccPortGrp" tDn="uni/infra/funcprof/accportgrp-L3-Example"
</lldpIfPol>
l
ows
Create an LACP Interface Policy
faces. These can be

Link Ag
gre
ga
tion Con
trol Pro
to
col is part of an IEEE spec
if
i
ca
tion (802.3ad) that al
r
a
tion re
quire
ments
an op
tional ne
go
ti
at
ion pro
to
col to be run on Port Chan
nel in
ter
pre-emp
tively de
fined to be used as needed for the var
io
us con
fig
u
in the data cen
ter.
.
In the Navigation pane, choose Interface Policies > Policies > LACP
In the Work pane, choose Actions > Create LACP Policy.
In the Create LACP Policy dialog box, perform the following actions:
c. Select the LACP mode required for the server. Note if LACP is enabled on the
leaf switch, LACP must also be enabled on the server or other connected
device.
d. Optionally, specify the minimum and maximum number of links in the Port
Channel.
Click Submit.
Fabric Connectivity
95
XML Ob
ject
<lacpLagPol childAction="" ctrl="fast-sel-hot-stdby,graceful-conv,susp-individual"
descr="" dn="uni/infra/lacplagp-LACP-Active" lcOwn="local" maxLinks="16" minLinks="1"
modTs="2015-02-24T11:58:36.547-08:00" mode="active" name="LACP-Active" ownerKey=""

ownerTag="" status="" uid="8131">
<lacpRtLacpPol childAction="" lcOwn="local" modTs="2015-02-24T14:59:11.154-
08:00" rn="rtinfraLacpPol-[uni/infra/funcprof/accbundle-ACI-VPC-IPG]" status=""

tCl="infraAccBndlGrp" tDn="uni/infra/funcprof/accbundle-ACI-VPC-IPG"/>
</lacpLagPol>
Create an LACP Member Profile (optional)
Op
tion
ally, the LACP mem
ber pro
file pro
vides the abil
ity to pro
vide pri
or
ity spec
if
i
ca
tions to mem
bers of an LACP group.
1
In the Navigation pane, choose Interface Policies > Policies > LACP Member.
In the Work pane, choose Actions > Create LACP Member Policy.
In the Create LACP Member Policy dialog box, perform the following actions:
c. If required, change the priority.
d. If required, change the transmit rate.
Click Submit.
Create a Spanning Tree Interface Policy (optional)
The Span
ning Tree pol
icy dic
tates the be
hav
ior of south
bound leaf port Span
ning Tree
fea
tures. It is a com
mon best prac
tice to en
able BPDU guard on in
ter
faces con
nected to
servers.
Note: ACI does not run Span
ning Tree on the fab
ric be
tween the leaves and spines. The
Span
ning Tree in
ter
face pol
icy sim
ply de
fines the port be
hav
ior.
1
In the Navigation pane, choose Interface Policies > Policies > Spanning Tree
Interface.
In the Work pane, choose Actions > Create Spanning Tree Interface Policy.
Fabric Connectivity
96
In the Create Spanning Tree Interface Policy dialog box, perform the following
actions:
c. Enable BPDU filter and/or BPDU guard.
Click Submit.
Create a Storm Control Policy (optional)
A traf
fic storm oc
curs when pack
ets flood the LAN, cre
at
ing ex
ces
sive traf
fic and de
grad
ing net
work per
for
mance. The traf
fic storm con
trol fea
ture can be used to pre
vent dis
rup
tions on ports by a broad
cast, mul
ti
cast, or uni
cast traf
fic storm on phys
i
cal in
ter
faces.
1
In the Navigation pane, choose Interface Policies > Policies > Storm Control.
In the Work pane, choose Actions > Create Storm Control Policy.
In the Create Storm Control Policy dialog box, perform the following actions:
c. Specify how the control policy is to be applied, either through percentage of
the total bandwidth or as a packet per second definition that matches the
requirement for the data center
Click Submit.
Creating a Layer 2 Interface Policy to enable per port-VLAN
In the Navigation pane, choose Interface Policies > Policies > L2 Interface.
In the Work pane, choose Actions > Create L2 Interface Policy.
In the Create L2 Interface Policy dialog box, perform the following actions:
a. Give the L2 Interface name and an optional description.
b. Select VLAN scope to Port Local scope to enable per port-VLAN.
Create Interface Policy Groups

The in
ter
face pol
icy groups com
prise the in
ter
face poli
cies as a func
tional group that is
as
so
ci
ated to an in
ter
face. The fol
low
ing di
ag
ram shows how pre
vi
ously cre
ated items
are grouped under the pol
icy group.
Fabric Connectivity
97
Policies contained in a policy group

Once all the in
ter
face poli
cies have been de
fined, the in
di
vid
ual poli
cies can be brought
to
gether to form a pol
icy group that will be linked to the in
ter
face pro
file. The pol
icy
group is de
fined from a mas
ter de
f
i
n
i
tion that en
com
passes being one of the fol
low
ing:
Access Policy Group
Port Channel Policy Group
VPC Policy Group
Create Access Port Policy Group
The ac
cess port pol
icy is de
fined for an in
di
vid
ual link (non-Port Chan
nel or vPC).
1
In the Navigation pane, choose Interface Policies > Policy Groups.
In the Work pane, choose Actions > Create Access Policy Group.
In the Create Access Policy Group dialog box, perform the following actions:
a. Define a meaningful name for the policy group.
b. Optionally, provide a description for the policy group.
c. Use the profiles created previously that are relevant for this policy group.
Click Submit.
Fabric Connectivity
98
Create Port Channel Interface Policy Group
Port Chan
nel
ing also load-bal
ances traf
fic across the phys
ic
al in
ter
faces that are mem
bers of the chan
nel group. For every group of in
ter
faces that needs to be con
fig
ured
into a port chan
nel, a dif
fer
ent pol
icy group has to be cre
ated. This pol
icy group de
fines
the be
hav
iour. For ex
am
ple, if ports 1/1-4 are to be con
fig
ured into one port chan
nel, and ports 1/5-8 into a sep
ar
ate port chan
nel, each of those groups would re
quire
the cre
ation of a sep
ar
ate pol
icy group.
1
In the Work pane, choose Actions > Create PC Interface Policy Group.
In the Create PC Interface Policy Group dialog box, perform the following
actions:
c. Select the policies created previously that are relevant for this PC policy
group.
Click Submit.
Create VPC Interface Policy Group
Note: This ob
ject must be unique for each VPC cre
ated.
A vir
tual PortChan
nel (vPC) al
lows links that are phys
ic
ally con
nected to two dif
fer
ent
de
vices to ap
pear as a sin
gle Port Chan
nel to a third de
vice. In the world of ACI, pairs of
leaf switches may be con
fig
ured in a vPC do
main so that down
stream de
vices can be
ac
tive-ac
tive dual-homed.
For every group of in
ter
faces that are to be con
fig
ured into a vPC, a dif
fer
ent in
ter
face
pol
icy group needs to be cre
ated. The vPC pol
icy group con
tains both the de
f
in
i
t
ion for
the be
hav
iour of the port chan
nel, and the iden
ti
fier. For ex
am
ple, if ports 1/1-4 are to
be con
fig
ured into one vPC across two switches, and ports 1/5-8 into a sep
ar
ate vPC
across two switches, each of those groups would re
quire the de
f
in
i
t
ion of a sep
ar
ate
pol
icy group.
Note: For vPC you will also re
quire a unique vPC do
main de
f
in
i
t
ion be
tween the two
paired switches. More de
tails to fol
low.
1
Fabric Connectivity
In the Work pane, choose Actions > Create VPC Interface Policy Group.
In the Create VPC Interface Policy Group dialog box, perform the following
99
actions:
c. Choose the policies created previously that are relevant for this vPC policy
group.
5
Click Submit.
Interface Profile
The in
ter
face pro
file in ACI links the pol
icy groups that de
fine how the in
ter
face is
going to be
have, and as
signs them to spe
cific ports via the con
cept of in
ter
face se
lec
tor. In turn, the in
ter
face pro
file is even
tu
ally tied to a switch pro
file to spec
ify which
leaf switch the ref
er
enced ports should be con
fig
ured. As we con
tinue the process of
defin
ing the port pro
files, you can ob
serve how we have started at the bot
tom of this
ob
ject tree con
fig
ur
ing the dif
fer
ent pro
files. The pur
poses for all these in
di
vid
ual poli
cies that tie to
gether is to max
i
mize pol
icy re-use.
Interface Profile links to Interface Selector and Interface Policy Group

The di
a
gram in the pre
vi
ous sec
tion pro
vides a vi
sual de
scrip
tion of what can be ac
com
plished by group
ing the poli
cies that have been de
fined under the in
ter
face pro
file,
and then as
signed to ports with in
ter
face se
lec
tors and the ac
cess port pol
icy groups.
100 Fabric Connectivity
Create Interface Profile
The in
ter
face pro
file is com
posed of two pri
mary ob
jects. The in
ter
face se
lec
tor and
the ac
cess port pol
icy group. The in
ter
face se
lec
tor de
fines what in
ter
faces will apply
the ac
cess port pol
icy. The ports that share the same at
trib
utes can then be grouped
under the same in
ter
face pro
file.
1
In the Navigation pane, choose Interface Policies > Profiles.
In the Work pane, choose Actions > Create Interface Profile.
In the Create Interface Profile dialog box, perform the following actions:
b. Optionally, provide a description for the profile.
Click Submit.
Next, add the in

ter
face se
lec
tors that are as
so
ci
ated to this in
ter
face pro
file.
Create Interface Selector
In the Navigation pane, choose Interface Policies > Profiles

> Name_of_Interface_Profile_Created.
3
4
In the Work pane, choose Actions > Create Access Port Selector.
In the Create Access Port Selector dialog box, perform the following actions:
c. Enter the interface IDs.
d. Choose the interface policy group that should be associated to these ports.
Click Submit.
Create Interface Profile for Port Channel
If a server has two or more up

links to a leaf switch, the links can be bun
dled into a Port
Chan
nel for re
siliency and load dis
tri
bu
t
ion. In order to con
fig
ure this in ACI, cre
ate an
in
ter
face pol
icy group of type Port Chan
nel to bun
dle the in
ter
faces. Dif
fer
ent Port
Chan
nels re
quire dif
fer
ent pol
icy groups.
Fabric Connectivity 101
Port Channel Policy Group

1
Click Submit.
Next, cre
ate an in
ter
face port se
lec
tor. Since you will be con
fig
ur
ing a Port Chan
nel,
the op
er
a
tor will add all of the in
ter
faces re
quired in the Port Chan
nel in
ter
face. In this
ex
am
ple in
ter
faces Eth
er
net 1/1-2 will be con
fig
ured in one Port Chan
nel and in
ter
faces Eth
er
net 1/3-4 will be con
fig
ured in an
other Port Chan
nel.
1

c. Enter interface IDs for the first port channel.
d. Choose the interface policy group.
Click Submit.
Repeat this process for the second Port Channel (if you have another Port
Channel to add).
Create an Interface Profile for Virtual Port Channel
A vPC do
main is al
ways made up of two leaf switches, and a leaf switch can only be a
mem
ber of one vPC do
main. In ACI, that means that the de
f
i
n
i
tion of the poli
cies is sig
nif
i
cant be
tween the two switches. The same pol
icy can be reused be
tweeen the two
switches, and through the vPC do
main the pair as
so
ci
a
tion can be de
fined. vPC Switch
do
main mem
bers should be taken into con
sid
er
a
tion when con
fig
ur
ing firmware main
te
nance groups. By keep
ing this in mind, firmware up
grades should never im
pact both
vPC switch peers at the same time. More de
tails on this can be found in the Up
grad
ing
and Down
grad
ing Firmware sec
tion.
For this rea
son, a switch pro
file that would rep
re
sent two sep
a
rate switch IDs needs to
be cre
ated. The re
la
tion
ship of these switches to the two ports in the same pol
icy
group is de
fined through the in
ter
face pro
file.
vPC Policy Group
The same process would have to be re

peated for every grouped in
ter
face on each side
that will be a mem
ber of the vPC.
1
Click Submit.

7
8
c. Enter interface IDs.
d. Select of the interface policy group to be used for the vPC port behavior.
Click Submit.
Create a vPC Domain for Virtual Port Channel
When con
fig
ur
ing a vPC, there is one ad
di
tional step to be con
fig
ured once to put two
leaf switches into the same vPC do
main.
Creating a vPC Domain

1
In the Navigation pane, choose Switch Policies > VPC Domain > Virual Port
Channel default.
In the Work pane, choose Actions > Explicit VPC Protection Group.
In the Explicit VPC Protection Group dialog box, perform the following actions:
a. Define a meaningful name for the vPC domain.
b. Provide a unique ID to represent the vPC domain.
c. Choose the first switch you want to be part of the vPC domain.
d. Choose the second switch you want to be part of the vPC domain.
Click Submit.
Switch Profiles
A switch pro
file groups all the in
ter
face pro
files that de
fine the be
hav
ior of its re
spec
tive switch ports. A switch pro
file could be the de
f
in
i
t
ion of a sin
gle switch or it could
be the de
f
in
i
t
ion of mul
ti
ple switches. As a best prac
tice, there should be a switch pro
file for each leaf switch, and an ad
di
tional switch pro
file for each vPC do
main pair of
leaf switches.
The in
ter
face pro
files that you have cre
ated can be as
so
ci
ated to the switch through a
sin
gle switch pro
file or they can be as
so
ci
ated through dif
fer
ent switch pro
files. If you
have var
io
us racks that are iden
ti
cal in the way the in
ter
face ports are con
fig
ured, it
could be ben
e
fi
cial to uti
lize the same switch pro
file. This would make it pos
si
ble to
mod
ify the con
fig
ur
a
tion of many switches dur
ing op
er
at
ions with
out hav
ing to con
fig
ure each switch in
di
vid
ua
lly.
Reusability
The ca
pa
bil
ity of pol
icy reusabil
ity is cru
cial to re-em
pha
size from an op
er
at
ional per
spec
tive. If a pro
file has been de
fined to con
fig
ure a port as 1GB speed for ex
am
ple, that
pro
file can be reused for many in
ter
face pol
icy groups. When look
ing at whole switch
con
fig
ur
a
tions, the re-us
abil
ity of the pro
file can be ex
tended to sim
plify data cen
ter
op
er
at
ions and en
sure com
pli
ance. The fol
low
ing fig
ure il
lus
trates the reusabil
ity of
pro
files across racks of switches.
Policy re-use at scale

In the pre
vi
ous di
a
gram, each of the top of rack switches is based on the same switch
pro
file. If all these racks are con
fig
ured in the same fash
ion (mean
ing they are wired in
the same way) the same poli
cies could be reused by sim
ply as
sign
ing the switches to
the same switch pro
file. It would then in
herit the pro
file tree and be con
fig
ured the
exact same way as the other racks.
It is also im
por
tant to be aware of the im
pli
ca
tion of delet
ing pro
files. If a pro
file has
been reused across many de
vices, make sure to check where it is being used be
fore you
delete the pro
file or pol
icy.
Sample vPC Creation

The fol
low
ing pro
ce
dure demon
strates what a vPC bringup looks like and also API
POST con
fig
u
ra
tion as
s
es
ment of the vPC. The fol
low
ing topol
ogy will be con
fig
ured:
Sample Topology
Create VLAN Pools

1
In the Navigation pane, choose Pools > VLAN.
In the Work pane, choose Actions > Create VLAN Pool.
In the Create VLAN Pool dialog box, perform the following actions:
a. Define a meaningful name for the pool.
b. Optionally, provide a description for the pool.
c. Click Static Allocation for the allocation mode.
Note: For this example the pool will be from VLAN 100 to VLAN 199.
REST :: /api/node/class/fvnsVlanInstP.xml
CLI :: moquery -c fvnsVlanInstP
Create a Physical Domain

1
In the Navigation pane, choose Physical and External Domains > Physical
Domains.
3
4
In the Work pane, choose Actions > Create Physical Domain.

In the Create Physical Domain dialog box, perform the following actions:
a. Define a meaningful name for the domain.
b. Choose the VLAN pool that you previously created.
REST :: /api/node/class/physDomP.xml
CLI :: moquery -c physDomP
Create Access Entity Profile

1
In the Navigation pane, choose Global Policies > Attachable Access Entity
Profles.
3
4
In the Work pane, choose Actions > Create Attached Entity Profile.
In the Create Attached Entity Profile dialog box, perform the following actions:
a. Define a meaningful name for the AEP.
b. Provide a unique ID to represent the AEP.
c. Click + to add a domain that will be associated to the interfaces.
d. Choose the physical domain that you previously created.
Click Next.
Click Submit.
REST :: /api/node/class/infraAttEntityP.xml
CLI :: moquery -c infraAttEntityP
Interface Policies
Create Link Level Policy
In the Navigation pane, choose Interface Policies > Policies > Link Level.
In the Work pane, choose Actions > Create Link Level Policy.
In the Create Link Level Policy dialog box, perform the following actions:
a. Define a meaningful name for the pool.
c. Choose the auto negotiation for the interface.
d. Choose the speed to match the interface requirement.
Click Submit.
REST :: /api/node/class/fabricHIfPol.xml
CLI :: moquery -c fabricHIfPol
Create LACP Policy
In the Navigation pane, choose Interface Policies > Policies > LACP.
In the Work pane, choose Actions > Create LACP Policy.
In the Create LACP Policy dialog box, perform the following actions:
b. Optionally, provide a description for the pool.
c. In mode click on Active.
Click Submit.
REST :: /api/node/class/lacpLagPol.xml
CLI :: moquery -c lacpLagPol
Create vPC Interface Policy Group
In the Work pane, choose Actions > Create vPC Interface Policy Group.
In the Create vPC Interface Policy Group dialog box, perform the following
actions:
c. Choose a Link Level Policy.
d. Choose an LACP Policy.
Note: LACP is recommended for vPCs. However, ensure LACP is configured
on the device connected to the leaf switch.
e. Choose an AEP to associate the policy group to.
Click Submit.
REST :: /api/node/class/infraAccBndlGrp.xml
CLI :: moquery -c infraAccBndlGrp
Create an Interface Profile

1
Click Submit.
In the Navigation pane, choose Interface Policies > Profiles > Profiles > ACIVPC-int-profile.
7
8
b. Optionally, provide a description.
c. Select the proper interfaces.
d. Select an interface policy group.
Click Submit.
REST :: /api/node/class/infraAccPortP.xml
CLI :: moquery -c infraAccPortP
Create a Switch Profile

1
In the Navigation pane, choose Switch Policies > Profiles.
In the Work pane, choose Actions > Create Switch Profile.
In the Create Switch Profile dialog box, perform the following actions:
c. In switch selectors click on the + symbol.
i. Name: 103-104 (example node numbers).
ii. Blocks: Select switch 103 and switch 104.
Click Next.
Select the previously created interface selector.
Click Finish.
REST :: /api/node/class/infraNodeP.xml
CLI :: moquery -c infraNodeP
Create vPC domain

1
In the Navigation pane, choose Switch Policies > VPC Domain > Virual Port
Channel default.
In the Work pane, choose Actions > Explicit VPC Protection Group.
a. In the Explicit VPC Protection Group dialog box, perform the following
actions:
b. In the Explicit VPC Protection Groups section, click + to create a vPC
protection group.
c. Define a meaningful name for the vPC domain.
d. Provide a unique ID for the vPC domain.
e. Select the first switch ID that is part of this vPC pair: 103.
f. Select the second switch ID that is part of this vPC pair: 104.
Click Submit.
REST :: /api/node/class/fabricExplicitGEp.xml
CLI :: moquery -c fabricExplicitGEp
Validate Operation of Configured vPC

1
On the menu bar, choose Fabric > Inventory.
In the Navigation pane, choose POD 1 > Interfaces > vPC Interfaces.
In the Work pane, there will be a table that shows the status of the vPC
interface. If configured correctly, the status should be displayed and you should
see successful establishment of the vPC domain.
You can also val

id
ate the op
er
at
ion of the vPC di
rectly from the CLI of the switch it
self.
If you con
nect to the con
sole or the out of band man
age
ment in
ter
face of the leaf node
you should be able to see the sta
tus with the com
mand show vpc.
Leaf-3# show vpc
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id
: 100
vPC keep-alive status
: Disabled
Peer status
: peer adjacency formed ok
Configuration consistency status

Per-vlan consistency status
Type-2 consistency status
: success
: success
: success
vPC role
: primary
Number of vPCs configured
: 1
Peer Gateway
: Disabled
Dual-active excluded VLANs
: -
Graceful Consistency Check
: Enabled
Auto-recovery status
: Enabled (timeout = 240 seconds)
Operational Layer3 Peer
: Disabled
vPC Peer-link status
--------------------------------------------------------------------id
-1
Port
----
Status Active vlans
------ -------------------------------------------------up
vPC status
---------------------------------------------------------------------id
Port
Status Consistency Reason
Active vlans
Po1
up
--
----
------ ----------- -----success
success
------------
The fol
low
ing REST API call can be used to build vPCs and at
tach vPCs to sta
tic port
bind
ings.
URL: https://{{apic-ip}}/api/policymgr/mo/.xml
<polUni>
<infraInfra>

<infraNodeP name="switchProfileforVPC_201">
<infraLeafS name="switchProfileforVPC_201" type="range">

<infraNodeBlk name="nodeBlk" from_="201" to_="201"/>
</infraLeafS>
<infraRsAccPortP tDn="uni/infra/accportprof-intProfileforVPC_201"/>
</infraNodeP>
<infraNodeP name="switchProfileforVPC_202">
<infraLeafS name="switchProfileforVPC_202" type="range">

<infraNodeBlk name="nodeBlk" from_="202" to_="202"/>
</infraLeafS>
<infraRsAccPortP tDn="uni/infra/accportprof-intProfileforVPC_202"/>
</infraNodeP>

<infraAccPortP name="intProfileforVPC_201">
<infraHPortS name="vpc201-202" type="range">
<infraPortBlk name="vpcPort1-15" fromCard="1" toCard="1" fromPort="15"
toPort="15"/>
<infraRsAccBaseGrp tDn="uni/infra/funcprof/accbundle-intPolicyGroupforVPC"/>
</infraHPortS>
</infraAccPortP>
<infraAccPortP name="intProfileforVPC_202">
<infraHPortS name="vpc201-202" type="range">
<infraPortBlk name="vpcPort1-1" fromCard="1" toCard="1" fromPort="1"
toPort="1"/>
<infraRsAccBaseGrp tDn="uni/infra/funcprof/accbundle-intPolicyGroupforVPC"/>
</infraHPortS>
</infraAccPortP>

<infraFuncP>
<infraAccBndlGrp name="intPolicyGroupforVPC" lagT="node">
<infraRsAttEntP tDn="uni/infra/attentp-AttEntityProfileforCisco"/>
<infraRsCdpIfPol tnCdpIfPolName="CDP_ON" />
<infraRsLacpPol tnLacpLagPolName="LACP_ACTIVE" />

<infraRsHIfPol tnFabricHIfPolName="10GigAuto" />
</infraAccBndlGrp>
</infraFuncP>
</infraInfra>
</polUni>
https://{{hostName}}/api/node/mo/uni.xml
<polUni>
<fvTenant descr="" dn="uni/tn-Cisco" name="Cisco" ownerKey="" ownerTag="">

<fvAp descr="" name="CCO" ownerKey="" ownerTag="" prio="unspecified">
<fvAEPg descr="" matchT="AtleastOne" name="Web" prio="unspecified">

<fvRsPathAtt encap="vlan-1201" instrImedcy="immediate" mode="native"
tDn="topology/pod-1/protpaths-201-202/pathep-[vpc201-202] />
</fvAEPg>
<fvAEPg descr="" matchT="AtleastOne" name="App" prio="unspecified">
tDn="topology/pod-1/protpaths-201-202/pathep-[vpc201-202] />
</fvAEPg>
</fvAp>
</fvTenant>
</polUni>
Server Connectivity
Server con
nec
tiv
ity is nec
es
sary for all ap
pli
ca
tion work
loads to func
tion prop
erly
on the Cisco Ap
pli
ca
tion Cen
tric In
fra
struc
ture (ACI) fab
ric. The fab
ric con
nec
tiv
ity re
quire
ments that are dic
tated by the server in
fra
struc
ture must be care
fully con
sid
ered. In the case of Cisco Uni
fied Com
put
ing Sys
tem (UCS), fab
ric ac
cess poli
cies must
be pro
vi
soned to match these re
quire
ments. These poli
cies are all gov
erned by in
ter
face pol
icy groups. ACME Inc. has sev
eral dif
fer
ent mod
els of servers in their data cen
ters, such as Cisco UCS B and C se
ries, as well as some third party servers that all need
to be con
nected to the ACI fab
ric.
Cisco UCS B-Series Servers

If UCS B-se
ries Fab
ric In
ter
con
nects are being con
nected to your ACI fab
ric and Cisco
UCS, the Link Level Dis
cov
ery Pro
to
col (LLDP) must be dis
abled. Cisco Dis
cov
ery Pro
to
col (CDP) must be en
abled on any ACI fab
ric in
ter
faces that are con
nect
ing to UCS Fab
ric In
ter
con
nects. You can pre-pro
vi
sion these poli
cies as part of the ini
tial man
age
ment tasks. See the Con
fig
ur
ing Man
age
ment Pro
to
cols chap
ter for more in
for
ma
tion.
When con
nect
ing UCS to the ACI fab
ric, the type of Layer 2 con
nec
tion needed on the
Fab
ric In
ter
con
nect fac
ing ports must be de
ter
mined first. A best prac
tice is to lever
age
a vir
tual pri
vate cloud (vPC) to con
nect the UCS en
vi
ron
ment so as to cre
ate a multichas
sis ether
chan
nel. In this sce
nario, in
di
vid
ual link and fab
ric switch fail
ures are mit
i-
gated to main
tain a higher ex
pected up time.
For more in
for
ma
tion on the process needed to con
fig
ure links to UCS as ei
ther a vPC
or a tra
di
tional port chan
nel, see the Adding New De
vices to the Fab
ric sec
tion.
Standalone Rack Mount Servers or Non-Cisco Servers

Any non-UCS server ar
chi
tec
ture can also be con
nected di
rectly to the ACI fab
ric or to a
Cisco Nexus 2000 Fab
ric Ex
ten
der (FEX). When being con
nected to the ACI fab
ric, the kind
of traf
fic ex
pected out of the server links needs to be de
ter
mined. If the work
load is a
bare metal server, traf
fic can be clas
si
fied on a per port basis and as
so
ci
ated AEPs and
EPGs can be mapped ap
pro
pri
ately to match the en
cap
su
lated traf
fic. If a sup
ported hy
-
per
vi
sor is to be used, a Vir
tual Ma
chine Man
ager (VMM) do
main must be prop
erly con
fig
ured, and then as
so
ci
ated with the cor
re
spond
ing ports on the fab
ric as a hy
per
vi
sor that
is fac
ing through EPG and AEP map
ping. The key is to map the ex
pected traf
fic clas
si
fi
ca
tion to the ports that are con
nected to the server in
fra
struc
ture.
Uti
liz
ing a FEX is an al
ter
na
tive way to con
nect host de
vices into the ACI fab
ric. Re
stric
tions that are pre
sent in NX-OS mode such that non-host-fac
ing ports are not
sup
ported, are still true. Ports must only be con
nected to hosts, and con
nec
tiv
ity to any
other net
work de
vice will not func
tion prop
erly. When uti
liz
ing a FEX, all host-fac
ing
ports are treated the same way as if they were di
rectly at
tached to the ACI fab
ric.
Virtual Machine Networking

Understanding VM Networking in ACI
One of the most com
mon uses of the Cisco Ap
pli
ca
tion Cen
tric In
fra
struc
ture (ACI) will
be to help man
age and de
ploy ap
pli
ca
tions in vir
tual en
vi
ron
ments. The ACI pro
vides
the abil
ity to man
age both vir
tual and phys
ic
al end
points with the same set of poli
cies.
This chap
ter will look at var
io
us op
er
at
ional tasks that will be per
formed through
out
the daily op
er
at
ions.
The fol
low
ing list de
scribes some vir
tual ma
chine man
ager (VMM) sys
tem terms:
A virtual machine controller is an external VMM entity, such as VMware

vCenter, VMware vShield, and Microsoft Systems Center Virtual Machine
Manager (SCVMM). The Application Policy Infrastructure Controller
(APIC) communicates with the VMM to publish network policies that are
applied to virtual workloads. A virtual machine controller administrator
provides an APIC administrator with a virtual machine controller authentication
credential; multiple controllers of the same type can use the same credential.
Credentials represent the authentication credentials to communicate with

virtual machine controllers. Multiple controllers can use the same credentials.
A pool represents a range of traffic encapsulation identifiers, such as VLAN

and VXLAN IDs, and multicast addresses. A pool is a shared resource and can be
consumed by multiple domains, such as VMM and Layer 4 to Layer 7 services. A
leaf switch does not support overlapping VLAN pools. Different overlapping
VLAN pools must not be associated with the same attachable entity profile
(AEP).
The two types of VLAN-based pools are as follows:

Dynamic pools - Managed internally by the APIC to allocate VLANs for
endpoint groups (EPGs). A VMware vCenter domain can associate only to a
dynamic pool. This is the pool type that is required for VMM integration.
Static pools - The EPG has a relation to the domain, and the domain has a
relation to the pool. The pool contains a range of encapsulated VLANs and
VXLANs. For static EPG deployment, the user defines the interface and the
encapsulation. The encapsulation must be within the range of a pool that is
associated with a domain with which the EPG is associated.
When cre
at
ing dy
namic VLAN pools for VMM in
te
gra
tion, the VLAN range must also be
cre
ated on any in
ter
me
di
ate de
vices, such as tra
di
tional switches or blade switches.
This in
cludes cre
at
ing the VLANs on Uni
fied Com
put
ing Sys
tem (UCS).

For de
tailed in
for
ma
tion on how to de
ploy the VMware vSphere Dis
trib
uted Switch
with the Cisco APIC, see the fol
low
ing doc
u
ments:
Cisco APIC Getting Started Guide
VMware Integration
When in
te
grat
ing ACI into your VMware in
fra
struc
ture you have two op
tions for de
ploy
ing net
work
ing. VMware do
mains can be de
ployed, lever
ag
ing the VMware
vSphere Dis
trib
uted Vir
tual Switch (DVS) or the Cisco Ap
pli
ca
tion Vir
tual Switch (AVS).
Both pro
vide sim
i
lar basic vir
tual net
work
ing func
tion
al
ity; how
ever, the AVS pro
vides
ad
di
tional ca
pa
bil
i
ties, such as VXLAN and mi
croseg
men
ta
tion sup
port. ACME Inc. has
cho
sen to lever
age the ad
di
tional fea
tures pro
vided by AVS. For or
ga
ni
za
tions in
ter
ested in using the stan
dard DVS pro
vided by VMware, please refer to the fol
low
ing doc
u
ments:
http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
getting-started/
video/
cisco_
apic_
create_
vcenter_
domain_
profile_
using_
gui.
html

Shown below are some of the var
i
ous ACI poli
cies which are in
volved with set
ting up
VM In
te
gra
tion. This serves as a ref
er
ence for the ways the var
i
ous poli
cies are re
lated
to each other.
Publishing EPGs to a VMM Domain

This sec
tion will de
tail how to pub
lish an ex
ist
ing end
point group (EPG) to a Vir
tual Ma
chine Man
ager (VMM) do
main. For de
tails on how to cre
ate EPGs, see the Ten
ants sec
tion.
For an EPG to be pushed to a VMM do
main, a do
main bind
ing within the ten
ant EPG
must be cre
ated.
1
On the menu bar, choose Tenants > ALL TENANTS
In the Work pane, choose the Tenant_Name.
In the Navigation pane, choose Tenant_Name > Application

Profiles > Application_Profile_Name > Application
EPGs > Application_EPG_Name > Domains (VMs and Bare-Metals).
In the Work pane, choose Actions > Add VM Domain Association.
In the Add VM Domain Association dialog box, choose the VMM Domain Profile
that you previously created.
a. For the Deployment & Resolution Immediacy, Cisco recommends keeping
the default option of On Demand. This provides the best resource usage in
the fabric by only deploying policies to Leaf nodes when endpoints assigned
to this EPG are connected. There is no communication delay or traffic loss
by keeping the default selections.
Click Submit.
Note: The EPG will now be available as a Port Group to your VMM.
Connecting VMs to the EPG Port Groups on vCenter

1
Connect to your vCenter using the VI Client.
From the Host and Clusters view, right click on your Virtual Machine and select
"Edit Settings".
Click on the Network Adapter, and in the Network Connection dropdown

box select the Port Group which corresponds to your EPG. It should display in
the format of
[TENANT|APPLICATION_PROFILE|EPG|VMM_DOMAIN_PROFILE]
If you do not see your ACI EPG in the Net

work Con
nec
tion list, it means one of the fol
low
ing:
The VM is running on a host which is not attached to the Distributed Switch
There may be a communication between your APIC and vCenter either through
managed by the APIC.

the OOB or INB management network.
Verifying Virtual Endpoint Learning

Once the VMs are con
nected to the ap
pro
pri
ate port group/EPG, you should ver
ify the
APIC has learned your vir
tual end point.
1
On the menu bar, choose Tenants > ALL TENANTS.
In the Navigation pane, choose Tenant_Name > Application

Profiles > Application_Profile_Name > Application
EPGs > Application_EPG_Name.

Note: The current tab should display CLIENT ENDPOINTS. All endpoints either
virtual or physical will be displayed. From here you should be able to find your
Virtual Machine by filtering the "Learning Source" column for rows with values
of "Learned VMM".
Verifying VM Endpoint Learning on the APIC from the CLI

You can ver
ify the same info as above from the CLI by using the 'mo
query' (Man
aged Ob
ject Query) com
mand and adding two fil
ters. One for the Dis
tin
guished Name (DN) name
of your EPG, and one for the Class Name of 'fvCEp' (Fab
ric Vec
tor Client End
point)
moquery -c fvCEp --dn uni/tn-<TENANT_NAME>/ap-<APP_PROFILE_NAME>/epg-<EPG_NAME>
You can de
ter
mine the DN of your EPG by right click
ing on the EPG in the GUI, se
lect
ing "Save As" and look
ing at the XML ob
ject. From this file you will see the DN entry for
the par
tic
ul
ar EPG:
<imdata totalCount="1"><fvAEPg uid="15374" triggerSt="triggerable" status=""

scope="2588672" prio="unspecified" pcTag="49159" name="epg-od"
monPolDn="uni/tn-common/monepg-default" modTs="2015-02-06T06:46:24.729+11:00"
matchT="AtleastOne" lcOwn="local" dn="uni/tn-mb-tennant1/ap-mb-app-pro/epg-epg-od"
descr="" configSt="applied" configIssues="" childAction=""/></imdata>
Next, use this DN with the mo

query to re
turn the list of client En
points for this EPG:
admin@apic1:~> moquery -c fvCEp --dn uni/tn-mb-tennant1/ap-mb-app-pro/epg-epg-od
Total Objects shown: 1
# fv.CEp
name
: 00:50:56:BB:8C:6A
childAction
dn
: uni/tn-mb-tennant1/ap-mb-app-pro/epg-epg-od/cep-00:50:56:BB:8C:6A
encap
: vlan-211
id
: 0
idepdn
ip
: 10.10.10.10
lcC
: learned,vmm
lcOwn
: local
mac
: 00:50:56:BB:8C:6A
mcastAddr
: not-applicable
modTs
: 2015-02-06T06:48:52.229+11:00
rn
: cep-00:50:56:BB:8C:6A
status
uid
: 0
uuid
VMware Integration Use Case

A VMWare ad
min
is
tra
tor in ACME re
quests the net
work team to trunk a set of VLANs
down to the ESX hosts to pro
vide con
nec
tiv
ity to their DVS switches. Rather than
trunk
ing VLANs on a per server basis, the net
work team de
cides to lever
age a new
method
ol
ogy to be more agile and lever
age the on-de
mand pro
vi
sion
ing of re
sources
where and when they are needed, as well as pro
vid
ing un
lim
ited Layer 2 mo
bil
ity for all
the VM hosts within the ACI fab
ric.
To do so, the net

work ad
mins work with the VMware ad
mins to de
cide on a range of
VLANs that will be pro
vided dy
nam
ic
ally by APIC to the ESX hosts that need them. They
de
cide on an un
used VLAN range of (600 - 800). This is their dy
namic VLAN pool. Once
this is de
cided, the APIC ad
min
is
tra
tor pro
ceeds to con
fig
ure VMM in
te
gra
tion in the
APIC GUI by pro
vid
ing the vCen
ter cre
den
tials to APIC. APIC dy
nam
ic
ally pro
vi
sions all
EPGs and makes them avail
able to the ESX hosts as a port-group.
Note: The APIC does not au
to
mat
ic
ally move VM
NICs into the port-group. This al
lows
VMware ad
mins to main
tain con
trol and move vir
tual NICs into these port-groups on
de
mand.
As the VMware admin pro
vi
sions ESX hosts and se
lects the ap
pro
pri
ate port-groups for
VMs, the APIC dy
nam
ic
ally com
mu
ni
cates with vCen
ter to make EPGs avail
able through
port groups. The APIC also con
fig
ures VLAN IDs on the leaf-switches as needed.
Dur
ing a vMo
tion event, APIC is au
to
mat
ic
ally in
formed of the VM move and then up
dates the end
point track
ing table to allow seam
less com
mu
ni
ca
tion. VMs are al
lowed to
move any
where within the ACI fab
ric with no re
stric
tions other than those im
posed by
vCen
ter.
It is im
por
tant to note that ACME can still choose to de
ploy tra
di
tional VLAN trunk
ing
down to VMware DVS switches by sta
t
i
cally pro
vi
sion
ing EPGs on a per-port basis, and
still reap the ad
van
tages of the Layer 2-any
where ACI fab
ric. How
ever, ACME chose VMM
in
te
gra
tion as the pre
ferred de
ploy
ment model as it is the most ef
fec
tive method of
break
ing down or
ga
ni
za
tional chal
lenges, doing on-de
mand re
source al
lo
ca
tion, and get
ting en
hanced vis
i
bil
ity and teleme
try into both the vir
tual and phys
i
cal en
vi
ron
ments.
Deploying the Application Virtual Switch

Prerequisites
All switch nodes have been discovered by the fabric
INB or OOB management connectivity is configured.
VMware vCenter is installed, configured, and available.
One or more vSphere hosts are available for deployment to the AVS.
A DNS server policy has been configured to enable connection to a VMM using
a hostname.
A dynamic VLAN pool has been created with enough VLAN IDs to accommodate
one VLAN per EPG you plan on deploying to each VMM domain.
Getting Started
The AVS soft
ware was de
signed to op
er
ate in
de
pen
dently of the APIC soft
ware ver
sion.
This al
lows ei
ther de
vice to be up
graded in
de
pen
dently. Al
ways refer to the AVS re
lease notes to con
firm if any spe
cial con
sid
er
at
ions may exist.
Just like any soft
ware, new ver
sions of the AVS will be re
leased to in
clude new fea
tures
and im
prove
ments. The ini
tial AVS soft
ware re
leased was ver
sion 4.2.1, fol
lowed by
tem Com
pat
i
bil
ity List doc
um
ent to en
sure your
ver
sion 5.2.1. Refer to the ACI Ecosys
de
sired ver
sion of AVS is com
pat
ib
le with the APIC and vSphere ver
sions being run.
The AVS pack
age for ei
ther ver
sion will in
clude vSphere In
stal
la
tion Bun
dles (VIBs).
Each ver
sion of AVS soft
ware in
cludes the VIB files for all sup
ported vSphere ver
sions.
As of this pub
li
ca
tion there are two VIBs to sup
port vSphere ver
sions 5.1 and 5.5
(vSphere 5.0 is not sup
ported). These can be down
loaded from CCO at the fol
low
ing
lo
ca
tion:
Down
loads Home > Prod
ucts > Switches > Vir
tual Net
work
ing > Ap
pli
ca
tion Vir
tual
Switch
The VIBs can be iden

ti
fied as fol
lows:
AVS 4.2.1 Bundle
cross_cisco-vem-v165-4.2.1.2.2.3.0-3.1.1.vib
AVS 5.2.1 Bundle
Install the AVS VIB

Be
fore set
ting up the AVS con
fig
ur
a
tion on the APIC, the AVS soft
ware must be in
stalled in vSphere, re
ferred to as the Vir
tual Eth
er
net Mod
ule (VEM). This can be
achieved in a va
ri
ety of ways, all of which are dis
cussed in Cisco Ap
pli
ca
tion Vir
tual
stal
la
tion Guide. For a few hosts, this can eas
ily be done man
ua
lly, but for 10+
Switch In
hosts it may be eas
ier to lever
age the Vir
tual Switch Up
date Man
ager (VSUM) to help
au
to
mate the in
stal
la
tion process.
Manual Installation
1
Copy the VIB file to a host. The easiest way to copy the VIB to the host is to
leverage the VMware VI Client, navigate to the Host > Configuration > Storage
> Datastore_X. Right click on the desired datastore and select Browse. From
here,VIB can be uploaded directly to the host's datastore.
SSH into the vSphere host on which the AVS VIB is to be installed.
Install or upgrade the VIB using the esxcli command:

To in
stall the AVS VIB:
esxcli software vib install -v /<path>/<vibname> --maintenance-mode --no-sig-check
To Up
grade an ex
ist
ing AVS VIB:
esxcli software vib update -v /<path>/<vibname> --maintenance-mode --no-sig-check
A sam
ple out
put is shown below:
# esxcli software vib install -v /vmfs/volumes/datastore1/cross_cisco-vem-v1725.2.1.3.1.3.0-3.2.1.vib --maintenance-mode --no-sig-check
Installation Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed: Cisco_bootbank_cisco-vem-v172-5.2.1.3.1.3.0-3.2.1
VIBs Removed:
VIBs Skipped:
/vmfs/volumes/53cab6da-55209af3-0ef2-24e9b391de3e # vem version
Running esx version -1623387 x86_64
VEM Version: 5.2.1.3.1.3.0-3.2.1
VSM Version:
System Version: VMware ESXi 5.5.0 Releasebuild-1623387
Confirm the VEM is loaded and running.
# vem status
VEM modules are loaded
Switch Name
Num Ports
Used Ports
Configured Ports
MTU
Uplinks
vSwitch0
3072
128
1500
VMNIC0
VEM Agent (vemdpa) is running
DHCP Relay
The first task is to cre
ate a DHCP relay pol
icy in the infra ten
ant on the APIC. This will
allow the AVS to cre
ate a vmk Vir
tual Tun
nel End
point (VTEP) in
ter
face on each host.
This VTEP in
ter
face will be used for the Open
Flex con
trol chan
nel and/or the VXLAN
tun
nel source be
tween the VTEP and ACI fab
ric.
If the de
fault TEP Ad
dress pool was used dur
ing ini
tial setup (10.
0.
0.
0/
16), this will
allow the AVS VTEP in
ter
faces to pull an ad
dress from this pool. The way APIC would
do it is through over the infra vlan. For this rea
son it is crit
ic
al to ex
tend the Infra
VLAN from a leaf through to each vSphere host that will host the AVS.
Create the Infra DHCP Relay Policy

Cre
ate a DHCP relay pol
icy that will setup the DHCP server (the APIC in this case) and a
label using the Infra Ten
ant and de
fault EPG. The APIC IP ad
dresses au
to
mat
ic
ally
func
tion as the DHCP providers and DO NOT need to be ex
plic
itly added.
STEPS
1
In the Work pane, choose the infra.
In the Navigation pane, choose infra > Networking > Protocol Policies > DHCP
> Relay Policies.
4
5
In the Work pane, choose Actions > Create DHCP Relay Policy.
In the Create DHCP Relay Policy dialog box, perform the following actions:
a. Provide a name for the Relay policy, such as "avs-dhcp-relay-pol".
b. Leave other fields blank and click OK.
Click Submit.
Create the DHCP Relay Label

1
In the Work pane, choose the infra.
In the Navigation pane, choose infra > Networking > Bridge

Domains > default > DHCP Relay Labels.
4
5
In the Work pane, choose Actions > Create DHCP Relay Policy.
In the Create DHCP Relay Policy dialog box, perform the following actions:
a. Change the Scope to TENANT.
b. From the Name drop-down list, choose the DHCP Relay Policy that you
created previously.
Click Submit.
Attachable Access Entity Profile (AEP) and AVS

An im
por
tant com
po
nent used by the AVS is the At
tach
able En
tity Pro
file (AEP). Re
gard
less of using an ex
ist
ing AEP or cre
at
ing a new one, the En
able In
fra
struc
ture
VLAN check box must be checked for the AEP pol
icy. This is to en
sure that the traf
fic
of in
ter
est (DHCP re
quest/offer can flow through the in
fra
struc
ture VLAN to the AVS).
The AEP de
fines which VLANs will be per
mit
ted on a host fac
ing in
ter
face. This can be
com
pared to a "switch
port trunk al
lowed VLAN xxx" com
mand in tra
di
tional net
work
ing. Re
fer
ring back to the "VMM Pol
icy Model In
ter
ac
tion" di
ag
ram from the "VM Net
work
ing Overview" chap
ter, the AEP is what ties the VMM do
main to the phys
ic
al in
ter
faces where the vSphere hosts are con
nected. The AEP can be cre
ated on-the-fly dur
ing the cre
ation of the VMM do
main it
self, but this guide will de
tail cre
at
ing the AEP
sep
ar
ately first.
Create a New AEP

1
Profile.
In the Work pane, choose Actions > Create Attachable Access Entity Profile.
In the Create Attachable Access Entity Profile dialog box, perform the
following actions:
a. Fill in the AEP wizard information then click Next.
i. Name: Provide any name to identify the AEP, such as "AVS-AEP".
ii. Enable Infrastructure VLAN: Check this box.
iii. Domains (VMs or Baremetal): Leave blank. This will be covered later in the
Publishing EPGs to VMM Domains chapter.
b. From the next page of the wizard, select the Interface Policy Group your
AEP will be associated to. This procedure assumes your Interface Policy
Group has already been created. Click the All Interfaces radio button for the
desired Interface Policy Group.
Note: In
ter
face Pol
icy Group cre
ation is cov
ered else
where in this guide. Es
sen
tially
the In
ter
face Pol
icy Group is a col
lec
tion of In
ter
face poli
cies which de
fine In
ter
faces
Se
lec
tors and prop
er
ties, such as speed/ne
go
ti
at
ion, LLDP, and CDP. See the "Adding
New De
vices to the Fab
ric" chap
ter for more de
tail on cre
at
ing the in
ter
face pol
icy
group and in
ter
face pro
files.
Modify an Exisiting AEP

1
Profile.
a. In the Navigation pane, choose the existing AEP
b. In the Work pane, check the Enable Infrastructure VLAN check box.
Note: As men
tioned early in this chap
ter, the In
fra
struc
ture VLAN is re
quired for AVS
com
mu
ni
ca
tion to the fab
ric using the Open
Flex con
trol chan
nel.
VMM Domains for vCenter

A Vir
tual Ma
chine Man
ager (VMM) do
main de
fines a vir
tual in
fra
struc
ture that will be
in
te
grated into ACI. This al
lows the same poli
cies ap
plied to phys
ic
al end
points, to also
be ap
plied to vir
tual end
points. vCen
ter VMM Do
mains are cre
ated using ei
ther the
VMware DVS or Cisco AVS. You can
not change from one to the other. A new VMM Do
main will be cre
ated from scratch to sup
port AVS de
ploy
ment.
AVS Switching Modes

The AVS can op
er
ate in the fol
low
ing switch
ing modes:
Local Switching: Supports VXLAN encapsulation or VLAN encapsulation.

This switching mode allows Inter-EPG traffic to be switched locally
on the AVS.
No Local Switch only supports VLAN encapsulation.

This switching mode sends all traffic (Inter-EPG included) to the
Leaf switch.
AVS Switching Modes: Non-Local and Local switching mode

The de
ci
sion be
tween using VLAN or VXLAN en
cap
su
la
tion will man
date dif
fer
ent VLAN ex
ten
sion re
quire
ments out
side of the fab
ric. When using VXLAN en
cap
su
la
tion, only the infra VLAN is re
quired to be ex
tended to the AVS hosts. All
traf
fic be
tween the AVS up
links and ACI fab
ric will be en
cap
su
lated by VXLAN
and trans
fered using the in
fra
struc
ture VLAN.
If VLAN en
cap
su
la
tion is pre
ferred, you will need to en
sure every VLAN in the
VM Do
main VLAN pool has been ex
tended be
tween the fab
ric and AVS host.
This in
cludes cre
at
ing the VLANs on in
ter
me
di
ate de
vices such as UCS and the
vNICs for any AVS vSphere hosts.
Create the VMM Domain for AVS

Now that the DHCP server pol
icy has been cre
ated and AEP cre
ated/mod
i
fied, you can
cre
ate the VMM do
main for the AVS.
1
On the menu bar, choose VM NETWORKING.
In the Navigation pane, choose the Policies tab.
In the Work pane, choose Actions > Create VCenter Domain.
In the Create vCenter Domain dialog box, perform the following actions:
a. Name: This value will be used as the AVS "Switchname" displayed in vCenter.
b. Virtual Switch: Cisco AVS
c. Switching Preference: <Choose Local or No Local Switching>
For No Local Switching mode:

Multicast Address: <Assign a multicast address to represent your AVS>
Multicast Address Pool: <Create a unique Multicast Address Pool large
enough to include each AVS vSphere host.>
For Local Switching mode:

Encapsulation: <Choose VLAN or VXLAN based on preference>
For VLAN Encapsulation
VLAN Pool: <Choose/Create a VLAN pool>
For VXLAN Encapsulation
Multicast Address: <Assign a multicast address to represent your
AVS>
Multicast Address Pool: <Create a unique Multicast Address Pool
large enough to include each AVS vSphere host.>
d. Attachable Access Entity Profile: <Choose the AEP previously

created/modified>
e. vCenter Credentials: Create a credential set with administrator/root acces
to vCenter
f. vCenter: Add the vCenter details.
Name: Friendly name for this vCenter
Hostname/IP Address: <DNS or IP Address of vCenter>
DVS Version: vCenter Default
Datacenter: <Enter the exact Datacenter name displayed in vCenter>
Management EPG: <Set to oob or inb Management EPG>
Associated Credentials: <Choose the Credential set previously created>
Click OK to complete the creation of the vCenter.
Click Submit.
Verify AVS Deployment on vCenter

1
In the vCenter client, navigate to HOME > INVENTORY > NETWORKING and
confirm a new Distributed Virtual Switch folder has been created.
Expand this folder to find your AVS, and a few default Port Groups including
"uplink" and "vtep".
Add vSphere Hosts to the AVS

After the AVS has been cre
ated in vCen
ter, you then need to at
tach hosts to it. To do
this you will need at least one un
used phys
ic
al in
ter
face (VMNIC) to act as the up
link on
each host. AVS up
links can not be shared with any other ex
ist
ing vSwitch or vDS.
1
From the vCenter client, navigate to HOME > INVENTORY > NETWORKING.
Right click on the newly created AVS switch (not the folder) and choose Add
Host....
In the Add Host dialog box, choose any vSphere hosts to add to the AVS, and
select an unassigned VMNIC uplink.
Click Next until the wizard completes, skipping the migration of any virtual
adapters or virtual machine networking at this time.
Note: For blade switch sys

tems such as UCS, the VMNIC in
ter
face used must have all
nec
es
sary VLANs al
lowed on the in
ter
face. In UCS terms, this re
quires the vNIC within
the ser
vice pro
file to have all rel
e
vant VLANs ac
tive on the vNICs.
Adding Virtual host physical NICs to participate in Virtual Switch

5
Assuming the ACI fabric can reach the vSphere host over the infra VLAN, you
should see a new vmk interface created on your distributed switch within
vCenter and assigned to the 'vtep' port group. This vmk is your Virtual Tunnel
Endpoint (vtep) interface and should have pulled a DHCP address from the APIC
from the TEP subnet. As can be seen from the screenshot below, we see that
the VMkernel port has received the IP address from the APIC. The APIC uses
the same 10.0.0.0/16 pool that is created during the APIC setup to provision the
IP address. This implies that we are ready for Opflex communication in
between the AVS and the APIC.
~ # esxcfg-VMKNIC -l
Interface
Port Group/DVPort
Netmask
Broadcast
vmk0
Management Network
255.255.255.0
vmk1
255.255.255.0
vmk2
255.255.0.0
172.16.176.255
vmotion
192.168.99.255
9
10.0.255.255
IP Family IP Address
MAC Address
IPv4
MTU
TSO MSS
Enabled Type
65535
true
STATIC
true
STATIC
true
DHCP
172.16.176.54
00:25:b5:00:00:29 1500
IPv4
192.168.99.54
00:50:56:61:1c:92 1500
IPv4
65535
10.0.16.95
00:50:56:65:3d:b3 1500
65535
Verify AVS on ESX

On the ESX com
mand line, issue the vem
cmd show open
flex com
mand.
Ver
ify that the sta
tus: 12 (Ac
tive) is seen as well as the switch
ing mode. Also ver
ify that
the GIPO ad
dress is the same as the mul
ti
cast ad
dress that was used while cre
at
ing the
VMM do
main.
~ # vemcmd show openflex
Status: 12 (Active)
Dvs name: comp/prov-VMware/ctrlr-[AVS-TEST]-VC/sw-dvs-87
Remote IP: 10.0.0.30 Port: 8000
Infra vlan: 4093
FTEP IP: 10.0.0.32
Switching Mode: LS
NS GIPO: 225.127.1.1
Ver
ify on the AVS host - there should be one mul
ti
cast group per de
ployed EPG on the
host. In the out
put below, there are three dif
fer
ent Vir
tual Ma
chines con
nected to dif
fer
ent EPGs.
~ # vemcmd show epp multicast
Number of Group Additions 3
Number of Group Deletions 0
Multicast Address
EPP Ref Count
225.0.0.58
225.0.0.76
225.0.0.92
These mul
ti
cast ad
dresses will cor
re
spond to the EPG de
tails found in the APIC GUI
antX > Ap
pli
ca
tion Pro
files > Ap
pli
ca
tion
Pro
fileX > End Point
under Ten
ants > Ten
Groups > End
Point
GroupX and click the Op
er
a
tional Tab > Client End Points.
VXLAN Load Balancing

VXLAN load bal
anc
ing is au
to
mat
ic
ally en
abled as soon as more than one VMKNIC is
con
nected to the Cisco AVS. Each VMKNIC can use only one up
link port; you should
have an equal num
ber of VMKNICs and up
links. A max
im
um of eight VMKNICs can be
at
tached to a Cisco AVS switch. Each of the VMKNICs that you cre
ate has its own soft
ware-based MAC ad
dress. In VXLAN load bal
anc
ing, the VMKNICs pro
vide a unique
MAC ad
dress to pack
ets of data that can then be di
rected to use cer
tain phys
i
cal NICs
(VM
NICs).
You need to have as many VMKNICs as the host has VM
NICs, up to a max
i
mum of eight.
For ex
am
ple, if the host has five VM
NICs, you need to add four VMKNICs to en
able
VXLAN load-bal
anc
ing; the Cisco Ap
pli
ca
tion Pol
icy In
fra
struc
ture Con
troller (APIC) al
ready cre
ated one VMKNIC when the host was added to the dis
trib
uted vir
tual switch
(DVS).
In VMware vSphere Client, you will need to cre
ate an ad
di
tional vir
tual adapter (VMK)
for each AVS up
link. Each vmk in
ter
face cre
ated for the AVS should be at
tached to the
vtep port group and con
fig
ured for DHCP. In the screen
shot below you can see the
four VMNIC up
links to the AVS and the four vmk vir
tual in
ter
faces to pro
vide equal load
bal
anc
ing traf
fic across all up
links. Note that each VMK in
ter
face added is as
signed a
unique DHCP ad
dress from the fab
ric TEP pool.
Distributed Switch configured with APIC port groups
IGMP Snooping Policy for AVS

Cisco UCS-B Series Considerations with AVS Deployments
This sec
tion of the ar
ti
cle will focus the nec
es
sary steps to en
able AVS through the
Cisco UCS-B se
ries.
By de
fault, USC-B FI has IGMP snoop
ing en
abled. Due to this, we have to con
fig
ure an
IGMP querier pol
icy on the APIC. The IGMP snoop
ing pol
icy needs to be en
abled on the
infra ten
ant.
If we dis
able IGMP snoop
ing on UCS or other in
ter
me
di
ate blade switches, then IGMP
pol
icy is not needed since the blade switch will flood the mul
ti
cast traf
fic on all the rel
e
vant ports.
Create an IGMP Snooping Policy for AVS
In the Work pane, choose infra.
In the Navigation pane, choose infr > Networking > Bridge Domain > default.
a. In the IGMP Snoop Policy drop-down list, choose Create IGMP Snooping
Policy.
b. Provide a name for the policy, such as "IGMP_Infra".
c. Click the Fast Leave check box.
d. Click the Enable Querier check box.
Click Submit.
Note: Verify if IGMP snooping is working properly on the vSphere host CLI
using 'vemcmd show epp multicast' as shown above.
The al
ter
nate method would be to cre
ate an IGMP pol
icy on UCS to dis
able IGMP
snoop
ing. This will cause flood
ing of the mul
ti
cast traf
fic to all end
points.
External Connectivity
As men
tioned in the in
tro
duc
tion of this book, ACME Inc. is a multi
na
tional com
pany
with mul
ti
ple data cen
ters. There
fore, ACME Inc. must con
fig
ure some Layer 2 con
nec
tiv
ity. This is nec
es
sary for ex
tend
ing Layer 2 con
nec
tiv
ity to a Data Cen
ter In
ter
con
nect (DCI) plat
form, to fur
ther ex
tend con
nec
tiv
ity to a re
mote data cen
ter, or sim
ply
to ex
tend a Layer 2 do
main out
side of the fab
ric to con
nect in an ex
ist
ing Layer 2 net
work in a non-ACI fab
ric.
Extending Endpoint Groups Outside the ACI Fabric

The sim
plest way to ex
tend an end
point group (EPG) out
side of the ACI fab
ric is to sta
t-
i
cally as
sign a leaf port and VLAN ID to an ex
ist
ing end
point group. After doing so, all of
the traf
fic re
ceived on this leaf port with the con
fig
ured VLAN ID will be mapped to the
EPG, and as such, the con
fig
ured pol
icy for this EPG will be en
forced. The end
points do
not need to be di
rectly con
nected to the ACI leaf, as the traf
fic clas
si
fi
ca
tion will be
based on the en
cap
su
la
tion re
ceived on a port.
To as
sign a Layer 2 con
nec
tion sta
ti
c
ally on an ACI leaf port to an EPG:
1
In the Navigation pane, choose Tenant_Name > Application Profiles >

App_Profile_Name > Application EPGs > EPG_Name > Static Bindings
(Paths).
In the Work pane, choose Action > Deploy Static EPG on PC, vPC or Interface.
In the Deploy Static EPG on PC, vPC or Interface dialog box, perform the
following actions:
a. In the Path field, specify a port as well as a VLAN ID.
b. Click one of the Deployment Immediacy radio buttons. Deployment
immediacy determines when the actual configuration will be applied on the
leaf switch hardware. The immediacy also determines when the hardware
resource, such as a VLAN resource and policy content-addressable memory
(CAM) to support the related contract for this EPG, will be consumed on the
leaf switch. The option Immediate means that the EPG configuration and its
related policy configuration will be programmed in the hardware right away.
The option On Demand instructs the leaf switch to program the EPG and its
related policy in the hardware only when traffic matching this policy is
received for this EPG.
c. Click one of the Mode radio buttons. The mode option specifies whether the
ACI leaf expects incoming traffic to be tagged with a VLAN ID or not.
i. Tagged - The tagged option means that the leaf node expects incoming
traffic to be tagged with the specified VLAN ID previously established. This
is the default deployment mode. Choose this mode if the traffic from the
host is tagged with a VLAN ID. Multiple EPGs can be statically bound to
the same interface as long as the encap VLAN/VXLAN ID is unique.
ii. Untagged - The untagged option means that the leaf expects untagged
traffic without a VLAN ID. Similar to the switchport access
vlan vlan_ID command, with this option you can only assign the interface
to one EPG. This option can be used to connect a leaf port to a bare metal
server whose network interface cards (NICs) typically generate untagged
traffic. A port can have only one EPG statically bound to a port as
untagged.
iii. 802.1P - The 802.1P option refers to traffic tagged with 802.1P headers.
802.1P mode is useful when its necessary to handle the traffic on one EPG
as untagged to the interface (similar to the switchport trunk native vlan
vlan_ID command), but (unlike the untagged mode) 802.1P will allow other
'tagged' EPGs to be statically bound to the same interface. Any traffic
received on links with this mode classification will have the following
conditions applied to them:
d. Create a physical domain and VLAN pool that are associated to this physical
domain.
e. Associate the physical domain to the EPG in question.
f. Create an attachable access entity profile (AEP) to map the interfaces and
policies together.
See the Adding New De
vices to the Fab
ric sec
tion for more in
for
ma
tion on how to con
fig
ure an AEP and a phys
ic
al do
main.
Extending an ACI Bridge Domain Outside of the Fabric

A Layer 2 out
side con
nec
tion is as
so
ci
ated with a bridge do
main and is de
signed to ex
tend the whole bridge do
main, not just an in
di
vid
ual EPG under the bridge do
main, to
the out
side net
work.
To ac
com
plish an ex
ten
sion of the bridge do
main to the out
side, a Layer 2 out
side con
nec
tion must be cre
ated for the bridge do
main. Dur
ing this process, cre
ate a new ex
ter
nal EPG to clas
sify this ex
ter
nal traf
fic. This new EPG will be part of the ex
ist
ing
bridge do
main. Clas
sify any out
side con
nec
tions or end
points into this new ex
ter
nal
EPG. With two sep
ar
ate EPGs, you will also need to se
lect which traf
fic you would like
to tra
verse be
tween the two EPGs. Sim
il
ar to the pre
vi
ous ex
am
ple of adding an end
point to a pre-ex
ist
ing EPG, this method will also allow the end
points to share the same
sub
net and de
fault gate
way.
To cre
ate an ex
ter
nal Layer 2 do
main:
1
In the Navigation pane, choose Tenant_Name > Networking > External

Bridged Network.
4
5
In the Work pane, choose Action > Create Bridged Outside.

In the Create Bridged Outside dialog box, perform the following actions:
a. Associate the Layer 2 outside connection with the bridge domain and a VLAN
ID. This VLAN must be configured on the external Layer 2 network. This
Layer 2 outside connection will put this VLAN and the bridge domain of the
ACI fabric under the same Layer 2 domain. This VLAN ID must be in the range
of the VLAN pool that is used for the Layer 2 outside connection.
i. For the External Bridged Domain drop-down list, create a Layer 2 domain
if one does not already exist.
ii. While creating the Layer 2 domain, if it does not already exist, create a
VLAN pool to associate to the VLAN on the Layer 2 outside
connection. This is a means to specify the range of the VLAN IDs that will
be used for creating a Layer 2 outside connection. This helps avoid any
overlapping in the VLAN range between VLANs used for an EPG and those
in use for a Layer 2 outside connection.
b. Add a Layer 2 border leaf node and Layer 2 interface for a Layer 2 outside
connection.
c. After adding a Layer 2 border leaf and Layer 2 interface, click Next to start
creating a Layer 2 EPG. Simply provide a name for the Layer 2 EPG. All of the
traffic entering the ACI fabric with the designated VLAN (the VLAN ID
provided in step 1) will be classified into this Layer 2 EPG.
d. Configure a contract to allow communication between your existing
endpoints in the existing EPG and your new external Layer 2 EPG. In the
Navigation pane, choose External Bridged Networks > Networks and specify
a contract to govern this policy as the consumed contract. After specifying
this contract as the provided contract for your internal EPG, the
communication between this external Layer 2 EPG and your existing internal
EPG will be allowed.
e. Create an AEP. This is a policy object that tells the APIC to allow certain
encap (VLANs) on selected ports. For more information on how to create a
Access Attachable Entity Profile, see the Adding New Devices to the
Fabric section.
You should now have the de
sired reach
ab
il
ity be
tween the in
side and out
side
Layer 2 seg
ments.

The most im
por
tant com
po
nent of any ap
pli
ca
tion is the user or cus
tomer, which gen
er
ally does not di
rectly at
tach to the fab
ric, and there
fore there must be con
nected to
the ex
ter
nal net
work. ACME must be able to con
nect to both their in
ter
nal cor
po
rate
back
bone, as well as to the In
ter
net to pro
vide ac
cess to the mo
bile ap
pli
ca
tion. This
in
te
gra
tion is pos
si
ble with Cisco Ap
pli
ca
tion Cen
tric In
fra
struc
ture (ACI) at the ten
ant
pol
icy level. Layer 3 con
nec
tiv
ity to a de
vice like a router is known as an Ex
ter
nal
Routed Net
work, and pro
vides IP con
nec
tiv
ity be
tween a ten
ant pri
vate net
work and
an ex
ter
nal IP net
work. Each Layer 3 ex
ter
nal con
nec
tion is as
so
ci
ated with one ten
ant
pri
vate net
work. The re
quire
ment of the Layer 3 ex
ter
nal net
work is only needed when
a group of de
vices in the ap
pli
ca
tion pro
file re
quire Layer 3 con
nec
tiv
ity to a net
work
out
side of the ACI fab
ric.
An ap
pli
ca
tion pro
file en
ables an op
er
a
tor to group the dif
fer
ent com
po
nents, or tiers,
of an ap
pli
ca
tion into end
point groups (EPGs). These ap
pli
ca
tion com
po
nents might
have re
quire
ments for ex
ter
nal con
nec
tiv
ity into them. The fol
low
ing fig
ure shows part
of a sim
pli
fied fab
ric:
A sample application profile for a three-tiered application with contracts between

the tiers
For ex
am
ple, web servers need a con
nec
tion to the out
side world for users to reach
them. With ACI, the con
nec
tiv
ity is de
fined by a con
tract to a de
fined ex
ter
nal Layer 3
end
point group. As the op
er
a
tor of the fab
ric, you can pro
vide the ten
ant ad
min
is
tra
tor
with the abil
ity to in
ter
face to an ex
ter
nal Layer 3 con
nec
tion in var
i
ous ways by using
a uniquely-de
fined Layer 3 con
struct for the ten
ant ap
pli
ca
tion pro
file or a shared
com
mon in
fra
struc
ture.
Ex
ter
nal Layer 3 con
nec
tions are usu
ally es
tab
lished on the bor
der leaf con
struct of the
ACI. Any ACI leaf can be
come a bor
der leaf. In large scale ACI de
signs it might be pro
duc
tive to have ded
i
cated ACI leafs as bor
der leafs. It is im
por
tant to note that the
spine nodes can
not have con
nec
tions to ex
ter
nal routers. A bor
der leaf is sim
ply ter
mi
nol
ogy to refer to a leaf that hap
pens to be con
nected to a Layer 3 de
vice. Other de
vices, like servers, can still con
nect to the bor
der leaves. In the ACI fab
ric, the ex
ter
nal
Layer 3 con
nec
tion can be one of the fol
low
ing types:
1
Physical Layer 3 interface
Subinterface with 8021.Q tagging
Switched Virtual Interface (SVI)
Bridge do
mains con
tain one or more sub
nets. These sub
nets can be clas
si
fied as pri
vate, pub
lic, or shared:
Public - Indicates that this subnet will be advertised to the external router.
Private - Indicates that this subnet will be contained only within the ACI fabric
private network.
Shared - Indicates that this subnet will be leaked to one or more private
networks inside of the ACI fabric.
The fol
low
ing fig
ure de
picts the logic of pub
lic and pri
vate net
works:
Application profile with external consumers, public and private networks annotated
With de
vices con
nect
ing through the ex
ter
nal Layer 3 con
nec
tion, the ex
ter
nal net
work has learned of the in
ter
nal ACI net
work 10.
1.
1.
0/
24, as it is ad
ver
tised to the ad
ja
cent router through the Layer 3 ex
ter
nal con
nec
tion. For the pri
vate net
works, ACI
does not ad
ver
tise the net
works through the rout
ing pro
to
col to the ad
ja
cent Layer 3
router, and the net
works are not reach
able to de
vices ex
ter
nal to the fab
ric.
In re
leases prior to ver
sion 1.1 of Cisco Ap
pli
ca
tion Pol
icy In
fra
struc
ture Con
troller
(APIC), the fab
ric only ad
ver
tises sub
nets that are marked pub
lic in the as
so
ci
ated
bridge do
main. Routes that are learned ex
ter
nally from the fab
ric are not ad
ver
tised
through other ports. This be
hav
ior is known as a non-tran
sit fab
ric. In re
lease ver
sion
1.1 and later, ACI is ca
pa
ble of act
ing as a tran
sit net
work, and routes learned from one
ex
ter
nal Layer 3 con
nec
tion can be ad
ver
tised out to a dif
fer
ent ex
ter
nal Layer 3 con
nec
tion, not just fab
ric routes.
The net
work team will pro
vide the ex
ter
nal Layer 3 con
nec
tiv
ity for their ten
ants. One
com
mon mech
a
nism is to use sub-in
ter
faces on a router to cre
ate dif
fer
ent Layer 3 do
mains since each ten
ant will likely not have their own ex
ter
nal router.
Supported Routing Protocols

The fol
low
ing rout
ing pro
to
cols are sup
ported at time of writ
ing:
Static routesYou can define static routes to the outside world. Using static
routes reduces the sizing and complexity of the routing tables in the leaf nodes,
but increases administrator overhead. With static routes, you must also
configure the static path back to the interanal network that you wish to be
reachable from the outside world.
OSPF NSSAUsing not-so-stubby area (NSSA) reduces the size of the Open
Shortest Path First (OSPF) database and the need to maintain the overhead of
the routing protocols with large tables of routes. With OSPF NSSA, the router
learns only a summarization of routes, including a default path out of the fabric.
OSPF NSSA advertises to the adjacent router the internal public subnets part of
the Layer 3 external.
iBGP peering leaf and external routerWith internal Border Gateway Protocol
(iBGP), ACI supports only one autonomous system (AS) number that has to
match the one that is used for the internal Multiprotocol Border Gateway
Protocol (MP-BGP) route reflector. Without MP-BGP, the external routes
(static, OSPF, or BGP) for the Layer 3 outside connections are not propagated
within the ACI fabric, and the ACI leaves that are not part of the border leaf
does not have IP connectivity to any outside networks. Given that the same AS
number is used for both cases, the user must find out the AS number on the
router to which the ACI border leaf will connect, and use that AS number as the
BGP AS number for the ACI fabric.
Configure MP-BGP Spine Route Reflectors

The ACI fab
ric route re
flec
tors use mul
ti
pro
to
col bor
der gate
way pro
to
col (MP-BGP) to
dis
trib
ute ex
ter
nal routes within the fab
ric so a full mesh BGP topol
ogy is not re
quired.
To en
able route re
flec
tors in the ACI fab
ric, the fab
ric ad
min
is
tra
tor must se
lect at least
one spine switch that will be a route re
flec
tor, and pro
vide the au
tonomous sys
tem (AS)
num
ber for the fab
ric. Once route re
flec
tors are con
fig
ured, ad
min
is
tra
tors can setup
con
nec
tiv
ity to ex
ter
nal net
works.
To con
nect ex
ter
nal Layer 3 de
vices to the ACI fab
ric, the fab
ric in
fra
struc
ture op
er
at
or
must con
fig
ure a Route Re
flec
tor pol
icy to des
ig
nate which spines act as the route re
flec
tor(s). For re
dun
dancy pur
poses, con
fig
ure more than one spine as a router re
flec
tor node.
When a ten
ant re
quires a Layer 3 con
nec
tion, the in
fra
struc
ture op
er
at
or con
fig
ures
the leaf node to which the WAN router is being con
nected as bor
der leaf node, which
pairs the bor
der leaf node with one of the route re
flec
tor nodes as a BGP peer. After
the route re
flec
tors are con
fig
ured, they can ad
ver
tise routes in the fab
ric.
Each leaf node can store up to 4000 routes at time of writ
ing. If a WAN router must ad
ver
tise more than 4000 routes, the router should peer with mul
ti
ple leaf nodes. The in
fra
struc
ture op
er
at
or con
fig
ures each of the paired leaf nodes with the routes (or route
pre
fixes) that the nodes can ad
ver
tise.
To con
fig
ure the Route Re
flec
tor pol
icy:
1
In the Navigation pane, choose Pod Policies > Policies > BGP Route
Deflector default.
In the Work pane, perform the following actions:

a. Change the Autonomous System Number to match the required number for
your network.
b. Add the two spine nodes that will be members of this reflector policy and
c. Click Submit.
In the Navigation pane, choose Pod Policies.
In the Work pane, choose Actions > Create Pod Policy Group.
In the Create Pod Policy Group dialog box, perform the following actions:
a. In the BGP Route Reflector Policy drop-down list, choose default.
b. In the Navigation pane, choose Pod Policies > Profiles > default.
c. In the Work pane, in the Fabric Policy Group drop-down list, choose Create
Pod Policy Group.
d. In the Create Pod Policy Group dialog box, in the Date Time Policy dropdown list, choose default.
e. In the BGP Route Reflector Policy drop-down list, choose default.
f. Complete the remainder of the dialog box as appropriate to your setup.
Click Submit.
The fol
low
ing fig
ure il
lus
trates the ob
jects and their re
la
tion
ships for ex
ter
nal Layer 3
con
nec
tions:
Object relationships for Layer 3 outside objects
Layer 3 Integration Through Tenant Network with OSPF NSSA

The fol
low
ing fig
ure shows a sim
ple in
te
gra
tion of a Layer 3 ex
ter
nal into ACI using
OSPF:
Logical topology for an external OSPF router communicating with two border leafs
The setup in
cludes a sin
gle router with two in
ter
faces con
nected to leaf switches.
Note: See the "Adding New De
vices to The Fab
ric" sec
tion to setup the ac
cess poli
cies
for the in
ter
faces of the leaves that are con
nected to the router.
To in
te
grate Layer 3 through a ten
ant net
work with OSPF/NSSA:
1
In the Work pane, choose the tenant.
In the Navigation pane, choose Tenant_Name > Networking > External Routed
Networks.
4
5
In the Work pane, choose Action > Create Routed Outside.

In the Create Routed Outside dialog box, perform the following actions:
a. In the Name field, enter a name for the profile.
b. In the Private Network drop-down list, choose the private network for this
tenant.
c. Click the OSPF check box.
d. In the OSPF Area ID field, enter the OSPF area ID, such as "1".
e. In the OSPF Area Control section, click the Send redistributed LSAs into
NSSA area check box.
f. In the OSPF Area Type section, click the NSSA Area radio button.
g. In the Nodes and Interfaces Protocol Profiles section, click + to add a profile.
h. In the Create Node Profile dialog box, perform the following actions:
i. In the Name field, enter a name for the profile.
ii. In the Nodes section, click + to add a node.
iii. In the Select Node dialog box, perform the following actions:
1. In the Node ID drop-down list, choose a node, such as Leaf-1.
2. In the Router ID field, enter the router's IP address as the ID, such
as "10.0.1.1".
3. Uncheck the Router ID as Loopback Address check box.
4. In the Loopback Addresses section, click + to add a loopback address.
5. Enter the loopback address, such as "10.254.254.1", and click Update.
6. Click OK.
iv. In the OSPF Interface Profiles section, click + to create an OSPF interface
profile.
v. In the Create Interface Profile dialog box, perform the following actions:
1. In the Name field, enter a name for the profile.
2. In the OSPF Policy drop-down list, choose Create OSPF Interface
Policy. When defining the interaction with another OSPF router, you
must specify the policy interaction. This document does not explain the
different OSPF parameters.
3. In the Create OSPF Interface Policy dialog box, perform the following
actions:
a. In the Name field, enter a name for the OSPF policy, such as "OSPFPoint2Point".
b. In the Network Type section, click the radio button that matches the
adjacent router, such as Point to Point.
c. Complete the remainder of the dialog box as appropriate to your
setup.
d. Click Submit.
4. In the Interfaces section, click on the Routed Intefaces tab.
5. Click the + sign to select a routed interface.
6. In the Select Routed Interface dialog box, perform the following actions:
a. In the Path drop-down list, choose the interface on the leaf, such as
e1/9 on Leaf-1.
b. In the IP Address field, enter the IP address of the path that is
attached to the layer 3 outside profile, such as "10.0.1.1/24".
c. In the MTU (bytes) field, enter the maximum MTU of the external
network, such as "1500" to match the example peering router.
d. Complete the remainder of the dialog box as appropriate to your
setup and click OK.
7. Click OK.
vi. Click OK.
i. Click Next.
j. In the External EPG Networks section, click + create an external network.
k. In the Create External Network dialog box, perform the following actions:
i. In the IP Address field, enter 0.0.0.0/0 to permit the learning of any

subnet and click OK.
l. Click Finish.
Next, you must configure the external network EPG.
External Layer 3 for Multiple Tenants

In ACI, you can use var
io
us mech
an
isms to reuse the same ex
ter
nal Layer 3 router for
mul
ti
ple ten
ants. If the ad
ja
cent router is a Cisco Nexus Se
ries Switch with a Layer 2
trunk in
ter
face, the ex
ter
nal Layer 3 con
nec
tion can be con
fig
ured to route via SVI. For
routers ca
pa
ble of using sub-in
ter
faces, those can be used to pro
vide mul
ti
ple ex
ter
nal
Layer 3 con
nec
tion for mul
ti
ple VRFs and/or ten
ants. The fab
ric op
er
at
or can con
fig
ure mul
ti
ple ex
ter
nal Layer 3 con
nec
tions using ei
ther sub-in
ter
face or SVI and pro
vide
that to each ten
ant.
Application Migration Use Case

When op
er
at
ing the ACI fab
ric, there can be oc
ca
sions when you will have to mi
grate
work
loads, servers or vir
tu
al
iza
tion hosts from out
side the ACI fab
ric, onto the fab
ric.
One com
mon ex
am
ple is when mi
grat
ing from a tra
di
tional data cen
ter con
fig
ur
a
tion
over to a pol
icy-dri
ven data cen
ter using ACI. As ACME starts to use ACI in more of
their data cen
ters, it will be come nec
es
sary to per
form these mi
gra
tions. In this ex
am
ple, ACME must man
age the mi
gra
tion of SVI in
ter
faces as well as the pol
icy al
low
ing
traf
fic to tra
verse a Layer 2 out
side net
work and then on to the ACI fab
ric.
At a high level, you must start by con
fig
ur
ing the Layer 2 out
side net
work to allow traf
fic from the source VLAN to com
mu
ni
cate with the same VLAN re
sid
ing on the ACI fab
ric. You will also need to con
fig
ure Layer 3 con
nec
tiv
ity from the fab
ric out to the ex
ist
ing Layer 3 net
works in prepa
ra
tion for full con
nec
tiv
ity after SVI mi
gra
tion.
Fur
ther in
for
ma
tion and steps on how to cre
ate this Layer 2 and Layer 3 con
nec
tiv
ity in
clud
ing the pol
icy, can be found in the Fab
ric Con
nec
tiv
ity chap
ter of this book.
Once they have suc
cess
fuly es
tab
lished con
nec
tiv
ity be
tween the out
side Layer 2 net
work (where the work
load or host is com
ing from) and the ex
ist
ing in
ter
nal fab
ric EPG,
you can then start the mi
gra
tion process of mov
ing ap
pli
ca
tion work
loads onto the fab
ric. One key con
sid
er
at
ion should be when to switch over the SVI in
ter
faces from the
ex
ist
ing en
vi
ron
ment into the ACI fab
ric and when to start ad
ver
tis
ing routes to this
SVI net
work. As
sum
ing that the SVIs re
side on the ex
ter
nal Layer 2 net
work, Cisco rec
om
mends that you move the SVIs over to the ACI fab
ric once a ma
jor
ity of the hosts
have been mi
grated over.
Extending the Network to ACI

One of the ACME sites would like to mi
grate from the legacy data cen
ter ar
chi
tec
ture
to the next gen
er
at
ion ACI Fab
ric. They would like to mi
grate with min
im
al ser
vice in
ter
rup
tion while tak
ing ad
van
tage of ACI in
no
va
tions where ap
plic
ab
le. ACME would
like to per
form the mi
gra
tion in mul
ti
ple stages.
The Legacy data cen

ter prior to mi
gra
tion:
Traditional pre-migration data center

The ACI data cen
ter fol
low
ing mi
gra
tion:
Post-migration ACI based data center topology
The first stage will pro

vide con
nec
tiv
ity from the legacy data cen
ter to the ACI fab
ric.
In this state, you log
ic
ally map a VLAN=EPG. The in
ter
con
nect from the legacy net
work
to the ACI fab
ric will be ac
com
plished through stan
dard Layer 2 ex
ten
sions
(VLAN/VXLAN).
Pro
vide phys
ic
al con
nec
tiv
ity from the ex
ist
ing ag
gre
ga
tion layer to the ACI bor
der
leafs. This con
nec
tiv
ity can be ac
com
plished in ei
ther the form of a Vir
tual Port Chan
nel, Port Chan
nel, or a sin
gle in
ter
face.
1
Provide a physical connection from aggregation switch #1 to the ACI border

leaf #1.
Provide a physical connection from aggregation switch #2 to the ACI border

leaf #1.
Note: Be
fore con
nect
ing ex
ter
nal phys
ic
al con
nec
tions into the fab
ric, the Fab
ric Ac
cess Poli
cies for the ac
cess ports that you will be used for the DCI must be con
fig
ured.
For de
tails on con
fig
ur
ing the ac
cess poli
cies please ref
er
ence the Fab
ric Con
nec
tivty
sec
tion of this book.
Con
fig
ure the ag
gre
ga
tion links as a Layer 2 trunk.
1
Trunk the VLAN representing the host connectivity. This allows for the host
VLAN to be extended into the fabric.
In the APIC con

troller you will now con
fig
ure a sin
gle ten
ant. The cre
ated ten
ant will
rep
re
sent the legacy data cen
ter into the ACI fab
ric.
1
2
On the menu bar, choose Tenants > Add Tenant.

In the Create Tenant dialog box perform the following actions:
a. In the Name field enter a name for the tenant.
b. Click Next.
Click Finish.
Con
fig
ure a sin
gle pri
vate net
work.
1
In the Navigation pane choose Tenant_Name > Networking > Private

Networks.
In the Work pane, choose Actions > Create Private Network.
In the Create Private Network dialog box, perform the following actions:
a. In the Name field enter a name for the private network.
b. Click Next.
c. In the Name field enter a name for the bridge domain.
d. In the Forwarding drop-down list, choose Custom.
e. For the Layer 2 Unknown Unicast radio buttons, click Flood.
f. For the Multi Destination Flooding radio buttons, click Flood in BD.
g. Click the ARP Flooing check box.
Click Finish.
Note: The con

cept of flood
ing the un
known uni
cast and arp within the Fab
ric is to
allow for the Layer 2 se
man
tics from the legacy data cen
ter to be ex
tended into the ACI
Fab
ric. When a host in the legacy data cen
ter sends an ARP re
quest and/or floods an
un
known uni
cast frame, the bridge do
main will then mimic the be
hav
ior in the ACI Fab
ric. By de
fault BPDU frames are flooded within EPG.
Con
fig
ure a sin
gle ap
pli
ca
tion pro
file:
1
On the menu bar, choose Tenants > ALL TENANTS
In the Navigation pane choose Application Profiles.
In the Work pane, choose Actions > Create Application Profile.
In the Create Application Profile dialog box, perform the following actions:
a. In the Name field enter a name for the Application Profile.
b. Click Submit.
Con
fig
ure a sin
gle end
point group:
1
In the Navigation pane choose Tenant_Name > Application Profiles >

Application_Profile_Name.
4
5
In the Work pane, choose Actions > Create Application EPG.

In the Create Application EPG dialog box, perform the following actions:
a. In the Name field, enter a name for the endpoint group.
b. In the Bridge Domain field, choose the appropiate bridge domain.
c. Click Finish.
Note: The EPG within the fab

ric will map to a sin
gle VLAN in the legacy data cen
ter.
The use of a sin
gle VLAN per EPG will pro
vide a path for a net
work-cen
tric mi
gra
tion
min
im
iz
ing im
pact while in
tro
duc
ing fab
ric in
no
va
tions.
Con
fig
ure a VPC for the con
nec
tiv
ity to the legacy data cen
ter. See the "Fab
ric Con
nec
tiv
ity" sec
tion. Then, con
fig
ure a sta
tic trunk bind
ing using the VPC under the EPG,
Acme
Out
Side. The en
cap
su
la
tion VLAN should match the de
fined legacy data cen
ter
VLAN de
f
in
i
t
ion.
1
In the Navigation pane choose Tenant_Name > Application

Profiles > Application_Profile_Name > Application EPGs > EPG_Name
> Static Bindings (Paths).
In the Work pane, choose Actions > Deploy Static EPG on PC, VPC or
Interface.
In the Deploy Static EPG on PC, VPC or Interface dialog box, perform the
following actions:
a. Choose the Path Type.
b. Choose the Path.
c. Enter the encapsulation VLAN.
d. Click Submit.
Fol
low
ing the Stage 1 mi
gra
tion, the Legacy host VLAN is now ex
tended to the ACI fab
ric and all hosts from the Fab
ric point of view are in the EPG, Acme
Out
Side. From the
ACI Fab
ric per
spec
tive, the local VLAN num
ber is not sig
nif
ic
ant as it will be mapped to
the tagged VLAN on the VPC to
ward the Legacy data cen
ter. This is what is known as
nor
mal
iza
tion where ACI uses the tagged vlan to map ex
ter
nal Layer 2 con
nec
tions into
ACI End Point Groups.
The con
nec
tiv
ity noted below rep
re
sents a BD=EPG=VLAN. The Layer 3 gate
way for the
ACI fab
ric and the legacy data cen
ter or pro
vided by the Legacy data cen
ter.
An EPG as a VLAN datacenter deployment

The sec
ond stage of the mi
gra
tion takes ad
van
tage of the fab
ric pol
icy model to de
fine
fur
ther the ap
pli
ca
tion mod
el
ing. Dur
ing this stage, mul
ti
ple EPGs will be cre
ated with
APIC con
tracts that de
fine reach
a
bil
ity. The fol
low
ing list is an overview of the steps
that you must per
form:
1
In the APIC:
a. Configure additional Tenants (optional).
b. Configure additional Private Networks (optional).
c. Configure additional Bridge Domains (optional).
d. Configure additional Application Profiles (optional).
e. Configure additional End Point Groups.
i. Create the endpoint group "AcmeInSide".
f. Configure contracts for inter-EPG communications.
i. For the Scope drop-down list, choose Tenant.
ii. For the Allow drop-down list, choose Any-Any.
Begin migrating the host to the fabric EPG, "AcmeInSide".

a. Create a static binding for each physical host to be migrated.
b. Create a VMM domain to deploy the host within the ACI Fabric (optional).
Note: En
sure the ap
pro
pri
ate re
sources have been cre
ated to sup
port the con
nec
tiv
ity, VLAN pools, phys
i
cal do
main or VMM Do
main, AEP, and switch or in
ter
face pro
files. For more in
for
ma
tion, see the "Fab
ric Con
nec
tivty" sec
tion.
Fol
low
ing the stage 2 mi
gra
tion, the host con
nec
tiv
ity across both the legacy data cen
ter and ACI fab
ric are now gov
erned by the APIC pol
icy (con
tracts).
Note: The Layer 3 gate
way for the ACI fab
ter are pro
vided by
the legacy data cen
ter.
Datacenter migration with Layer 3 provided by existing DC and Layer 2 extended to

new ACI datacenter
In the third stage of the mi
gra
tion the Layer 3 gate
way will move from the legacy data
cen
ter to the ACI Fab
ric. The fol
low
ing list is an overview of the steps that you must
per
form:
1
In the APIC:
a. Configure a Layer 3 out.
b. Migrate the gateway from the legacy data center to the fabric.
i. In the Bridge Domain drop-down list, choose AcmeBD.
ii. In the Flood Layer 2 drop-down list, choose Unknown Unicast.
iii. In the ARP drop-down list, choose Flooding.
iv. In the Unicast drop-down list, choose Routing.
Note: The con

cept of uni
cast rout
ing within the bridge do
main al
lows for the con
fig
u
ra
tion of the per
va
sive gate
way across the fab
ric. The Layer 3 gate
way for the ACI fab
ter are pro
vided by the ACI fab
ric.
159
Tenants
Tenants 161
Section Content
ACI Tenancy Models

Application Profile
Application Profile Configuration
Create a New Application Profile
Modify Application Profile
Remove Application Profile
Verify Application Profile
Endpoint Group
Endpoint Group Configuration
Create a New Endpoint Group
Modify Endpoint Group
Remove Endpoint Group
Verify Endpoint Group
Endpoint
Private Network
Private Network Configuration Parameters
Creating a New Private Network
Modify Private Network
Remove Private Network
Verify Private Network
162 Tenants
Bridge Domain
Bridge Domain Configuration Parameters
Create a new Bridge Domain
Modify a Bridge Domain
Remove a Bridge Domain
Verify Bridge Domain
Tenant Networking Use Cases

Common Private Network for All Tenants
Multiple Private Networks with Intra-Tenant Communication
Multiple Private Networks with Inter-Tenant Communication
Tenants 163
ACI Tenancy Models

ACME Inc. will be using ten
ancy for a cou
ple of use cases. They will be using ten
ant
con
structs for the ap
pli
ca
tion life
cy
cle of their cur
rent de
ploy
ment, main
tain
ing a sep
a
rate ten
ant for the re
sources that de
vel
op
ers will be using to build the ap
pli
ca
tion, a
ten
ant that will be used for the au
to
mated test
ing, and fi
nally a pro
duc
tion ten
ant. Ad
di
tion
ally, as men
tioned in the in
tro
duc
tion, they are also look
ing to build an in
fra
struc
ture which can be lever
aged for sim
il
ar ini
tia
tives in the fu
ture. Ten
ants will be
used to draw vir
tual bound
aries for dif
fer
ent lines of busi
ness. The in
for
ma
tion se
cu
rity team will be able to in
te
grate this into the cor
po
rate LDAP sys
tem, and pre
vent
changes which would im
pact other groups.
ACI has been de
signed from the be
gin
ning to be multi-ten
ant. This means dif
fer
ent
things to dif
fer
ent peo
ple (much like the term Cloud) based on their per
spec
tive. In the
case of a clas
sic ser
vice provider, a ten
ant is a unique cus
tomer, while in a typ
ic
al endcus
tomer en
vi
ron
ment a ten
ant could be an op
er
at
ing group, busi
ness unit, ap
pli
ca
tion
owner, etc.
The de
ci
sion on how to lever
age ten
ancy mod
els is dri
ven by a num
ber of fac
tors:
1
Overall IT operations and support models in your organization to manage

application, networking, servers, security, and so on.
Separation of environments from a software development lifecycle perspective:

Development, Quality Assurance, and Production.
Separation of duties by domain owner, such as web, app, and database owners.
Fault domain size and scope to limit the impact of failures, such as different
business units.
In tra
di
tional net
work
ing en
vi
ron
ments, mak
ing a rout
ing pro
to
col change on a router
or Layer 3 switch could po
ten
tially af
fect hun
dreds of unique VLANs/sub
nets. This in
tro
duces a war
ranted level of cau
tion around change con
trol and ap
pli
ca
tion im
pact.
Lever
ag
ing the ACI pol
icy model, the phys
ic
al hard
ware is ab
stracted from the log
ic
al
con
structs. The ten
ant ob
ject gives us the abil
ity to draw a box around the log
ic
al and
con
crete ob
jects that we use to pro
vide a uni
fied view of the con
fig
ur
a
tion de
pen
den
cies for un
der
lay and over
lay net
works.
164 Tenants
A Ten
ant in the ACI Ob
ject model rep
re
sents the high
est-level ob
ject. In
side, you can
dif
fer
en
ti
ate be
tween the ob
jects that de
fine the ten
ant net
work
ing, such as Pri
vate
Net
works, Bridge Do
mains and sub
nets; and the ob
jects that de
fine the ten
ant poli
cies
such as Ap
pli
ca
tion Pro
files and End
point Groups.
In ACI, the ten
ant poli
cies are where you de
fine ap
pli
ca
tions. An ap
pli
ca
tion could con
sist of a com
bi
na
tion of phys
ic
al servers or VMs that we will call servers from now on.
For ex
am
ple, a web
site could use a 3-tier ap
pli
ca
tion model, com
prised of web servers,
ap
pli
ca
tion servers and data
base servers. When a user browses the web site, they
might ac
tu
ally be com
mu
ni
cat
ing with a vir
tual IP ad
dress on a load bal
ancer that in
turn can dis
trib
ute the web re
quest to a num
ber of dif
fer
ent web servers. The web
servers in turn com
mu
ni
cate with core ap
pli
ca
tions that can be di
vided amongst sev
eral ap
pli
ca
tions servers for load bal
anc
ing or high avail
abil
ity pur
poses. Fi
nally, the
ap
pli
ca
tion servers com
mu
ni
cate with the data
base which could also be a clus
ter of
servers.
Each server is re
ferred to as an End
point in ACI. End
points are clas
si
fied in ACI to apply
poli
cies. You cre
ate end
point groups with end
points that share the same type of poli
cies, such as with whom are they going to com
mu
ni
cate and what type of com
mu
ni
ca
tion or re
stric
tions are re
quired. There
fore, an ap
pli
ca
tion can be formed by sev
eral end
point groups and they are grouped in an Ap
pli
ca
tion Pro
file.
The ten
ant net
work
ing is used to de
fine net
work
ing poli
cies and will be ap
plied to the
un
der
ly
ing hard
ware in a trans
par
ent way thanks to the layer of ab
strac
tion pro
vided
by ACI using pri
vate net
works, bridge do
mains and sub
nets. In the next sec
tions of this
chap
ter these con
cepts will be cov
ered in de
tail. Below you can find an il
lus
tra
tion with
the dif
fer
ent ob
jects that com
pound a ten
ant and how they are re
lated.
Al
though the ten
ant net
work
ing and the ten
ant poli
cies are de
fined sep
ar
ately, the
net
work
ing poli
cies used by an ap
pli
ca
tion are de
fined with a re
la
tion
ship be
tween the
End
point Groups and the Bridge Do
main.
The fol
low
ing image shows all of the com
po
nents that can be con
fig
ured within a ten
ant. In the fol
low
ing sec
tions each di
ag
ram shows the progress of how ACME Inc. adds
each com
po
nent.
Tenants 165
Tenant Logical Model

There are 3 Ten
ants pre
con
fig
ured in the sys
tem by de
faut:
1
Common a special tenant with the purpose of providing common services to

other tenants in the ACI fabric. Global reuse is a core principle in the common
tenant. Some examples of common services are:
a. Shared Private Networks
b. Shared Bridge Domains
c. DNS
d. DHCP
e. Active Directory
Infra The Infrastructure tenant that is used for all internal fabric
communications, such as tunnels and policy deployment. This includes switch
to switch (Leaf, Spine, Application Virtual Switch (AVS)) and switch to APIC.
The infra tenant does not get exposed to the user space (tenants) and it has its
own private network space and bridge domains. Fabric discovery, image
management, and DHCP for fabric functions are all handled within this tenant.
Mgmt - The management tenant provides convenient means to configure

access policies for fabric nodes. While fabric nodes are accessible and
configurable through the APIC, they can also be accessed directly using in-band
and out-of band connections. In-band and out-of-band policies are configured
under the mgmt tenant:
In-Band Management Access
Out-of-Band Management Access
Tenants 167
Application Profile
An ap
pli
ca
tion pro
file is a con
ve
nient log
i
cal con
tainer for mul
ti
ple hosts (phys
i
cal or
vir
tual). You can cre
ate Ap
pli
ca
tion Pro
file con
tain
ers based on a va
ri
ety of cri
te
ria,
such as what func
tion the ap
pli
ca
tion pro
vides, how the ap
pli
ca
tion looks from the
end-user per
spec
tive, where they are lo
cated within the con
text of the data cen
ter, or
any other log
i
cal group
ing rel
a
tive to the im
ple
men
ta
tion. Ap
pli
ca
tion Pro
file servers
are grouped in EPGs de
pend
ing on the use of com
mon poli
cies.
Ap
pli
ca
tion Pro
files pro
vide a mech
a
nism to un
der
stand groups of servers as a sin
gle
ap
pli
ca
tion. This ap
proach makes an ACI ap
pli
ca
tion aware and al
lows us to check the
op
er
a
tional state for an ap
pli
ca
tion mon
i
tor
ing all the servers that are part of an ap
pli
ca
tion as a whole and be
come in
formed about rel
e
vant faults and health sta
tus for that
par
tic
u
lar ap
pli
ca
tion. Each Ap
pli
ca
tion Pro
file cre
ated can have a unique mon
i
tor
ing
pol
icy and QOS pol
icy.
An Ap
pli
ca
tion Pro
file is a child ob
ject of the Ten
ant and a sin
gle Ten
ant can con
tain
mul
ti
ple Ap
pli
ca
tion Pro
files.
Adding components to a Tenant - 1. Application Profile
168 Tenants
Application Profile Configuration

Name - The name of the ap
pli
ca
tion pro
file.
Tags - A tag or meta
data is a non-hi
er
ar
chi
cal key
word or term as
signed to the fab
ric
mod
ule.
Mon
i
tor
ing Pol
icy - The mon
it
or
ing pol
icy name for the EPG se
man
tic scope (op
tional).
Create a New Application Profile

1
In the Navigation pane choose Tenant_Name > Application Profiles.
a. Enter an Application Profile Name.
b. Enter a TAG (optional).
c. Choose a Monitoring Policy (optional).
Click Submit.
Modify Application Profile

1

Application_Profile.
4
5
In the Work pane, choose policy.

a. Enter an Application Profile Name.
b. Enter an appropriate TAG (optional).
c. Choose the Monitoring Policy (optional).
Click Submit.
Tenants 169
Remove Application Profile

1
In the Navigation pane choose Tenant_Name > Application Profiles

> Application_Profile.
In the Work pane, choose Actions > Delete.
Verify Application Profile

REST :: /api/node/class/fvAp.xml
CLI :: moquery -c fvAp
Tenants 171
Endpoint Group
End
point Groups are used to cre
ate log
i
cal group
ings of hosts or servers that per
form
sim
i
lar func
tions within the fab
ric. Each End
point Group cre
ated can have a unique
mon
i
tor
ing pol
icy and QoS pol
icy and must be as
so
ci
ated with a Bridge Do
main.
An End
point group is a child ob
ject of the Ap
pli
ca
tion Pro
file and an Ap
pli
ca
tion Pro
file can con
tain mul
ti
ple End
point Groups. Each end
point within an End
point Group is
sus
cep
ti
ble to the same pol
icy in the Fab
ric.
Adding components to a tenant - 2. End Point Group in the Application Profile

All the End
points in
side an EPG can com
mu
ni
cate with each other. How com
mu
ni
ca
tions be
tween EPGs con
tracts will be re
quired is gov
erned by con
tracts and not tra
di
tional Layer 2/Layer 3 for
ward
ing con
structs. For ex
am
ple, Host-A in EPG-A can have
the IP ad
dress/mask of 10.
1.
1.
10/
24 and Host B in EPG B can have the IP ad
dress/mask 10.
1.
1.
20/
24 (note that both hosts be
lieve they are "in the same sub
net"). In
this case they would not be al
lowed to com
mu
ni
cate un
less a con
tract that per
mit
ted
con
nec
tiv
ity ex
isted be
tween EPG-A and EPG-B. Con
tracts will be ex
plained in greater
de
tail in a fol
low
ing sec
tion.
Note that there are some types of End
point Groups within the fab
ric that are not con
tained under Ap
pli
ca
tion Pro
files such as, Ap
pli
ca
tion End
point Group, Ex
ter
nal Bridge
Net
works (aka Lay
er2 Ex
ter
nal), Ex
ter
nal Routed Net
works (aka as Lay
er3 Ex
ter
nal) and
Man
age
ment End
point Groups. These End
point Groups might have spe
cial re
quire
-
172 Tenants
ments, for ex
am
ple, in Ex
ter
nal Bridge Net
works, MAC ad
dresses of the end
points are
not learnt by the leaf switches.
End
point Groups are linked to Bridge Do
mains but they will re
ceive a VLAN ID dif
fer
ent
from the bridge do
main, un
less Bridge Do
main legacy mode is used.
It is im
por
tant to un
der
stand that a sin
gle sub
net can be ex
tended across sev
eral EPGs.
Each EPG is iden
ti
fied by an en
cap
su
la
tion VLAN or VXLAN so that the same sub
net will
be using dif
fer
ent en
cap
su
la
tion IDs across the fab
ric. This con
cept is dif
fer
ent from
tra
di
tional net
work
ing.
Endpoint Group Configuration

Name - The name for the end
point group.
Tag - A tag or meta
data is a non-hi
er
ar
chi
cal key
word or term as
signed to the fab
ric
mod
ule.
Qos Class - The QoS pri
or
ity class iden
ti
fier.
The class can be:
Unspecified
Level1-Class 1 Differentiated Services Code Point (DSCP) value.
Level2-Class 2 DSCP value.
Level3-Class 3 DSCP value.
Cus
tom Qos - The QoS traf
fic pri
or
ity class iden
ti
fier. The Cus
tom class is a user-con
fig
urable DSCP value.
Bridge Do
main - The name of the bridge do
main as
so
ci
ated with this ob
ject.
it
or
ing pol
icy name for the EPG se
man
tic scope (op
tional).
Mon
it
or
ing Pol
icy The mon
As
so
ci
ated Do
main Pro
file A source re
la
tion to an in
fra
struc
ture do
main pro
file as
so
ci
ated with ap
pli
ca
tion end
point groups.
Tenants 173
Sub
net - An End
point Group sub
net is rel
e
vant when and only when con
fig
ur
ing route
leak
ing be
tween VRF's/Pri
vate Net
work within a Ten
ant (op
tional).
Sta
tic End
point - The sta
tic client end
point rep
re
sents a silent client end
point at
tached
to the net
work (op
tional).
Create a New Endpoint Group

1

Application_Profile > Application EPGs.
4
5

In the Create Application EPG dialog box, perform the following.
a. Enter an Application EPG Name.
b. Enter an Tag (optional).
c. Enter an Qos Class (optional).
d. Enter an Custom Qos (optional).
e. Enter a Bridge Domain Name.
f. Choose a Monitoring Policy (optional).
g. Enter an Associated Domain Profile Name.
Click Finish.
Modify Endpoint Group

1

> Application_Profile > Application EPGs > Application_EPG.
In the Work pane, select policy.

a. Enter an Application EPG Name.
b. Enter an Tag (optional).
c. Enter an Qos Class (optional).
d. Enter an Custom Qos (optional).
174 Tenants
e. Enter a Bridge Domain Name.

f. Choose the appropriate Monitoring Policy if applicable (optional).
g. Enter an Associated Domain Profile Name.
5
Click Finish.
Remove Endpoint Group

1

> Application_Profile > Application EPGs > Application_EPG..

REST :: /api/node/class/fvAEPg.xml
CLI :: moquery -c fvAEPg
Tenants 175
Endpoint
End
points are de
vices that are con
nected to the net
work ei
ther di
rectly or in
di
rectly.
End
points have an ad
dress (iden
tity), a lo
ca
tion, and at
trib
utes, and can be ei
ther vir
tual or phys
ic
al. Each end
point has a path, an en
cap
su
la
tion, and a de
ploy
ment Im
me
di
acy mode as
so
ci
ated with it.
An End
point is a child ob
ject of the End
point Group and an End
point Group con
struct
can con
tain mul
ti
ple End
points. The End
points ref
er
enced within the fab
ric can be ei
ther sta
tic (de
fined within the APIC) or dy
namic (au
to
mated by vCen
ter/Open
stack).
You can add Sta
tic End
points by cre
at
ing Sta
tic Bind
ings within the End
point Group.
Below is an ex
am
ple of a sta
tic bind
ing. See the VVM sec
tion for an ex
am
ple of a dy
namic bind
ing.
1

Profiles > Application_Profile_Name > Application EPGs > EPG_Name
> Static Bindings (Paths).
In the Work pane, choose Actions > Deploy Static EPG on PC, VPC or
Interface.
In the Deploy Static EPG on PC, VPC or Interface dialog box, perform the
following actions:
a. Choose the Path Type.
b. Choose the Path.
c. Enter the encapsulation VLAN.
d. Click Submit.
In order to show the end

points that are con
nected to the fab
ric under cer
tain EPGs:
1
176 Tenants
In the Navigation Pane choose Tenant_Name > Application Profiles

> Application_Profile > Application EPGs > Application_EPG.
In the Work pane, choose Operational.
Verify Endpoint
REST :: /api/node/class/fvCEp.xml
CLI
:: moquery -c fvCEp
Tenants 177
Private Network
A Pri
vate Net
work is also re
ferred to as a VRF, pri
vate Layer 3 net
work, or con
text. It is
a unique Layer 3 for
ward
ing and ap
pli
ca
tion pol
icy do
main. Pri
vate net
works are a
child of the Ten
ant ob
ject. All of the end
points within the Pri
vate Net
work must have
unique IP ad
dresses be
cause it is pos
si
ble to for
ward pack
ets di
rectly be
tween these
de
vices if the pol
icy al
lows it. One or more bridge do
mains are as
so
ci
ated with a pri
vate
net
work.
Adding components to a Tenant - 3. Private Network

as part of the Tenant Logical Model
The most com
mon method to share Pri
vate Net
works be
tween ten
ants is through the
com
mon ten
ant. For more in
for
ma
tion about com
mon ten
ants, see the overview sec
tion of this chap
ter. Pri
vate Net
works cre
ated in the com
mon ten
ant are shared glob
ally within the fab
ric. How
ever, a Pri
vate Net
work that is in
tended to be used by mul
ti
ple ten
ants and is not cre
ated in the com
mon ten
ant re
quires ex
plicit con
fig
u
ra
tion to
be shared.
When there is a re
quire
ment to route traf
fic be
tween sep
a
rate Pri
vate Net
work in
stances, spe
cial con
sid
er
a
tion for sub
net con
fig
u
ra
tion is needed. This will be dis
cussed
in de
tail in the Bridge Do
main and EPG con
fig
u
ra
tion sec
tions.
178 Tenants
Private Network Configuration Parameters

Name - The name of the Pri
vate Net
work.
Pol
icy En
force
ment - The pre
ferred pol
icy con
trol. The val
ues can be en
forced or un
forced is cho
sen, con
tracts be
tween EPGs are re
quired to allow
en
forced. When en
traf
fic. Un
en
forced al
lows all traf
fic within the Pri
vate Net
work. The de
fault is en
forced.
icy as
so
ci
ated with this ob
ject.
BGP Timers - Name of the BGP timers pol
OSPF Timers - Name of the OSPF timers pol
icy as
so
ci
ated with this ob
ject.
End Point Re
ten
tion Pol
icy - The end point re
ten
tion pol
icy name (op
tional).
i
tor
ing pol
icy name for the Ten
ant se
man
tic scope (op
tional).
Mon
i
tor
ing Pol
icy - The mon
DNS Label - The net
work do
main name label. La
bels en
able clas
si
fy
ing which ob
jects
can and can
not com
mu
ni
cate with one an
other (op
tional).
Creating a New Private Network

1

Networks.
4
5

a. Enter an Private Network Name.
b. Choose a Policy Enforcement (optional).
c. Choose a BGP Policy Name (optional).
d. Choose an OSPF Policy Name (optional).
e. Choose an End Point Retention Policy Name (optional).
g. Choose the appropriate DNS Label if applicable (optional).
Click Finish.
Tenants 179
Modify Private Network

1

Networks > Private_Network.
In the Work pane, select policy.

a. Choosesa Policy Enforcement (optional).
b. Choose a BGP Policy Name (optional).
c. Choose an OSPF Policy Name (optional).
d. Choose an EIGRP Policy Name (optional).
e. Choose an End Point Retention Policy Name (optional).
g. Choose the appropriate DNS Label if applicable (optional).
Click Finish.
Remove Private Network

1

Networks > Private_Network.
Verify Private Network

REST :: /api/node/class/fvCtx.xml
CLI :: moquery -c fvCtx
Tenants 181
Bridge Domain
A Bridge Do
main is the ab
stract rep
re
sen
ta
tion of a Layer 2 for
ward
ing do
main within
the fab
ric. A Bridge Do
main is a child of the Ten
ant ob
ject and must be linked to a Pri
vate Net
work.
The bridge do
main de
fines the unique Layer 2 MAC ad
dress space and a Layer 2 flood
do
main if flood
ing is en
abled. While a Pri
vate Net
work de
fines a unique IP ad
dress
space, that ad
dress space can con
sist of mul
ti
ple sub
nets. Those sub
nets will be spread
across one or more bridge do
mains con
tained in the Pri
vate Net
work.
Bridge do
mains will span all switches in which as
so
ci
ated EPG are con
fig
ured. A bridge
do
main can have mul
ti
ple sub
nets. How
ever, a sub
net is con
tained within a sin
gle
bridge do
main.
Adding components to a Tenant - 4. Bridge Domain

as part of the Tenant Application Profile
The fol
low
ing image pro
vides an ex
am
ple of a ten
ant to show how bridge do
mains are
con
tained in
side of Pri
vate Net
works and how they are linked to EPGs and the other el
e
ments.
182 Tenants
End Point Group as part of the Tenant Application Profile

It is im
por
tant to un
der
stand that a bridge do
main is NOT a VLAN, al
though it can act
sim
i
lar to a VLAN. You in
stead should think of it as a dis
trib
uted switch, which, on a
leaf, can be trans
lated lo
cally as a VLAN with local sig
nif
i
cance.
From a prac
ti
cal per
spec
tive, each Bridge Do
main will exist in a par
tic
u
lar leaf if there
is a con
nected end
point that be
longs to that EPG. Each Bridge Do
main re
ceives a VLAN
ID in the leaf switches. This VLAN ID is sig
nif
i
cant lo
cally in the leaf switches and there
fore it might be dif
fer from one to other leaf switch.
The VLAN ID used is also called Plat
form In
de
pen
dent VLAN or PI VLAN. This VLAN
con
cept is dif
fer
ent from tra
di
tional net
work
ing and it is not used to for
ward traf
fic but
as an iden
ti
fier. Each PI VLAN is then linked to a VXLAN ID that will be used for for
ward
ing pur
poses in
side of the fab
ric.
In the fol
low
ing ex
am
ple, under the Ten
ant Acme, the bridge do
main Acme-Ap
pli
ca
tions-BD was as
signed the PI VLAN ID 42 in the Leaf-1.
Tenants 183
VLAN output from Leaf node

EPGs are also as
signed with a PI VLAN ID that is lo
cally sig
nif
i
cant in each leaf. This
VLAN ID is dif
fer
ent from the Bridge Do
main. There
fore in ACI, sev
eral VLANs will be
used for EPs in
side on one bridge do
main. For more de
tails refer to the EP sec
tion in
this chap
ter.
In some sit
u
a
tions where tra
di
tional net
work
ing is re
quired, for ex
am
ple dur
ing some
mi
gra
tion processes, it might be re
quired to have only one en
cap
su
la
tion VLAN per
Bridge do
main across all the leaf switches and EPGs that ref
er
ence that Bridge Do
main.
For this sit
u
a
tion, there is a fea
ture called Bridge do
main legacy mode.
Bridge do
main in legacy mode is not rec
om
mended under ACI best prac
tices. It is
thought to use tra
di
tional net
work
ing within ACI but it lim
its the ACI fea
tures that can
be used. EPGs con
tained in bridge do
mains in legacy mode can
not com
mu
ni
cate with
EPGs in ACI bridge do
mains.
When a Sub
net is de
fined in a Bridge Do
main, the leaf switches will be the de
fault gate
way
for the EPGs using that sub
net. If the EPGs have end
points on mul
ti
ple leaves, each
leaf will con
fig
ure the de
fault gate
way. In that way, the de
fault gate
way for the end
points
will al
ways be the first switch of the fab
ric that is reached, also know as a per
va
sive gate
way. This means that an SVI will be con
fig
ured under the VRF that rep
re
sents the pri
vate
184 Tenants
net
work that the Bridge Do
main is linked to. If a Bridge Do
main has sev
eral sub
nets, there
will be only one SVI per Bridge Do
main but it will use sec
ondary IP ad
dresses.
Bridge Domain Configuration Parameters

Name - The name of the Bridge Do
main.
Net
work - The as
so
ci
ated layer 3 con
text.
For
ward
ing - Op
ti
mize/Cus
tom
L2 Un
known Uni
cast - The for
ward
ing method for un
known layer 2 des
ti
na
tions. De
fault is Proxy.
Un
known Mul
ti
cast Flood
ing - The node for
ward
ing pa
ra
me
ter for un
known
Mul
ti
cast des
ti
na
tions.
erty to spec
ify whether ARP flood
ing is en
abled. If flood
ARP Flood
ing - A prop
ing is dis
abled, uni
cast rout
ing will be per
formed on the tar
get IP ad
dress.
Flood
ing is dis
abled by de
fault but can be en
abled by click
ing the check box.
ward
ing method based on pre
de
fined for
ward
ing cri
Uni
cast Rout
ing - The for
te
ria (IP or MAC ad
dress). Uni
cast rout
ing is en
abled by de
fault but can be dis
abled by click
ing the check box.
Con
fig BD MAC Ad
dress - The MAC ad
dress of the bridge do
main (BD) or
switched vir
tual in
ter
face (SVI). Every BD by de
fault takes the fab
ric wide de
fault
mac ad
dress.
ing pol
icy name. By ex
am
in
ing (snoop
ing) IGMP
IGMP Snoop Pol
icy - The IGMP Snoop
mem
ber
ship re
port mes
sages from in
ter
ested hosts, mul
ti
cast traf
fic is lim
ited to the
sub
set of VLAN in
ter
faces on which the hosts re
side.
As
so
ci
ated L3 Outs - The name of the Layer 3 out
side in
ter
face as
so
ci
ated with this
ob
ject.
L3 Out for Route Pro
file - The Layer 3 out
side in
ter
face iden
ti
fier con
trol
ling con
nec
tiv
ity to out
side net
works.
Tenants 185
Route Pro
file - The as
so
ci
ated route pro
file name.
Mon
i
tor
ing Pol
icy - The mon
it
or
ing pol
icy name for the Ten
ant se
man
tic scope (op
tional).
Sub
nets - The IP ad
dress and mask of the de
fault gate
way. It will be con
fig
ured in the
leaf nodes which have EPGs in that bridge do
main.
The net
work vis
ib
il
ity of the sub
net. The sub
net is a por
tion of a net
work shar
ing a par
tic
ul
ar sub
net ad
dress. The scope can be:
Shared - Defines subnets under an endpoint group, with the Shared option
Public - Defines subnets under a bridge domain, with the Public option
configured, to route leak to other Tenants within the Fabric.

configured, to share with Layer 3 outbound.
Private - Defines subnets under a bridge domain, with the Private option
configured, to only be used in that Tenant (will not be leaked). The default
is Private.
Sub
net Con
trol - The con
trol can be spe
cific pro
to
cols ap
plied to the sub
net
such as IGMP Snoop
ing. The con
trol can be:
Unspecified - The default is Unspecified.
Querier IP - Enables IGMP Snooping on the subnet.

DHCP La
bels - The net
work do
main name label.
Create a new Bridge Domain

1
In the Navigation pane choose Tenant_Name > Networking > Bridge Domains.
In the Work pane, choose Actions > Create Bridge Domain.
In the Create Bridge Domain dialog box, perform the following actions:
a. Enter an Bridge Domain Name.
b. Choose the Network.
c. Choose the Forwarding Semantics (optional).
186 Tenants
d. Choose the IGMP Snoop Policy (optional).

e. Choose the Associated L3 Outs (optional).
f. Choose the L3 Out for Route Profile (optional).
g. Choose the Route Profile (optional).
h. Choose the Monitoring Policy if applicable (optional).
i. Choose the Subnets (optional).
j. Choose the DNS Label if applicable (optional).
6
Click Submit.
Modify a Bridge Domain

1
In the Navigation pane choose Tenant_Name > Networking > Bridge Domains
> Bridge_Domain_Name.
In the Work pane, choose the Policy tab and perform the following actions:
a. Choose the Network.
b. Choose the Forwarding Semantics (optional).
c. Choose the IGMP Snoop Policy (optional).
d. Choose the Associated L3 Outs (optional).
e. Choose the L3 Out for Route Profile (optional).
f. Choose the Route Profile (optional).
g. Choose the Monitoring Policy if applicable (optional).
h. Choose the Subnets (optional).
i. Choose the DNS Label if applicable (optional).
Click Finish.
Remove a Bridge Domain

1
In the Navigation Pane choose Tenant_Name > Networking > Bridge Domain >
Bridge Domain_Name.
Tenants 187
Verify Bridge Domain

REST :: /api/node/class/fvBD.xml
CLI :: moquery -c fvBD
Tenants 189
Tenant Networking Use Cases

Common Private Network for All Tenants
This use case may be typ
i
cal for en
vi
ron
ments where an ACI ad
min
is
tra
tor wishes to
cre
ate mul
ti
ple ten
ants, but place all within a sin
gle pri
vate net
work in the fab
ric.
This method has the fol
low
ing ad
van
tages and dis
ad
van
tages:
Ad
van
tages:
Ability to use a single private network for all internal and external fabric
No route leaking needed between EPGs in different VRFs
Single Layer 3 Outside can be used by all tenants
connectivity
Dis
ad
van
tages:
Changes to routing will impact all tenants
From a con
tain
ment and re
la
tion
ship per
spec
tive, this topol
ogy looks as fol
lows:
Common Private Network for all Tenants

To Con
fig
ure the com
mon Ten
ant pri
vate net
work:
1
In the Work pane, choose the common.
190 Tenants
In the Navigation pane choose common > Networking > Private Networks >
default.

a. Choose a Policy Enforcement (optional).
b. Choosea BGP Policy Name (optional).
c. Choose an OSPF Policy Name (optional).
d. Choosean End Point Retention Policy Name (optional).
e. Choose the appropriate Monitoring Policy if applicable (optional).
f. Choosethe appropriate DNS Label if applicable (optional).
Click Finish.
The Ten
ant has been cre
ated. Now the net
work ad
min
is
tra
tor will have to as
so
ci
ate the
com
mon pri
vate net
work to the Ten
ant by first cre
at
ing a bridge do
main.
1
In the Navigation pane choose Tenant_Name > Networking > Bridge Domains.
In the Work pane, choose Actions > Create Bridge Domain.
In the Create Bridge Domain dialog box, perform the following actions:
a. Enter a Bridge Domain Name.
b. Choose network.
c. In the Subnets field, click +.
d. In the Gateway IP field enter the IP address for this subnet.
e. In the Scope field you have the option to select Private, Public and Shared.
Note: By default the Private option is selected. For more information on what
to select please reference the External Layer 3 section.
Click OK.
Click Finish.
Tenants 191
The con
fig
ur
a
tion for this use case can be ap
plied via the fol
low
ing CLI con
fig
ur
a
tion:
CLI : Ten
ant Cisco
# tenant
cd '/aci/tenants'
mocreate 'Cisco'
moconfig commit
# bridge-domain
cd '/aci/tenants/Cisco/networking/bridge-domains'
mocreate 'Cisco'
cd 'Cisco'
moset network 'default'
moconfig commit
# subnet
cd '/aci/tenants/Cisco/networking/bridge-domains/Cisco/subnets'
mocreate '172.16.0.1/24'
moconfig commit
# application-profile
cd '/aci/tenants/Cisco/application-profiles'
mocreate 'App1'
moconfig commit
# application-epg
cd '/aci/tenants/Cisco/application-profiles/App1/application-epgs'
mocreate 'EPG1'
cd 'EPG1'
moset bridge-domain 'Cisco'
moconfig commit
# criterion
cd '/aci/tenants/Cisco/application-profiles/App1/application-epgs/EPG1/vm-attributescriteria'
mocreate 'default'
moconfig commit
This con
fig
ur
a
tion can also be ap
plied using the fol
low
ing XML posted to the APIC
REST API
192 Tenants
XML : Ten
ant Cisco
<fvTenant name="Cisco">
<fvBD arpFlood="no" multiDstPktAct="bd-flood" name="Cisco" unicastRoute="yes"
unkMacUcastAct="proxy" unkMcastAct="flood">
<fvRsCtx tnFvCtxName="default"/>
<fvSubnet ctrl="nd" descr="" ip="172.16.0.1/24" preferred="no"
scope="private"/>
</fvBD>
<fvAp name="App1">
<fvAEPg matchT="AtleastOne" name="EPG1">
<fvRsBd tnFvBDName="Cisco"/>
</fvAEPg>
</fvAp>
<fvRsTenantMonPol tnMonEPGPolName=""/>
</fvTenant>
For many multi-ten

ant en
vi
ron
ments it is de
sir
able to allow each ten
ant to man
age and
own their own ad
dress space and not be con
cerned with over
laps be
tween other ten
ants. This par
tic
ul
ar use case demon
strates how a pri
vate net
work can be as
so
ci
ated
with each ten
ant. One Pri
vate Net
work per Ten
ant with In
tra-EPG com
mu
ni
ca
tions
Ad
van
tages:
Allow for maximum isolation between tenants
Ability to address hosts in tenants with overlapping IP addresses
Dis
ad
van
tages:
Increased complexity when needing to allow EPG communication between

different tenants with dedicated VRF
Tenants 193
The ob
ject con
tain
ment for this par
tic
u
lar setup can be de
picted as shown below:
Private Network per Tenant

To cre
ate the ten
ant:
1
On the menu bar, choose Tenants > ADD TENANT.
In the Create Tenant dialog box, perform the following actions:

a. Enter a Tenant Name.
b. Click Next.
Click Finish.
The Ten
ant has been cre
ated. Now the ten
ant ad
min
is
tra
tor can cre
ate the pri
vate net
work.
1

Networks.
4
5

a. Enter Private Network Name.
b. Click Next.
194 Tenants
c. Enter Associated Bridge Domain Name.

d. In the Subnets field, click +.
e. In the Gateway IP field enter the IP address for this subnet.
f. In the Scope field you have the option to select Private, Public and Shared.
Note: By default the Private option is selected. For more information on what
to select please reference the External Layer 3 section.
6
Click OK.
Click Finish.
The con
fig
ur
a
plied via the fol
low
ing CLI con
fig
ur
a
tion:
CLI : Ten
ant Cisco
# tenant
cd '/aci/tenants'
mocreate 'Cisco'
moconfig commit
# bridge-domain
mocreate 'Cisco'
cd 'Cisco'
moset network 'Cisco'
moconfig commit
# subnet
cd '/aci/tenants/Cisco/networking/bridge-domains/Cisco/subnets'
mocreate '172.16.0.1/24'
moconfig commit
# private-network
cd '/aci/tenants/Cisco/networking/private-networks'
mocreate 'Cisco'
moconfig commit
mocreate 'App1'
moconfig commit
# application-epg
mocreate 'EPG1'
cd 'EPG1'
moconfig commit
Tenants 195
This con
fig
ur
a
tion can also be ap
plied using the fol
low
REST API:
XML : Ten
ant Cisco
<fvTenant name="Cisco">
<fvBD arpFlood="no" multiDstPktAct="bd-flood" name="Cisco" unicastRoute="yes"
unkMacUcastAct="proxy" unkMcastAct="flood">
<fvRsCtx tnFvCtxName="Cisco"/>
<fvSubnet ctrl="nd" ip="172.16.0.1/24" name="" preferred="no" scope="private"/>
</fvBD>
<fvCtx knwMcastAct="permit" name="Cisco" pcEnfPref="enforced"/>
<fvAp name="App1" prio="unspecified">
<fvAEPg name="EPG1">
<fvRsBd tnFvBDName="Cisco"/>
</fvAEPg>
</fvAp>
</fvTenant>
Multiple Private Networks with Intra-Tenant Communication

An
other use case that may be de
sir
able to sup
port is the op
tion to have a sin
gle ten
ant
with mul
ti
ple pri
vate net
works. This may be a re
sult of need
ing to pro
vide multi-ten
ancy at a net
work level, but not at a man
age
ment level. It may also be caused by need
ing to sup
port over
lap
ping sub
nets within a sin
gle ten
ant, due to merg
ers and ac
qui
si
tions or other busi
ness changes.
low
ing ad
van
tages and dis
ad
van
tages:
Ad
van
tages:
Ability to have overlapping subnets within a single tenant
Dis
ad
van
tages:
EPGs residing in overlapping subnets cannot have policy applied between one
another
196 Tenants
The ob
ject con
tain
ment for this par
tic
u
lar setup can be de
picted as shown below:
Multiple Private Networks with Intra-Tenant communication

To cre
ate the ten
ant:
1
On the menu bar, choose Tenants > ADD TENANT.
In the Create Tenant dialog box, perform the following actions:

a. Enter a Tenant Name.
Click Next.
Click Finish.
The Ten
ant has been cre
ated. Now the ten
ant ad
min
is
tra
tor can cre
ate the pri
vate net
work.
1

Networks.
4
5

a. In the Name field enter a name for the Private Network.
b. Click Next.
Tenants 197
d. In the Subnets field click +.

f. In the Scope field select Shared.
Note: The shared subnet type causes what is known in ACI as a route
leak between two Private Networks (VRF).
6
Click OK.
Click Finish.

Networks.
9
10

a. In the Name field enter a name for the second Private Network.
b. Click Next.
c. In the Name field enter a name for the second bridge domain.
11
Click OK.
12
Click Finish.
Now that the two pri

vate net
works and bridge do
mains have been cre
ated, you can
move to the Ap
pli
ca
tion Pro
file.
1
In the Navigation pane choose Tenant_Name > Networking > Application

Profiles.
4
5
In the Action Tab select Create Application Profile.

Click Submit.
198 Tenants
To cre
ate the two end
point groups:
1

Profiles > Application Profile_Name.
4
5

a. In the Name field enter a name for the End Point Group.
b. In the Bridge Domain field, select the appropiate bridge domain.
Click Finish.
Re
peat these steps for the sec
ond EPG.
The ten
ant ad
min
is
tra
tor will have to now cre
ate a con
tract and fil
ter be
tween the two
EPGs.
1
In the Navigation pane choose Tenant_Name > Networking > Security

Policies > Filters.
4
5
In the Work pane, choose Actions > Create Filters.

In the Create Filter dialog box, perform the following actions:
a. In the Name field enter a name for the Filter.
b. Click +.
c. In the Name column enter ICMP.
d. In the Ethertype select IP.
e. In the IP Protocol column select ICMP.
Click Update.
Click Submit.
As the ten
ant ad
min
is
tra
tor, you would have the knowl
edge of the fil
ters re
quired to
per
mit traf
fic across the two EPGs. In the fil
ter, you can re
peat the process of defin
ing
Tenants 199
var
io
us dif
fer
ent net
work pro
to
cols as re
quired for your ap
pli
ca
tions. Now you have to
de
fine the con
tract that will be con
sumed and pro
vided by the two EPGs.
1

Policies > Contracts.
4
5
In the Work pane, choose Actions > Create Contract.

In the Create Contract dialog box, perform the following actions:
b. In the Scope field select Global.
c. Click Subjects field click +.
d. In the Name field enter a name for the Subject.
e. In the Filter Chain field click +.
f. Select the Filter you just created.
Click Update.
Click OK.
Click Submit.
As
sign the Con
tract be
tween the EPGs. A con
tract is as
signed to an EPG as ei
ther a
con
sumed or a pro
vided con
tract. Each EPG that you have cre
ated will then ei
ther con
sume or pro
vide that con
tract to es
tab
lish a re
la
tion
ship be
tween both EPGs.
1

Application Profile Name > Application EPGs > EPG_Name > Contract.
In the Work pane, choose Actions > Add Provided Contract.
In the Add Provided Contract dialog box, perform the following actions:
a. Select the created contract.
b. Click Submit..
6
7
In the Navigation pane under the second EPG select Contracts.

In the Action Tab select Add Consume Contract.
a. Select the created contract
200 Tenants
Click Submit.
The con
fig
ur
a
plied via the fol
low
ing CLI con
fig
ur
a
tion:
CLI : Ten
ant Cisco
# tenant
cd '/aci/tenants'
mocreate 'Cisco'
moconfig commit
# bridge-domain
mocreate 'Cisco'
cd 'Cisco'
moconfig commit
# bridge-domain
mocreate 'Cisco1'
cd 'Cisco1'
moset network 'Cisco1'
moconfig commit
# private-network
mocreate 'Cisco'
moconfig commit
# private-network
mocreate 'Cisco1'
moconfig commit
mocreate 'App1'
moconfig commit
# application-epg
mocreate 'EPG1'
cd 'EPG1'
Tenants 201
moconfig commit
# fv-rscon
cd '/aci/tenants/Cisco/application-profiles/App1/applicationepgs/EPG1/contracts/consumed-contracts'
mocreate 'ICMP'
moconfig commit
# fv-subnet
cd '/aci/tenants/Cisco/application-profiles/App1/application-epgs/EPG1/subnets'
mocreate '172.16.1.1/24'
cd '172.16.1.1:24'
moset scope 'private,shared'
moconfig commit
# application-epg
mocreate 'EPG2'
cd 'EPG2'
moset bridge-domain 'Cisco1'
moconfig commit
# fv-rsprov
cd '/aci/tenants/Cisco/application-profiles/App/applicationepgs/EPG2/contracts/provided-contracts'
mocreate 'ICMP'
moconfig commit
# fv-subnet
cd '/aci/tenants/Cisco/application-profiles/CCO/application-epgs/EPG2/subnets'
mocreate '172.16.2.1/24'
cd '172.16.2.1:24'
moconfig commit
This con
fig
ur
a
tion can also be ap
plied using the fol
low
REST API:
XML : Ten
ant Cisco
<fvTenant dn="uni/tn-Cisco" name="Cisco">
<vzBrCP name="ICMP" scope="tenant">
<vzSubj consMatchT="AtleastOne" name="icmp" provMatchT="AtleastOne"
202 Tenants
revFltPorts="yes">
<vzRsSubjFiltAtt tnVzFilterName="icmp"/>
</vzSubj>
</vzBrCP>
<fvCtx knwMcastAct="permit" name="CiscoCtx" pcEnfPref="enforced"/>
<fvCtx knwMcastAct="permit" name="CiscoCtx2" pcEnfPref="enforced"/>
<fvBD arpFlood="yes" mac="00:22:BD:F8:19:FF" name="CiscoBD2" unicastRoute="yes"
unkMacUcastAct="flood" unkMcastAct="flood">
<fvRsCtx tnFvCtxName="CiscoCtx2"/>
</fvBD>
<fvBD arpFlood="yes" mac="00:22:BD:F8:19:FF" name="CiscoBD" unicastRoute="yes"
<fvRsCtx tnFvCtxName="CiscoCtx"/>
</fvBD>
<fvAp name="CCO">
<fvAEPg matchT="AtleastOne" name="Web">
<fvRsCons tnVzBrCPName="ICMP"/>
tDn="topology/pod-1/paths-201/pathep-[eth1/16]"/>
<fvSubnet ip="172.16.2.1/24" scope="private,shared"/>
<fvRsDomAtt instrImedcy="lazy" resImedcy="lazy" tDn="uni/physPhysDomainforCisco"/>
<fvRsBd tnFvBDName="CiscoBD2"/>
</fvAEPg>
<fvAEPg matchT="AtleastOne" name="App">
<fvRsBd tnFvBDName="CiscoBD"/>
<fvRsProv matchT="AtleastOne" tnVzBrCPName="ICMP"/>
</fvAEPg>
</fvAp>
</fvTenant>
Tenants 203

i
cal for en
vi
ron
min
is
tra
tor wishes to
cre
ate mul
ti
ple ten
ants with the abil
ity to sup
port in
ter-ten
ant com
mu
ni
ca
tions.
low
ing ad
van
tages and dis
ad
van
tages:
Ad
van
tages
Each tenant container can be managed separately
Allows for maximum isolation between tenants
Dis
ad
van
tages
Tenant address space must be unique
From a con
tain
ment and re
la
tion
ship per
spec
tive, this topol
ogy looks as fol
lows:

To cre
ate the ten
ant:
1
In the GUI Navigate to Tenants > ADD TENANT.

a. In the Name field enter a name for the first Tenant.
Click Next.
204 Tenants
Click Finish.
The ten
ant has been cre
ated. Now the ten
ant ad
min
is
tra
tor can cre
ate the pri
vate net
work.
1
In the Navigation Pane choose Tenant_Name > Networking > Private

Networks.
b. Click Next.
Click OK.
Click Finish.
To cre
ate the ap
pli
ca
tion pro
file:
1
In the Navigation Pane choose Tenant_Name > Networking > Application

Profiles.
4
5

Click Submit.
Tenants 205
To cre
ate the end
point group:
1

Profiles > Application Profile_Name
4
5

Click Finish.
To cre
ate the sec
ond ten
ant and ap
pli
ca
tion pro
file:
1
In the GUI Navigate to Tenants > ADD TENANT.

a. In the Name field enter a name for the first Tenant.
Click Next.
Click Finish.
The ten
ant has been cre
ated. Now the ten
ant ad
min
is
tra
tor can cre
ate the pri
vate net
work.
1

Networks.
b. Click Next.
206 Tenants

6
Click OK.
Click Finish.
To cre
ate the ap
pli
ca
tion pro
file:
1

Profiles.
4
5

Click Submit.
To cre
ate the end
point group:
1

Profiles > Application Profile_Name
4
5

Click Finish.
The ten
ant ad
min
is
tra
tor will have to now cre
ate a con
tract and fil
ter be
tween the two
EPGs.
1
Tenants 207

Policies > Filters.
In the Work pane, choose Actions > Create Filters.

b. Click +.
c. In the Name column enter ICMP.
d. In the Ethertype select IP.
e. In the IP Protocol column select ICMP.
Click Update.
Click Submit.
As the ten
ant ad
min
is
tra
tor you would have the knowl
edge of the fil
ters re
quired to
per
mit traf
fic across the two EPGs. In the fil
ter you can re
peat the process of defin
ing
var
io
us dif
fer
ent net
work pro
to
cols as re
quired for your ap
pli
ca
tions. Now you have to
de
fine the con
tract that will be con
sumed and pro
vided by the two EPGs.
1

Policies > Contracts.
4
5

a. In the Name field enter a name for the filter.
b. In the Scope field choose Global.
c. Click Subjects field click +.
d. In the Name field enter a name for the Subject.
e. In the Filter Chain field click +.
f. Select the filter you just created.
Click Update.
Click OK.
Click Submit.
208 Tenants
As
sign the Con
tract be
tween the EPGs. A con
tract is as
signed to an EPG as ei
ther a
con
sumed or a pro
vided con
tract. Each EPG that you have cre
ated will then ei
ther con
sume or pro
vide that con
tract to es
tab
lish a re
la
tion
ship be
tween both EPGs.
1

Profiles > Application Profile Name > Application EPGs > EPG_Name >
Contracts.
In the Work pane, choose Actions > Add Provided Contract.
In the Add Provided Contract dialog box, perform the following actions:
a. Select the created Contract.
b. Click Submit.
In the GUI Navigate to Tenants > ALL TENANTS.
Select the second created Tenant.

Profiles > Application Profile Name > Application EPGs > EPG_Name >
Contracts.
In Work pane, choose Action > Add Consume Contract.

a. Select the created Contract.
10
Click Submit.
The con
fig
ur
a
plied via the fol
low
ing CLI con
fig
ur
a
tion:
CLI : TEN
ANT Cis
co1
# tenant
cd '/aci/tenants'
mocreate 'Cisco1'
moconfig commit
# bridge-domain
cd '/aci/tenants/Cisco1/networking/bridge-domains'
mocreate 'Cisco1'
cd 'Cisco1'
moset network 'Cisco1'
moconfig commit
Tenants 209
# private-network
cd '/aci/tenants/Cisco1/networking/private-networks'
mocreate 'Cisco1'
moconfig commit
cd '/aci/tenants/Cisco1/application-profiles'
mocreate 'App1'
moconfig commit
# application-epg
cd '/aci/tenants/Cisco1/application-profiles/App1/application-epgs'
mocreate 'EPG1'
cd 'EPG1'
moset bridge-domain 'Cisco1'
moconfig commit
# fv-rsprov
cd '/aci/tenants/Cisco/application-profiles/CCO/applicationepgs/App/contracts/provided-contracts'
mocreate 'ICMP'
moconfig commit
# fv-subnet
cd '/aci/tenants/Cisco/application-profiles/CCO/application-epgs/App/subnets'
mocreate '172.16.1.1/24'
cd '172.16.1.1:24'
moconfig commit
# contract
cd '/aci/tenants/Cisco/security-policies/contracts'
mocreate 'ICMP'
cd 'ICMP'
moset scope 'global'
moconfig commit
# contract-subject
cd '/aci/tenants/Cisco/security-policies/contracts/ICMP/subjects'
mocreate 'icmp'
moconfig commit
# vz-rssubjfiltatt
cd '/aci/tenants/Cisco/security-policies/contracts/ICMP/subjects/icmp/common-filters'
mocreate 'icmp'
moconfig commit
210 Tenants
CLI : TEN
ANT Cis
co2
# tenant
cd '/aci/tenants'
mocreate 'Cisco'
moconfig commit
# bridge-domain
mocreate 'Cisco'
cd 'Cisco'
moconfig commit
# private-network
mocreate 'Cisco'
moconfig commit
mocreate 'App1'
moconfig commit
# application-epg
cd '/aci/tenants/Cisco2/application-profiles/App1/application-epgs'
mocreate 'EPG1'
cd 'EPG1'
moconfig commit
# fv-rsconsif
cd '/aci/tenants/Cisco1/application-profiles/CCO/applicationepgs/Web/contracts/consumed-contract-interfaces'
mocreate 'CiscoInterTenantICMP'
moconfig commit
# fv-subnet
cd '/aci/tenants/Cisco1/application-profiles/CCO/application-epgs/Web/subnets'
mocreate '172.16.2.1/24'
cd '172.16.2.1:24'
moset scope 'shared-subnet'
moconfig commit
# imported-contract
cd '/aci/tenants/Cisco1/security-policies/imported-contracts'
mocreate 'CiscoInterTenantICMP'
cd 'CiscoInterTenantICMP'
moset contract 'tenants/Cisco/security-policies/contracts/ICMP'
moconfig commit
Tenants 211
This con
fig
ur
a
tion can also be ap
plied using the fol
low
REST API:
XML : TEN
ANT Cis
co1
<fvTenant dn="uni/tn-Cisco1" name="Cisco1">
<vzBrCP name="ICMP" scope="global">

<vzSubj consMatchT="AtleastOne" name="icmp" provMatchT="AtleastOne"
revFltPorts="yes">
<vzRsSubjFiltAtt tnVzFilterName="icmp"/>
</vzSubj>
</vzBrCP>
<vzCPIf dn="uni/tn-Cisco1/cif-ICMP" name="ICMP">
<vzRsIf consMatchT="AtleastOne" name="icmp" provMatchT="AtleastOne"

revFltPorts="yes">
<vzRsSubjFiltAtt tDn="uni/tn-Cisco2/brc-default"/>
</vzRsIf>
</vzCPIf>

<fvRsCtx tnFvCtxName="CiscoCtx2"/>
</fvBD>
<fvBD arpFlood="yes" name="CiscoBD" unicastRoute="yes" unkMacUcastAct="flood"
unkMcastAct="flood">
</fvBD>
<fvAp name="CCO">
212 Tenants

<fvRsProv matchT="AtleastOne" tnVzBrCPName="ICMP"/>
</fvAEPg>
</fvAp>
</fvTenant>
XML : TEN
ANT Cis
co2
<fvTenant dn="uni/tn-Cisco2" name="Cisco2">
</fvBD>
<fvBD arpFlood="yes" name="CiscoBD" unicastRoute="yes" unkMacUcastAct="flood"
unkMcastAct="flood">
</fvBD>
<fvAp name="CCO">
Tenants 213
<fvRsConsIf matchT="AtleastOne" tnVzBrCPIfName="ICMP"/>
</fvAEPg>
</fvAp>
</fvTenant>
215
Working with Contracts
Working with Contracts 217
Section Content
Contracts
Contract Configuration Parameters
Create/Modify/Remove Contracts
Create Contracts
Modify Contracts
Remove Contracts
Verify Contracts
Apply/Remove EPG Contracts

Apply a Contract to an EPG
Remove a Contract from EPG
Verify Contract on EPG
Apply/Remove External Network Contracts

Apply a Contract to an External Network
Remove a Contract from an External Network
Verify External Network Contracts
Apply/Remove Private Network Contracts

Apply a Contract to a Private Network (vzAny)
Remove a Contract From a Private Network (vzAny)
Verify Private Network Contracts
Filters
Filter Entry Configuration Parameters
Create Filters
Modify Filters
Remove Filters
Verify Filters
218 Working with Contracts
Taboo Contracts
Taboo Contract Configuration Parameters
Create, Modify, or Delete Taboo Contracts
Create Taboo Contracts
Modify Taboo Contracts
Delete Taboo Contracts
Verify Taboo Contracts
Apply/Remove Taboo Contracts

Apply Taboo Contract to an EPG
Remove Taboo Contract from EPG
Verify Taboo Contracts Applied to EPG
Inter-Tenant Contracts
Configuration Parameters
Create/Modify/Remove Export Contracts

Export Contract
Modify Exported Contracts
Remove Exported Contracts
Verify Exported Contracts
Contracts Use Cases

Tenant Cisco-1/EPG-1
Inter-Private Network Contracts Communication

Single Contract Bidirectional Reverse Filter
Single Contract Unidirectional with Multiple Filters
Multiple Contracts Uni-Directional Single Filter
Contracts
Con
tracts pro
vide a way for the ACI ad
min
is
tra
tor to con
trol traf
fic flow within the ACI
fab
ric be
tween EPGs. These con
tracts are built using a provider-con
sumer model
where one EPG pro
vides the ser
vices it wants to offer and an
other EPG con
sumes
them.
Con
tracts are com
prised of the fol
low
ing items:
Subjects - A group of filters for a specific application or service
Filters - Used to classify traffic based upon layer 2 to layer 4 attributes (such as
Actions - Permit, deny, mark, log, redirect, copy, and service graph to perform
Ethernet type, protocol type, TCP flags and ports)

matches based upon filters
Labels - Used optionally to group objects such as subjects and EPGs for the
purpose of further defining policy enforcement
EPGs can only com

mu
ni
cate with other EPGs based upon the con
tract rules de
fined. There is no con
tract re
quired for in
tra-EPG com
mu
ni
ca
tion: in
tra-EPG com
mu
ni
ca
tion is al
lowed by de
fault.
The ex
am
ple below, shows how con
tracts would con
trol traf
fic flow be
tween EPGs in a
3-tiered ap
pli
ca
tion. The Web EPG pro
vides a con
tract which is con
sumed by the Ap
pli
ca
tion EPG, and the Ap
pli
ca
tion EPG pro
vides a con
tract which the Data
base EPG
would con
sume. EPGs may acts a both a provider and con
sumer of the same con
tract
eas
ily by mak
ing the con
tract bidi
rec
tional.
Contract Policies Between End Point Groups

Con
tracts gov
ern the fol
low
ing types of end
point group com
mu
ni
ca
tions:
Between application EPGs
Between application EPGs and external networks
Between application EPGs and in-band management EPG, for example if inband management is configured for the ACI fabric and certain EPGs are to be
allowed to access it
Contract Configuration Parameters

When con
fig
ur
ing con
tracts you can de
fine the fol
low
ing op
tions:
Con
tract Scope - The scope of a ser
vice con
tract be
tween two or more par
tic
i
pat
ing
peer en
ti
ties.
The states are:
Context - This contract will be applied for endpoint groups associated with the
same private network.
Application-profile - This contract will be applied for endpoint groups in the
Tenant - This contract will be applied for endpoint groups within the same tenant.
same application profile.
Global - This contract will be applied for endpoint groups throughout the
fabric.
The de
fault is Con
text.
QoS Class - The pri

or
ity level of the ser
vice con
tract.
The pri
or
ity level can be:
Unspecified
Level1 - Class 1 Differentiated Services Code Point (DSCP) value.
Level2 - Class 2 DSCP value.
Level3 - Class 3 DSCP value.

The de
fault is Un
spec
i
fied.
Tags - The search key

word or term that is as
signed to the ap
pli
ca
tion pro
file. A tag al
lows you to group mul
ti
ple ob
jects by a de
scrip
tive name. You can as
sign the same tag
name to mul
ti
ple ob
jects and you can as
sign one or more tag names to an ob
ject.
Match - The sub
ject match cri
te
ria across con
sumers. The op
tions are:
AtleastOne
AtmostOne
None
All
The de
fault is Atlea
st
O
ne.
Create/Modify/Remove Contracts
Create Contracts
1
In the Navigation pane choose Tenant_Name > Security Policies > Contracts.
4
5

a. Enter a Contract Name.
b. Choose a Contract Scope (optional).
c. Choose a QoS Class (optional).
d. Click + next to the Subject to add a Contract Subject.
i. In the Create Contract Subject dialog box, perform the following actions:
1. Enter a Contract Subject Name.
2. Click + in the Filter Chain field.
For information regarding filter creation, see the "Filters" section.
Click Update.
Click OK.
Click Submit.
Modify Contracts
1
In the Navigation pane choose Tenant_Name > Security Policies > Contracts >
Contract_Name.
In the Work pane, choose the Policy tab.

a. Choose a Contract Scope (optional).
b. Choose a Qos Class (optional).
c. Click + next to the Subject field. to add a Contract Subject.
i. In the Create Contract Subject dialog box, perform the following actions:
1. Enter a Contract Subject Name.
2. Click + next to Filter Chain.
Note: For information regarding filter creation, see the "Filters" section.
Click Update.
Click OK.
Click Submit.
Remove Contracts
Remove Contracts
1
In the Navigation pane choose Tenant_Name > Security Policies > Contracts
> Contract_Name.
Verify Contracts
REST :: /api/node/class/vzBrCP.xml
CLI :: moquery -c vzBrCP
Apply/Remove EPG Contracts

Apply a Contract to an EPG
1

Application_Profile_Name > Application EPGs > EPG_Name > Contracts.
In the Work pane, choose Actions > Add Provided Contract or Actions > Add
Consumed Contract.
Note: Choose the action depending on how the contract is to be deployed.
In the Add Contract dialog box, perform the following actions:

a. Enter a Contract_Name.
b. Choose a QOS policy (optional).
c. Choose a Label (optional).
Click Submit.
Remove a Contract from EPG

1

> Application_Profile_Name .> Application EPGs > EPG_Name > Contracts >
Contract_Name.
Verify Contract on EPG

Provider
REST :: /api/node/class/fvRsProv.xml
CLI :: moquery -c fvRsProv
Consumer
REST :: /api/node/class/fvRsCons.xml
CLI :: moquery -c fvRsCons
Apply/Remove External Network Contracts

Apply a Contract to an External Network
1
In the Navigation pane choose Tenant_Name > Networking > External Routed
Networks > Routed Outside_Name > Networks >
External_Network_Instance_Profile.
In the Work pane, click + next to either Add Provided Contract or Add
Consumed Contract.
Note: Make a selection depending on how the contract is to be deployed.
a. Choose a Contract_Name.
b. Choose a QOS Type.
c. Choose a Match Criteria.
Click Update.
Remove a Contract from an External Network

1
In the Navigation pane choose Tenant_Name > Networking > External Routed
Networks > Routed Outside_Name > Networks
> External_Network_Instance_Profile.
In the Work pane, choose the Contract_Name and click x.
Verify External Network Contracts

Provider
Consumer
Apply/Remove Private Network Contracts

In order to apply con
tracts to all end
point groups within a pri
vate net
work, con
tracts
can be ap
plied di
rectly to the pri
vate net
work. This con
cept is also re
ferred as "vzAny"
end
point group
. It eases con
tract man
age
ment by al
low
ing the con
tract con
fig
ur
a
tion
for all end
point groups within a pri
vate net
work from a sin
gle lo
ca
tion as well as op
ti
miz
ing hard
ware re
source con
sump
tion.
For in
stance, if an ACI Ad
min
is
tra
tion has 100 EPGs that are all part of the same pri
vate
net
work, they can apply the con
tracts to this one vzAny group under the pri
vate net
work, rather than to each EPG.
Apply a Contract to a Private Network (vzAny)

1

Networks > Private_Network_Name > EPG Collection for Context.
In the Work pane, click + next to either Add Provided Contract or Add
Consumed Contract.
Note: Make a selection depending on how the contract is to be deployed.
a. Enter a Contract_Name.
b. Choose a QOS Type.
c. Choose a Match Criteria.
Click Update.
Remove a Contract From a Private Network (

vzAny)
1

Networks > Private_Network_Name > EPG Collection for Context.
In the Work pane, choose the Contract_Name and click x.
Verify Private Network Contracts

REST :: /api/node/class/vzBrCP.xml
CLI :: moquery -c vzBrCP
Filters
A fil
ter pol
icy is a group of re
solv
able fil
ter en
tries. Each fil
ter entry is a com
bi
na
tion of
net
work traf
fic clas
si
fi
ca
tion prop
er
ties.
Fil
ters are Layer 2 to Layer 4 fields, TCP/IP header fields such as Layer 3 pro
to
col type,
Layer 4 ports, and so on. Ac
cord
ing to its re
lated con
tract, an EPG provider dic
tates the
pro
to
cols and ports in both the in and out di
rec
tions. Con
tract sub
jects con
tain as
so
ci
a
tions to the fil
ters (and their di
rec
tions) that are ap
plied be
tween EPGs that pro
duce
and con
sume the con
tract. See the Use Cases below for an ex
am
ple.
Filter Entry Configuration Parameters

When con
fig
ur
ing a Fil
ter, the fol
low
ing op
tions can be de
fined:
Name - The name of a fil
ter entry.
Ether
Type - The Ether
Type of the fil
ter entry. The Ether
Types are:
ARP
FCOE
IP
MAC Security
MPLS Unicast
Trill
Unspecified
The de
fault is ARP.
Arp Flag - The Ad

dress Res
ol
u
tion Pro
to
col flag for a fil
ter entry. The fil
ter entry is a
com
bi
na
tion of net
work traf
fic clas
si
fi
ca
tion prop
er
ties.
IP Pro
to
col - The IP pro
to
col for a fil
ter entry. The fil
ter entry is a com
bi
na
tion of net
work traf
fic clas
si
fi
ca
tion prop
er
ties.
Allow Frag
ment - The start of the source port range. The start of the port range is de
ter
mined by the server type. The range is 0 to 0xffff. The port can be for the fol
low
ing
server types:
Source Port: From - The start of the source port range. The start of the port range is
de
ter
low
ing server types:
Unspecified
ftpData
SMTP
DNS
HTTP
POP3
HTTPS
RTSP
Source Port: To - The end of the source port range. The end of the port range is de
ter
low
ing
server types:
Unspecified
ftpData
smtp
DNS
HTTP
POP3
HTTPS
RTSP
ti
na
tion port range. The end of the port
Des
ti
na
tion Port: From - The end of the des
range is de
ter
mined by the server type. The range is 0 to 0xffff. The port state set
tings
are:
Unspecified
ftpData
SMTP
DNS
HTTP
POP3
HTTPS
RTSP
Des
ti
na
tion Port: To - The start of the des
ti
na
tion port range. The port range is de
ter
mined by the server type. The range is 0 to 0xffff. The des
ti
na
tion port can be set for
the fol
low
ing server types:
unspecified
ftpData
smtp
dns
http
pop3
https
rtsp
The de
fault is un
spec
i
fied.
sion rules for a fil

ter entry. The fil
ter entry is a com
bi
TCP Ses
sion Rules - The TCP ses
na
tion of net
work traf
fic clas
si
fi
ca
tion prop
er
ties.
Create Filters
1
In the Navigation pane choose Tenant_Name > Security Policies > Filters.
In the Work pane, choose Actions > Create Filter.

a. Enter an Filter Name.
b. Click + to add a Filter Entry.
i. In the Filter Entry dialog box, perform the following actions:

1. Enter a Filter Entry Name.
2. In the drop-down list select an Ethertype.
3. In the drop-down list select an ARP Flag (optional).
4. In the drop-down list select an IP Protocol (optional).
5. If required, check the Allow Fragment check box (optional).
6. In the drop-down list select the Source Port From (optional).
7. In the drop-down list select the Source Port To (optional).
8. In the drop-down list select the Destination Port From (optional).
9. In the drop-down list select the Destination Port To (optional).
10. In the drop-down list select the TCP Session Rules (optional).
6
Click Update.
Click Submit.
Modify Filters
1
In the Navigation pane choose Tenant_Name > Security Policies > Filters >
Filter_Name.
In the Work pane, double click the filter entry.

a. In the drop-down list select an Ethertype (optional).
b. In the drop-down list select an ARP Flag (optional).
c. In the drop-down list select an IP Protocol (optional).
d. If required, click the Allow Fragment check box (optional).
e. In the drop-down list select the Source Port From (optional).
f. In the drop-down list select the Source Port To (optional).
g. In the drop-down list select the Destination Port From (optional).
h. In the drop-down list select the Destination Port To (optional).
i. In the drop-down list select the TCP Session Rules (optional).
Click Update.
Click Submit.
Remove Filters
1
In the Navigation pane choose Tenant_Name > Security Policies > Filters >
Filter_Name.
Verify Filters
REST :: /api/node/class/vzFilter.xml
CLI :: moquery -c vzFilter
Taboo Contracts
There may be times when the ACI ad
min
is
tra
tor might need to deny traf
fic that is al
lowed by an
other con
tract. Taboos are a spe
cial type of con
tract that an ACI ad
min
is
tra
tor can use to deny spe
cific traf
fic that would oth
er
wise be al
lowed by an
other con
tract. Taboos can be used to drop traf
fic match
ing a pat
tern (any EPG, a spe
cific EPG,
match
ing a fil
ter, and so forth). Taboo rules are ap
plied in the hard
ware be
fore the
rules of reg
ul
ar con
tracts are ap
plied.
Taboo con
tracts are not rec
om
mended as part of the ACI best prac
tices but they can
be used to tran
si
tion from tra
di
tional net
work
ing to ACI. To im
it
ate the tra
di
tional net
work
ing con
cepts, an "al
low-all-traf
fic" con
tract can be ap
plied, with taboo con
tracts
con
fig
ured to re
strict cer
tain types of traf
fic.
Taboo Contract Configuration Parameters

When con
fig
ur
ing Taboo Con
tracts you can de
fine the fol
low
ing op
tions:
Name - The name of the con
tract or con
tract ob
ject.
Sub
jects - The net
work do
main name label. La
bels en
able clas
si
fi
ca
tion of the ob
jects
which can and can
not com
mu
ni
cate with one an
other (op
tional).
Di
rec
tive - The fil
ter di
rec
tives as
signed to the taboo con
tract.
Create, Modify, or Delete Taboo Contracts

Create Taboo Contracts
1
In the Navigation pane choose Tenant_Name > Security Policies > Taboo
Contracts.
4
5
In the Work pane, choose Action > Create Taboo Contract.

In the Create Taboo Contract dialog box, perform the following actions:
a. Enter a Taboo Contract Name.
b. Click + to next to the Subject field to add a Taboo Subject.
i. Enter a Filter Name.
ii. Choose Directives.
Click Update.
Click OK.
Click Submit.
Modify Taboo Contracts

1
In the Navigation pane choose Tenant_Name > Security Policies > Taboo
Contracts > Taboo_Contract_Name.

a. Click + to next to the Subject field.
b. In the Create Taboo Contract Subject dialog box, perform the following
actions:
i. Enter a Taboo Contract Subject Name.
ii. Click + in the Filter Chain field.
1. Enter a Filter Name.
2. Choose Directives.
Click Submit.
Delete Taboo Contracts

1
In the Navigation pane choose Tenant_Name > Security Policies

> Taboo Contracts > Taboo_Contract_Name.
In the Work pane, choose Action > Delete.
Verify Taboo Contracts

REST :: /api/node/class/vzTaboo.xml
CLI :: moquery -c vzTaboo
Apply/Remove Taboo Contracts

Apply Taboo Contract to an EPG
1

> Application_Profile_Name > Application EPGs > EPG_Name > Contracts.
In the Work pane, choose Actions > Add Taboo Contract.
In the Add Taboo Contract dialog box,

a. Choose the Taboo Contract.
Click Submit.
Remove Taboo Contract from EPG

1

> Application_Profile_Name > Application EPGs > EPG_Name > Contracts.
In the Work pane, choose the Taboo Contract_Name > Actions > Delete.
Verify Taboo Contracts Applied to EPG

Provider
C
onsumer
There may be times when the ACI ad
min
is
tra
tor might need to allow traf
fic be
tween
two ten
ants. In
ter
face con
tracts are a spe
cial type of con
tract that an ACI ad
min
is
tra
tor can use to allow spe
cific traf
fic through the use of a con
tract ex
port. The con
tract
in essence is ex
ported in the source ten
ant and im
ported into the tar
get ten
ant. Sim
il
ar
to tra
di
tional con
tracts, the source EPG will be of type provider. How
ever, in the tar
get
ten
ant, the con
tract is im
ported as type con
tract in
ter
face. Some use case ex
am
ples
show the com
plete process in the next chap
ter.
Configuration Parameters
When Im
port
ing a Con
tract, the fol
low
ing op
tions can be de
fined:
tract in
ter
face.
Name - The name of the con
Global Con
tract - Name of a ser
vice con
tract to be shared be
tween two or more par
tic
ip
at
ing peer en
ti
ties.
Ten
ant - The Ten
ant name of the tar
geted Ex
port con
tract.
Create/Modify/Remove Export Contracts

Export Contract
1
In the Navigation pane choose Tenant_Name > Security Policies > Contracts.
In the Work pane, choose Actions > Export Contract.
In the Export Contract dialog box, perform the following actions:

a. Enter an Export Contract Name.
b. Choose the Global Contract.
c. Enter the Tenant Name.

6
Click Finish.
Modify Exported Contracts

1
In the Navigation pane choose Tenant_Name > Security Policies > Contracts >
Contract_Name.

a. Enter an Export Contract Name.
b. Choose the Global Contract.
c. Enter the Tenant Name.
Click Finish.
Remove Exported Contracts

1
In the Navigation pane choose Tenant_Name > Security Policies > Contracts
> Imported Contracts > Contact_Name.
Verify Exported Contracts

REST :: /api/node/class/vzCPif.xml
CLI :: moquery -c vzCPif
Contracts Use Cases

These use cases all as
sume the ob
jec
tive is for a host in EPG-1 to talk to a host in EPG2, achiev
ing bidi
rec
tional traf
fic. How these sce
nar
ios are im
ple
mented will de
pend on
the op
er
at
ional model cho
sen, and whether the sys
tem is more fo
cused on ob
ject reuse or ten
ant au
ton
omy. Re
view the con
tracts sec
tion on Con
tract Scop
ing for a more
de
tailed dis
cus
sion.
There are four com
mon sce
nar
ios:
1
Inter-Tenant Contracts.
Inter-Private Network Contracts.
Single Contract Bidirectional forwarding with reverse filter.
Single Contract Unidirectional with multiple Filters.
Multiple Contracts Unidirectional with single Filter.
ACME Inc., as with most com
pa
nies, makes use of shared ser
vices such as DNS for name
res
o
lu
tion and Ac
tive Di
rec
tory for user man
age
ment. These ser
vices will be used across
most of their ten
ants and so ACME Inc. must allow this traf
fic across the whole fab
ric. Com
mu
ni
ca
tion be
tween EPGs that be
long to dif
fer
ent ten
ants is only al
lowed when
they share the same con
tract. To use the same con
tract, it will need to be ex
ported from
the source ten
ant to the ap
pro
pri
ate des
ti
na
tion ten
ant. That con
tract will ap
pear under
the Im
ported Con
tract sec
tion in the Se
cu
rity Poli
cies of the des
ti
na
tion ten
ant.
tract In
ter
face will be used to as
so
ci
ate an EPG from the des
ti
na
tion
A Con
sumed Con
ten
ant with the im
ported con
tract.
Note: A con
tract con
sump
tion in
ter
face rep
re
sents one or more sub
jects de
fined under
the con
tract. By as
so
ci
at
ing to an in
ter
face, an end
point group starts con
sum
ing all the
sub
jects rep
re
sented by the in
ter
face.
In the use case below, EPG-1 in ten
ant Cisco-1 re
quires com
mu
ni
ca
tion with EPG-2 in
ten
ant Cisco-2. This is ac
com
plished by uti
liz
ing con
tact in
ter
faces. In ten
ant Cisco-1
the user will ex

port the in
tended con
tract in
ter
faces. In ten
ant Cisco-1 the user will ex
port the in
tended con
tract and se
lect provider to pro
vide the con
trast to EPG-2. The
user will then con
firm the im
ported con
tract in ten
ant Cisco-2 and se
lect the con
tract
as con
sumed. To ad
ver
tise the routes from the source VRF to the in
tended VRF, the
user must cre
ate the sub
net within the EPG.
Exporting Contracts Between Tenants
1
Create an Export Contract under security policies.
Create the host subnet (default Gateway IP) under EPG1 - subnet scope shared.
Add the Contract under EPG1 - contract type provider.
Create the host subnet under the bridge domain - subnet scope private/public.
1
Confirm the exported contract is listed under Imported Contracts.
Add the Interface Contract under EPG2 - contract type consumed.
Create the host subnet (default Gateway IP) under the bridge domain - subnet
scope private/public.
Inter-Private Network Contracts Communication

In the use case below, EPG-1 in VRF Cisco-1 re
quires com
mu
ni
ca
tion with EPG-2 in VRF
Cisco-2. This is ac
com
plished by uti
liz
ing the sub
net field within the EPG. By cre
at
ing
the sub
net under the EPG and se
lect
ing shared, the route will be leaked to the VRF
noted within the Ten
ant scoped con
tract.
Exporting Contracts Between Private Networks

1
Create the contract under Security Policies - contract scope Tenant.
1
1
Single Contract Bidirectional Reverse Filter

This use case is use
ful when im
ple
ment
ing a con
tract with the op
tion to apply the con
tract sub
ject in both di
rec
tions and with the op
tion to apply the re
verse fil
ter. This is
the most com

mon of the use cases and al
lows for a sin
gle sub
ject/fil
ter to be im
ple
mented with a sin
gle Provider/Con
sumer re
la
tion
ship.
In the use case below, EPG-1 is pro
vid
ing a con
tract with a sub
ject of www and EPG-2
is con
sum
ing the con
tract. This al
lows the Web Client in EPG-2 to ac
cess the Web
Server in EPG-1. i.e. EPG-1 is pro
vid
ing a ser
vice to EPG-2.
Default Bi-directional Contract with Reverse Filter

Re
sult:
A sin
gle con
tract with (1) Sub
ject and (1) Fil
ter with a sin
gle provider and a sin
gle con
sumer. In this ex
am
ple, www.
Sample Contract using Single Bi-directional Subject, Single Filter with Reverse
Filtering Ports Enabled

This use case in
volves im
ple
ment
ing a con
tract with
out the op
tract sub
ject in both di
rec
tions. When se
lect
ing this op
tion the user no longer has the
op
tion to se
lect the re
verse fil
ter op
tion.
vid
ing a con
tract with a sub
ject of icmp and EPG-2 is
con
sum
ing the con
tract. This al
lows the Host in EPG-1 to ac
cess the Host in EPG-2 via
icmp. When uti
liz
ing a sin
gle sub
ject with
out the use of Apply Both Di
rec
tions, the
user must then con
fig
ure two fil
ters, one in each di
rec
tion.
Single Contract, Single Unidirectional Subject, Multiple Filters

Re
sult:
A sin
gle con
tract with (1) Sub
ject (2) Fil
ters and a sin
gle provider and a sin
gle con
sumer.
In this ex
am
ple, icmp.
Sample Single Contract, Single Unidirectional Subject, Multiple Filters
Multiple Contracts Uni-Directional Single Filter

This use case is use
ful when im
ple
ment
ing a con
tract with the op
tract sub
ject in both di
rec
tions, and with
out the op
tion to apply the re
verse fil
ter. This
al
lows the end-user the most gran
u
lar
ity when de
ploy
ing con
tracts, but is also the
most com
pre
hen
sive.
vid
ing a con
tract with a sub
ject of www and EPG-2
is con
sum
ing the con
tract. This al
lows the Web Client in EPG-2 to ac
cess the Web
Server in EPG-1. That is, EPG-1 is pro
vid
ing a ser
vice to EPG-2.
Multiple Contracts, Unidirectional Subjects, Single Filters

Re
sult:
Two con
tracts with (1) Sub
ject (1) Fil
ters. Each con
tract will have a sin
gle provider and a
sin
gle con
sumer ref
er
enc
ing the same con
tract. The dif
fer
ence here is that the con
tract is ex
plic
itly ap
plied in BOTH di
rec
tions.
Sample Contract with Single Bi-Directional Subjects and single Filter
249
Layer 4 to Layer 7
Services
Layer 4 to Layer 7 Services 251
Section Content
Understanding Layer 4 to Layer 7 Integration

Service Insertion Design Principles
Applying Service Graphs to EPG Communications
Rendering Service Graphs
Integration Support Function
Services Deployment Guide Reference

Import Device Package
Create a Device
Modify a Device
Create Layer 4 to Layer 7 Service Graph Template
Apply a Service Graph Template to EPGs
Service Graph Monitoring

Monitoring a Service Graph Instance
Monitoring and Resolving Service Graph Faults
Monitoring a Virtual Device
ASAv Sample Configuration

Verify ASAv VM configuration
Remove GW IP from Relevant Fabric Bridge Domains
Create a Layer 4 to Layer 7 Device
Create a Layer 4 to Layer 7 Service Graph Template
Apply the Service Graph Template
Associate the Web and App Bridge domains to the Interfaces
Verifying service node configuration
Display the Access List configuration pushed-down from APIC
Display the Interface configuration pushed-down from APIC
Display the current Interface names
Understanding Layer 4 to Layer 7 Integration

When ACME Inc. de
signed their new ap
pli
ca
tion, they knew they would need to in
cor
po
rate some ser
vices, such as fire
walls, load bal
ancers, IDS/IPS, or other types of
higher-layer ser
vice de
vice.
In tra
di
tional in
fra
struc
ture, ser
vice in
ser
tion would re
quire some in
line de
vice place
ment or redi
rec
tion in order to get traf
fic to the ser
vice de
vices. As an ex
am
ple, fire
walls might be placed di
rectly in
line as a "bump in the line" or might be an ad
junct ser
vice de
vice near a gate
way. Fire
walls are typ
ic
ally con
fig
ured per de
vice by build
ing up
blocks of sta
tic con
fig
ur
a
tion. These sta
tic con
fig
ur
a
tion blocks build up over time to
cre
ate a sit
ua
t
ion where the con
fig
ur
a
tion works, but can be
come in
flex
ib
le and frag
ile,
which can cause change man
age
ment to be
come chal
leng
ing.
One of the key tech
nol
ogy in
no
va
tions in Cisco's Ap
pli
ca
tion Cen
tric In
fra
struc
ture
(ACI) is pol
icy-based man
age
ment of ser
vice in
ser
tion through ap
pli
ca
tion of a Ser
vice
Graph. Through the rest of this chap
ter, we will dis
cuss the high level overview of how
the process works and how ACME will uti
lize Ser
vice Graphs for ef
fi
cient man
age
ment
of Layer 4 to Layer 7 ser
vices.
The main pur
pose of data cen
ter fab
ric equip
ment is fast and ef
fi
cient for
ward
ing
of traf
fic from ingress to the fab
ric, be
tween phys
ic
al and vir
tual hosts within the fab
ric
and egress back out of the fab
ric. Use
ful in
fra
struc
ture im
ple
men
ta
tions also uti
lize this
fast fab
ric in a smart way to also in
te
grate Layer 4 to Layer 7 ser
vices. Some pos
si
ble
Layer 4 to Layer 7 ser
vices in
clude:
Firewalls
Load balancers
Traffic inspection appliances
SSL offload functions
Application flow acceleration functions
In
te
grat
ing ser
vices with Cisco ACI Ser
vice Graphs will pro
vide ACME with the fol
low
ing ben
e
fits:
Policy based configuration management
254 Layer 4 to Layer 7 Services
Flexible configuration state abstraction through the ACI object model
Integrated configuration management using the APIC GUI, REST API or Python
scripts, all based on a consistent ACI object model
Complex topology modeling with logical flow stitching allowing abstracted links
Policy-based provisioning allowing rapid complex topology deployment
Configuration synchronization allowing dynamic workload provisioning and de-
between multiple service devices
provisioning without manual intervention
Application centric template-based configuration management and object
Infrastructure multi-tenancy within the fabric and the service devices
reuse to shorten infrastructure implementation timelines
Service Insertion Design Principles

With the spine-leaf ar
chi
tec
ture and holis
tic fab
ric man
age
ment as
pects of ACI, traf
fic
flow to/from ser
vice ap
pli
ances is man
aged very ef
fi
ciently, and thus, the ap
pli
ances
do not need to be placed at any spe
cific lo
ca
tion within the net
work fab
ric. Ser
vices ap
pli
ances can be phys
ic
al or vir
tual and can be con
nected to any leaf under man
age
ment
of the ACI fab
ric. Where ap
plic
ab
le, phys
ic
al ap
pli
ances can also be run in mul
ti
ple con
text mode, al
low
ing multi-ten
ant map
ping of fab
ric for
ward
ing and ten
ant-spe
cific ser
vice con
fig
ur
a
tions.
With vir
tual ser
vices ap
pli
ances, cur
rently only VMware Hy
per
vi
sors and VLAN trans
port modes are sup
ported.
Applying Service Graphs to EPG Communications

To allow com
mu
ni
ca
tions be
tween EPGs within an ACI fab
ric, a con
tract must be put in
to place. This con
tract may take the form of a spe
cific con
sumer/provider re
la
tion
ship
de
fined by spec
if
ied pro
to
col and port. There could also be an "allow all" con
tract that
al
lows com
pletely open com
mu
ni
ca
tions. This con
tract es
sen
tially con
trols com
mu
ni
ca
tions flow be
tween EPGs, and can be ex
tended to in
clude ser
vice in
ser
tion via at
tach
ment of a Ser
vice Graph to a con
tract. The Ser
vice Graph then ties the con
tract to
the re
solved ser
vice de
vice with the pol
icy-based con
fig
ur
a
tion in place.
Rendering Service Graphs

The ACI al
lows users to de
fine a pol
icy using the fol
low
ing ways:
APIC GUI
API with a programmatic tool, such as Python or a RESTful API post through
POSTman
These pol
icy ob
jects can be cre
ated, ma
nip
ul
ated, and reused. As it re
lates to the Layer
4 to Layer 7 ser
vices, these ob
jects ex
press the in
tent of use for that ob
ject in re
la
tion
to the ap
pli
ca
tion.
When an ap
pli
ca
tion pro
file is de
ployed and end
points are at
tached to the leaf
switches, the ser
vice graph ob
jects are then trans
lated into spe
cific de
vice con
fig
ur
a
tions that gets pushed the ser
vice nodes through a process called ren
der
ing. The APIC
also sets up the net
work for
ward
ing path to make sure the cor
rect for
ward
ing ac
tion is
taken to get traf
fic flow to the ser
vice nodes for treat
ment.
This ab
stracted process of con
fig
ur
a
tion man
age
ment works like a pol
icy tem
plate
where you can de
fine the ex
pected be
hav
ior, then link two groups and sub
ject their re
la
tion
ship to that pol
icy. This pol
icy can be copied, reused and repack
aged as nec
es
sary.
The ren
der
ing in
volves al
lo
ca
tion of the nec
es
sary bridge do
mains, con
fig
ur
a
tion of IP
ad
dresses on the fire
wall and load bal
ancer in
ter
faces, cre
ation of the VLAN on these
de
vices to cre
ate the path for the func
tions, and per
for
mance of all the work nec
es
sary
to make sure that the path be
tween EPGs is the path de
fined in the ser
vice graph.
As is the case with many cus
tomers, ACME has a few cookie cut
ter tem
plates for fire
wall and load-bal
anc
ing ser
vices. Though the ini
tial de
f
in
i
t
ion of these tem
plates can be
po
ten
tially cum
ber
some, sub
se
quently reusing the tem
plates is very straight
for
ward
sim
ply by re
plac
ing IP ad
dresses, ports, ob
ject-groups, and other val
ues.
Integration Support Function

In the ACI model, com
mu
ni
ca
tions with the ser
vice de
vices is sup
ported by im
port
ing
de
vice pack
ages. These de
vice pack
ages carry the de
vice de
scrip
tion, ex
posed func
tions, and have con
fig
ur
a
tion script con
tent. When the de
vice pack
age is im
ported, the
ACI fab
ric has full un
der
stand
ing of what a de
vice can do, how it con
nects to the fab
ric,
how to build path for

ward
ing to bring traf
fic into and get traf
fic back from the de
vice,
and how to trans
late pol
icy in
tent to a de
vice-spe
cific con
fig
ur
a
tion. This de
vice pack
age is a ven
dor-de
vel
oped pack
age that is read
ily avail
able from the orig
in
al ven
dor or
from Cisco on the soft
ware down
load page.
Some of the De
vice Pack
ages can be down
loaded at: http://
www.
cisco.
com/
c/
en/
us/
solutions/
collateral/
data-center-virtualization/
application-centricinfrastructure/
solution-overview-c22-732445.
html
De
vice pack
ages from mul
ti
ple ven
dors that lever
age the rich open API that ACI pro
vides are avail
able from ven
dors such as Cit
rix and F5 at the time of writ
ing of this
book.
An up to date list
ing of part
ners that lever
age the API are avail
able at: http://
www.
cisco.
com/
c/
en/
us/
solutions/
unified-fabric/
aci_
ecosystem.
html
Services Deployment Guide Reference

The di
a
gram below shows a high level overview of the Layer 4 to Layer 7 ser
vices work
flow when at
tempt
ing to in
te
grate a de
vice.
L4-L7 Workflow
For in
for
ma
tion about de
ploy
ing Layer 4 to Layer 7 ser
vices, see the Cisco APIC Layer 4
to Layer 7 Ser
vices De
ploy
ment Guide: http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
L4-L7_
Services_
Deployment/
guide/
b_
L4L7_
Deploy.
html.
The key sec
tions of the De
ploy
ment Guide are listed below:
Import Device Package

To begin the process of Ser
vice Node in
te
gra
tion, ACME will im
port the ven
dor/model-spe
cific de
vice pack
ages, as shown in this sec
tion of the De
ploy
ment
Guide.
Create a Device
Once the de
vice pack
age is im
ported, the de
vices will be added through a process of
cre
at
ing a log
i
cal de
vice clus
ter and cre
at
ing a re
la
tion
ship be
tween this log
i
cal de
vice
and the phys
i
cal ap
pli
ance. This can be done with a phys
i
cal or VM de
vice. The con
fig
u
ra
tion steps dif
fer slightly for phys
i
cal de
vices and vir
tual de
vices, but are very sim
i
lar.
Modify a Device
You can mod
ify a de
vice's con
fig
ur
a
tion through the GUI as de
scribed in this sec
tion of
the De
ploy
ment Guide.
Create Layer 4 to Layer 7 Service Graph Template

This sec
tion of the De
ploy
ment Guide ex
plains how to cre
ate a ser
vice graph.
Apply a Service Graph Template to EPGs

Once the ap
pli
ca
tion End
point Groups (EPGs) have been cre
ated, the process to apply
a ser
vice graph tem
plate to EPGs can be found in this sec
tion of the De
ploy
ment Guide.
Service Graph Monitoring

Once ACME Inc. has de
ployed Ser
vice Graphs for ap
pli
ca
tion ser
vice in
ser
tion, the re
quire
ment of ob
serv
ing the Ser
vice Graph be
comes an op
er
at
ional im
per
at
ive. To sup
port these ef
forts, there are a few tech
niques that can be em
ployed, in
clud
ing:

Once a ser
vice graph is con
fig
ured and as
so
ci
ated with a con
tract that is at
tached to an
EPG, there are some pri
mary mon
it
or
ing as
pects that should be con
sid
ered: State of
the Ser
vice Graph, Func
tions of a Graph in
stance, re
sources al
lo
cated to a func
tion and
pa
ra
me
ters spec
if
ied for a func
tion.
To mon
it
or a ser
vice graph in
stance:
1
In the Navigation pane, choose Tenant_Name > L4-L7 Services > Deployed
Graph Instances. The Work pane displays information about the deployed
graph instances, including a list of the deployed service graphs, the associated
contracts, and the current state of the graph policy. A state of "Applied" means
the graph has been applied and is active in the fabric and the service device.
For fur
ther de
tails of the pos
si
ble states and other rel
e
vant states, see the Cisco APIC
vices De
ploy
ment Guide at: http://
www.
cisco.
com/
c/
en/
us/
td/
Layer 4 to Layer 7 Ser
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
L4-L7_
Services_
Deployment/
guide/
b_
L4L7_
Deploy/
b_
L4L7_
Deploy_
chapter_
01010.
html#
task_
F2BFF7545D9142EFB208C10F5DFBB1B4

To mon
it
or a ser
vice graph's faults:
1
In the Navigation pane, choose Tenant_Name > L4-L7 Services > Deployed
Graph Instances.
In the Work pane, choose the Faults tab. The Work pane displays the faults that
are related to the active service graph.
To un
der
stand the faults listed and pos
si
ble res
o
lu
tions, see Ta
bles 1 and 2 in the Cisco
APIC Layer 4 to Layer 7 Ser
vices De
ploy
ment Guide at: http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
L4-L7_
Services_
Deployment/
guide/
b_
L4L7_
Deploy/
b_
L4L7_
Deploy_
chapter_
01010.
html#
concept_
307C0CA3EB57469EAF7EF87AAE5A240F

After you con
fig
ure a ser
vice graph and at
tach the graph to an end
point group (EPG)
and a con
tract, you can mon
it
or the vir
tual de
vices as
so
ci
ated with the ser
vice graphs
of a ten
ant. Mon
it
or
ing the vir
tual de
vices tells you what de
vice clus
ters are as
so
ci
ated,
which VLANs are con
fig
ured for a de
vice clus
ter, the func
tions in use and the func
tion
pa
ra
me
ters passed to the de
vices, the sta
tis
tics from the de
vices, and the health of the
de
vices in a de
vice clus
ter.
To mon
it
or a vir
tual de
vice:
1
In the Navigation pane, choose Tenant_Name > L4-L7 Services > Virtual
Devices. The Work pane displays information about the virtual devices, such as
faults and health scores.
ASAv Sample Configuration

One of the ser
vice nodes that ACME Inc. will in
te
grate is an ASAv de
ployed in sin
glenode, routed FW mode. This ex
am
ple de
tails the process they fol
lowed to con
fig
ure the
ASAv fire
wall vir
tual ser
vice ap
pli
ance as a sin
gle node in routed mode. As a routed
node, the ASAv will be
come the de
fault gate
way for the hosts in the 2 EPGs that are
con
nected by the con
tract that this ser
vice graph gets as
so
ci
ated to.
The high level steps are:
Create Logical and Concrete device clusters
Define a firewall ruleset/graph
Deploy the graph between 2 EPGs
Verify ASAv VM configuration

The first set of steps per
formed is to ver
ify ASAv VM con
fig
ur
a
tion and up
load the ASA
de
vice pack
age (or ver
ify if it has al
ready been done)
1
Log in to the ASAv VM.
Enter enable mode.
Issue the following command:
# show ip int brief
Verify that the ASAv has a correct IPv4 address on the management 0/0
interface.
Verify connectivity from the APIC to the management 0/0 interface of the
ASAv.
a. SSH to the APIC cluster IP address
b. Issue the following command:
# ping {ASAv management 0/0 interface}
c. The return response should be similar to the following example:

64 bytes from 172.16.10.10: icmp_seq=1 ttl=64 time=0.050 ms
If the re
sponse is dif
fer
ent, then there is likely some sort of con
nec
tiv
ity
issue. Ad
dress the con
nec
tiv
ity prob
lems be
fore mov
ing on.
6
In the APIC GUI, on the menu bar, choose Tenant > Tenant_Name.
In the Navigation Pane expand Tenant_Name > L4-L7 Services >

Packages > L4-L7 Service Device Types.
If an ASA device package is not listed, then perform the following actions:
a. Right click on the Device Types.
b. Choose Import Device Package.
c. Follow the prompts to upload the device package.
Remove GW IP from Relevant Fabric Bridge Domains

Once the ASAv pack
age and VM are ver
if
ied, the next step is to re
move SVI/GW IP ad
dresses from the fab
ric Bridge Do
mains so the Layer 3 routed fire
wall can be
come the
de
fault gate
way.
To re
move the rout
ing de
fault gate
way func
tion on the EPG1 and EPG2 bridge do
mains:
1
In the Navigation pane, choose Tenant_Name > Networking > Bridge Domains
> BD-EPG1.
In the Work pane, perform the following actions:

a. For the L2 Unknown Unicast radio buttons, click Flood.
b. For the L3 Unknown Multicast Flooding radio buttons, click Flood.
c. Uncheck the Unicast Routing check box.
d. Click Submit.
Re
peat this process for the Bridge Do
mains of the af
fected EPGs.

Per
form the fol
low
ing steps to add log
ic
al and con
crete de
vice clus
ters to a ten
ant:
1
In the Navigation Pane choose Tenant_Name > L4-L7 Services.
In the Work pane, click Create a L4-L7 Function Profile.
In the Create a L4-L7 Function Profile dialog box, perform the following
actions:
a. Enter a name that is relevant to the tenant and fuction, such as "TXX-ASAvFP".
b. In the Profile Group drop-down list, choose Create Function Profile Group.
c. In the Create L4-L7 Services Function Profile Group dialog box, perform the
following actions:
i. Enter a meaningful name, such as "TXX-FP-Group".
ii. Click SUBMIT.
d. In the Profile drop-down list, choose WebServiceProfileGroup
or WebPolicyForRoutedMode.
e. In the Feature and Parameters section, choose the All Parameters tab. You
will configure IP addressing under Interface Related Configuration for both
external and internal interfaces (externalIf and interalIf).
f. Expand Interface Related Configuration for externalIf.
g. Expand Interface Specific Configuration.
h. Double-click IPv4 Address.
i. Enter: ipv4 address in a.b.c.d/m.n.o.p format
j. Click Update.
k. Repeat steps 4f - 4j as needed.
Choose L4-7 Services > Function Profiles > ALL PARAMETERS.
Verify the external and internal interfaces IPv4 addresses.
The fol
low
ing steps will cre
ate a Layer 4 to Layer 7 de
vice:
1
In the Navigation pane, choose Tenant_Name > L4-L7 Services.
In the Work pane, click Create a L4-L7 virtual device.
In the Create L4-L7 Devices dialog box, perform the following actions:
a. Enter a meaningful name: Txx-ASAv-Cluster
b. Device Package: CISCO-ASA-1.1 Model: ASAv
c. Mode: Single Node
d. Function Type: Goto
e. Connectivity VMM Domain: Txx-vCenter
f. APIC to Device: Out of Band
g. Credentials: {uid/pwd}
h. Under Device 1, specify the following values:
i. Management IP address: ASAv IP address
ii. Management Port: https
iii. VM: Tenant ASAv Controller (in the dropdown box)
iv. Virtual Interfaces: Create two entries; click + twice; enter interface values
accordingly:
1. Name: GigabitEthernet0/0 vNIC: Network Adapter 2 Direction:
provider
2. Name: GigabitEthernet0/1 vNIC: Network Adapter 3 Direction:
consumer
v. Click UPDATE after each entry.
vi. Click NEXT.
vii. Click FINISH.
Ver
ify the Log
ic
al and Con
crete De
vice Clus
ters have been con
fig
ured:
1
In the Navigation Pane choose Tenant_Name > L4-L7 Services > L4-L7 Devices.
Expand both TXX-ASAv-Cluster and TXX-ASAv-Cluster_Device_1 to view the

logical and physical interfaces.
Select TXX-ASAv-Cluster_Device_1 to see a graphic view of the concrete

device.
Note This com

pletes the Log
ic
al and Con
crete de
vice cre
ation.
Create a Layer 4 to Layer 7 Service Graph Template

1
In the Navigation pane, choose Tenant_Name > L4-L7 Services > L4-L7 Service
Graph Templates.
In the Work pane, choose Actions > Create a L4-L7 Service Graph Template.
In the Create L4-L7 Devices dialog box, perform the following actions:
a. In the Name field, enter TXX-ASAv-L3-Routed-Template.
b. In the Type drop-down list, choose Single Node - Firewall in Routed Mode.
c. In the Device Function drop-down list, choose CISCO-ASA-1.1/Firewall.
d. In the Profile drop-down list, choose TXX-ASAv-FP, which is the functional
profile you created previously.
e. Click Submit.
You can verify that the template was created successfully by expanding
Tenant_Name > L4-L7 Services > L4-L7 Service Graph Templates > Txx-ASAvL3-Routed-Template > Function Node - Firewall.
Apply the Service Graph Template

1
In the Navigation pane, choose Tenant_Name > L4-L7 Services > L4-L7 Service
Graph Templates.
Right click Txx-ASAv-L3-Routed-Template and choose Apply L4-L7 Service

Graph Template.
In the Apply L4-L7 Service Graph Template to EPGs dialog box, perform the
following actions:
a. In the Consumer EPG / External Network drop-down list, choose EPG1.
b. In the Provider EPG / External Network drop-down list, choose EPG2.
c. In the Service Graph Template drop-down list, choose Txx-ASAv-L3Routed-Template.

d. For the Contract radio buttons, click Create A New One.
e. Check the No Filter (Allow All Traffic) check box.
f. In the Contract Name field, enter TXX-EPG1-to-EPG2.
g. Click Next.
h. In the L4-L7 Devices drop-down list, choose Txx-ASAv-Cluster.
i. In the Features and Parameters section, choose the All Parameters tab.
j. Double click Device Config > Interface Related Configuration externalIf >
Access Group > Inbound Access List.
k. In the Value drop-down list, choose access-list-inbound.
l. Click Update.
m. Expand Device Config > Interface Related Configuration externalIf >
Interface Related Configuration to verify the IP address assignment. If the
mask is missing, the configuration will not push to the ASA.
n. Double click Device Config > Interface Related Configuration internalIf >
Access Group > Inbound Access List.
o. Click Reset. This unassigns the inbound access list from the internal
interface. This inbound access list is not desirable for the lab traffic-flow.
p. Expand Device Config > Interface Related Configuration internalIf >
Interface Specific Configuration to verify the IP address assignment with
the mask.
q. Click Finish. The Devices Selection Polices folder and Deployed Graph
Instances folder are now populated.
6
Choose TXX-EPG1-to-EPG2-TXX-ASAv-L3-Router-Template-TXXProduction.
You can see that the Consumer and Provider EPGs are associated with the EPG1
and EPG2 server EPGs.
Associate the Web and App Bridge domains to the Interfaces

1
Expand Devices Selection Policies.
Expand TXX-EPG1-to-EPG2-TXX-ASAv-L3-Router-Template-Firewall.
Select External.
Assign the Bridge Domain to BD-EPG1.
Repeat the process to assign the Bridge Domain to BD-EPG2.
Click SUBMIT.
Expand Layer 4 to Layer 7 Service Graph Templates.
Expand TXX-ASAv-L3-Router-Template.
Expand Function Node Firewall > choose external.
10
Filters: common/default.
11
CTX Terms: TXX-ASAv-L3Routed Template/T1/Out.
12
Click SUBMIT.
13
Repeat the process for internal.
14
Filters: common/default.
15
CTX Terms: TXX-ASAv-L3Routed Template/T1/In.
16
Click SUBMIT.
The ASAv will now have IP ad

dresses asigned in the Ser
vice Graph Pro
file. This can be
ver
if
ied by going into the ASAv VM con
sole and issue the fol
low
ing com
mand:
# show ip int brief
Verifying service node configuration

Once the De
vice has been in
te
grated and the Ser
vice Graph has been con
fig
ured and
as
so
ci
ated, the re
sult
ing con
fig
ur
a
tion pushed to the ser
vice node can be ver
if
ied by
sim
ple show com
mands on the real ser
vice node de
vice.
Display the Access List configuration pushed-down from APIC

SSH to the ASAv ser
vice node for ac
cess list val
id
a
tion. Issue the fol
low
ing com
mand:
# show run | grep access
This will show the ac

cess list con
fig
ur
a
tion and you can re
late that to the con
fig
ur
a
tion
that was done in the APIC.
Display the Interface configuration pushed-down from APIC

SSH to the ASAv ser
vice node for in
ter
face con
fig
ur
a
tion val
id
a
tion. Issue the fol
low
ing
com
mand:
# show run interface
This will show the in

ter
face con
fig
ur
a
tion where the IP ad
dress con
fig
ur
a
tion was
pushed from the APIC.
Display the current Interface names

SSH to the ASAv ser
vice node for in
ter
face name con
fig
ur
a
tion val
id
a
tion. Issue the fol
low
ing com
mand:
# show nameif
This will show the in

ter
face names pushed from the APIC and will show the re
lated in
ter
face names to the log
ic
al in
ter
face names that were con
fig
ured in the APIC above.
269
Health Scores
Health Scores 271
Section Content
Understanding Health Scores
Understanding Faults
How Are Health Scores Calculated
Health Score Use Cases

Using Health Scores for Proactive Monitoring
Using Health Scores for Reactive Monitoring
Health Scores 273
Understanding Health Scores

ACME's Op
er
at
ions team has been chal
lenged on a reg
ul
ar basis to an
swer basic ques
tions re
gard
ing the cur
rent sta
tus, per
for
mance, and avail
abil
ity of the sys
tem they are
re
spon
si
ble for op
er
at
ing. To ad
dress these chal
lenges they can now uti
lize the Ap
pli
ca
tion Cen
tric In
fra
struc
ture (ACI), which pro
vides Health Scores that make in
for
ma
tion on sta
tus, per
for
mance, and avail
abil
ity read
ily avail
able. It is worth not
ing that
while pro
vid
ing such an
swers may be easy as it re
lates to an in
de
pen
dent de
vice or link,
this in
for
ma
tion by it
self is of lit
tle to no value with
out ad
di
tional data on its ef
fect on
the over
all health of the net
work. To man
ua
lly col
lect and cor
re
late in
for
ma
tion would
have pre
vi
ously been a long and te
dious task, but with health scores, data through
out
the fab
ric is col
lected, com
puted, and cor
re
lated by the APIC in real time and then pre
sented in an un
der
stand
able for
mat.
Tra
di
tional net
work mon
it
or
ing and man
age
ment sys
tems at
tempt to pro
vide a model
of in
fra
struc
ture that has been pro
vi
sioned, and de
scribe the re
la
tion
ship be
tween the
var
io
us de
vices and links in at
tempt to pro
vide a cor
re
la
tion. The ob
ject model at the
heart of ACI is in
her
ent to the in
fra
struc
ture, and there
fore the cur
rent sta
tus of all of
the ob
jects in
clud
ing links, de
vices, their re
la
tion
ships, as well as the real time sta
tus of
their uti
liza
tion, can be rep
re
sented in a sin
gle con
sol
id
ated health score.
The vis
ib
il
ity pro
vided by health scores give the op
er
at
or a quick at-a-glance as
sess
ment of the cur
rent sta
tus of the en
tire sys
tem, or any sub
set of the sys
tem. This vis
i-
bil
ity has a num
ber of prac
ti
cal use cases, and in this chap
ter we will clas
sify these use
cases as re
ac
tive and proac
tive. ACI also pro
vides the flex
ib
il
ity to mon
it
or some as
pects of how the health scores are cal
cu
lated, and how var
io
us faults im
pact the cal
cu
la
tion of the health score.
Most ob
jects in the model will have an as
so
ci
ated health score, which can be found
from the Dash
board or Pol
icy tabs of the ob
ject from the GUI. Ad
di
tion
ally, all health
scores are in
stan
ti
ated from the health
Inst class and can be ex
tracted through the API.
274 Health Scores
Tenant Health Score

In a re
ac
tive ca
pac
ity, ACI health scores pro
vide a quick check as to whether an issue
being re
ported is con
firmed in a degra
da
tion of the health score. If so, the root cause
of the issue can be found by ex
plor
ing the faults and how these get rolled up in the
larger model. Health scores also pro
vide a real-time cor
re
la
tion in the event of a fail
ure
sce
nario, im
me
di
ately pro
vid
ing feed
back as to which ten
ants, ap
pli
ca
tions, and EPGs
are im
pacted by that fail
ure.
As a ex
am
ple, if you nav
i
gate to the ap
pli
ca
tion pro
file it has a Health tab. In this tab is
a tree that will show the var
i
ous ob
jects in a tree form to re
veal faults.
Health Scores 275
Object with a fault

Proac
tively, ACI health scores can help iden
tify po
ten
tial bot
tle
necks in terms of hard
ware re
sources, band
width uti
liza
tion, and other ca
pac
ity plan
ning ex
er
cises. Op
er
a
tions teams also stand a bet
ter chance of iden
ti
fy
ing is
sues be
fore they im
pact cus
tomers or users.
Ide
ally, the health of all ap
pli
ca
tion and in
fra
struc
ture com
po
nents would al
ways be at
100%, how
ever, this is not al
ways re
al
is
tic given the dy
namic na
ture of data cen
ter en
vi
ron
ments. Links, equip
ment, and end
points have fail
ures. In
stead the health score
should be seen as a met
ric that will change over time, with the goal of in
creas
ing the
av
er
age health score of a given set of com
po
nents over time.
Health Scores 277
Understanding Faults
From a man
age
ment point of view we look at the Ap
pli
ca
tion Pol
icy In
fra
struc
ture Con
troller (APIC) from two per
spec
tives:
1
Policy Controller - Where all fabric configuration is created, managed and

applied. It maintains a comprehensive, up-to-date run-time representation of
the administrative or configured state.
Telemetry device - All devices (Fabric Switches, Virtual Switches, integrated

Layer 4 to Layer 7 devices) in an ACI fabric report faults, events and statistics to
the APIC.
Faults, events, and sta

tis
tics in the ACI fab
ric are rep
re
sented as a col
lec
tion of Man
aged Ob
jects (MOs) within the over
all ACI Ob
ject Model/Man
age
ment In
for
ma
tion
Tree (MIT). All ob
jects within ACI can be queried, in
clud
ing faults. In this model, a fault
is rep
re
sented as a mu
ta
ble, state
ful, and per
sis
tent MO.
Fault Lifecycle
278 Health Scores
When a spe
cific con
di
tion oc
curs, such as a com
po
nent fail
ure or an alarm, the sys
tem
cre
ates a fault MO as a child ob
ject to the MO that is pri
mar
ily as
so
ci
ated with the fault.
For a fault ob
ject class, the fault con
di
tions are de
fined by the fault rules of the par
ent
ob
ject class. Fault MOs ap
pear as reg
ul
ar MOs in MIT; they have a par
ent, a DN, RN,
and so on. The Fault code is an al
phanu
mer
ic
al string in the form FXXX. For more in
for
ma
tion about fault codes, see the Cisco APIC Faults, Events, and Sys
tem Mes
sages
age
ment Guide.
Man
The fol
low
ing ex
am
ple is a REST query to the fab
ric that re
turns the health score for a
ten
ant named "3tier
app":
https://hostname/api/node/mo/uni/tn-3tierapp.xml?query-target=self&rsp-subtreeinclude=health
The fol
low
ing ex
am
ric that re
turns the sta
tis
tics for a ten
ant named "3tier
app":
https://hostname/api/node/mo/uni/tn-3tierapp.xml?query-target=self&rsp-subtreeinclude=stats
The fol
low
ing ex
am
ric that re
turns the faults for a leaf
node:
https://hostname/api/node/mo/topology/pod-1/node-103.xml?query-target=self&rspsubtree-include=faults
As you can see, MOs can be queried by class and DN, with prop
erty fil
ters, pag
i
na
tion, and
so on.
In most cases, a fault MO is au
to
mat
ic
ally cre
ated, es
ca
lated, de-es
ca
lated, and deleted
by the sys
tem as spe
cific con
di
tions are de
tected. There can be at most one fault with a
given code under an MO. If the same con
di
tion is de
tected mul
ti
ple times while the
cor
re
spond
ing fault MO is ac
tive, no ad
di
tional in
stances of the fault MO are cre
ated.
In other words, if the same con
di
tion is de
tected mul
ti
ple times for the same af
fected
ob
ject, only one fault is raised while a counter for the re
cur
rence of that fault will be
in
cre
mented. A fault MO re
mains in the sys
tem until the fault con
di
tion is cleared. To
re
move a fault, the con
di
tion rais
ing the fault must be cleared, whether by con
fig
ur
a
-
Health Scores 279
tion, or a change in the run time state of the fab

ric. An ex
cep
tion to this is if the fault is
in the cleared or re
tained state, in which case the fault can be deleted by the user by
ac
knowl
edg
ing it.
Sever
ity pro
vides an in
di
ca
tion of the es
ti
mated im
pact of the con
di
tion on the ca
pa
bil
ity of the sys
tem or com
po
nent to pro
vide ser
vice.
Pos
si
ble val
ues are:
Warning (possibly no impact)
Minor
Major
Critical (system or component completely unusable)
The cre
ation of a fault MO can be trig
gered by in
ter
nal processes such as:
Finite state machine (FSM) transitions or detected component failures
Conditions specified by various fault policies, some of which are user

configurable
For ex
am
ple, you can set fault thresh
olds on sta
tis
ti
cal mea
sure
ments such as health
scores, data traf
fic, or tem
per
at
ures.
Health Scores 281
How Health Scores Are Calculated

Health scores exist for sys
tems and pods, ten
ants, man
aged ob
jects (such as switches
and ports), as well as an over
all health score for the over
all sys
tem. All health scores are
cal
cu
lated using the num
ber and im
por
tance of faults that apply to it. Sys
tem and pod
health scores are a weighted av
er
age of the leaf health scores, di
vided by the total
num
ber of learned end points, mul
ti
plied by the spine co
ef
fi
cient which is de
rived from
the num
ber of spines and their health scores. In other words:
Health Score calculation

Ten
ant health scores are sim
i
lar, but con
tain health scores of log
i
cal com
po
nents
within that ten
ant. For ex
am
ple, it will only be weighted by the end points that are in
cluded in that ten
ant.
You can see how all of these scores are ag
gre
gated by look
ing at how man
aged ob
ject
scores are cal
cu
lated, which is di
rectly by the faults they have as
so
ci
ated with them.
Each fault is weighted de
pend
ing on the level of im
por
tance. Crit
i
cal faults might have a
282 Health Scores
high fault level at 100%, while warn

ings might have a low fault level at only 20%. Faults
that have been iden
ti
fied as not im
pact
ing might even be re
as
signed a per
cent
age value
of 0% so that it does not af
fect the health score com
pu
ta
tion.
Luck
ily there is re
ally no need to un
der
stand the cal
cu
la
tions of the health scores to
use them ef
fec
tively, but there should be a basic un
der
stand
ing of whether faults
should have high, medium, low, or none fault lev
els. Though faults in ACI come with
de
fault val
ues, it is pos
si
ble to change these val
ues to bet
ter match your en
vi
ron
ment.
Keep in mind, be
cause of the role-based ac
cess con
trol, not all ad
min
is
tra
tors will be
able to see all of the health scores. For ex
am
ple, a fab
ric admin will be able to see all
health scores, but a ten
ant admin would only be able to see the health scores that per
tain to the ten
ants to which they have ac
cess. In most cases, the ten
ant admin should
be able to drill into the health scores that are vis
ib
le to them, but it is pos
si
ble a fault
may be oc
cur
ring that is af
fect
ing more than that one ten
ant. In this case the fab
ric ad
min
is
tra
tor may have to start trou
bleshoot
ing. The ten
ant and fab
ric ad
mins may also
see health scores of any layer four through seven de
vices, such as fire
walls, load bal
ancers, and in
tru
sion pre
ven
tion/de
tec
tion sys
tems. These, along with faults within
our VMM do
mains will all roll up into our ten
ant, pod, and over
all sys
tem health scores.
For more in
for
ma
tion on how to use faults, see the Trou
bleshoot
ing Cisco Ap
pli
ca
tion
tric In
fra
struc
ture book: http://
datacenter.
github.
io/
aci-troubleshooting-book/.
Cen
Health Scores 283
Health Score Use Cases

Using Health Scores for Proactive Monitoring
While ACME ad
min
is
tra
tors have tra
di
tion
ally spent a lot of time re
act
ing to is
sues on
the net
work, ACI health scores will allow them to start pre
vent
ing is
sues. Health scores
not only act as in
di
ca
tors of faults, they are es
sen
tially base
lines to which you can make
com
par
isons later. If you see that one of the leaf switches is at 100% (green for good)
one week, and the next week the leaf is show
ing a warn
ing, you can drill down to see
what changed. In this sce
nario, it is pos
si
ble the links are over
sub
scribed and so it can
be time to ei
ther move some of of the work
load to an
other leaf or maybe to add more
band
width by con
nect
ing more ca
bles. Since it is still only a warn
ing, there is time to
re
solve the issue be
fore any bot
tle
necks on the net
work are no
tice
able.
The same sce
nario can ob
served with a load bal
ancer or fire
wall that is get
ting over
loaded. In these cases adding an
other load bal
ancer, or fire
wall, or maybe even op
ti
miz
ing the rules may be needed to make traf
fic flow more ef
fi
cient. As shown in the
above ex
am
ples, this baselin
ing method can be used as a ca
pac
ity plan
ning tool.
Other ways health scores can be used to proac
tively mon
it
or your ACI en
vi
ron
ment are
by giv
ing vis
ib
il
ity of cer
tain com
po
nents to other groups. Since you can ex
port the
scores and faults, it is pos
si
ble to send these no
ti
fi
ca
tions to ap
pli
ca
tion own
ers,
VMware ad
min
is
tra
tors, Data
base Ad
min
is
tra
tor, and so on. This would pro
vide mon
i-
tor
ing of the en
vi
ron
ment across the net
work that has not pre
vi
ously been avail
able
and which is not able to be re
trieved by any other means.
Using Health Scores for Reactive Monitoring

Re
ac
tively, health scores can be used to di
ag
nose prob
lems with the ACI fab
ric. Upon
no
ti
fi
ca
tion that a health score has been de
graded, an op
er
at
or can use the GUI to eas
ily nav
ig
ate the re
la
tion
ships and faults that are con
tribut
ing to that health score. Once
the root cause faults have been iden
ti
fied, the fault it
self will con
tain in
for
ma
tion about
pos
si
ble re
me
di
at
ion steps.
Most ob
jects will have a Health tab which can be used to ex
plore the re
la
tion
ship be
tween
ob
jects, and their as
so
ci
ated faults. This pro
vides the abil
ity to dou
ble-click to root cause.
285
Monitoring
Monitoring 287
Section Content
Proactive Monitoring - Tenant and Fabric Policies

Stats Collection Policies
Stats Export Policies
Diagnostics Policies
Call Home/SNMP/syslog
Event Severity and Fault Severity Assignments
Fault Lifecycle Policies
TCAM Policy Usage
Create TCAM Policy Monitor
TCAM Prefix Usage
Health Score Evaluation Policy
Communication Policy
Proactive Monitoring - Infrastructure

Monitoring APICs
CPU utilization and Memory
Disk Utilization
Physical and Bond Interface Statistics
APIC Fan Status
Temperature Status
Power Supply Status
Monitoring Leaf Switches
Monitoring Switch CPU Utilization
Monitoring Switch Memory Utilization
Monitoring File System Health
Monitoring CoPP (Control Plane Policing) Statistics
Physical Interface Statistics and Link State
Module Status
Switch Fan Status
288 Monitoring
Power Supply Status

LLDP Neighbor Status
GOLD Diagnostic Results
Proactive Monitoring Use Cases

Monitoring Workload Bandwidth
EPG Level Statistics
Reactive Monitoring
Reactive Monitoring Tools
Switch Port Analyzer (SPAN)
Traceroute
Atomic Counters
Traffic Map
Enhanced Troubleshooting Wizard
IPing
Audit Logs
Reactive Monitoring Use Cases

Loss of Connectivity to Endpoint
Users Report that an Application Running in the Fabric is Slow
Monitoring 289
Proactive Monitoring Tenant and Fabric Policies

Proac
tive mon
it
or
ing is a very im
por
tant piece of the net
work ad
min
is
tra
tor's job, but
is often ne
glected be
cause putting out fires in the net
work usu
ally takes pri
or
ity. How
ever, since the APIC makes it in
cred
ib
ly easy to gather sta
tis
tics and per
form analy
ses,
this will save net
work ad
min
is
tra
tors both time and frus
tra
tion. Since sta
tis
tics are
gath
ered au
to
mat
ic
ally and poli
cies are used and can be re-used in other places, the
human error and ef
fort is min
im
al.
Sta
tis
tics gath
er
ing has been a some
what man
ual and even re
source in
ten
sive process
for ACME in the past. Even when they have used tools to gather data on layer one
through seven de
vices, it has still been nec
es
sary to man
ua
lly spec
ify which de
vices are
to be mon
it
ored and how they should be mon
it
ored. For ex
am
ple, SNMP and a third
party tool may have been used to mon
it
or the CPU of switches or band
width uti
liza
tion
on ports, but they strug
gled with en
ter
ing cor
rect SNMP in
for
ma
tion on each de
vice,
or often for
got to add a new de
vice to their Net
work Mon
it
or
ing Sys
tem (NMS). ACI
pro
vides an APIC which will do all of the sta
tis
tics gath
er
ing, and pro
vides the abil
ity to
proac
tively mon
it
or your en
tire en
vi
ron
ment with
out all of the has
sle of main
tain
ing a
third party mon
it
or
ing tool.
The APIC, whether ac
cessed through the GUI, CLI, or API, can be used to drill into any
of the com
po
nents and pro
vides the abil
ity to click on a Stats tab to dis
play on-de
mand
sta
tis
tics, but more im
por
tantly it en
ables the setup of poli
cies to keep per
sis
tent data
to an
al
yze trends in the en
vi
ron
ment, as well as to trou
bleshoot or pre
dict any is
sues
that may be aris
ing. When plan
ning to move an ap
pli
ca
tion from a legacy net
work to
the ACI in
fra
struc
ture, it is sen
si
ble to start by test
ing be
fore going straight to pro
duc
tion. Add test VMs to port groups on ei
ther a DVS or AVS as
so
ci
ated with the APIC,
and add phys
ic
al test servers to VPCs on the leaf switches. This could also be in a test
ing ten
ant which is com
pletely sep
ar
ate from the pro
duc
tion en
vi
ron
ment. At this
point the APIC is al
ready gath
er
ing sta
tis
tics for the VMM do
main and the phys
ic
al de
vices. The next step is to con
fig
ure a pol
icy for trend analy
sis.
There are four dif
fer
ent scopes for sta
tis
tics gath
er
ing: Com
mon or Fab
ric Wide, Fab
ric, Ten
ant, or Ac
cess. A Fab
ric Wide pol
icy would be cre
ated as a de
fault pol
icy to be
ap
plied to all ten
ants. How
ever, to over
ride that pol
icy for a par
tic
ul
ar ten
ant, the ten
-
290 Monitoring
ant pol
icy will over
ride the Fab
ric pol
icy. In the fol
low
ing test
ing ex
am
ple, a Ten
ant
pol
icy is cre
ated to gather sta
tis
tics. Even if this ten
ant is shared with other ap
pli
ca
tions, cus
tomers, test cases, it will pro
vide a real world ex
am
ple of how the ap
pli
ca
tion
will be
have in a pro
duc
tion en
vi
ron
ment.
Create Tenant Monitor policy

To cre
ate a ten
ant mon
it
or
ing pol
icy:
1
In the Navigation pane, choose Tenant_Name > Monitoring Policies.
In the Work pane, choose Actions > Create Monitoring Policies.
In the Create Monitoring Policies dialog box, perform the following actions:
a. In the Name field enter a name for the Monitoring Policy.
b. Click Submit.
In the Navigation pane, choose Tenant_Name > Monitoring Policies >

Policy_Name to display the following information:
Callhome, SNMP, and syslog
Event Severity Assignment Policies

Click
ing on Stats Col
lec
tion Poli
cies will dis
play the de
fault re
ten
tion pe
ri
ods and
admin states (En
abled/Dis
abled) for ALL Mon
it
ored Ob
jects. Most likely the de
faults
will be kept, but a dou
ble click on them will change the admin state or re
ten
tion pe
ri
ods. For ex
am
ple, to have it poll a com
po
nent every 5 min
utes, but be re
tained for 2
hours, just click on the pol
icy that spec
if
ies a 5 minute gran
ul
ar
ity and change the re
ten
tion pe
riod to 2 hours. It is sim
il
arly pos
si
ble to change the poli
cies for spe
cific
Mon
it
or
ing Ob
jects. A mon
it
or
ing ob
ject tells the APIC which com
po
nents to gather
Monitoring 291
sta
tis
tics about. For ex
am
ple, to change the in
for
ma
tion gath
ered for Bridge Do
mains,
use the Bridge Do
main (infra.
RSOInfraBD) Mon
it
or
ing Ob
ject.
To add mon
it
or
ing ob
jects:
1
In the Navigation pane choose Tenant_Name > Monitoring

Policies > Monitoring Policy_Name > Stats Collection Policies
a. Click on the Pencil icon to edit the Monitoring Objects.
b. Put a checkmark next to the Monitoring Objects to be included, and remove
any checkmarks next to Monitoring Objects to be left out.
c. Click Submit.
For this ex
am
ple, changes might be made to Mon
it
or
ing Ob
ject poli
cies for Ten
ant,
VXLAN Pool, Leaf Port, and/or Taboo Con
tract. There are sev
eral op
tions and this will
all de
pend on what is im
por
tant to mon
it
or in the en
vi
ron
ment. Click on the pull down
menu to se
lect a mon
it
or
ing ob
ject and add a re
ten
tion pol
icy to it.
To add a pol
icy to a Mon
it
or
ing Ob
ject:
1
In the Navigation pane choose Monitoring Policies > Monitoring

Policy_Name > Stats Collection Policies.
In the Work pane, in the Stats Collection Policy dialog box, perform the
following actions:
a. Select the Monitoring Object.
b. Click + to add the policy.
c. Select the granularity with which it is to poll.
d. Leave the state as inherited to stick with the defaults as set for ALL, or
explicitly select enabled or disabled.
e. The retention policy may either be inherited or explicitly specified as enabled
or disabled as well.
f. Click Update.
292 Monitoring

It is de
sir
able to col
lect these on
go
ing sta
tis
tics as well as to see how this data be
haves
over time. Use the Stats Ex
port Poli
cies op
tion in the left nav
ig
a
tion pane, lo
cated
under the mon
it
or
ing pol
icy. Much like the Stats Col
lec
tion Poli
cies, it is pos
si
ble to
cre
ate a pol
icy for ALL mon
it
or
ing ob
jects, or se
lect spe
cific mon
it
or
ing ob
jects and
spec
ify where this in
for
ma
tion will be saved.
To cre
ate a Stats Ex
port Pol
icy:
1
In the Navigation pane choose Tenant_Name > Monitoring

Policies > Monitoring Policy_Name > Stats Export Policies.
In the Work pane, in the Stats Export Policy dialog box, perform the following
actions:
a. Select ALL or a specific monitoring object from the drop-down list.
b. Click + to add the policy.
c. Now define the Stats Export Policy in the wizard.
d. Choose either JSON or XML as the format. There's really no difference other
than personal preference, or it may be dictated by the tool used to read it.
e. Choose to compress it using GZIP, or leave it uncompressed.
f. Click + under Export Destinations to specify a server where this information
is to be collected. Another wizard will pop up to enable specification of the
protocols and credentials used to connect to this server.
g. Click Ok.
Click Submit.
Next are the di
ag
nos
tics poli
cies in the nav
ig
a
tion pane on the left. This is a re
ally slick
fea
ture that al
lows the setup of di
ag
nos
tics test for the Mon
it
or
ing Ob
jects that were
spec
if
ied in the Stats Col
lec
tion Poli
cies. Next to the Mon
it
or
ing Ob
ject is the Pen
cil
but
ton which en
ables se
lec
tion of the mon
it
or
ing ob
jects to be con
fig
ured with di
ag
nos
tics poli
cies. There are two dif
fer
ent kind of poli
cies for con
fig
ur
a
tion - Boot-Up di
ag
nos
tics or On
go
ing di
ag
nos
tics.
Monitoring 293
To con
fig
ure di
ag
nos
tic poli
cies:
1
In the Navigation pane choose Tenant_Name > Monitoring Policies > default
> Diagnostics Policies.
In the Work pane, in the Diagnostic Policies dialog box, perform the following
actions:
Note: Click on the Pencil Icon and put checks next to the Monitoring Objects
which diagnostics tests are to be added to.
a. Select one of the Monitoring Objects.
b. Click + to add an Object.
i. select either Boot-Up or Ongoing.
ii. Boot-Up runs the tests while the devices are booting, and Ongoing will run
the tests as often as specified within the wizard.
iii. In the wizard give it a name and select the admin state.
iv. There are five different diagnostics tests available: ASIC, CPU, Internal
Connectivity, Peripherals, and System Memory. Double-click on each to
obtain the option of specifying no tests, full tests, or recommended tests.
v. Click Submit.
The di
ag
nos
tics found here can be use
ful in find
ing failed com
po
nents be
fore they
cause major is
sues within your en
vi
ron
ment.
Call Home/SNMP/syslog
There are a few dif
fer
ent ways to setup no
ti
fi
ca
tion or alert poli
cies. The Call
Home/SNMP/sys
log pol
icy will allow alert
ing to be con
fig
ured in a flex
ib
le man
ner.
Cisco Call Home is a fea
ture in many Cisco prod
ucts that will pro
vide email or webbased no
ti
fi
ca
tion alerts in sev
eral dif
fer
ent for
mats for crit
ic
al events. This al
lows ad
min
is
tra
tors to re
solve is
sues be
fore they turn into out
ages. SNMP or sys
log poli
cies
can also be used with cur
rent no
ti
fi
ca
tion sys
tems. Dif
fer
ent log
ging lev
els may be se
lected for no
ti
fi
ca
tions and alert lev
els spec
if
ied for Mon
it
or
ing Ob
jects from which
alerts are to be re
ceived.
294 Monitoring
Event Severity and Fault Severity Assignments

Event and fault sever
it
ies can be changed for events raised by Mon
it
or
ing Ob
jects.
Most likely, the de
fault sever
ity as
sign
ments for Events and Faults will be kept, but
there are ex
am
ples where an ACI ad
min
is
tra
tor may de
cide the event or fault is more
or less se
vere than the de
fault value. For ex
am
ple, if only crit
ic
al faults are being no
ti
fied, but there is a major fault you'd also like to be no
ti
fied about im
me
di
ately, you can
change the sever
ity for that par
tic
ul
ar fault code.
1
In the Navigation pane, choose Tenant_Name > Monitoring Policies

> Monitoring_Policy > Fault Lifecycle Policies.
In the Work pane, in the Fault Severity Assignment Policies dialog box,
perform the following actions:
a. Select a Monitoring Object, which will dictate the fault codes for which you
are changing the fault severity.
b. Click + to add an Object.
c. Select the particular fault code for which severity is to be changed.
d. Select the severity: Cleared, Critical, Major, Minor, Squelched, Inherit,
Warning, Info.
Note: Squelched gives it a weight of 0%, meaning it does not affect health
scores.
Click Update.
The Event Sever

ity As
sign
ment Poli
cies are con
fig
ured in the same way.

Fault Life
cy
cle is the term Cisco uses to de
scribe the life of a fault. Once a fault is de
tected it is in the "soak
ing" state. After a cer
tain amount of time, re
ferred to as the
"soak
ing in
ter
val" it will move on to the "raised" state. "Raised" means the fault is still
pre
sent after the soak
ing in
ter
val. After the fault clears it's in a state called "raised
clear
ing." It is only in this state briefly and moves on to the "clear
ing time" state. It re
mains in the "clear
ing time" state for the amount of time spec
if
ied in the "clear
ing in
ter
val." Lastly it moves on to the "re
tain
ing" state and does not get re
moved until the
end of the "re
tain
ing in
ter
val."
Monitoring 295
To change Fault Life

cy
cle In
ter
vals:
1
In the Navigation pane, choose Tenant_Name > Monitoring Policies >

Monitoring_Policy > Fault Lifecycle Policies.
In the Work pane, in the Fault Lifecycle Policies dialog box, perform the
following actions:
a. Select a Monitoring Object, which will dictate the fault codes for which you
are changing the default intervals.
b. Click +.
c. Specify times for the Clearing Interval, Retention Interval, and Soaking
Interval (all in seconds).
Note: The default for the Clearing Interval is 120 seconds; the Retention
Interval is 3600 seconds; and the Soaking Interval is 120 seconds.
At this point there will be a fully work

ing ten
ant mon
it
or
ing pol
icy. ACME will have
other poli
cies to con
fig
ure in the fab
ric as out
lined in the fol
low
ing sec
tions.
TCAM Policy Usage

The phys
ic
al ternary con
tent-ad
dress
able mem
ory (TCAM) in which pol
icy is stored for
en
force
ment is an ex
pen
sive com
po
nent of switch hard
ware and there
fore tends to
lower pol
icy scale or raise hard
ware costs. Within the Cisco ACI fab
ric, pol
icy is ap
plied based on the EPG rather than the end
point it
self. This pol
icy size can be ex
pressed as n*m*f, where n is the num
ber of sources, m is the num
ber of des
ti
na
tions,
and f is the num
ber of pol
icy fil
ters. Within the Cisco ACI fab
ric, sources and des
ti
na
tions be
come one entry for a given EPG, which re
duces the num
ber of total en
tries re
quired.
TCAM is a fab
ric re
source that should be mon
it
ored. There is a a sys
tem wide view of
avail
able TCAM re
sources. To see this click on Fab
ric > In
ven
tory > Pod1 and then se
lect the Op
er
a
tional tab in the work pane, and you will see a table sum
ma
riz
ing ca
pac
i-
tiy for all nodes.
296 Monitoring
Switch TCAM capacity dashboard

TCAM is a crit
i
cal sys
tem re
source in an ACI fab
ric and should be mon
i
tored for uti
liza
tion. The ar
chi
tec
ture/de
sign team should ar
tic
u
late what the as
sump
tions were for
TCAM uti
liza
tion. There is a Fab
ric Re
source Cal
cu
la
tion tool on Github that will help
with es
ti
ma
tion of nor
mal op
er
at
ing pa
ra
me
ters:
https://
github.
com/
datacenter/
FabricResourceCalculation
As a gen
eral rule, the de
fault mon
i
tor
ing poli
cies will alert you to a re
source short
age
and lower over
all fab
ric health score. If your en
vi
ron
ment has a high rate of change, or
you an
tic
i
pate the pos
si
bil
ity of con
sis
tently being over
sub
scribed, you may want to set
dif
fer
ent thresh
olds.
Create TCAM Policy Monitor

1
In the Navigation pane, choose Monitor Policies > default > Stats Collection
Policies.
Monitoring 297
In the Work pane, in the Stats Collection Policies dialog box, perform the
following actions:
a. Select the Monitoring Object Equipment Capacity Entity
(eqptcapacity.Entity).
b. Select the Stats Type Policy Entry.
c. Click + under Config Thresholds.
d. In the Thresholds For Collection 5 Minute window, select the blue pencil
icon next to policy CAM entries usage current value.
TCAM Prefix Usage

This pro
ce
dure man
ages a TCAM Pre
fix Usage.
1
In the Navigation pane, choose Monitor Policies > default > Stats Collection
Policies.
In the Work pane, in the Stats Collection Policies dialog box, perform the
following actions:
a. Select the Monitoring Object Equipment Capacity Entity
(eqptcapacity.Entity).
b. Select the Stats TypeLayer3 Entry.
c. Click + under Config Thresholds.
d. In the Thresholds For Collection 5 Minute window, select the blue pencil
icon next to policy CAM entries usage current value.
Health Score Evaluation Policy

1
In the Navigation pane, choose Monitor Policies > Common Policies > Health
Score Evaluation Policy > Health Score Evaluation Policy.
In the Work pane, in the Properties dialog box, perform the following actions:
a. In the Penalty of fault severity critical dropdown menu, select the desired %.
b. In the Penalty of fault severity major dropdown menu, select the desired %.
c. In the Penalty of fault severity minor dropdown menu, select the desired %.
d. In the Penalty of fault severity warning dropdown menu, select the desired %.
298 Monitoring
Click Submit.
Communication Policy
1
In the Navigation pane, expand Pod Policies > Policies > Communication.
In the Work pane, choose Actions > Create Communication Policy.
In the Create Communication Policy dialog box, perform the following actions:
a. Enter Communication Policy Name.
b. From the HTTP Admin State dropdown menu select the desired state.
c. From the HTTP Port dropdown menu select the desired port.
d. Select the desired HTTP redirect state.
e. From the HTTPS Admin State dropdown menu select the desired state.
f. From the HTTPS Port dropdown menu select the desired port.
g. Select the desired HTTPS redirect state.
h. From the SSH Admin State dropdown menu select the desired state.
i. From the Telnet Admin State dropdown menu select the desired state.
j. From the Telnet Port dropdown menu select the desired port.
Click Submit.
Monitoring 299
Proactive Monitoring - Infrastructure

While health scores pro
vide a com
pre
hen
sive view of the health sta
tus of var
io
us ob
jects, and fab
ric and ten
ant poli
cies show us a nar
rower view of fab
ric and ten
ant
health, ACME will also need to set up poli
cies to mon
it
or spe
cific re
sources.
This sec
tion of the book at
tempts to cover some key per
for
mance in
di
ca
tors that
should be mon
it
ored, pro
ce
dures for mon
it
or
ing them, and when pos
si
ble, sug
gest rec
om
mended thresh
olds for trig
ger
ing alerts or alarms from Net
work Mon
it
or
ing Sys
tems (NMS).
There are sev
eral meth
ods to ob
tain this data, and ref
er
ences are pro
vided to ways of
ob
tain
ing the data when pos
si
ble. Op
er
at
ors can use a wide array of tools in
clud
ing
Sys
log, SNMP, GUI, REST API calls and CLI.
A full list of MIBs sup
ported on the 1.x is avail
able at:
http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
mib/
list/
mib-support.
html
It is im
por
tant to note that this list changes as newer soft
ware ver
sions are made avail
able, and more vari
ables may be ex
posed via SNMP MIBs in the fu
ture.
Monitoring APICs
CPU utilization and Memory
GUI
The eas
ie
st way to quickly ver
ify the health of the con
trollers is the APIC. When log
ging
into the sys
tem dash
board, the health of the APICs and the health of the clus
ter it
self
are dis
played right at the dash
board.
The screen
shot shows where this in
for
ma
tion is vis
ib
le. In this ex
am
ple, two out of the
three APICs are in a sub-op
ti
mal state and APIC 1 is also ex
pe
ri
enc
ing is
sues.
300 Monitoring
System health dashboard

The nor
mal state for these is to have them all green in a "fully fit" state im
ply
ing the
APICs are syn
chro
nized with each other.
A more de
tailed drill
down is avail
able by click
ing on Sys
tem > Con
trollers.
REST API
Con
trollers pro
vide in
for
ma
tion re
gard
ing the cur
rent sta
tus of CPU and mem
ory uti
liza
tion by cre
at
ing in
stances of the pro
cEn
tity class. pro
cEn
tity is a con
tainer of
processes in the sys
tem. This ob
ject holds de
tailed in
for
ma
tion about var
i
ous processes
run
ning on the APIC. The pro
cEn
tity ob
jects con
tain the fol
low
ing use
ful prop
er
ties:
cpuPct - CPU uti
liza
tion
maxMemAl
loc - The max
i
mum mem
ory al
lo
cated for the sys
tem
mem
Free - The max
i
mum amount of avail
able mem
ory for the sys
tem
Sam
ple Usage: This in
for
ma
tion can be re
trieved for all APIC con
trollers using the fol
low
ing REST call:
http[s]://apic_ip/api/node/class/procEntity.xml?
Monitoring 301
CLI
The Linux Top util

ity also comes built into the APIC con
trollers and can be used for
trou
bleshoot
ing and/or ver
if
i
ca
tion.
user@apic1:~> top
top - 11:41:51 up 16:50,
Tasks: 354 total,
Cpu(s):
Mem:
0.3%us,
0.4%sy,
131954932k total,
Swap:
4 users,
load average: 4.19, 4.27, 4.29
1 running, 353 sleeping,
0 stopped,
0.0%ni, 99.3%id,
0.0%wa,
0 zombie
0.0%hi,
7473180k used, 124481752k free,
0k total,
0k used,
RES
SHR S %CPU %MEM
0.0%st
1952656k cached
PID USER
PR
NI
32102 root
20
556m 205m
85m S
3.3
0.2
38:11.04 svc_ifc_applian
32120 ifc
20
660m 343m
86m S
2.0
0.3
27:58.73 svc_ifc_dbgr.bi
32121 ifc
20
631m 286m
86m S
2.0
0.2
17:41.92 svc_ifc_topomgr
32105 root
20
659m 258m
85m S
1.7
0.2
17:08.35 svc_ifc_bootmgr
32113 ifc
20
0 1083m 721m
69m S
1.7
0.6
20:03.37 svc_ifc_observe
32128 ifc
20
639m 315m
69m S
1.7
0.2
16:28.34 svc_ifc_reader.
32132 ifc
20
657m 252m
71m S
1.7
0.2
17:13.74 svc_ifc_scripth
20
834m 419m
94m S
1.3
0.3
20:35.24 nginx.bin
1291 root
VIRT
0k free,
0.0%si,
409540k buffers
TIME+
COMMAND
Disk Utilization
GUI
There are sev
eral disks and file sys
tems pre
sent on the APICs. The GUI pro
vides ready
ac
cess to disk space uti
liza
tion of all par
ti
tions on the sys
tem and can be used for mon
i
tor
ing this in
for
ma
tion.
The disk uti
liza
tion can be viewed by click
ing on Sys
tem > Con
trollers > Apic-X > Stor
age
The work pane dis
plays the uti
liza
tion of all par
ti
tions in the sys
tem.
302 Monitoring
CLI
user@apic1:~> df
Filesystem
1K-blocks
/dev/dm-1
41282880
10518960
Used Available Use% Mounted on

28666872
tmpfs
4194304
56456
4137848
tmpfs
65977464
964
65976500
1% /tmp
10518960
28666872
27% /data
13860672
25327104
36% /firmware
27% /
2% /dev/shm
/dev/mapper/vg_ifc0-data
41282880
/dev/mapper/vg_ifc0-firmware
41284928
/dev/mapper/vg_ifc0-data2
583149656
1281104 552246280
1% /data2
* Note that not all file sys

tems are vis
ib
le from the CLI as some re
quire root ac
cess to
reach the mount points. The GUI should be used as a sin
gle source of truth for file sys
tem uti
liza
tion.
REST API
This in
for
ma
tion can be re
low
ing REST call:
http[s]://apic-ip/api/node/class/eqptStorage.xml?
Physical and Bond Interface Statistics

APICs use a bonded in
ter
face that is typ
ic
ally dual-homed to two leaves for con
nec
tiv
ity to the ACI fab
ric and have the abil
ity to use a bonded in
ter
face that can be dual
homed to the out-of-band man
age
ment net
work.
Bond0 is the bond in
ter
face used to con
nect to the fab
ric it
self (to con
nect to leaves
that con
nect into the fab
ric).
Bond1 is the bond in
ter
face used to con
nect to the out-of-band seg
ment (to con
nect to
an OOB seg
ment that al
lows setup of the APIC it
self).
Monitoring 303
The bond in
ter
faces rely on un
der
ly
ing phys
ic
al in
ter
faces and it is im
por
tant to note
that the GUI pro
vides link in
for
ma
tion for both the phys
ic
al and log
ic
al bond in
ter
faces.
GUI
To view in
ter
face sta
tus for the in
ter
faces on the APICs, nav
ig
ate to Sys
tem > Con
trollers > Apic-x > In
ter
faces
CLI
Both "if
con
fig" and the "ip link" CLI com
mands can be used to ver
ify link state. The CLI
also pro
vides in
for
ma
tion on de
tailed in
ter
face sta
tis
tics such as RX and TX coun
ters.
REST API
This in
for
ma
tion can be re
low
ing REST call:
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-1/sys.json?querytarget=subtree&target-subtree-class=l3EncRtdIf
APIC Fan Status

The fol
low
ing sec
tion de
scribes method
olo
gies to re
trieve the sta
tus of the fan trays on
the APICs.
GUI
To view in
ter
face sta
tus for the in
ter
ig
ate to Sys
tem > Con
trollers > Apic-x > Equip
ment-Fans
REST API
This in
for
ma
tion can be re
low
ing REST call:
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-1.json?querytarget=subtree&target-subtree-class=eqptFan
CLI
304 Monitoring
CLI
The Fan sta
tus for the APICs can be mon
it
ored using the CLI on the CIMC port of the
APIC. To ob
tain this, login to the CIMC using the cre
den
tials used for set
ting up the
CIMC (may not be the same as the cre
den
tials used for APIC). If this has not been setup
pre
vi
ously, the de
fault user
name is admin and the de
fault pass
word is pass
word.
The CIMC port is the in
te
grated lights-out man
age
ment port that can be used to re
cover an APIC in the event of a cat
as
trophic fail
ure.
user@apic1:~> ssh -l admin 172.16.176.179
Warning: Permanently added '172.16.176.179' (RSA) to the list of known hosts.
[email protected]'s password:
C220-FCH1807V02V# scope sensor
C220-FCH1807V02V /sensor # show fan
Name
Sensor
Reading
Units
Status
----------- ------- --------
Min.
Max.
Min.
Max.
Warning
Warning
Failure
Failure
----- --------- --------- -------- ---------
FAN1_TACH1
Normal
7490
RPM
1712
N/A
1284
N/A
FAN1_TACH2
Normal
7490
RPM
1712
N/A
1284
N/A
FAN2_TACH1
Normal
7490
RPM
1712
N/A
1284
N/A
FAN2_TACH2
Normal
7276
RPM
1712
N/A
1284
N/A
FAN3_TACH1
Normal
7704
RPM
1712
N/A
1284
N/A
FAN3_TACH2
Normal
7276
RPM
1712
N/A
1284
N/A
FAN4_TACH1
Normal
7704
RPM
1712
N/A
1284
N/A
FAN4_TACH2
Normal
7276
RPM
1712
N/A
1284
N/A
FAN5_TACH1
Normal
7704
RPM
1712
N/A
1284
N/A
Temperature Status
To mon
it
or the tem
per
at
ure state of the var
io
us sen
sors avail
able on the APICs use the
fol
low
ing steps.
GUI
To view in
ter
face sta
tus for the in
ter
ig
ate to Sys
tem > Con
trollers > Apic-x > Equip
ment-Sen
sors
REST API
Monitoring 305
REST API
This in
for
ma
tion can be re
low
ing REST call:
https://{{apic-ip}}//api/node/mo/topology/pod-1/node-1.json?querytarget=subtree&target-subtree-class=eqptSensor
CLI
C220-FCH1807V02V /sensor # show temperature
C220-FCH1807V02V /sensor # show temperature
Name
Sensor
Reading
Units
Status
------------------ -------- --------
Min.
Max.
Min.
Max.
Warning
Warning
Failure
Failure
------- ------- --------- -------- ---------
P1_TEMP_SENS
Normal
49.5
N/A
81.0
N/A
86.0
P2_TEMP_SENS
Normal
50.5
N/A
81.0
N/A
86.0
RISER1_INLET_TMP
Normal
45.0
N/A
60.0
N/A
70.0
RISER2_INLET_TMP
Normal
41.0
N/A
60.0
N/A
70.0
RISER1_OUTLETTMP
Normal
50.0
N/A
60.0
N/A
70.0
RISER2_OUTLETTMP
Normal
41.0
N/A
60.0
N/A
70.0
FP_TEMP_SENSOR
Normal
37.0
N/A
60.0
N/A
70.0
DDR3_P1_A1_TEMP
Normal
42.0
N/A
65.0
N/A
85.0
DDR3_P1_B1_TEMP
Normal
43.0
N/A
65.0
N/A
85.0
DDR3_P1_C1_TEMP
Normal
44.0
N/A
65.0
N/A
85.0
DDR3_P1_D1_TEMP
Normal
44.0
N/A
65.0
N/A
85.0
DDR3_P2_E1_TEMP
Normal
43.0
N/A
65.0
N/A
85.0
DDR3_P2_F1_TEMP
Normal
43.0
N/A
65.0
N/A
85.0
DDR3_P2_G1_TEMP
Normal
42.0
N/A
65.0
N/A
85.0
DDR3_P2_H1_TEMP
Normal
41.0
N/A
65.0
N/A
85.0
VICP81E_0_TMP3
Normal
56.0
N/A
85.0
N/A
90.0
PSU1_TEMP
Normal
37.0
N/A
60.0
N/A
65.0
PCH_TEMP_SENS
Normal
51.0
N/A
80.0
N/A
85.0
Power Supply Status

To mon
it
or the tem
per
at
ure state of the var
io
us sen
sors avail
able on the APICs use the
fol
low
ing steps.
306 Monitoring
CLI
C220-FCH1807V02V /sensor # show psu
Name
Sensor
Reading
Units
Status
------------------ -------- --------
--------
Min.
Max.
Min.
Max.
Warning
Warning
Failure
Failure
------- --------- -------- ---------
P1_TEMP_SENS
Normal
49.5
N/A
81.0
N/A
86.0
POWER_USAGE
Normal
160
Watts
N/A
N/A
N/A
800
PSU1_POUT
Normal
136
Watts
N/A
624
N/A
648
PSU1_PIN
Normal
160
Watts
N/A
720
N/A
744
PSU1_STATUS
Normal
present
PSU2_STATUS
Normal
absent
PSU1_PWRGD
Normal
good
PSU1_AC_OK
Normal
good
Monitoring Leaf Switches

Leaf switches in the ACI fab
ric typ
ic
ally equate to the Nexus 9300 fam
ily of switches
(with the ex
cep
tion of the Nexus 9336 switch).
The leaf switches pro
vide first hop con
nec
tiv
ity to any
thing that at
taches to the fab
ric.
Note that un
like tra
di
tional two or three tier de
signs where the "WAN" layer at
taches to
ei
ther the dis
tri
bu
t
ion/ag
gre
ga
tion layer, in the ACI fab
ric, all ex
ter
nal con
nec
tiv
ity is
pro
vided through a set of leaf switches that pro
vide high den
sity 10gig or 40gig con
nec
tiv
ity to el
e
ments out
side the fab
ric.
To ac
cess the dash
board to mon
it
or switches nav
ig
ate to Fab
ric > In
ven
tory > Pod-1 >
Leaf-*
Monitoring 307
Leaf switch monitoring dashboard

No
tice that the dash
board for this switch de
faults to pre
sent
ing the health score of the
switch at a node level on the dash
board, and the sub tabs on the right hand side (topol
ogy/gen
eral/stats/health/faults/trou
bleshoot
ing/his
tory) can be used to quickly drill
down into the var
i
ous prop
er
ties of the switch to un
der
stand how the switch is de
ployed from a hard
ware con
fig
u
ra
tion stand
point, re
me
di
ate faults on the switch, and
trou
bleshoot the switch from a hard
ware per
spec
tive.
Monitoring Switch CPU Utilization

There are sev
eral meth
ods to poll CPU uti
liza
tion and trend it over dif
fer
ent pe
ri
ods of
time. The fol
low
ing sec
tions de
scribe a few of the meth
ods avail
able.
REST API
Spine and Leaf switches CPU uti
liza
tion can be mon
i
tored using the fol
low
ing classes,
based on the de
sired timescale and gran
u
lar
ity.
proc:SysCPU5min A class that rep
re
sents the most cur
rent sta
tis
tics for Sys
tem cpu in
a 5 minute sam
pling in
ter
val. This class up
dates every 10 sec
onds.
308 Monitoring
proc:SysCPU15min A class that rep

re
sents the most cur
rent sta
tis
tics for Sys
tem cpu in
a 15 minute sam
pling in
ter
val. This class up
dates every 5 min
utes.
proc:SysCPU1h A class that rep
re
sents the most cur
rent sta
tis
tics for Sys
tem cpu in a 1
hour sam
pling in
ter
val. This class up
dates every 15 min
utes.
re
sents the most cur
rent sta
tis
tics for Sys
tem cpu in a 1
proc:SysCPU1d A class that rep
day sam
pling in
ter
val. This class up
dates every hour.
proc:SysCPU1w A class that rep
re
sents the most cur
rent sta
tis
tics for Sys
tem cpu in a 1
week sam
pling in
ter
val. This class up
dates every day.
proc:SysCPU1mo A class that rep
re
sents the most cur
rent sta
tis
tics for Sys
tem cpu in a
1 month sam
pling in
ter
val. This class up
dates every day.
proc:SysCPU1qtr A class that rep
re
sents the most cur
rent sta
tis
tics for Sys
tem cpu in a
1 quar
ter sam
pling in
ter
val. This class up
dates every day.
proc:SysCPU1year A class that rep
re
sents the most cur
rent sta
tis
tics for Sys
tem cpu in
a 1 year sam
pling in
ter
val. This class up
dates every day.
ACME would like to see the av
er
age CPU uti
liza
tion of all of the fab
ric switches over the
last day.
http[s]://apic_ip//api/node/class/procSysCPU1d.xml?
CLI
Leaf-1# show proc cpu sort
PID
Runtime(ms)
Invoked
uSecs
1Sec
Process
-----
-----------
--------
-----
------
-----------
4012
69510
493837
140
1.3%
t2usd_tor
4065
7239
27609
262
1.3%
python
4292
3841
134758
28
0.8%
svc_ifc_opflexe
4391
2355
4423
532
0.4%
nginx
4067
1911
206
9278
0.4%
svc_ifc_policye
4302
1904
1862
1022
0.3%
svc_ifc_observe
Monitoring 309
4311
1811
1018
1779
0.3%
svc_ifc_confele
4123
1407
251
5606
0.3%
svc_ifc_eventmg
4310
1802
689
2616
0.3%
svc_ifc_dbgrele
4846
119693
36527
3276
0.2%
stats_manager
3923
15406
2645
5824
0.1%
pfmclnt
4864
2361
2812
839
0.1%
ospfv3
4865
2402
2717
884
0.1%
ospf
13606
435
211
2065
0.0%
bgp
4296
6263
7413
844
0.0%
snmpd
4297
6667
4542
1467
0.0%
dc3_sensor
4299
8559
8225
1040
0.0%
policy_mgr
4301
1860
19152
97
0.0%
plog
4866
2792
3269
854
0.0%
isis
5025
1611
1743
924
0.0%
mcecm
In order to ob
tain a his
tor
ic
al view of CPU uti
liza
tion from the CLI it may be nec
es
sary
to jump into an al
ter
na
tive shell from the switch bash prompt. This shell is called vsh
(or v-shell).
Leaf-1# show processes cpu history
1
33
746554661885833055376572545534667663554785033943645665335644
100
90
80
70
60
50
40
30
##
20
##
10 # ### #######
######## # ##
##### ## ####
# ####
##
0....5....1....1....2....2....3....3....4....4....5....5....
0
CPU% per second (last 60 seconds)

# = average CPU%
310 Monitoring
32 13311134111111111311111111131 11111113 1111 11231 1111111

749513800432206328353370732175609342000769025791144192680117
100
90
80
70
60
50
40 *
30 *
**
20 ** ****
**
**
*
*
* ** * *
***
**
*
**
**
**
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0
CPU% per minute (last 60 minutes)

* = maximum CPU%
# = average CPU%
1
440
030
100
90
80
70
60
50
40 ***
30 ***
20 ***
10 ###
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0
CPU% per hour (last 72 hours)

* = maximum CPU%
# = average CPU%
Monitoring 311
Monitoring Switch Memory Utilization

There are sev
eral meth
ods to poll mem
ory uti
liza
tion and trend it over dif
fer
ent pe
ri
ods of time. The fol
low
ing sec
tions de
scribe a few of the meth
ods avail
able.
REST API
Spine and Leaf switches mem
ory uti
liza
tion can be mon
it
ored using the fol
low
ing
classes, based on the de
sired timescale and gran
ul
ar
ity.
proc:Sys
Mem5min A class that rep
re
sents the most cur
rent sta
tis
tics for Sys
tem mem
ory in a 5 minute sam
pling in
ter
val. This class up
dates every 10 sec
onds.
proc:Sys
Mem15min A class that rep
re
sents the most cur
rent sta
tis
tics for Sys
tem mem
ory in a 15 minute sam
pling in
ter
val. This class up
dates every 5 min
utes.
proc:Sys
Mem1h A class that rep
re
sents the most cur
rent sta
tis
tics for Sys
tem mem
ory
in a 1 hour sam
pling in
ter
val. This class up
dates every 15 min
utes.
proc:Sys
Mem1d A class that rep
re
sents the most cur
rent sta
tis
tics for Sys
tem mem
ory
in a 1 day sam
pling in
ter
val. This class up
dates every hour.
proc:Sys
Mem1w A class that rep
re
sents the most cur
rent sta
tis
tics for Sys
tem mem
ory
in a 1 week sam
pling in
ter
val. This class up
dates every day.
re
sents the most cur
rent sta
tis
tics for Sys
tem mem
proc:Sys
Mem1mo A class that rep
ory in a 1 month sam
pling in
ter
val. This class up
dates every day.
proc:Sys
Mem1qtr A class that rep
re
sents the most cur
rent sta
tis
tics for Sys
tem mem
ory in a 1 quar
ter sam
pling in
ter
val. This class up
dates every day.
proc:Sys
Mem1year A class that rep
re
sents the most cur
rent sta
tis
tics for Sys
tem mem
ory in a 1 year sam
pling in
ter
val. This class up
dates every day.
ACME would like to mon
it
or mem
ory over the last day, and would use the fol
low
ing
REST call:
http[s]://apic_ip/api/node/class/procSysMem1d.xml?
312 Monitoring
CLI
Leaf-1# show system resources
Load average:
1 minute: 1.21
Processes
513 total, 2 running
CPU states
4.1% user,
Memory usage:
5 minutes: 1.14
2.5% kernel,
16401072K total,
15 minutes: 0.80
93.4% idle
9054020K used,
7347052K free
Current memory status: OK
SNMP
As men
tioned in the URL for the SNMP ref
er
ence guide for ACI re
lease 1.x, the fol
low
ing
SNMP ob
jects are sup
ported from an SNMP polling per
spec
tive. See http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
mib/
list/
mibsupport.
html
Monitoring File System Health

CLI
Cur
rently, the CLI is only way to mon
it
or the uti
liza
tion of the file sys
tem on the leaves.
It is nor
mal to see a higher % uti
liza
tion on some of the mount points in the file sys
tem
hi
er
ar
chy. The crit
ic
al vol
umes to keep track of in terms of uti
liza
tion are /volatile,
boot
flash and logflash
Leaf-1# df
df: `/nxos/tmp': No such file or directory
df: `/var/home': No such file or directory
df: `/var/tmp': No such file or directory
df: `/nginx': No such file or directory
df: `/debugfs': No such file or directory
df: `/recovery': No such file or directory
df: `/cfg0': No such file or directory
df: `/cfg1': No such file or directory
df: `/logflash/INXOS_SYSMGR/startup-cfg': No such file or directory
df: `/mnt/plog': No such file or directory
Filesystem
1K-blocks
Used Available Use% Mounted on
Monitoring 313
rootfs
512000
1064
510936
1% /
rootfs
512000
1064
510936
1% /
none
512000
1064
510936
1% /isan
none
512000
1064
510936
1% /var
none
51200
2288
48912
5% /etc
none
51200
108
51092
1% /var/log
none
3145728
336664
2809064
11% /dev/shm
512000
512000
0% /volatile
7782036 1080636
6306088
15% /bootflash
none
/dev/sda4
/dev/sda5
60485
5356
52006
10% /mnt/cfg/0
/dev/sda6
60485
5356
52006
10% /mnt/cfg/1
/dev/sda3
120979
13349
101384
/dev/sda7
512000
1064
510936
/dev/sda9
15748508
591216
14357292
4% /mnt/ifc/cfg
/dev/sda8
15748508
991204
13957304
7% /mnt/ifc/log
/dev/sda8
15748508
991204
13957304
7% /var/log/dme/oldlog
/dev/sda9
15748508
591216
14357292
716800
665728
51072
93% /mnt/ifc/cfg/mgmt/opt/controller/sbin
93% /controller/sbin
rootfs
rootfs
/dev/sda8
rootfs
/dev/sda4
rootfs
/dev/sda9
rootfs
rootfs
716800
665728
51072
15748508
991204
13957304
716800
665728
51072
7782036 1080636
6306088
12% /mnt/pss
1% /logflash
4% /controller
7% /data/techsupport
93% /bin
15% /bootflash
716800
665728
51072
15748508
591216
14357292
93% /data/challenge.old.plugin
716800
665728
51072
93% /controller/sbin
93% /dev
4% /controller
716800
665728
51072
none
3145728
336664
2809064
none
51200
2288
48912
none
2097152
682360
1414792
33% /isan/plugin/0/isan/utils
none
2097152
682360
1414792
33% /isan/plugin/0/lc/isan/utils
none
2097152
682360
1414792
33% /isan/plugin/0/lc/isan/lib
none
2097152
682360
1414792
33% /isan/plugin/0/isan/lib
none
2097152
682360
1414792
33% /isan/lib
none
2097152
682360
1414792
33% /isan/plugin/0/lib
none
2097152
682360
1414792
33% /isan/utils
rootfs
716800
665728
51072
93% /lc/isan/utils
rootfs
716800
665728
51072
93% /lib
rootfs
716800
665728
51072
93% /mnt/cfg
60485
5356
52006
10% /mnt/cfg/0
/dev/sda5
11% /dev/shm
5% /etc
314 Monitoring
/dev/sda6
60485
5356
52006
10% /mnt/cfg/1
716800
665728
51072
93% /mnt/ifc
15748508
591216
14357292
716800
665728
51072
/dev/sda8
15748508
991204
13957304
/dev/sda3
120979
13349
101384
rootfs
/dev/sda9
rootfs
rootfs
/dev/sda8
none
rootfs
none
4% /mnt/ifc/cfg
93% /mnt/ifc/cfg/mgmt/opt/controller/sbin
7% /mnt/ifc/log
12% /mnt/pss
716800
665728
51072
15748508
991204
13957304
93% /sbin
1572864
39444
1533420
3% /tmp
716800
665728
51072
93% /usr
7% /data/techsupport
51200
108
51092
15748508
991204
13957304
51200
108
51092
rootfs
716800
665728
51072
93% /var/log/dme
rootfs
716800
665728
51072
93% /var/log/dme/nginx
rootfs
716800
665728
51072
93% /usr/share/vim
/dev/sda7
11811760
375608
10836140
4% /var/log/dme/log
/dev/sda8
15748508
991204
13957304
none
512000
1064
510936
1% /var/run/mgmt/log
none
512000
1064
510936
1% /var/run/utmp
11811760
375608
10836140
40960
40952
11811760
375608
10836140
rootfs
716800
665728
51072
none
512000
512000
/dev/sda8
none
/dev/sda7
none
/dev/sda7
1% /var/log
1% /var/log/messages
4% /var/sysmgr
1% /var/sysmgr/startup-cfg
4% /logflash/core
93% /usb
0% /volatile
Monitoring CoPP (Control Plane Policing) Statistics

CLI
CoPP is en
abled by de
fault and the pa
ra
me
ters can
not be changed at this time. CoPP
sta
tis
tics are avail
able through the CLI.
To show the CoPP pol
icy that is pro
gramed by the sys
tem use the fol
low
ing CLI com
mand:
Leaf-1# show copp policy
COPP Class
COPP proto
COPP Rate
COPP Burst
Monitoring 315
mcp
mcp
500
500
ifc
ifc
5000
5000
igmp
igmp
1500
1500
nd
nd
1000
1000
cdp
cdp
1000
1000
pim
pim
500
500
dhcp
dhcp
1360
340
lacp
lacp
1000
1000
ospf
ospf
2000
2000
arp
arp
1360
340
lldp
lldp
1000
1000
acllog
acllog
500
500
stp
stp
1000
1000
eigrp
eigrp
2000
2000
coop
coop
5000
5000
traceroute
traceroute
500
500
isis
isis
1500
5000
icmp
icmp
500
500
bgp
bgp
5000
5000
To show drops against spe

cific CoPP classes, use the fol
low
ing CLI com
mand:
Leaf-1# show copp policy stats
COPP Class
COPP proto
COPP Rate
COPP Burst
AllowPkts
AllowBytes
DropPkts
DropBytes
mcp
mcp
500
500
ifc
ifc
5000
5000
195072
161613961
igmp
igmp
1500
1500
192
nd
nd
1000
1000
564
cdp
cdp
1000
1000
494
140543
pim
pim
500
500
dhcp
dhcp
1360
340
1400
lacp
lacp
1000
1000
ospf
ospf
2000
2000
arp
arp
1360
340
1284
90068
lldp
lldp
1000
1000
5029
1717208
acllog
acllog
500
500
stp
stp
1000
1000
eigrp
eigrp
2000
2000
coop
coop
5000
5000
4722
470546
316 Monitoring
traceroute
traceroute
500
500
isis
isis
1500
5000
17141
2167565
icmp
icmp
500
500
bgp
bgp
5000
5000
864
73410
Physical Interface Statistics and Link State

GUI
To ac
cess in
ter
face link state in
for
ma
tion, in the APIC GUI, nav
i
gate to Fab
ric > In
ven
tory > Pod-1 > Leaf-X > In
ter
faces > Phys
i
cal In
ter
faces. In the work pane, the oper
state col
umn dis
plays the op
er
a
tional state of the link. Note that there are other tabs
avail
able in the work pane that ref
er
ence other types of in
ter
faces like port-chan
nels,
vir
tual port-chan
nels, routed in
ter
faces, loop
backs, etc.
To ac
cess in
ter
face sta
tis
tics, in the APIC GUI, nav
i
gate to Fab
ric > In
ven
tory > Pod-1 >
Leaf-X > In
ter
faces > Phys
i
cal in
ter
faces > Eth X/Y, and then click on the Stats tab in
the work pane on the right-hand side.
Physical interface throughput statistics

Note that click
ing on the check icon en
ables you to se
lect ad
di
tional sta
tis
tics that can
be graphed sim
i
lar to the fol
low
ing screen
shot.
Monitoring 317
Granular physical interface statistics
REST API
For cus
tomers that pre
fer the REST API in
ter
face to poll for in
ter
face sta
tis
tics, sev
eral
ob
jects are avail
able. There are sev
eral such coun
ters that are avail
able (e.g. RX/TX,
input/out
put / du
plex, 30 sec
ond rates, 5 minute rate, uni
cast pack
ets, mul
ti
cast
pack
ets, etc.). As a pointer, the par
ent man
aged ob
ject is pro
vided below, as the chil
dren can be de
rived from it.
It is ex
pected that the reader has a good un
der
stand
ing of the ob
ject model and is able
to nav
i
gate through the model to ob
tain the in
for
ma
tion de
siered using the ex
am
ple
below, in
for
ma
tion pro
vided in pre
ced
ing sec
tions and the tools de
scribed therein
An ex
am
ple of the base API call for phys
i
cal in
ter
face sta
tis
tics is:
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/phys-[eth1/1].json
318 Monitoring
For ex
am
ple, to de
ter
mine the total ingress bytes on Leaf 101 port Eth1/1, the ACME
op
er
at
or could issue the fol
low
ing API call:
/topology/pod-1/node-101/sys/phys-[eth1/1].json
Vi
sore al
lows the op
er
at
or to dig deeper into the hi
er
ar
chi
cal tree. From the prior com
mand, the op
er
at
or could see chil
dren of the in
ter
face ob
ject, such as ingress and
egress bytes. One of the child ob
jects in
cludes the fol
low
ing:
topology/pod-1/node-101/sys/phys-[eth1/1]/dbgEtherStats&#10
CLI
The show in
ter
face eth x/y com
mand can be used to mon
it
or in
ter
faces from the CLI.
Other sup
ported com
mands in
clude "show in
ter
face port-chan
nel x/y"
Leaf-1# show int e1/1
Ethernet1/1 is up
admin state is up, Dedicated Interface
Hardware: 1000/10000/auto Ethernet, address: 7c69.f60f.8771 (bia 7c69.f60f.8771)
MTU 9000 bytes, BW 1000000 Kbit, DLY 1 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 1000 Mb/s, media type is 1G
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped 04:19:13
Last clearing of "show interface" counters never
1 interface resets
30 seconds input rate 169328 bits/sec, 97 packets/sec
30 seconds output rate 424528 bits/sec, 115 packets/sec
Monitoring 319
Load-Interval #2: 5 minute (300 seconds)

input rate 644416 bps, 134 pps; output rate 365544 bps, 114 pps
RX
2474537 unicast packets
8434 multicast packets
2482973 input packets

0 jumbo packets
0 runts
0 storm suppression bytes
0 giants
0 input error
0 watchdog
2 broadcast packets
1686129815 bytes
0 CRC
0 no buffer
0 short frame
0 bad etype drop
0 input with dribble
0 overrun
0 underrun
0 bad proto drop
0 ignored
0 if down drop
712 input discard
0 Rx pause
TX
1673907 unicast packets
1674489 output packets
575 multicast packets
7 broadcast packets
455539518 bytes
0 jumbo packets
0 output error
0 collision
0 deferred
0 lost carrier
0 no carrier
0 babble
0 late collision
0 output discard
0 Tx pause
SNMP
As men
er
lease 1.x, the fol
low
ing
SNMP ob
jects are sup
spec
tive for in
ter
faces
See: http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
mib/
list/
mib-support.
html
Module Status
Even though the leaves are con
sid
ered fixed switches, they have a su
per
vi
sor com
po
nent which refers to the CPU com
plex. From a for
ward
ing per
spec
tive, there
are two data plane com
po
nents, viz. the NFE (Net
work For
ward
ing En
gine ASIC) which
pro
vide the front panel ports, and the ALE or ALE2 (Ap
pli
ca
tion Leaf En
gine ASIC) de
pend
ing on the gen
er
at
ion of switch hard
ware, which pro
vides up
link con
nec
tiv
ity to
the spines. The fol
low
ing meth
ods can be used to de
ter
mine the sta
tus of the mod
ules
in the switch.
320 Monitoring
GUI
To ac
cess mod
ule sta
tus for the NFE and the CPU com
plex, in the APIC GUI, nav
ig
ate to
Fab
ric > In
ven
tory > Pod-1 > Leaf-X > Chas
sis > Mod
ule > Su
per
vi
sor mod
ules and the
sta
tus of the mod
ule is dis
played in the work pane.
To ac
cess mod
ule sta
tus for the ALE/ALE2, in the APIC GUI, nav
ig
ate to Fab
ric > In
ven
sis > Mod
ule > Line mod
ules and the sta
tus of the mod
ule
is dis
REST API
The fol
low
ing REST API call(s) can be used to mon
it
or the state of the su
per
vi
sor and
the mod
ule.
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/ch/supslot-1/sup
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/ch/lcslot-1/lc
CLI
The show mod
ule com
mand can be used to ob
tain the sta
tus of the base mod
ule and
the up
link mod
ule.
Leaf-1# show module
Mod
Ports
Module-Type
---
-----
----------------------------------- ------------------ ----------
Model
Status
48
1/10G Ethernet Module
N9K-C9396PX
active
GEM
12
40G Ethernet Expansion Module
N9K-M12PQ
ok
Mod
Sw
Hw
---
--------------
------
11.1(0.152)
0.2050
Mod
MAC-Address(es)
Serial-Num
---
--------------------------------------
----------
7c-69-f6-0f-87-71 to 7c-69-f6-0f-87-ad
SAL17267Z9U
Monitoring 321
Mod
Online Diag Status
---
------------------
pass
SNMP
As men
er
lease 1.x, the fol
low
ing
SNMP ob
jects are sup
spec
tive for mod
ules.
See: http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
mib/
list/
mib-support.
html
Switch Fan Status

The fol
low
ing sec
tion de
scribes method
olo
gies to re
trieve the sta
tus of the fan trays on
the leaf switches.
GUI
To ac
cess fan sta
tus for the leaf switch, in the APIC GUI, nav
ig
ate to Fab
ric > In
ven
tory
> Pod-1 > Leaf-X > Chas
sis > Fan Tray and the sta
tus of the mod
ules is dis
played in the
work pane.
REST API
The fol
low
ing REST API call(s) and their child ob
jects can be used to mon
it
or the state
of the fans on a leaf switch (note that there are 3 slots on this par
tic
ul
ar switch).
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/ch/ftslot-1
CLI
The fol
low
ing CLI's can be used to mon
it
or the state of the fans on a leaf switch.
Leaf-1# show environment fan
Fan:
322 Monitoring
-----------------------------------------------------Fan
Model
Hw
Status
-----------------------------------------------------Fan1(sys_fan1)
N9K-C9300-FAN1-B
--
ok
Fan2(sys_fan2)
N9K-C9300-FAN1-B
--
ok
Fan3(sys_fan3)
N9K-C9300-FAN1-B
--
ok
Fan_in_PS1
--
--
unknown
Fan_in_PS2
--
--
ok
Fan Speed: Zone 1:
0x5f
Fan Air Filter : Absent
SNMP
As men
er
lease 1.x, the fol
low
ing
SNMP ob
jects are sup
spec
tive for fan trays (http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
mib/
list/
mib-support.
html).
Power Supply Status

The fol
low
ing sec
tions de
scribe method
olo
gies to re
trieve the sta
tus of the power sup
plies on the leaf switches
GUI
To ac
cess power sup
ply sta
tus for the leaf switch, in the APIC GUI, nav
ig
ate to Fab
ric >
In
ven
sis > Power Sup
ply Units and the sta
tus of the mod
ules is dis
REST API
The fol
low
ing REST API call(s) and their child ob
jects can be used to mon
it
or the state
of the fans on a leaf switch (note that there are 3 slots on this par
tic
ul
ar switch).
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/ch/psuslot-1
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/ch/psuslot-2
Monitoring 323
CLI
The fol
low
ing CLI com
mands can be used to mon
it
or the state of the fans on a leaf
switch:
Leaf-1# show environment power
Power Supply:
Voltage: 12.0 Volts
Power
Supply
Model
Actual
Total
Output
Capacity
(Watts )
(Watts )
Status
-------
-------------------
-----------
-----------
UCSC-PSU-650W
0 W
648 W
shut
UCSC-PSU-650W
168 W
648 W
ok
Actual
Power
Draw
Allocated
Module
Model
--------------
Status
(Watts )
(Watts )
-----------
-----------
168 W
456 W
Powered-Up
--------
-------------------
N9K-C9396PX
--------------
fan1
N9K-C9300-FAN1-B
N/A
N/A
Powered-Up
fan2
N9K-C9300-FAN1-B
N/A
N/A
Powered-Up
fan3
N9K-C9300-FAN1-B
N/A
N/A
Powered-Up
N/A - Per module power not available

Power Usage Summary:
-------------------Power Supply redundancy mode (configured)
Non-Redundant(combined)
Power Supply redundancy mode (operational)
Non-Redundant(combined)
Total Power Capacity (based on configured mode)
648
Total Power of all Inputs (cumulative)
648
Total Power Output (actual draw)
168
Total Power Allocated (budget)
N/A
Total Power Available for additional modules
N/A
SNMP
As men
er
lease 1.x, the fol
low
ing
SNMP ob
jects are sup
spec
tive for power sup
plies.
324 Monitoring
See: http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
mib/
list/
mib-support.
html
LLDP Neighbor Status

The APIC pro
vides a sin
gle pane of glass to query and de
ter
mine all LLDP neigh
bors in a
fab
ric.
To ob
tain a list of LLDP neigh
bors on an in
ter
face, nav
ig
ate to Fab
ric > In
ven
tory >
Pod-1 > Leaf-X > Pro
to
cols > LLDP > Neigh
bors > eth x/y
A full list
ing of all LLDP neigh
bors on the in
ter
face can be ob
tained in the work pane.
In the above work
flow click
ing on Neigh
bors (in
stead of eth x/y) gives you a list of all
LLDP neigh
bors on the switch.
REST API
The fol
low
ing rest API call can be used to ob
tain the same in
for
ma
tion:
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/lldp/inst/if-[eth1/1]
CLI
Leaf-1# show lldp neighbors
Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID
Local Intf
Hold-time
Capability
Port ID
apic2
Eth1/1
120
apic3
Eth1/4
120
4c:4e:35:09:77:2f
5548-2
Eth1/7
120
Eth1/3
Spine-1
Eth1/49
120
BR
Eth4/1
Spine-2
Eth1/50
120
BR
Eth4/1
Spine-3
Eth1/51
120
BR
Eth1/3
Spine-4
Eth1/52
120
BR
Eth1/3
Spine-5
Eth1/53
120
BR
Eth1/5
90:e2:ba:4b:fa:d4
Monitoring 325
Spine-6
Eth1/54
120
BR
Eth1/5
Total entries displayed: 9
SNMP
As men
er
lease 1.x, the fol
low
ing
SNMP ob
jects are sup
spec
tive for LLDP. See: http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
mib/
list/
mib-support.
html
APIC pro
vides a sin
gle pane of glass to query and de
ter
mine all CDP neigh
bors in a fab
ric. CDP Neigh
bor Sta
tus:
GUI
To ob
tain a list of CDP neigh
bors on an in
ter
face, nav
ig
ate to Fab
ric > In
ven
tory > Pod1 > Leaf-X > Pro
to
cols > CDP > Neigh
bors > eth x/y
A full list
ing of all CDP neigh
bors on the in
ter
face can be ob
tained in the work pane.
In the above work
flow click
ing on Neigh
bors (in
stead of eth x/y) gives you a list of all
LLDP neigh
bors on the switch.
REST API
The fol
low
ing rest API call can be used to ob
tain the same in
for
ma
tion:
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/cdp/inst/if-[eth1/1]
CLI
Leaf-1# show cdp neighbors
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
S - Switch, H - Host, I - IGMP, r - Repeater,
V - VoIP-Phone, D - Remotely-Managed-Device,
s - Supports-STP-Dispute
Device-ID
Local Intrfce
Hldtme
Capability
Platform
Port ID
326 Monitoring
Services-UCS-A(SSI15450J63)
Eth1/5
129
S I s
UCS-FI-6248UP
Eth1/17
Eth1/7
123
S I s
N5K-C5548UP
Eth1/3
5548-2(SSI154300VL)
SNMP
As men
er
lease 1.x, the fol
low
ing
SNMP ob
jects are sup
spec
tive for CDP. See: http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
mib/
list/
mib-support.
html
GOLD Diagnostic Results

GOLD is cov
ered in greater de
tail in the Hard
ware Re
place
ment sec
tion.
GOLD di
ag
nos
tics pro
vide an easy and quick way for op
er
at
ions teams to con
firm that
bootup and non-dis
rup
tive tests that run dur
ing nor
mal op
er
at
ions have ex
e
cuted
prop
erly, as well as the abil
ity to run on de
mand di
ag
nos
tics to iso
late po
ten
tial hard
ware at fault.
GUI
To view GOLD Di
ag
nos
tic test re
sults in the GUI for the Su
per
vi
sors, click on Fab
ric >
In
ven
tory > Pod-1 > Leaf-1 > Chas
sis > Su
per
vi
sor Mod
ules > Slot-1. Then click trou
bleshoot
ing in the work pane.
To view the same for mod
ules, click on Fab
ric > In
ven
tory > Pod-1 > Leaf-1 > Chas
sis >
Line Mod
ules > Slot-x. Then click Trou
bleshoot
ing in the work pane.
Monitoring 327
CLI
Leaf-1# show diagnostic result module all
Current bootup diagnostic level: bypass
Module 1: 1/10G Ethernet Module
(Active)
Test results: (. = Pass, F = Fail, I = Incomplete,

U = Untested, A = Abort, E = Error disabled)
1) bios-mem----------------------->
2) mgmtp-lb----------------------->
4) nsa-mem------------------------>
6) fabp-prbs:
Port
9 10 11 12
----------------------------------------.
22) cpu-cache---------------------->
23) mem-health--------------------->
24) ssd-acc------------------------>
25) act2-acc----------------------->
26) ge-eeprom---------------------->
29) usb-bus------------------------>
30) cons-dev----------------------->
31) obfl-acc----------------------->
32) nvram-cksum-------------------->
33) fpga-reg-chk------------------->
34) asic-scratch------------------->
40) rtc-test----------------------->
41) pcie-bus----------------------->
Monitoring 329
Proactive Monitoring Use Cases

Monitoring Workload Bandwidth
ACME would like to proac
tively mon
it
or con
nec
tions to servers to de
ter
mine whether
ad
e
quate band
width is avail
able to a given work
load. This en
ables them to an
swer
ques
tions about whether a server needs to be up
graded from 10G to 40G, or mul
ti
ple
in
ter
faces need to be bonded to pro
vide more band
width to the server. The fol
low
ing
will pro
vide an ex
am
ple of how sta
tis
tics poli
cies and thresh
olds can be used to alert
the op
er
at
ors when ad
di
tional band
width is re
quired to a server.
ACME's server team has de
ter
mined that they would like to mon
it
or link uti
liza
tion
over a 5 minute pe
riod, and when the av
er
age uti
liza
tion is above 80%, they would like
to raise a fault for the af
fected in
ter
face. To ac
com
plish these tasks re
quires con
fig
ur
ing a mon
it
or
ing pol
icy with two dis
tinct types of poli
cies. First, ACME will con
fig
ure a
stats col
lec
tion pol
icy to col
lect the av
er
age uti
liza
tion of a link over the de
sired 5
minute in
ter
val. Next they will con
fig
ure a thresh
old pol
icy to gen
er
ate a fault when
the link uti
liza
tion ex
ceeds var
io
us thresh
olds ac
cord
ing to the fol
low
ing table.
Cre
ate an in
ter
face mon
it
or
ing pol
icy.
1
In the Navigation pane, choose Monitoring Policies.
In the Work pane, choose Actions > Create Monitoring Policy.
In the Create Monitoring Policy dialog box, perform the following actions:
a. Expand the newly created monitoring policy and select Stats Collection
Policy.
b. Click on the edit button next to Monitoring Object, and select L1 Physical
Interface Configuration.
c. From the Monitoring Object drop-down list, choose L1 Physical Interface
Configuration.
d. Click the edit button next to Stats Type and select Ingress and Egress.
e. From the stats type drop-down list, choose Ingress.
f. Click +.
g. Select 5 minutes from the granularity column, and click on update.
330 Monitoring
We have now cre

ated a pol
icy which will mon
it
or as
so
ci
ated in
ter
faces and ingress
traf
fic rate at 5 minute in
ter
vals. Next we will con
fig
ure a thresh
old which will alert us
if the uti
liza
tion ex
ceeds 80% and as
sign an de
fault fault sever
ity to it.
1
Click + in the Config Thresholds column.
In the Thresholds for Collection 5 minute window, click +.
Click on Ingress Link utilization average value.
Enter 0 for the normal value and click the Rising radio button.
Note: In this ex
am
ple, we are in
ter
ested in an ab
solute per
cent
age, these poli
cies can
be fur
ther cus
tomized to pro
vide a nor
mal value, and look for de
vi
at
ions above or
below that nor
mal value.
In this sce
nario, 80% uti
liza
tion does not nec
es
sar
ily mean that the ap
pli
ca
tion per
for
mance is de
graded, so we will flag this as a warn
ing. Ad
di
tional lev
els/sever
it
ies can
also be spec
if
ied if de
sired.
The set col
umn spec
if
ies the level at which the fault will be raised, and the reset col
umn will spec
ify the the level at which the fault will be cleared. For ex
am
ple, we will be
rais
ing a warn
ing when the uti
liza
tion goes above 80, and clear the warn
ing when the
uti
liza
tion falls below 75. Re
peat these steps for the egress sta
tis
tics as well.
Fi
nally, we will as
so
ci
ate the newly cre
ated pol
icy with an in
ter
face pol
icy group that
rep
re
sents the in
ter
faces we to mon
it
or with this pol
icy.
For our ex
am
ple, we will apply the pol
icy to the UCS-10G-PG
1
2

In the Work pane, choose Interface Policies > Policy Groups.
a. Select UCS-10G-PG.
b. From the Monitoring Policy drop down menu, select the newly created
monitoring policy.
EPG Level Statistics

The ap
pli
ca
tion owner would like to be able to mon
it
or net
work-re
lated in
for
ma
tion
for their ap
pli
ca
tion, such as the ag
gre
gate amount of traf
fic to a spe
cific tier. As an ex
am
ple, we will mon
it
or the amount of traf
fic to the web tier of a given ap
pli
ca
tion. In
Monitoring 331
this ex
am
ple, the de
fault mon
i
tor
ing poli
cies are ap
pro
pri
ate, and they are sim
ply ex
tract
ing them from the sys
tem to be con
sumed ex
ter
nally. This in
for
ma
tion is use
ful in
sce
nar
ios such as a new re
lease being pushed, and to make sure that no traf
fic anom
alies are cre
ated after the push.
This can be ac
com
plished by nav
i
gat
ing to the EPG, and se
lect
ing the Stats tab:
EPG-level throughput statistics

Ad
di
tion
ally, this in
for
ma
tion can be gath
ered from the API:
http[s]://apic_ip/api/node/mo/uni/tn-mynewproject/ap-app1/epg-web-epg.xml?querytarget=self&rsp-subtree-include=stats
Monitoring 333
Reactive Monitoring
It is cru
cial that the ACME op
er
a
tional staff are able to react to any in
di
ca
tion of some
thing going wrong. If there is a no
ti
fi
ca
tion that some
thing has gone wrong, such as a
fault no
ti
fi
ca
tion, a low health score, or a ticket/re
port that end-user func
tion
al
ity has
been im
pacted, knowl
edge of the avail
able mon
i
tor
ing tools is im
por
tant for the iden
ti
fi
ca
tion and col
lec
tion of ev
i
dence. This ev
i
dence can then be used to iden
tify and an
a
lyze the root cause of the prob
lem be
fore tak
ing cor
rec
tive ac
tion. For more in
for
ma
tion
re
gard
ing faults and health scores please refer to those spe
cific sec
tions within this book.
A deep dive into the processes of trou
bleshoot
ing is out of the scope of this book.
bleshoot
ing Cisco Ap
pli
ca
tion Cen
tric In
fra
struc
ture: An
a
lyt
i
cal
Please refer to "Trou
lem solv
ing ap
plied to the pol
icy dri
ven data cen
ter" avail
able at: http://
datacenter.
prob
github.
io/
TenantTroubleshoot Policies
Within the APIC GUI, under each Ten
ant you can find a Trou
bleshoot Pol
icy sec
tion.
This sec
tion will allow con
fig
ur
a
tion of poli
cies that are spe
cific to one ten
ant, and the
mon
it
or
ing of traf
fic and test con
nec
tiv
ity be
tween end
points.
As seen in the image above, the fol
low
ing trou
bleshoot
ing poli
cies can be con
fig
ured:
SPAN (Switched Port ANalyzer)Configuration of SPAN and ERSPAN sources

and destinations to be used in external monitoring of Tenant traffic flows
Endpoint-To-Endpoint TracerouteConfiguration of a path validation tool for

verifying validity of communications between Tenant endpoints in an ACI fabric
Atomic CountersConfiguration of a set of customizable counters to collect

and report on information between a definable set of objects. As shown in the
image below, a policy can be configured to collect statistics between EPs,
between EPGs, between EPGs and specific IPs and other special objects, such
as Any or External
traffic flows
FabricTroubleshoot Policies
For trou
bleshoot
ing within the en
tire fab
ric, there are the fol
low
ing tools and poli
cies:
334 Monitoring
SPAN (Switched Port Analyzer)Configuration of SPAN and ERSPAN sources
On-demand DiagnosticsConfiguration of a policy for collection of diagnostic
and destinations to be used in external monitoring of fabric traffic flows

information that can be executed at a point in time and which will return a set
of valuable output for investigation
Leaf Nodes TracerouteConfiguration of a path validation tool for verifying
Traffic MapAt-a-glance hotspot map of node-to-node traffic flow in an ACI
validity of communications between ACI fabric nodes

fabric

From ver
sion 1.1, APIC pro
vides a trou
bleshoot
ing graphic tool to find rel
e
vant faults
and sta
tis
tics, re
cent changes, and run con
nec
tiv
ity tests in a sim
ple man
ner.
It also gives the op
tion to gen
er
ate a re
port to save the re
sults so it can be used as a
ref
er
ence.
Other Tools
iPingA troubleshooting tool in the ACI fabric that can be used to verify
reachability of a device connected to the fabric utilizing the fabric as the
pervasive source
Audit LogsAudit logs are continually collected on all actions taken in an ACI
fabric and can give a quick indication of which user took which actions at what
time

At the end of this chap
ter, we are going to de
scribe two sit
ua
t
ions where ACME ran
into is
sues and how they made use of the tools pre
vi
ously de
scribed.
1
No access to application: An end-user calls and reports that they can no longer
access a web application running within the fabric.
Users report that an application running in the fabric is slow or there is a report
of slowness to the web application running within the fabric.
Monitoring 335
Reactive Monitoring Tools

Switch Port Analyzer (SPAN)
SPAN is gen
er
ally used in two ways:
Proactively as part of a third party or offline analysis requirement.

Security tools (IDS/IPS, Data Loss Prevention)
Call Recording
Troubleshooting application and networking issues within the fabric.
It may be help
ful to per
form a cap
ture of some par
tic
ul
ar traf
fic to see what is going on
within the stream of data. Look
ing through traf
fic flows will allow in
ves
ti
ga
tion of
packet and pro
to
col level de
tails, such as traf
fic re
sets, mis
be
hav
ing pro
to
cols, im
proper host re
quests or re
sponses, or node level com
mu
ni
ca
tions. This will pro
vide deeper in
sight into how de
vices are using the net
work than sim
ple traf
fic flow and
fab
ric con
fig
ur
a
tion re
view.
Switched Port AN
al
yzer, or SPAN, is a stan
dard fea
ture that al
lows copy and repli
ca
tion
of traf
fic to a net
work an
al
yzer for fur
ther de
cod
ing and in
ves
ti
ga
tion. It can be used to
copy traf
fic from one or more ports, VLANs, or end
point groups (EPGs).
The SPAN fea
ture process is non-dis
rup
tive to any con
nected de
vices and is fa
cil
it
ated
in hard
ware, which pre
vents any un
nec
es
sary CPU load.
SPAN ses
sions can be con
fig
ured to mon
it
or traf
fic re
ceived by the source (ingress
traf
fic), traf
fic trans
mit
ted from the source (egress traf
fic), or both. By de
fault, SPAN
mon
it
ors all traf
fic, but fil
ters can be con
fig
ured to mon
it
or only se
lected traf
fic.
Multinode SPAN
APIC traf
fic mon
it
or
ing poli
cies can en
force SPAN at the ap
pro
pri
ate places to copy
traf
fic from mem
bers of each End Point Group wher
ever they are con
nected. If a mem
ber moves, APIC au
to
mat
ic
ally pushes the pol
icy to the new leaf switch. For ex
am
ple,
when a VMo
tion event re
lo
cates an End
point to a new leaf switch, the SPAN fea
ture
con
fig
ur
a
tion au
to
mat
ic
ally ad
justs.
336 Monitoring
SPAN Guidelines and Restrictions
Use SPAN for troubleshooting. SPAN traffic competes with user traffic for
switch resources. To minimize the load, configure SPAN to copy only the
specific traffic that you want to analyze.
An l3extLIfP Layer 3 subinterface cannot be configured as a SPAN source. The

entire port must be used for monitoring traffic from external sources.
Tenant and access SPAN use the encapsulated remote extension of SPAN
(ERSPAN) type I, while fabric SPAN uses ERSPAN type II. For information
regarding ERSPAN headers, refer to the IETF Internet Draft at this URL: https:/
/
tools.ietf.org/
html/
draft-foschiano-erspan-00.
See the Verified Scalability Guide for Cisco ACI document for SPAN-related
limits, such as the maximum number of active SPAN sessions.
Configuring a SPAN Session

This pro
ce
dure shows how to con
fig
ure a SPAN pol
icy to for
ward repli
cated source
pack
ets to a re
mote traf
fic an
al
yzer.
1
In the Navigation pane, choose Tenant_Name > Troubleshooting Policies >

SPAN > SPAN Destination Groups.
In the Work pane, choose Actions > Create SPAN Destination Group.
In the Create SPAN Destination Group dialog box, perform the following
actions:
a. In the Name field, enter a name for the SPAN destination group.
b. In the Destination EPG dropdowns, select the destination Tenant,
Application Profile, and EPG.
c. Enter the Destination IP.
d. Enter the Source IP Prefix.
e. Optionally, modify the other fields as needed.
f. Click OK.
g. If needed, add additional destinations.
h. Click Submit.
Monitoring 337
Under SPAN, right-click SPAN Source Groups and choose Create SPAN Source
Group.
In the Create SPAN Source Group dialog box, perform the following actions:
a. In the Name field, enter a name for the SPAN source group.
b. From the Destination Group drop-down list, choose the SPAN destination
group that you configured previously.
c. In the Create Sources table, click the + icon to open the Create Sources
dialog box.
d. In the Name field, enter a name for the source.
e. In the Direction field, choose the radio button based on whether you want to
replicate and forward packets that are incoming to the source, outgoing from
the source, or both incoming and outgoing.
f. From the Source EPG drop-down list, choose the EPG (identified by
Tenant/ApplicationProfile/EPG) whose packets will be replicated and
forwarded to the SPAN destination. Click OK to save the SPAN source.
g. Click Submit to save the SPAN source group.
Traceroute
Tracer
oute is a use
ful fea
ture in tra
di
tional net
work
ing. In ACI this fea
ture is im
ple
mented tak
ing into ac
count the way the fab
ric works.
Tracer
oute sup
ports a va
ri
ety of modes, in
clud
ing end
point-to-end
point, and leaf-toleaf (tun
nel end
point, or TEP to TEP). It dis
cov
ers all paths across the fab
ric, dis
cov
ers
point of exits for ex
ter
nal end
points, and helps to de
tect if any path is blocked.
A tracer
oute that is ini
ti
ated from the ten
ant end
points shows the de
fault gate
way as
an in
ter
me
di
ate hop that ap
pears at the ingress leaf switch.
Note: If tracer
oute is done from the OS of a con
nected server or VM, it will show the
hops for the leaves and spines as un
known, and will keep record
ing the in
for
ma
tion
after the packet gets out of the fab
ric. For more pre
cise in
for
ma
tion, please use tracer
oute from the APIC (GUI or CLI)
338 Monitoring
Traceroute Guidelines and Restrictions
When the traceroute source or destination is an endpoint, the endpoint must

be dynamic and not static. Unlike a dynamic endpoint (fv:CEp), a static endpoint
(fv:StCEp) does not have a child object (fv:RsCEpToPathEp) that is required for
traceroute.
See the Verified Scalability Guide for Cisco ACI document for traceroute-related
limits.
Traceroute results will display the IP address of the remote node and interface
which it came it came in on. (Browse on the APIC GUI to Fabric | Inventory |
Fabric Management to view the IP address information for correlation.)
Traceroute cannot be used for endpoints that reside in an external EPG.
Performing a Traceroute Between Endpoints

1

Endpoint-to-Endpoint Traceroute Policies.
In the Work pane, choose Actions > Create Endpoint-to-Endpoint Traceroute

Policy.
In the Create Endpoint-to-Endpoint Traceroute Policy dialog box, perform the

following actions:
a. In the Name field, enter a name for the traceroute policy.
b. In the Source End Points table, click the + icon to edit the traceroute source.
c. From the Source MAC drop-down list, choose or enter the MAC address of
the source endpoint and click Update.
d. In the Destination End Points table, click the + icon to edit the traceroute
destination.
e. From the Destination MAC drop-down list, choose or enter the MAC address
of the destination endpoint and click Update.
f. In the State field, click the Start radio button.
g. Click Submit to launch the traceroute.
In the Navigation pane or the Traceroute Policies table, click the traceroute
policy. The traceroute policy is displayed in the Work pane.
Monitoring 339
In the Work pane, click the Operational tab, click the Source End Points tab,
and click the Results tab.
In the Traceroute Results table, verify the path or paths that were used in the trace.
a. More than one path might have been traversed from the source node to the
destination node.
b. For readability, increase the width of one or more columns, such as the Name
column.
Atomic Counters
Atomic Coun
ters are use
ful for trou
bleshoot
ing con
nec
tiv
ity be
tween end
points, EPGs,
or an ap
pli
ca
tion within the fab
ric. A user re
port
ing ap
pli
ca
tion may be ex
pe
ri
enc
ing
slow
ness, or atomic coun
ters may be needed for mon
it
or
ing any traf
fic loss be
tween
two end
points. One ca
pa
bil
ity pro
vided by atomic coun
ters is the abil
ity to place a
trou
ble ticket into a proac
tive mon
it
or
ing mode, for ex
am
ple when the prob
lem is in
ter
mit
tent, and not nec
es
sar
ily hap
pen
ing at the time the op
er
at
or is ac
tively work
ing
the ticket.
Atomic coun
ters can help de
tect packet loss in the fab
ric and allow the quick iso
la
tion of the
source of con
nec
tiv
ity is
sues. Atomic coun
ters re
quire NTP to be en
abled on the fab
ric.
Leaf-to-leaf (TEP to TEP) atomic coun
ters can pro
vide the fol
low
ing:
Counts of drops, ad
mits, and ex
cess pack
ets
Short-term data col

lec
tion such as the last 30 sec
onds, and long-term data col
lec
tion such as 5 min
utes, 15 min
utes, or more
A break
down of per-spine traf
fic (avail
able when the num
ber of TEPs, leaf or
On
go
ing mon
it
or
ing
VPC, is less than 64)
Leaf-to-leaf (TEP to TEP) atomic coun

ters are cu
mu
la
tive and can
not be cleared. How
ever, be
cause 30 sec
ond atomic coun
ters reset at 30 sec
ond in
ter
vals, they can be used
to iso
late in
ter
mit
tent or re
cur
ring prob
lems.
Ten
ant atomic coun
ters can pro
vide the fol
low
ing:
Application-specific counters for traffic across the fabric, including drops,

admits, and excess packets
340 Monitoring
Modes include the following:
Endpoint to endpoint MAC address, or endpoint to endpoint IP address. Note

that a single target endpoint could have multiple IP addresses associated with it.
EPG to EPG with optional drill down
EPG to endpoint
EPG to * (any)
Endpoint to external IP address
***Atomic coun
ters track the amount pack
ets of be
tween the two end
points and
use this as a mea
sure
ment. They do not take into ac
count drops or error coun
ters in a
hard
ware level.***
Dropped pack
ets are cal
cu
lated when there are less pack
ets re
ceived by the des
ti
na
tion
than trans
mit
ted by the source.
Ex
cess pack
ets are cal
cu
lated when there are more pack
ets re
ceived by the des
ti
na
tion
than trans
mit
ted by the source.
Configuring Atomic Counters

1

Atomic Counter Policy.
Choose a policy (traffic topology). Traffic can be measured between a

combination of endpoints, endpoint groups, external interfaces, and IP
addresses.
5
6
In the Work pane, choose Actions > Add Policy_Name Policy.

In the Add Policy dialog box, perform the following actions:
a. In the Name field, enter a name for the policy.
b. Choose or enter the identifying information for the traffic source. The
required identifying information differs depending on the type of source
(endpoint, endpoint group, external interface, or IP address).
c. Choose or enter the identifying information for the traffic destination.
Monitoring 341
d. Optional: (Optional) In the Filters table, click + to specify filtering of the

traffic to be counted. In the resulting Create Atomic Counter Filter dialog
box, specify filtering by the IP protocol number (TCP=6, for example) and by
source and destination IP port numbers.
e. Click Submit to save the atomic counter policy.
7
In the Navigation pane, under the selected topology, choose the new atomic
counter policy. The policy configuration is displayed in the Work pane.
In the Work pane, choose the Operational tab and choose the Traffic subtab to
view the atomic counter statistics.
Traffic Map
Low per
for
mance and con
ges
tion can be iden
ti
fied by use of Traf
fic maps.
Traf
fic maps make use of atomic coun
ters to con
tin
u
ously mon
i
tor and dis
play traf
fic
be
tween leaf switches to help with quick de
bug
ging and iso
la
tion of ap
pli
ca
tion con
nec
tiv
ity is
sues.
Configuring Traffic Map

1
In the Navigation pane, choose Troubleshooting Policies > Traffic Map.
342 Monitoring
Set the drop-down menu options to view the source Leaf to destination Leaf
traffic paths.
Last interval and cumulative
Sent, received, dropped and excess
All spines and a specific spine switch

The per
cent
age is shown in rel
at
ive terms to all traf
fic by Source or re
ceived by
Des
ti
na
tion.
Clicking on a cell opens a table with all data for all trails and links.

When trou
bleshoot
ing con
nec
tiv
ity be
tween end
points within the fab
ric, the En
hanced
Trou
bleshoot
ing Wiz
ard may be used to quickly iden
tify con
nec
tiv
ity is
sues. The En
hanced Trou
bleshoot
ing Wiz
ard pro
vides a sin
gle lo
ca
tion that in
cludes sev
eral com
monly used tools and out
puts re
quired for trou
bleshoot
ing end point con
nec
tiv
ity.
1
In the menu bar, click the wrench icon to launch the enhanced troubleshooting
wizard.
In the Create SPAN Destination Group dialog box, perform the following
actions:
a. In the Name field, enter a name for the session.
b. Optional: If it is an external IP address, click the check box.
c. In the Source field, enter the MAC or IP address for the source endpoint, and
click search.
d. In the Destination field, enter the MAC or IP address for the destination
endpoint, and click search.
e. Select the Time Window duration for the debug. Optional: Generate a Report
to download the results.
f. Click Start.
In the next screen, you can click the following items for more information:
a. Show the faults in the path between the selected EPs, highlighting the
affected component.
b. Run traceroute between EPs.
c. Show relevant statistics for those EPs.
d. Show any related recent changes in the path between EPs.
Monitoring 343
e. Configure Atomic Counters between EPs.

f. Configure SPAN between EPs
g. Show the configured contracts between EPs.
IPing
IPing is used to test and val
i
date con
nec
tiv
ity within the from leaf node to end
points
within the fab
ric, tak
ing into ac
count the pri
vate net
work. IPing is a trou
bleshoot
ing
tool for net
work users sim
i
lar to the PING com
mand.
Using IPing
iping [ -V vrf ] [ -c count ] [ -i wait ] [ -p pattern ] [ -s packetsize ] [ -t timeout ] host
Syntax Description
Examples
pod1-leaf1# iping -V overlay-1 10.0.59.154
PING 10.0.59.154 (10.0.59.154): 56 data bytes

344 Monitoring
--- 10.0.59.154 ping statistics --5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.23/0.245/0.256 ms
Audit Logs
At times it may be re
quired to view changes which have taken place in the fab
ric. An
out
age re
ported on a host or ap
pli
ca
tion in the fab
ric may need to be tracked, or data
pulled for an audit re
quire
ment.
Audit logs are records of who made a change, when the change was made, and a de
scrip
tion of the ac
tion. Audit logs also record when users logon and lo
goff.
Audit logs can be found in sev
eral places within the GUI, fil
tered to show only those
events rel
e
vant to the cur
rent GUI con
text. Wher
ever a His
tory tab ap
pears in the GUI
Work pane, the audit log can be viewed. This pro
ce
dure shows how to view ten
ant
events as an ex
am
ple.
Viewing Audit Logs

Pro
ce
dure (ex
am
ple for view
ing audit log on a ten
ant)
1
In the Work pane, choose Common.
In the Navigation pane, choose Common.
In the Work pane, choose the History tab.
Under the History tab, choose the Events subtab to view the event log.
Under the History tab, choose the Audit Log subtab to view the audit log.
Double-click a log entry to view additional details about the event.
Monitoring 345

This chap
ter will show some ex
am
ples of how ACME's op
er
at
ions teams
can use their ACI mon
it
or
ing tools to react to a few com
mon pos
si
ble sce
nar
ios.
Note: these ex
am
ples as
sume that basic low-level in
ves
ti
ga
tion has been done and the
issue has been iso
lated to an issue with traf
fic flows across the fab
ric. Ca
bles and con
nec
tiv
ity have been ver
if
ied, hosts are up, VMs are run
ning, processes are run
ning,
mem
ory and CPU uti
liza
tion has been checked, etc.
Loss of Connectivity to Endpoint

The ACME ap
pli
ca
tion owner has re
ported that two servers have lost con
nec
tiv
ity.
These two End Points (EPs) be
long to two dif
fer
ent End Point Groups (EPGs), within the
same sub
net, bridge do
main and Ten
ant, and each EP is con
nected to dif
fer
ent leaf
switches. The bridge do
main has uni
cast rout
ing en
abled, there
fore the de
fault gate
way
for both EPs exist in the leaf switches.
The fol
low
ing steps trou
bleshoot this sit
ua
t
ion:
1
Check that the EPs have been learned by the leaf switches.
Verify that the required contracts are in place between the EPs.
Check that the EPs have been learned by the leaf switches
1

App_Profile_Name > Application EPGs > EPG_Name.
In the Work pane, choose the Operational tab and verify that the endpoint is
present.
Repeat this procedure for the destination EPG.
Verify the required contracts are in place between the EPs
346 Monitoring
Verify the required contracts are in place between the EPs

1

App_Profile_Name > Application EPGs > EPG_Name.
Choose the Contracts subtab.
Check for a relationship between the source EPG and destination EPG, noting
the direction of the arrows.
Click on the contract to verify the contents of the contract. This displays the
filters that are present within that contract.
Inspect the contents of each filter by examining the contract under the
Security Policies folder and verifying that each filter contains the appropriate
filter entries.
If the endpoints are discovered within each EPG and the contract relationships
look correct, examine the troubleshooting policies. A good starting place is the
Enhanced Troubleshooting Wizard.
10
Alternate techniques are available to validate communications between

endpoints within the fabric. One method could be to use endpoint-to-endpoint
traceroute to show if there are paths available between those endpoints.
a. Another option inside the fabric could be to utilize the iPing tool to verify
connectivity between the default gateway and the endpoints. Each leaf has an
SVI used as default gateway. If this test is successful, the connectivity
between endpoints and leaf switches is not the problem. To check the
connectivity to the remote end, use iPing from each leaf to the remote
endpoint using the default gateway as source.
b. In the event that these things seem valid, it then might be necessary to
use SPAN to verify where the traffic is entering and leaving the fabric, and
make sure frames have the right format.
Monitoring 347
Users Report that an Application Running in the Fabric is Slow

ACME test users re
port slow re
sponse of the web servers run
ning in the fab
ric. This
per
for
mance issue could be caused due to la
tency, packet drops, or in
ter
mit
tent con
nec
tiv
ity loss. In this case, the end-user is try
ing to ac
cess the web por
tal from a test
VM. The VM and the web por
tal be
long to a dif
fer
ent EPGs in the same bridge do
main
and ten
ant.
First the op
er
at
ions staff should make sure end points are learned by the leaf switches
in a con
sis
tent way so that the EPs are vis
ib
le in the op
er
at
ional tab under the Ten
ant->
Ap
pli
ca
tion Pro
file-> EPGs-> Op
er
at
ional Tab
Once they ver
ify end
points have been learned, they can use EP-to-EP Tracer
oute to
show the path va
lid
ity. Atomic coun
ters can also be de
ployed to check if there are any
drops or ir
reg
ul
ar
it
ies be
tween the de
fined de
vices. Look
ing through the Traf
fic Map
can also show an at-a-glance view in the fab
ric and high
light any pos
si
ble hotspots or
con
ges
tion in cer
tain el
e
ments of the fab
ric. Check spe
cific coun
ters for CRC er
rors on
the in
ter
faces used to trans
mit EP spe
cific traf
fic.
If every
thing in the fab
ric looks fine, SPAN may be used to ver
ify where the traf
fic is en
ter
ing and leav
ing the fab
ric and make sure frames have the right for
mat. Cor
rupted
frames could cause drops in dif
fer
ent points, from the EPs on a OS level to the
switches.
349
Scripting
Scripting 351
Section Content
Leveraging Network Programmability
Reference to Object Model

Programmatic Interfaces
REST
Read Operations
Write Operations
Authentication
Filters
API Inspector
Development Techniques
POSTman
Installation
Collections
Build Login request
Make Query to APIC
Make Configuration Change in APIC
Use API Inspector for Query Guidance
Cobra SDK and Arya

Establish Session
Work with Objects
Cisco APIC REST to Python Adapter
352 Scripting
ACI Toolkit
ACI Toolkit Applications
Endpoint Tracker
ACI Lint
GitHub
Source Control
GitHub
"It's on github"
Scripting 353
Leveraging Network Programmability

The in
dus
trial rev
ol
u
tion mod
ern
ized the tech
niques used to man
uf
ac
ture goods, going
from hand pro
duc
tion meth
ods to mech
an
ized man
uf
ac
tur
ing. This move
ment from
man
ual to au
to
mated op
er
at
ions changed human pro
duc
tiv
ity, al
low
ing peo
ple to free
them
selves from repet
it
ive tasks that could be more eas
ily ac
com
plished by a ma
chine.
The as
so
ci
ated de
crease in costs, in
crease in speed and in
creased qual
ity al
lowed for
more work to be done for less money in less time, yield
ing a higher qual
ity prod
uct.
Pro
gram
ma
bil
ity promises to offer the same out
come for net
works as the in
dus
trial
rev
ol
u
tion did for goods.
The in
evitable move to
ward au
toma
tion in the IT in
dus
try has pro
vided peo
ple and
busi
nesses a faster way to achieve their de
sired goals, a more cost-ef
fec
tive way to
rapidly pro
vi
sion in
fra
struc
ture in a timely fash
ion ac
cord
ing to de
mand, and
yielded more con
sis
tency in the con
fig
ured re
sults. ACI is able to take ad
van
tage of all
of these ben
e
fits by com
pletely ex
pos
ing all of the na
tive func
tion
al
ity in pro
gram
ma
ble
ways, using com
mon tools and lan
guages to pro
vide net
work en
gi
neers, de
vel
op
ers and
even novices an ap
proach
able path to
ward au
toma
tion. Though ACME is just get
ting
started with true De
vOps in their IT or
ga
ni
za
tion, they re
al
ize that these ben
e
fits will
allow them to keep up with the pace of busi
ness.
Given the com
pre
hen
sive
ness of the pro
gram
ma
bil
ity fea
tures avail
able on ACI, every
one can ben
e
fit. ACME's net
work en
gi
neer
ing and de
sign teams can ben
e
fit from the
quick time to pro
vi
sion large con
fig
ur
a
tions, and the con
sis
tency pro
vided by the abil
ity to au
to
mate all of the mov
ing parts. Their op
er
at
ions teams can uti
lize the plethora
of in
for
ma
tion con
tained within the APIC to stream
line their processes, gather bet
ter
met
rics and cor
re
late events more ac
cu
rately, yield
ing faster time to res
ol
u
tion and
higher cus
tomer sat
is
fac
tion.
Scripting 355
ACI and Scripting

The goals for net
work pro
gram
ma
bil
ity are clear, how
ever the meth
ods by which these
goals may be re
al
ized have been more dif
fi
cult to grasp. Tra
di
tional net
work
ing de
vices
pro
vide out
put that is meant for vi
sual con
sump
tion by peo
ple, and con
fig
u
ra
tions are
dri
ven using text input that is sim
pler for a per
son to type, how
ever these goals stand
in con
trast to an au
toma
tion-dri
ven ap
proach. Ma
chines are able to more eas
ily
process data that is pro
vided in some struc
tured form. Struc
tured data that may not be
vi
su
ally ap
peal
ing can be rapidly parsed, and also can eas
ily rep
re
sent the full de
tail
that a com
pre
hen
sive ob
ject-ori
ented con
fig
u
ra
tion model may rep
re
sent.
ACI uses an ad
vanced ob
ject model that rep
re
sents net
work con
fig
u
ra
tion with ap
pli
ca
tion-based se
man
tics which can be con
sumed and posted against using a well doc
u
mented REST API. In ad
di
tion to pro
vid
ing this in
ter
face into the ob
ject model, ACI also
pro
vides a num
ber of ac
cess meth
ods to read and ma
nip
u
late this data, at a va
ri
ety of
lev
els that will cater to the level of com
fort the user has with pro
gram
ming, all of which
use open stan
dards and open source.
Reference to Object Model
Representation of the top levels of the Object Model

While a com
pre
hen
sive overview of the Ob
ject Model is out
side of this book, from a
pro
gram
ma
bil
ity per
spec
tive it is im
por
tant to note that every as
pect of ACI func
tion
al
-
356 Scripting
ity is en
com
passed within the ob
ject model. This means that all of the con
fig
ur
a
tion
that can be made on the fab
ric, can be made pro
gram
mat
ic
ally using the REST API. This
in
cludes in
ter
nal fab
ric net
work
ing, ex
ter
nal net
work
ing, vir
tu
al
iza
tion in
te
gra
tion,
com
pute in
te
gra
tion, and all other facets of the prod
uct.
This data is stored within the Man
age
ment In
for
ma
tion Tree, with every piece of the
model rep
re
sented as a pro
gram
matic ob
ject with prop
er
ties, iden
tity, and con
sis
tency
rules that are en
forced. This en
sures that the con
fig
ured state of the model will never
get out of hand with stale nodes or en
tries, and every as
pect can be in
spected, ma
nip
u-
lated, and made to cater for the user's needs.
Programmatic Interfaces
APIC is very flex
ib
le in terms of how it can ac
cept con
fig
ur
a
tion and pro
vide ad
min
is
tra
tive and op
er
ab
le states, as well as ex
tend
ing that con
fig
ur
a
tion into sub
or
di
nate
com
po
nents. There are two pri
mary cat
e
gories of in
ter
faces that fa
cil
it
ate these func
tions: the north
bound REST API and the south
bound pro
gram
matic in
ter
faces.
The north
bound REST API is re
spon
si
ble for ac
cept
ing con
fig
ur
a
tion, as well as pro
vid
ing ac
cess to man
age
ment func
tions for the con
troller. This in
ter
face is a cru
cial com
po
nent for the GUI and CLI, and also pro
vides a touch point for au
toma
tion tools, pro
vi
sion
ing scripts and third party mon
it
or
ing and man
age
ment tools. The REST API is a
sin
gu
lar entry point to the fab
ric for mak
ing con
fig
ur
a
tion changes, and as such is a
crit
ic
al as
pect of the ar
chi
tec
ture for being able to pro
vide a con
sis
tent pro
gram
matic
ex
pe
ri
ence.
South
bound in
ter
faces on APIC allow for the de
clar
at
ive model of in
tent to be ex
tended
be
yond the fab
ric, into sub
or
di
nate de
vices. This is a key as
pect to the open
ness of the
ACI fab
ric, in that pol
icy can be pro
grammed once via APIC and then pushed out to hy
per
vi
sors, L4-7 de
vices and po
ten
tially more in the fu
ture, with
out the need to in
di
vid
u
ally con
fig
ure those de
vices. This south
bound ex
ten
sion is re
al
ized through two
meth
ods: L4-7 De
vice Pack
ages and OpFlex.
The L4-7 de
vice pack
age in
ter
face al
lows for ACI to apply pol
icy to ex
ist
ing L4-7 de
vices that do not have an im
plicit knowl
edge of ACI pol
icy. These de
vices can be from
any ven
dor, so long as the de
vice has some form of in
ter
face which is ac
ces
si
ble via IP.
The ac
tual im
ple
men
ta
tion of de
vice pack
ages is done via Python scripts which run on
the APIC within a con
tained ex
e
cu
tion en
vi
ron
ment, which can reach the de
vice
through their na
tive con
fig
ur
a
tion in
ter
faces, be that REST, CLI, SOAP or oth
ers. As a
Scripting 357
user makes changes to ser

vice graphs or EPG pol
icy, the de
vice pack
age will trans
late
the APIC pol
icy into API calls on the L4-7 de
vice.
OpFlex is de
signed to allow a data ex
change of a set of man
aged ob
jects that is de
fined
as part of an in
for
ma
tional model. OpFlex it
self does not dic
tate the in
for
ma
tion model,
and can be used with any tree-based ab
stract model in which each node in the tree has
a uni
ver
sal re
source iden
ti
fier (URI) as
so
ci
ated with it. The pro
to
col is de
signed to sup
port XML and JSON (as well as the bi
nary en
cod
ing used in some sce
nar
ios) and to use
stan
dard re
mote pro
ce
dure call (RPC) mech
an
isms such as JSON-RPC over TCP. In ACI,
OpFlex is cur
rently used to ex
tend pol
icy to the Ap
pli
ca
tion Vir
tual Switch as well as
ex
tend Group Based Pol
icy into Open
Stack.
REST
The Cisco APIC REST API is a pro
gram
matic in
ter
face for Cisco APIC that uses REST ar
chi
tec
ture. The API ac
cepts and re
turns HTTP (not en
abled by de
fault) or HTTPS mes
sages that con
tain JavaScript Ob
ject No
ta
tion (JSON) or Ex
ten
si
ble Markup Lan
guage
(XML) doc
um
ents. You can use any pro
gram
ming lan
guage to gen
er
ate the mes
sages
and the JSON or XML doc
um
ents that con
tain the API meth
ods or MO de
scrip
tions.
The REST API is the in
ter
face into the MIT and al
lows ma
nip
ul
a
tion of the ob
ject model
state. The same REST in
ter
face is used by the Cisco APIC com
mand-line in
ter
face (CLI),
GUI, and SDK, so that when
ever in
for
ma
tion is dis
played, it is read through the REST
API, and when con
fig
ur
a
tion changes are made, they are writ
ten through the REST API.
The REST API also pro
vides an in
ter
face through which other in
for
ma
tion can be re
trieved, in
clud
ing sta
tis
tics, faults, and audit events, and it even pro
vides a means of
sub
scrib
ing to push-based event no
ti
fi
ca
tion, so that when a change oc
curs in the MIT,
an event can be sent through a web socket.
Stan
dard REST meth
ods are sup
ported on the API, which in
cludes POST, GET, and
DELETE op
er
at
ions through HTTP. The POST and DELETE meth
ods are idem
po
tent,
mean
ing that there is no ad
di
tional ef
fect if they are called more than once with the
same input pa
ra
me
ters. The GET method is nul
lipo
tent, mean
ing that it can be called
zero or more times with
out mak
ing any changes (or that it is a read-only op
er
at
ion).
Pay
loads to and from the REST in
ter
face can be en
cap
su
lated through ei
ther XML or
JSON en
cod
ing. In the case of XML, the en
cod
ing op
er
at
ion is sim
ple: the el
e
ment tag is
the name of the pack
age and class, and any prop
er
ties of that ob
ject are spec
if
ied as at
trib
utes of that el
e
ment. Con
tain
ment is de
fined by cre
at
ing child el
e
ments.
358 Scripting
For JSON, en
cod
ing re
quires de
f
i
n
i
tion of cer
tain en
ti
ties to re
flect the tree-based hi
er
ar
chy; how
ever, the de
f
i
n
i
tion is re
peated at all lev
els of the tree, so it is fairly sim
ple
to im
ple
ment after it is ini
tially un
der
stood.
All objects are described as JSON dictionaries, in which the key is the name of the
package and class, and the value is another nested dictionary with two keys:
attribute and children.
The attribute key contains a further nested dictionary describing key-value pairs
that define attributes on the object.
The children key contains a list that defines all the child objects. The children in
this list are dictionaries containing any nested objects, which are defined as
described here.
Read Operations
After the ob
ject pay
loads are prop
erly en
coded as XML or JSON, they can be used in
cre
ate, read, up
date, or delete op
er
a
tions on the REST API. The fol
low
ing di
a
gram
shows the syn
tax for a read op
er
a
tion from the REST API.
REST syntax
Be
cause the REST API is HTTP based, defin
ing the uni
ver
sal re
source iden
ti
fier (URI) to
ac
cess a cer
tain re
source type is im
por
tant. The first two sec
tions of the re
quest URI
sim
ply de
fine the pro
to
col and ac
cess de
tails of Cisco APIC. Next in the re
quest URI is
the lit
eral string /api, in
di
cat
ing that the API will be in
voked. Gen
er
ally, read op
er
a
tions are for an ob
ject or class, as dis
cussed ear
lier, so the next part of the URI spec
i
fies
Scripting 359
whether the op
er
a
tion will be for an MO or class. The next com
po
nent de
fines ei
ther
the fully qual
i
fied Dn being queried for ob
ject-based queries, or the pack
age and class
name for class-based queries. The final manda
tory part of the re
quest URI is the en
cod
ing for
mat: ei
ther .xml or .json. This is the only method by which the pay
load for
mat
is de
fined (Cisco APIC ig
nores Con
tent-Type and other head
ers).
Write Operations
Cre
ate and up
date op
er
a
tions in the REST API are both im
ple
mented using the POST
method, so that if an ob
ject does not al
ready exist, it will be cre
ated, and if it does al
ready exist, it will be up
dated to re
flect any changes be
tween its ex
ist
ing state and de
sired state.
Both cre
ate and up
date op
er
a
tions can con
tain com
plex ob
ject hi
er
ar
chies, so that a
com
plete tree can be de
fined in a sin
gle com
mand so long as all ob
jects are within the
same con
text root and are under the 1MB limit for data pay
loads for the REST API. This
limit is in place to guar
an
tee per
for
mance and pro
tect the sys
tem under high load.
The con
text root helps de
fine a method by which Cisco APIC dis
trib
utes in
for
ma
tion to
mul
ti
ple con
trollers and helps en
sure con
sis
tency. For the most part, the con
fig
u
ra
tion
should be trans
par
ent to the user, though very large con
fig
u
ra
tions may need to be
bro
ken into smaller pieces if they re
sult in a dis
trib
uted trans
ac
tion.
REST Payload
360 Scripting
Cre
ate and up
date op
er
a
tions use the same syn
tax as read op
er
a
tions, ex
cept that they
al
ways are tar
geted at an ob
ject level, be
cause you can
not make changes to every ob
ject of a spe
cific class (nor would you want to). The cre
ate or up
date op
er
a
tion should
tar
get a spe
cific man
aged ob
ject, so the lit
eral string /mo in
di
cates that the Dn of the
man
aged ob
ject will be pro
vided, fol
lowed next by the ac
tual Dn. Fil
ter strings can be
ap
plied to POST op
er
a
tions; if you want to re
trieve the re
sults of your POST op
er
a
tion
in the re
sponse, for ex
am
ple, you can pass the rsp-sub
tree=mod
i
fied query string to
in
di
cate that you want the re
sponse to in
clude any ob
jects that have been mod
i
fied by
your POST op
er
a
tion.
The pay
load of the POST op
er
a
tion will con
tain the XML or JSON en
coded data rep
re
sent
ing the man
aged ob
ject the de
fines the Cisco API com
mand body.
Authentication
REST API user
name- and pass
word-based au
then
ti
ca
tion uses a spe
cial sub
set of re
quest URIs, in
clud
ing aaaLo
gin, aaaL
o
gout, and aaaRe
fresh as the Dn tar
gets of a POST
op
er
a
tion. Their pay
loads con
tain a sim
ple XML or JSON pay
load con
tain
ing the MO
rep
re
sen
ta
tion of an aaaUser ob
ject with the at
tribute name and pwd defin
ing the
user
name and pass
word: for ex
am
ple, <aaaUser name='admin' pwd='in
sieme'/>. The
re
sponse to the POST op
er
a
tion will con
tain an au
then
ti
ca
tion token as both a SetCookie header and an at
tribute to the aaaLo
gin ob
ject in the re
sponse named token,
for which the XPath is /im
data/aaaLo
gin/@to
ken if the en
cod
ing is XML. Sub
se
quent
op
er
a
tions on the REST API can use this token value as a cookie named APIC-cookie to
au
then
ti
cate fu
ture re
quests.
Filters
The REST API sup

ports a wide range of flex
i
ble fil
ters, use
ful for nar
row
ing the scope of
your search to allow in
for
ma
tion to be lo
cated more quickly. The fil
ters them
selves are
ap
pended as query URI op
tions, start
ing with a ques
tion mark (?) and con
cate
nated with
an am
per
sand (&). Mul
ti
ple con
di
tions can be joined to
gether to form com
plex fil
ters.
Scripting 361
The fol
low
ing query fil
ters are avail
able:
Scripting 363
API Inspector
All op
er
a
tions that are per
formed in the GUI in
voke REST calls to fetch and com
mit the
in
for
ma
tion being ac
cessed. The API In
spec
tor fur
ther sim
pli
fies the process of ex
am
in
ing what is tak
ing place on the REST in
ter
face as the GUI is nav
i
gated by dis
play
ing in
real time the URIs and pay
loads. When a new con
fig
u
ra
tion is com
mit
ted, the API In
spec
tor dis
plays the re
sult
ing POST re
quests, and when in
for
ma
tion is dis
played on the
GUI, the GET re
quest is dis
played.
To get started with the API In
spec
tor, it can be ac
cessed from the Ac
count menu, vis
i
ble at the top right of the Cisco APIC GUI. Click Wel
come, <user
name> and then choose
the Show API In
spec
tor op
tion
After the API In
spec
tor is brought up, time stamps will ap
pear along with the REST
method, URIs, and pay
loads. There may also be oc
ca
sional up
dates in the list as the GUI
re
freshes sub
scrip
tions to data being shown on the screen.
API Inspector
364 Scripting
From the out

put above it can see that the last logged item has a POST re
quest with the
JSON pay
load con
tain
ing a ten
ant named Cisco and some at
trib
utes de
fined on that ob
ject:
url: http://
172.
16.
176.
176/
api/
node/
mo/
uni/
tn-Cisco.
json
{
"fvTenant": {
"attributes": {
"name": "Cisco",
"status": "created"
},
"children": [
{
"fvBD": {
"attributes": {
"mac": "00:22:BD:F8:19:FF",
"name": "CiscoBd",
"status": "created"
},
"children": [
{
"fvRsCtx": {
"attributes": {
"tnFvCtxName": "CiscoVrf",
"status": "created,modified"
},
"children": []
}
}
]
}
},
{
"fvCtx": {
"attributes": {
"name": "CiscoVrf",
"status": "created"
Scripting 365
},
"children": []
}
}
]
}
}
Scripting 367
Development Techniques
ACI has a num
ber of meth
ods for de
vel
op
ing code that can be used by en
gi
neers who
have vary
ing lev
els of com
fort with pro
gram
ming or in
ter
act
ing with pro
gram
matic in
ter
faces.
The most basic and straight-for
ward tech
nique in
volves sim
ply tak
ing in
for
ma
tion
gleaned by the API in
spec
tor, Vi
sore, or by sav
ing XML/JSON di
rectly from the GUI,
and using com
mon freely avail
able tools, such as POST
man, to send this in
for
ma
tion
back to the REST API.
A step up from this method en
ables users to use com
mon ter
mi
nol
ogy and well un
der
stood net
work
ing con
structs, cou
pling these with the power and flex
ib
il
ity of the ACI
pol
icy lan
guage and the pop
ul
ar Python pro
gram
ming lan
guage to con
fig
ure ACI in a
pro
gram
matic fash
ion. ACI Toolkit is a util
ity de
vel
oped in open-source that ex
poses
the most com
mon ACI build
ing blocks, to en
able users to rapidly cre
ate ten
ants, ap
pli
ca
tion pro
files, EPGs and the as
so
ci
ated con
cepts to con
nect those to phys
ic
al in
fra
struc
ture. The stream
lined in
ter
face pro
vided makes it very quick to adopt and al
lows
users to begin to quickly de
velop their ap
pli
ca
tions.
The most pow
er
ful of the de
vel
op
ment tools avail
able is the Cobra SDK. With a com
plete rep
re
sen
ta
tion of the ACI ob
ject model avail
able, com
pre
hen
sive data val
id
a
tion,
and ex
ten
sive sup
port for query
ing and fil
ter
ing, Cobra en
sures that the com
plete ACI
ex
pe
ri
ence is avail
able to de
vel
op
ers and users alike.
Scripting 369
POSTman
POST
man is an open source ex
ten
sion for the Chrome web browser, which pro
vides
REST client func
tion
al
ity in an easy-to-use pack
age. POST
man can be used to in
ter
act
with the APIC REST in
ter
face, to both send and re
ceive data which may rep
re
sent con
fig
ur
a
tion, ac
tions, pol
icy and op
er
at
ional state data. For an in
di
vid
ual un
fa
mil
iar with
the struc
ture of REST, it is very sim
ple to uti
lize the API In
spec
tor to view what the un
der
ly
ing calls being made to the GUI are for cer
tain op
er
at
ions, cap
ture those, and then
use POST
man to re
play those op
er
at
ions. Fur
ther
more POST
man al
lows for the re
quests to be mod
if
ied: GUI op
er
at
ions can be made once, at
trib
utes changed in the
cap
tured data and then sent back to the REST API to make the mod
if
i
ca
tions.
Installation
To get started with POST
man, the first step is to down
load the plu
gin for the Chrome
web browser, which is avail
able at www.
getpostman.
com. Once the plu
gin is in
stalled,
it can be ac
cessed using the Chrome App launcher.
Ini
tially the user will be pre
sented with an in
ter
face that has two pri
mary sec
tions: the
side
bar on the left and the re
quest con
struc
tor on the right. Using the side
bar, the user
can switch be
tween the his
tory of REST re
quests sent by POST
man, as well as Col
lec
tions of re
quests that con
tain com
mon tasks.
Collections
A use
ful post to cre
ate in a col
lec
tion is a basic Login op
er
at
ion. In order to do this, the
user should first click into the Col
lec
tions tab in the side
bar. Within the side
bar, a small
folder with a plus (+) sign will be
come vis
ib
le, which should then be clicked, at which
point a popup will ap
pear prompt
ing the user to give a name to the col
lec
tion. For this
ex
am
ple, the col
lec
tion can be named APIC, after which the Cre
ate but
ton should be
clicked.
370 Scripting
Build Login request

Now a new re
quest can be built. In the re
quest con
struc
tor, where Enter re
quest URL
here is shown, the fol
low
ing re
quest URI should be en
tered, sub
sti
tut
ing APIC
IP
AD
DRESS with the IP of the APIC: https://<APIC-IP>/api/mo/aaaLogin.
xml
This re
quest URI will call the Login method in the REST API. Since a Login will re
quire
post
ing data, the HTTP method should be changed, which can be done by click
ing the
drop
down list to the right of the re
quest URL. By de
fault it will be a GET re
quest, but
POST will need to be se
lected from the drop down list.
With the POST method se
lected, it is now pos
si
ble to pro
vide the REST pay
load. Given
that the data will be sent via REST, the raw Re
quest body se
lec
tor should be picked.
Now the pay
load for the re
quest can be en
tered, which will be the sim
ple XML con
tain
ing the user
name and pass
word that will be used for au
then
ti
ca
tion. Note that the URL
is https, mean
ing that it will be en
crypted be
tween the web browser and the APIC, so
no data is being trans
mit
ted in clear text. The fol
low
ing re
quest body should be en
tered, sub
sti
tut
ing the cor
rect user
name and pass
word in place of USER
NAME and
PASS
WORD: <aaaUser name='USERNAME' pwd='PASSWORD'/>
With this re
quest built, it is now pos
si
ble to Send the re
quest, but since this will be a
com
monly used method, the re
quest should be added to a col
lec
tion. This can be ac
com
plished by click
ing the Add to col
lec
tion but
ton be
neath the re
quest body. Se
lect
the APIC col
lec
tion from the ex
ist
ing col
lec
tion list, and change the Re
quest name to
Login and then click Add to col
lec
tion.
By adding the re
quest to a col
lec
tion it can later be quickly ac
cessed to es
tab
lish a login
ses
sion with APIC as needed.
After com
plet
ing the above steps, the re
quest is ready to be sent. Click the Send but
ton in the re
quest con
struc
tor, and the REST API will re
turn the XML rep
re
sent
ing a
login ses
sion with the APIC. The fol
low
ing will be vis
ib
le in the POST
man GUI:
Scripting 371
Login request in POSTman
Make Query to APIC

The next re
quest that will be built is one that queries the APIC for a list of ten
ants on
the sys
tem. First click the Reset but
ton in the re
quest con
struc
tor, and pro
ceed with
the same steps as above, ex
cept that the re
quest URL will be shown as https://<APICIP>/api/class/fvTenant.
xml, and the re
quest method will be changed to GET.
Click Add to col
lec
tion and place the re
quest into the APIC col
lec
tion, and for the
name enter Query APIC for ten
ants
Now upon click
ing Send, this re
quest will re
turn an XML en
coded list of ten
ants in the
re
sponse body sec
tion of the con
struc
tor pane on the right.
Make Configuration Change in APIC

Mak
ing a con
fig
u
ra
tion change will use a POST re
quest sim
i
lar to log
ging in, how
ever
the re
quest URL and body will con
tain a dif
fer
ent set of in
for
ma
tion.
372 Scripting
For this ex
am
ple, a new ten
ant will be cre
ated in the fab
ric. Click the Reset but
ton in
the re
quest con
struc
tor to clear out all ex
ist
ing re
quest fields, and use URL
https://<APIC-IP>/api/mo/uni.
xml and change the method to POST.
In the re
quest pay
load, enter the fol
low
ing data:
<fvTenant name="Cisco"/>
The re
quest URL spec
if
ies that the tar
get for this query will be the pol
icy uni
verse,
which is where ten
ants live. With this tar
get prop
erly scoped, the data rep
re
sent
ing the
ten
ant can be pro
vided in the pay
load, in this case cre
at
ing a ten
ant named Cisco.
Use API Inspector for Query Guidance

As dis
cussed in the In
tro
duc
tion to Script
ing sec
tion, API in
spec
tor can be used as a
guide
line for build
ing cus
tom REST re
quests. Fur
ther
ing on the ex
am
ple in that sec
tion,
where the re
quest URL is https://<APIC-IP>/api/node/mo/uni/tn-Cisco.
json and
the pay
load is the fol
low
ing com
pacted ver
sion of JSON:
{"fvTenant": {"attributes": {"name": "Cisco", "status": "created"}, "children":
[{"fvBD": {"attributes": {"mac": "00:22:BD:F8:19:FF", "name": "CiscoBd", "status":
"created"}, "children": [{"fvRsCtx": {"attributes": {"tnFvCtxName": "CiscoVrf",
"status": "created,modified"}, "children": [] } } ] } }, {"fvCtx": {"attributes":
{"name": "CiscoVrf", "status": "created"}, "children": [] } } ] } }
It is pos
si
ble to mod
ify the re
quest URI and pay
load and sub
sti
tute the ten
ant name
Cisco with an
other ten
ant name, to cre
ate an en
tirely new ten
ant, with the same con
fig
ur
a
tion. The new re
quest URL and JSON would be: https://<APICIP>/api/node/mo/uni/tn-Acme.
json
{"fvTenant": {"attributes": {"name": "Acme", "status": "created"}, "children":
[{"fvBD": {"attributes": {"mac": "00:22:BD:F8:19:FF", "name": "AcmeBd", "status":
"created"}, "children": [{"fvRsCtx": {"attributes": {"tnFvCtxName": "AcmeVrf",
"status": "created,modified"}, "children": [] } } ] } }, {"fvCtx": {"attributes":
{"name": "AcmeVrf", "status": "created"}, "children": [] } } ] } }
Scripting 373
These val
ues can be placed into a POST re
quest in POST
man, and after es
tab
lish
ing a
Login ses
sion using the saved Login re
quest, the new ten
ant Acme can be cre
ated,
iden
ti
cal to the pre
vi
ously cre
ated Cisco ten
ant, with
out need
ing to man
ua
lly click
through the GUI or use other man
ual meth
ods.
Scripting 375
Cobra SDK and Arya

The com
plete Cisco ACI Python SDK is named Cobra. It is a pure Python im
ple
men
ta
tion of the API that pro
vides na
tive bind
ings for all the REST func
tions and also has a
com
plete copy of the ob
ject model so that data in
tegrity can be en
sured, as well as sup
port
ing the com
plete set of fea
tures and func
tions avail
able in ACI. Cobra pro
vides
meth
ods for per
form
ing lookups and queries and ob
ject cre
ation, mod
if
i
ca
tion, and
dele
tion that match the REST meth
ods used by the GUI and those that can be found
using API In
spec
tor. As a re
sult, pol
icy cre
ated in the GUI can be used as a pro
gram
ming tem
plate for rapid de
vel
op
ment.
The in
stal
la
tion process for Cobra is straight
for
ward, and you can use stan
dard Python
dis
tri
bu
t
ion util
it
ies. Cobra is dis
trib
uted on the APIC as an .egg file and can be in
stalled
using easy_in
stall, and is also avail
able on github at http://
github.
com/
datacenter/
cobra.
The com
plete doc
um
en
ta
tion for the Cobra SDK is avail
able at http://
cobra.
readthedocs.
org/
en/
latest/
Establish Session
The first step in any code that uses Cobra is es
tab
lish
ing a login ses
sion. Cobra cur
rently sup
ports user
name- and pass
word-based au
then
ti
ca
tion, as well as cer
tifi
catebased au
then
ti
ca
tion. The ex
am
ple here uses user
name- and pass
word-based au
then
ti
ca
tion.
import cobra.mit.access
import cobra.mit.session
apicUri = 'https://10.0.0.2'
apicUser = 'username'
apicPassword = 'password'
ls = cobra.mit.session.LoginSession(apicUri, apicUser, apicPassword)
md = cobra.mit.access.MoDirectory(ls)
md.login()
376 Scripting
This ex
am
ple pro
vides an MoDi
rec
tory ob
ject named md, which is logged into and au
then
ti
cated for Cisco APIC. If for some rea
son au
then
ti
ca
tion fails, Cobra will dis
play a
cobra.
mit.
request.
CommitEr
ror ex
cep
tion mes
sage. With the ses
sion logged in, you are
ready to pro
ceed.
Work with Objects

Use of the Cobra SDK to ma
nip
ul
ate the MIT gen
er
ally fol
lows this work
flow:
1
Identify the object to be manipulated.
Build a request to change attributes or add or remove children.
Commit the changes made to the object.
For ex
am
ple, if you want to cre
ate a new ten
ant, you must first iden
tify where the ten
ant will be placed in the MIT, where in this case it will be a child of the pol
icy Uni
verse man
aged ob
ject (pol
Un
iMo):
import cobra.model.pol
polUniMo = cobra.model.pol.Uni('')
With the pol

Un
iMo ob
ject de
fined, you can cre
ate a ten
ant ob
ject as a child of pol
U-
niMo:
import cobra.model.fv
tenantMo = cobra.model.fv.Tenant(polUniMo, 'cisco')
All these op
er
at
ions have re
sulted only in the cre
ation of Python ob
jects. To apply the
con
fig
ur
a
tion, you must com
mit it. You can do this using an ob
ject called a Con
fi
gRe
quest. Con
fi
gRe
quest acts as a con
tainer for MO-based classes that fall into a sin
gle
con
text, and they can all be com
mit
ted in a sin
gle atomic POST op
er
at
ion.
import cobra.mit.request
config = cobra.mit.request.ConfigRequest()
config.addMo(tenantMo)
md.commit(config)
Scripting 377
The Con
fi
gRe
quest ob
ject is cre
ated, then the ten
antMo ob
ject is added to the re
quest,
and then you com
mit the con
fig
ur
a
tion through the MoDi
rec
tory ob
ject.
For the pre
ced
ing ex
am
ple, the first step builds a local copy of the pol
Uni ob
ject. Be
cause it does not have any nam
ing prop
er
ties (re
flected by the empty dou
ble sin
gle
quo
ta
tion marks), you dont need to look it up in the MIT to fig
ure out what the full Dn
for the ob
ject is; it is al
ways known as uni.
If you wanted to post some
thing deeper in the MIT, where the ob
ject has nam
ing prop
er
ties, you would need to per
form a lookup for that ob
ject. For ex
am
ple, if you wanted
to post a con
fig
ur
a
tion to an ex
ist
ing ten
ant, you could query for that ten
ant and cre
ate
ob
jects be
neath it.
tenantMo = md.lookupByClass('fvTenant', propFilter='eq(fvTenant.name, "cisco")')
tenantMo = tenantMo[0] if tenantMo else None
The re
sult
ing ten
antMo ob
ject will be of class cobra.
model.
fv.
Tenant and will con
tain
prop
er
ties such as .dn, .sta
tus, and .name, all de
scrib
ing the ob
ject it
self. The lookup
By
Class() entry re
turns an array, be
cause it can re
turn more than one ob
ject. In this
case, the com
mand is spe
cific and is fil
ter
ing on an fv
Tenant ob
ject with a par
tic
ul
ar
name. For a ten
ant, the name at
tribute is a spe
cial type of at
tribute called a nam
ing at
tribute. The nam
ing at
tribute is used to build the rel
at
ive name, which must be unique
in its local name
space. As a re
sult, you can be as
sured that lookup
By
Class on an fv
Tenant ob
ject with a fil
ter on the name al
ways re
turns ei
ther an array of length 1 or
None, mean
ing that noth
ing was found.
To en
tirely avoid a lookup, you can build a Dn ob
ject and make an ob
ject a child of that
Dn. This method works only in cases in which the par
ent ob
ject al
ready ex
ists.
topDn = cobra.mit.naming.Dn.fromString('uni/tn-cisco')
fvAp = cobra.model.fv.Ap(topMo, name='AppProfile')
These fun
da
men
tal meth
ods for in
ter
act
ing with Cobra pro
vide the build
ing blocks
nec
es
sary to cre
ate more com
plex work
flows that can help au
to
mate net
work con
fig
u-
ra
tion, per
form trou
bleshoot
ing, and man
age the net
work.
378 Scripting
Cisco APIC REST to Python Adapter

The process of build
ing a re
quest can be time con
sum
ing, be
cause you must rep
re
sent
the ob
ject data pay
load as Python code re
flect
ing the ob
ject changes that you want to
make. Be
cause the Cobra SDK is di
rectly mod
eled on the Cisco ACI ob
ject model, you
should be able to gen
er
ate code di
rectly from what re
sides in the ob
ject model. As ex
pected, you can do this using a tool de
vel
oped by Cisco Ad
vanced Ser
vices. The tool is
the Cisco APIC REST to Python Adapter, known as Arya.
Sample REST to Python Adapter

The above fig
ure clearly shows how the input that might come from the API In
spec
tor,
Vi
sore, or even the out
put of a REST query and can then be quickly con
verted into
Cobra SDK code, to
k
enized, and reused in more ad
vanced ways.
In
stal
la
tion of Arya is rel
a
tively sim
ple, and the tool has few ex
ter
nal de
pen
den
cies. To
in
stall Arya, you must have Python 2.7.5 and git in
stalled. Use the fol
low
ing quick in
stal
la
tion steps to in
stall it and place it in your sys
tem Python.
git clone https://github.com/datacenter/ACI.git
cd ACI/arya
sudo python setup.py install
Scripting 379
After Arya has been in

stalled, you can take XML or JSON rep
re
sent
ing Cisco ACI mod
eled ob
jects and con
vert it to Python code quickly. For ex
am
ple, enter:
arya.py -f /home/palesiak/simpletenant.xml
The entry will yield the fol

low
ing Python code:
#!/usr/bin/env python
'''
Autogenerated code using arya.py
Original Object Document Input:
<fvTenant name='bob'/>
''' raise RuntimeError('Please review the auto generated code before ' +
'executing the output. Some placeholders will ' +
'need to be changed')
# list of packages that should be imported for this code to work
import cobra.mit.request
import cobra.model.fv
import cobra.model.pol
from cobra.internal.codec.xmlcodec import toXMLStr
# log into an APIC and create a directory object
ls = cobra.mit.session.LoginSession('https://1.1.1.1', 'admin', 'password')
md.login()
# the top level object on which operations will be made
topMo = cobra.model.pol.Uni('')
# build the request using cobra syntax
fvTenant = cobra.model.fv.Tenant(topMo, name='bob')
# commit the generated code to APIC
print toXMLStr(topMo)
c = cobra.mit.request.ConfigRequest()
c.addMo(topMo)
md.commit(c)
380 Scripting
The place
holder rais
ing a run
time error must first be re
moved be
fore this code can be
ex
e
cuted; it is pur
posely put in place to help en
sure that any other to
ke
nized val
ues
that must be up
dated are cor
rected. For ex
am
ple, the Cisco APIC IP ad
dress, which de
faults to 1.
1.
1.
1, should be up
dated to re
flect the ac
tual Cisco APIC IP ad
dress. The same
ap
plies to the cre
den
tials and any other place
hold
ers.
Note that if you pro
vide input XML or JSON that does not have a fully qual
if
ied hi
er
ar
chy, Arya may not be able to iden
tify it through heuris
tics. In this case, a place
holder
will be pop
ul
ated with the text RE
PLACEME, which you will need to re
place with the
cor
rect Dn. You can find this Dn by query
ing for the ob
ject in Vi
sore or in
spect
ing the
re
quest URI for the ob
ject shown in the API In
spec
tor.
Scripting 381
ACI Toolkit
The com
plete ACI ob
ject model con
tains many en
ti
ties, which may be daunt
ing for a
user being first in
tro
duced to net
work pro
gram
ma
bil
ity. The ac
it
oolkit makes avail
able
a sim
pli
fied sub
set of the model that can act as an in
tro
duc
tion to the con
cepts in ACI,
and give users a way to quickly bring up com
mon tasks and work
flows. In ad
di
tion, a
num
ber of ap
pli
ca
tions have been built on top of ACI toolkit.
The com
plete doc
um
en
ta
tion for ac
it
oolkit is avail
able at http://
datacenter.
github.
io/
acitoolkit/
ACI Toolkit Applications

Endpoint Tracker
The end
point tracker ap
pli
ca
tion cre
ates a sub
scrip
tion to the end
point class (fvCEp)
and pop
ul
ates a MySQL data
base with per
ti
nent de
tails about each end
point pre
sent on
the fab
ric (for ex
am
ple servers, fire
walls, load bal
ancers, and other de
vices). In
stalling
MySQL is out
side the scope of this book, so we will as
sume you have ac
cess to cre
ate a
new data
base on a MySQL server. The end
point tracker ap
pli
ca
tion has two pri
mary
com
po
nents that are both python scripts.
aci-endpoint-tracker.pyThis script creates the subscription to the endpoint

class and populates the MySQL database
aci-endpoint-tracker-gui.pyThis script creates a web interface that provides

a way to present the contents of the database to the operator. A sample is
shown below:
382 Scripting
To launch Endpoint Tracker run the following python scripts. The first script, aciendpoint-tracker.py, will actually connect to the APIC and populate the database. The
second script enables the content to be viewed in an understandable web UI.
user@linuxhost:~/acitoolkit/applications/endpointtracker$ ./aci-endpoint-tracker.py
MySQL IP address: 127.0.0.1
MySQL login username: root
MySQL Password:
user@linuxhost::~/acitoolkit/applications/endpointtracker$ python aci-endpointtracker-gui.py
MySQL IP address: 127.0.0.1
MySQL login username: root
MySQL Password:
* Running on http://127.0.0.1:5000/
* Restarting with reloader
After run
ning those python scripts you can now bring up a browser and go the Web
UI. Using the ACI End
point Tracker is sim
ply a mat
ter of in
putting an IP or MAC ad
dress into the search field, and the table is fil
tered ac
cord
ingly. In the ex
am
ple below,
the IP ad
dress 192.
168.
5.
20 has been input into the search field, and the match
ing re
sults are dis
played.
Scripting 383
One more interesting usage of the endpoint tracker applications is a series of

visualizations that can represent how various endpoints are mapped to other fabric
constructs including Tenants, Applications, and EPGs.
Some sam
ple screen
shots are shown below. These are rep
re
sen
ta
tions of where end
points are within the ACI fab
ric and how they re
late to or de
pend on other ob
jects in
the en
vi
ron
ment.
Pie chart view of endpoint distribution
384 Scripting
Tree view of endpoint relationships
Force diagram of endpoint location
ACI Lint
Scripting 385
ACI Lint
In com
puter pro
gram
ming, Lint is a term that refers to iden
ti
fy
ing dis
crep
an
cies, or
sim
ple debug tool for com
mon er
rors. In the sense that ACI pro
vides in
fra
struc
ture as
code, it is ap
pro
pri
ate for ACI to also have a Lint ap
pli
ca
tion. ACI Toolkit pro
vides just
that. ACI Lint is an ap
pli
ca
tion that checks and no
ti
fies an op
er
at
or of mis
con
fig
ur
a
tion
er
rors in two pri
mary ca
pac
it
ies:
Se
cu
rity Is
sues - sup
ports the abil
ity to tag EPGs as ei
ther se
cure or in
se
cure, and
then runs a val
id
a
tion that con
tracts are not used to cross se
cu
rity bound
aries.
Con
fig
u
ra
tion Is
sues - checks for com
mon con
fig
ur
a
tion er
rors and re
ports them to
the user.
A sam
ple out
put is pro
vided here for ref
er
ence:
user@linuxhost:~/acitoolkit/applications/lint$ ./acilint.py
Getting configuration from APIC....
Processing configuration....
Critical 001: EPG 'default' in tenant 'infra' app 'access' is not assigned security
clearance
Critical 001: EPG 'x' in tenant 'common' app 'default' is not assigned security
clearance
Warning 001: Tenant 'Cisco' has no Application Profile.
Warning 001: Tenant 'Books' has no Application Profile.
Warning 001: Tenant '3tierapp' has no Application Profile.
Warning 001: Tenant 'mgmt' has no Application Profile.
Warning 002: Tenant 'Books' has no Context.
Warning 002: Tenant '3tierapp' has no Context.
Warning 004: Context 'oob' in Tenant 'mgmt' has no BridgeDomains.
Warning 005: BridgeDomain 'CiscoBd' in Tenant 'Cisco' has no EPGs.
Warning 005: BridgeDomain 'inb' in Tenant 'mgmt' has no EPGs.
Warning 006: Contract 'default' in Tenant 'common' is not provided at all.
Warning 006: Contract 'WebServers' in Tenant 'Acme' is not provided at all.
Warning 006: Contract 'External' in Tenant 'Acme' is not provided at all.
Warning 007: Contract 'default' in Tenant 'common' is not consumed at all.
Warning 007: Contract 'WebServers' in Tenant 'Acme' is not consumed at all.
Warning 007: Contract 'External' in Tenant 'Acme' is not consumed at all.
Warning 007: Contract 'outside-to-web' in Tenant 'roberbur' is not consumed at all.
386 Scripting
While the ACI Toolkit pro

vides some use
ful tools for an op
er
at
or to im
me
di
ately use,
the real value is in the abil
ity to take these ex
am
ples as a start
ing point, and mod
ify or
ex
tend these sam
ples to suit your par
tic
ul
ar needs. Give it a try! Be sure to share your
work back with the com
mu
nity!
Scripting 387
GitHub
Source Control
Open source soft
ware has been a pop
ul
ar move
ment in IT, and has been the mo
ti
va
tion
be
hind many suc
cess
ful pro
jects, in
clud
ing con
sumer soft
ware, web servers, data
bases
and even en
tire op
er
at
ing sys
tems. One of the key as
pects to the suc
cess of open
source is the abil
ity for many de
vel
op
ers around the globe to col
lab
or
ate to
gether on a
sin
gle pro
ject. Pre
vi
ous tools like Con
cur
rent Ver
sion Con
trol (CVS), and Sub
ver
sion
(SVN) were used to allow many de
vel
op
ers to work to
gether, with a cen
tral server
main
tain
ing a com
mon data
base of source code. While these tools have and con
tinue to
work well, there has been a slow mi
gra
tion away from those server-based tools to de
cen
tral
ized util
it
ies, with the fore
most being Git. Git was cre
ated by Linus Tor
valds, the
au
thor of the pop
ul
ar open-source op
er
at
ing sys
tem Linux. Git has a num
ber of ad
van
tages over most other source con
trol tools: com
plete local repos
it
ory copies, dis
trib
uted ar
chi
tec
ture, and more ef
fi
cient sup
port for branches.
GitHub
GitHub is a host
ing plat
form based around git, which pro
vides both free and paid host
ing ser
vices, that allow for in
di
vid
ua
ls to col
lab
o
rate with over eight-mil
lion other
GitHub users on pro
jects to
gether. Aside from being a wrap
per around git, GitHub also
pro
vides tech
niques for track
ing is
sues, se
cur
ing ac
cess to pro
jects, and built-in pro
ject doc
um
en
ta
tion. The com
bi
na
tion of all of these fea
tures has made GitHub a very
com
mon place for mem
bers of the com
mu
nity to share code with one an
other, build on
each other's work, and con
tribute their ef
forts back into larger pro
jects.
What is stored on GitHub is usu
ally source code, not lim
ited to any spe
cific lan
guage,
how
ever the git pro
to
col it
self sup
ports stor
age and ver
sion con
trol of any file type, so
its not un
com
mon for users to store doc
um
en
ta
tion or other con
stantly chang
ing files
in git. The pri
mary ad
van
tage is that the ver
sion con
trol pro
vided by git al
lows a user to
re
vert a file back to any pre
vi
ously stored ver
sion, or al
ter
nately move for
ward to a
newer ver
sion. Git also main
tains an audit of changes that have been made to files and
even has ad
vanced sup
port for branch
ing ver
sions of files to allow mul
ti
ple con
cur
rent
mod
if
i
ca
tions to a file to take place, and allow for them to be merged after work ef
forts
have com
pleted.
388 Scripting
"It's on github"
A com
mon phrase in mod
ern IT jar
gon is, Its on github, and for users fa
mil
iar with
GitHub, this is an in
vi
ta
tion to down
load, mod
ify and con
tribute to the pro
ject, how
ever for those who have not had an in
tro
duc
tion it can seem like a com
plex topic.
GitHub is ac
tu
ally a very sim
ple tool to use, and the sim
plest way to begin to take ad
van
tage of the in
for
ma
tion stored on GitHub is to sim
ply ac
cess a pro
jects main page
and look for the Down
load ZIP but
ton at the bot
tom right of any pro
ject's main page.
The re
sult
ing down
loaded file will con
tain the lat
est ver
sion of the files in the pro
ject.
What a user does with these files will greatly de
pend on what the con
tents are, how
ever one of the most highly en
cour
aged be
hav
iors on GitHub is to pro
vide clear and
ob
vi
ous doc
um
en
ta
tion for a pro
ject, so if a new user ac
cesses the front page of a pro
ject on Git, they will typ
ic
ally be able to find in
struc
tions on how to down
load and in
stall the pro
ject, right on the first page they see.
For users look
ing to con
tribute back to a pro
ject, the next step would be to sign up for
an ac
count on GitHub, and down
load a graph
ic
al-based client to pro
vide a sim
pler in
ter
face to the com
mand line-based git tool. GitHub it
self has a graph
ic
al client with the
Win
dows ver
sion avail
able at http://
windows.
github.
com and the Mac ver
sions at
http://
mac.
github.
com. Other com
mon source con
trol tools in
clude Source
Tree from
At
lass
ian, avail
able at http://
sourcetreeapp.
com.
Once a user has an ac
count and a github client, they can Fork, or split off a pro
ject
that is avail
able into their own pri
vate repos
it
ory, make changes and com
mit those
back to their pri
vate branch. If those changes work, and the user wishes to con
tribute
them back into the orig
in
al pro
ject, it is pos
si
ble to sub
mit a Pull re
quest, which es
sen
tially means that the user is propos
ing their ef
forts should be pulled back into the
orig
in
al pro
ject. The process can be that sim
ple, though many more ad
vanced pro
jects
have stan
dards and rules for con
tribut
ing to those pro
jects that put in place re
quire
ments around how work is com
mit
ted back into the pro
jects, which may re
quire some
read
ing be
fore at
tempt
ing to con
tribute.
389
Hardware Expansion
and Replacement
Hardware Expansion and Replacement 391
Section Content
Expanding and Shrinking the Fabric

Switches
Add Connected Switch
Pre-Provision Switch Before Connection
Decommission Existing Switch
APICs
Add New APIC
Decommission Existing APIC
Hardware Diagnostics and Replacement

Identify Hardware Failure
Resolve Leaf Hardware Failure
Resolve APIC Hardware Failure
Diagnose Equipment Failures
Expanding and Shrinking the Fabric

ACME may de
cide to ex
pand their ACI fab
ric as their data cen
ter grows, which means
adding new leaf and spine switches, and pos
si
bly APICs. Gen
er
ally, spine switches are
added for more through
put, and leaf switches are added for more ac
cess ports. APICs
are added as the num
ber of poli
cies and end
points in
crease. Ad
di
tion
ally, some
switches or APICs may need to be de
com
mis
sioned. Also, there may be times when you
need to re
place failed hard
ware, which is dis
cussed in the Hard
ware Re
place
ments
chap
ter.
This sec
tion will walk through the op
er
at
ions of adding and re
mov
ing switches and
APICs in your ex
ist
ing ACI fab
ric. This is done the same way for both spine and leaf
switches. Adding APICs will also be cov
ered.
Switches
There are two ways switches can be added to ACME's ex
ist
ing fab
ric: by dis
cov
er
ing the
switches au
to
mat
ic
ally in the APIC after they have been ca
bled to the fab
ric, or by prepro
vi
sion
ing the switches by adding their se
ri
al num
bers and later con
nect
ing them
phys
ic
ally to the fab
ric when the switches ar
rive. Both meth
ods have the same out
come: an ex
panded fab
ric in the mat
ter of min
utes. This sec
tion will also cover de
com
mis
sion
ing switches.
Add Connected Switch

To add a switch that has al
ready been at
tached to the fab
ric go through the fol
low
ing
steps in the APIC GUI:
1
In the case of a leaf switch, cable it to all of the spine switches. In the case of a
spine switch, cable it to all the leaf switches. Ideally, a best-practice ACI fabric
is connected in a full mesh topology with every leaf cabled to every spine. All
devices should connect to the leaf switches, leaves should never connect to
other leaves, and spines should never connect to other spines.
On the APIC click on Fabric at the top of the screen.
Click on Fabric Membership in the left navigation pane.
394 Hardware Expansion and Replacement
When the new switch appears, you'll see a node with a serial number but no
Node ID or Node Name configured. Double click the switch and assign a Node
ID and a Node Name. As a best practice, number leaf nodes starting with 101,
and spine nodes with 201. Lower numbers are reserved for APICs.
Optionally, add a Rack Name name. This is commonly used to identify the
physical location of the switch in the data center.
Click Submit.
Repeat this process for all new switches connected to the fabric.
Pre-Provision Switch Before Connection

Pre-pro
vi
sion
ing a switch is a handy op
er
at
ionally proac
tive step to get the switch reg
is
tered be
fore it even ar
rives to your data cen
ter. You will need to know the se
ri
al num
ber of the switch you will re
ceive to pre-pro
vi
sion. The fol
low
ing steps walk you
through switch pre-pro
vi
sion
ing for both leaves and spines:
1
On the menu bar, choose Fabric.
In the Navigation pane, choose Fabric Membership.
In the Work pane, choose Actions > Create Add Fabric Node Member.
In the Create Add Fabric Node Member dialog box, perform the following
actions:
a. In the pop-up window, enter the serial number of the switch that will be
arriving.
b. Assign a Node ID and a Switch Name. As a best practice, number leaf nodes
starting with 101, and spine nodes with 201. Lower numbers are reserved for
APICs.
Click Submit.
Note: Repeat this process for all switches you wish to pre-provision.
The new entry in the Fab

ric Mem
ber
ship win
dow will show Un
sup
ported in the Role
col
umn until the switch is ac
tu
ally con
nected to the fab
ric, but the switch will im
me
di
ately be
come a mem
ber of the fab
ric once it ar
rives and is ca
bled.
To be proac
tive, you can also pre-pro
vi
sion fab
ric poli
cies. Fab
ric poli
cies are cov
ered
in the Fab
ric Con
nec
tiv
ity chap
ter. For more in
for
ma
tion on pre-pro
vi
sion
ing poli
cies,
refer to the fol
low
ing white paper:
http://
www.
cisco.
com/
c/
en/
us/
solutions/
collateral/
application-centric-infrastructure/
html#_
Toc405844675
Decommission Existing Switch

De
com
mis
sion
ing a switch can ei
ther re
move it from the fab
ric en
tirely, or re
move a
switch tem
porar
ily to per
form main
te
nance. En
sure you do not have any de
vices con
nected, as the switch will not for
ward traf
fic after de
com
mis
sion
ing. There are two
types of switch de
com
mis
sion
ing: Reg
ul
ar, and Re
move from Con
troller.
Reg
u
lar de
com
mis
sion
ing can be used for main
te
nance and es
sen
tially si
lences the switch
from re
port
ing faults and send
ing SNMP in
for
ma
tion as a tem
po
rary so
lu
tion, while keep
ing the switch's node ID and fab
ric mem
ber
ship. The switch will show up under the Dis
abled In
ter
faces and De
com
mis
sioned Switches folder in the left nav
i
ga
tion pane.
The Re
move from Con
troller op
tion will com
pletely re
move the switch from the ACI fab
ric
and all APICs. The switch will no longer show up in the fab
ric mem
ber
ship as a reg
is
tered
node and the in
fra
struc
ture VTEP IP ad
dresses it was as
signed will be re
moved.
Per
form the fol
low
ing steps from the APIC GUI to de
com
mis
sion a switch from the ACI
fab
ric:
1
On the menu bar, choose Fabric.
In the Navigation pane, choose Inventory > Pod 1.
Click the switch to decommission in the Navigation pane.

a. Click the General tab.
b. Chose the Actions > Decommission.
c. In the pop-up, choose either Regular or Remove from Controller.
Click Submit.
APICs
Add New APIC
Be
fore mak
ing any changes to an APIC clus
ter, en
sure each APIC in the clus
ter is fully
fit and change the clus
ter size to re
flect the new con
troller you are adding to the clus
ter. Per
form the fol
low
ing steps to ver
ify clus
ter health:
1
2
On the menu bar, choose System Controllers.

In the Navigation pane, choose Controllers.
a. Expand the first APIC in the folder.
b. Click the Cluster folder.
c. Verify every controller shows Fully Fit under the Heath State column.
If any of the APICs are not fully fit, refer to the fol
low
ing trou
bleshoot
ing guide:
http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
troubleshooting/
b_
APIC_
Troubleshooting.
html
Per
form the fol
low
ing steps to change the APIC clus
ter size:
1
On the menu bar, choose System > Controllers.
In the Navigation pane, choose Controllers > APIC_Name > Cluster.
In the Work pane, choose Actions > Change Cluster Size.

a. Change the Target Cluster Administrative Size to reflect the new APIC(s)
being added.
Note: A cluster size of two is not permitted as that does not allow for quorum
amongst APICs.
Click Submit.
Per
form the fol
low
ing steps to add a new APIC to the clus
ter:
1
Install and stage the APIC by connecting it to the fabric by following the
hardware installation guide:
http://
www.
cisco.
com/
c/
en/
us/
support/
application-policy-infrastructure-controller-apic/
products-installationguides-list.
html

a. The APIC controllers are added one by one and displayed in the sequential order
starting with N + 1 and continuing until the target cluster size is achieved.
b. Verify that the APIC controllers are in operational state, and the health state of
each controller is Fully Fit from the Cluster folder under the new controller.
Note: It will take sev

eral min
utes for the APICs to syn
chro
nize and join the new APIC to
the clus
ter. Fab
ric op
er
at
ion will con
tinue nor
mally.
Decommission Existing APIC

When de
com
mis
sion
ing APICs, they must be de
com
mis
sioned se
quen
tially in re
verse
order. For ex
am
ple, APIC5 must be de
com
mis
sioned be
fore APIC4. Again, be
fore mak
ing any changes to an APIC clus
ter, en
sure each APIC in the clus
ter is fully fit with the
ex
cep
tion of the faulty APIC being de
com
mis
sioned. You can
not de
com
mis
sion a pow
ered on fully fit APIC.
Per
form the fol
low
ing steps to de
com
mis
sion an APIC that needs to be re
moved from
the fab
ric:
1

Note: Select an APIC that is NOT being decommissioned.
In the Work pane, choose Actions > Change Cluster Size.

a. Change the Target Cluster Administrative Size to reflect the new APIC(s)
being added.
Note: A cluster size of two is not permitted as that does not allow for quorum
amongst APICs.
Click Submit.

Note: In the main pane, click the APIC to be decommissioned.
In the Work pane, choose Actions > Actions > Decommission.

a. Verify the APIC no longer appears in the Cluster folder under any of the
remaining APICs.
Hardware Diagnostics and Replacement

The Cisco Ap
pli
ca
tion Cen
tric In
fra
struc
ture (ACI) fab
ric em
ploys a com
bi
na
tion of key
soft
ware and hard
ware fea
tures that are specif
ic
ally de
signed to re
duce the mean time
be
tween fail
ures (MTBF) and the mean time to re
pair (MTTR). Re
gard
ing hard
ware,
there are sev
eral hot-swap
pable com
po
nents on both the leaf and spine switches in ad
di
tion to a few com
po
nents that are fixed on the chas
sis. If ACME ever ex
pe
ri
ences
some sort of power surge or sees a com
po
nent of their switches go bad, the hot-swap
pable com
po
nents en
able them to re
place failed hard
ware quickly and non-dis
rup
tively.
Ex
am
ples of hot-swap
pable com
po
nents on both the leaves and spines in
clude:
Power supplies
Fan trays
Ex
am
ples of hot-swap
pable com
po
nents on the spines in
clude:
Power supplies
Fan trays
Supervisors
System Controller cards
Linecard modules
De
spite sig
nif
ic
ant ad
vances in the above com
po
nents that re
duce the MTBF, there is
al
ways the pos
si
bil
ity of a fail
ure on a leaf switch ei
ther in switch hard
ware or soft
ware,
or a com
bi
na
tion of the two that ne
ces
si
tates a leaf re
place
ment. In such an event, the
state
less na
ture of the ACI fab
ric pro
vides sig
nif
ic
ant ad
van
tages to ad
min
is
tra
tors
from an op
er
at
ions stand
point.
Identify Hardware Failure

When a hard
ware fail
ure oc
curs in the fab
ric, faults are raised in the sys
tem dash
board
and are pre
sented to the ad
min
is
tra
tor. For cases where there is a com
po
nent level
fail
ure with re
dun
dant com
po
nents pre
sent in the sys
tem, sys
log mes
sages and SNMP
traps are gen
er
ated.
Ex
am
ples of hard
ware events that gen
er
ate sys
log mes
sages and SNMP traps in
clude:
Linecard failure on a spine switch
Supervisor failure on a spine switch
System controller failure on a spine switch
Power supply or fan failures on a leaf or a spine switch
While Cisco Ap
pli
ca
tion Pol
icy In
fra
struc
ture Con
troller (APIC) is a cen
tral point of
man
age
ment for the en
tire fab
ric, op
er
at
ions teams can lever
age their ex
ist
ing NMS
tools. Log
ging mes
sages can be sent to sys
log servers, such as Splunk, or SNMP mes
sages can be sent to NMS sys
tems, such as ZenOSS, to pro
vide alert
ing. The leaf and
spine switches in the ACI fab
ric also sup
port tra
di
tional meth
ods of de
tect
ing fail
ures,
such as SNMP polling at a set in
ter
val. If re
sponses are not re
ceived from the switch in
a cer
tain time
frame, there is a pos
si
bil
ity that the hard
ware has failed.
How
ever, while the leaf and spine switches re
port SNMP and Sys
log mes
sages for com
po
nent level fail
ures, the APICs them
selves do not have the abil
ity to gen
er
ate alerts
using SNMP or sys
log. For ex
am
ple a power sup
ply fail
ure on the APIC will not gen
er
ate
an SNMP or sys
log mes
sage and must be mon
it
ored and re
me
di
ated using the APIC
dash
board.
Resolve Leaf Hardware Failure

As an ex
am
ple of a leaf fail
ure, a Nexus 9396 leaf switch that is a part of the fab
ric is un
reach
able, per
haps due to a hard
ware fail
ure on the up
link mod
ules. You can use the
GUI to de
ter
mine the node health to con
firm that the leaf has failed.
To view the node health score:
1
In the Navigation pane, choose Pod 1.

Note: The pod health displays in the Work pane and is zero.
After con
firm
ing that the leaf node has failed, you want to re
move the failed switch and
pro
vi
sion a new switch as part of the fab
ric. The first step in re
plac
ing the failed switch
is to get the failed switch's unique ID (node ID). Each node is as

signed an ID in the fab
ric, which is the ref
er
ence ob
ject that al
lows a re
place
ment switch with a new se
ri
al
num
ber to in
herit the same state
less con
fig
ur
a
tion that was as
signed to the old node.
To view the fab
ric node IDs using the GUI:
1
You can also use a sin

gle REST API call to pe
ri
od
ic
ally poll for a full list of nodes that are
at or below a cer
tain health level, as shown in the fol
low
ing ex
am
ple:
{{protocol}}://{{apic}}/api/class/topSystem.xml?rsp-subtree-include=health&rspsubtree-filter=le(healthInst.cur,"0")
In the case of a tra

di
tional op
er
at
ions model where each switch was man
aged as an in
de
pen
dent en
tity, the fol
low
ing high-level pro
ce
dure re
places the switch:
1
Stand up the replacement switch.
Load the correct version of code.
Attempt to obtain the latest version of configurations from a configuration

repository server.
Stage the device with the right configuration file and eliminate any errors. For
example, update the AAA, NTP, and syslog servers and the ACLs that are
associated with each of them.
Copy the old configuration over to the switch.
Bring up links one by one and verify if data traffic is flowing correctly.
In an ACI fab
ric, you can take ad
van
tage of the state
less na
ture of the hard
ware to in
stan
ti
ate the log
ic
al con
fig
ur
a
tion pro
files. Re
plac
ing the node is as sim
ple as de
com
mis
sion
ing the switch and recom
mis
sion
ing it.
To de
com
mis
sion and recom
mis
sion a switch:
1
In the Navigation pane, expand Pod 1.
Right click the failed node and choose Decommission.
Replace the failed leaf switch with the new leaf switch.
The new leaf appears with a node ID of 0 and an IP address of 0.0.0.0.
In the Work pane, click on the new leaf.
Choose Actions > Commission Switch.
10
When prompted for the node ID, enter the old node's ID. In most cases, you
can also reuse the same leaf name.
11
Click Update.
If the new switch is not op

er
at
ional, the new switch's name and node ID are dif
fer
ent
from the name and ID that you en
tered. You can get the name and ID by view
ing the
un
reach
able nodes.
To view the un
reach
able nodes:
1
In the Navigation pane, choose Unreachable Nodes.
Find the new switch and record its name and node ID.
Repeat the "To decommission and recommission a switch" procedure, starting

with step 5. When prompted for the name and node ID, enter the information
that you recorded in this procedure.
When the new leaf switch is com

mis
sioned suc
cess
fully, the APIC au
to
mat
ic
ally loads
the cor
rect ver
sion of the firmware into the leaf.
To view which ver
sion of the firmware that the APIC will load:
1
In the Navigation pane, choose Fabric Node Firmware > Firmware Groups > All.
Note: In the Work pane, you can see the target firmware version, which
is automatically set to the latest firmware version.
In ad
di
tion, by lever
ag
ing the state
less ob
ject mod
el
ing that re
places the tra
di
tional run
ning con
fig
u
ra
tion on a de
vice, APIC au
to
mat
i
cally loads the cor
rect run
ning con
fig
u
ra
tion onto the de
vice, such as AAA, sys
log, SNMP, NTP, ACLs, bridge do
mains, and EPGs.
In the event that the re
place
ment switch runs stand
alone NX-OS soft
ware in
stead
of ACI switch soft
ware, you might need to copy the ACI switch soft
ware image to the
switch in ques
tion.
To copy the ACI switch soft
ware image to the switch:
1
Connect to the switch console.
Set the IP address on the mgmt0 interface to allow connectivity between the
switch and the APIC.
Enable SCP services:
# feature scp-server
Copy the firmware image from APIC to the switch:
# scp -r /firmware/fwrepos/fwrepo/switch_image_name
admin@switch_ip_address:switch_image_name
For dual supervisor systems, ensure that images are copied to the standby
supervisor in case of a full chassis replacement by using the command:
# copy bootflash:aci_image bootflash://sup-standby/
Configure the switch not to boot from Cisco NX-OS.
switch(config)# no boot nxos
Save the configuration.
switch(config)# copy running-config startup-config
Boot the active and standby supervisor modules with the ACI image.
switch(config)# boot aci bootflash:aci-image-name
Verify the integrity of the file by displaying the MD5 checksum.
switch(config)# show file bootflash:aci-image-name md5sum
10
Reload the switch.
switch(config)# reload
11
Log in to the switch as an administrator.
Login: admin
12
Verify whether you must install certificates for your device.
admin@apic1:aci> openssl asn1parse /securedata/ssl/server.crt
13
Look for PRINTABLESTRING in the command output. If "Cisco Manufacturing

CA" is listed, the correct certificates are installed. If something else is listed,
contact TAC to generate and install the correct certificates for your device.
Once you have con

firmed the that cer
tifi
cate is in
stalled and the switch is in ACI mode,
the switch should ap
pear as an un
man
aged fab
ric node when con
nected to the fab
ric.
Resolve APIC Hardware Failure

In this ex
am
ple, you must iden
tify and re
me
di
ate a hard
ware fail
ure on one of the
APICs in your APIC clus
ter.
From the GUI of an op
er
at
ional APIC,
1
On the menu bar choose System > Controllers.
In the Navigation pane, choose Controllers > apic_name > Cluster.

Note: In the Work pane, you should see the failed APIC in the "Unavailable"
operational state.
Record the fabric name, target size, node ID of the failed APIC, and the TEP
address space. This information is also available through the acidiag avread
command on APICs CLI.
In the Work pane, click the failed APIC to select it.
Choose Actions > Decommission. The APIC changes to an "Out of Service"

admin state.
Decommissioning a Failed APIC

6
Remove the failed APIC from your rack and install the replacement. The new
APIC should boot to the initial setup script.
Proceed through the setup script and enter the values of the failed APIC that
you recorded in step 3. Failure to configure the APIC with the same settings
could result in the fabric entering a partially diverged state.
Once the new APIC finishes booting, in the Navigation pane, choose Controllers
> apic_name > Cluster. You can choose any APIC.
In the Work pane, click the new APIC to select it.
10
Choose Actions > Commission.
Recommissioning an APIC
11
The new APIC will receive an IP address, which will be reflected in the APIC GUI. It
might take 5 to 10 minutes for this to occur. The new APIC might also cycle
between the Available and Unavailable operational states before becoming Fully Fit.
Waiting for Cluster Convergence

12
On the command line of the new APIC, you can verify that it has joined the
fabric by logging in using the credentials that are configured for the rest of the
fabric.
Diagnose Equipment Failures

The ACI fab
ric pro
vides bootup, run
time and on-de
mand di
ag
nos
tics to help as
sess the
hard
ware health of sev
eral sub-sys
tems on each leaf and spine switch.
Boot-up tests run when switch, card boots up. These are typically ONLY
disruptive tests. Comes with default set of tests that can be modified. Deployed
via selectors.
Health (aka On-going) tests run periodically. Can only run non-disruptive tests.
Comes with default set of tests that can be modified and are deployed via
selectors
On-Demand Tests are to be run on specific ports or cards for troubleshooting,

there are no defaults, and they can be disruptive.
By de
fault, tests are log
i
cally grouped into col
lec
tions.
To look at the de
fault di
ag
nos
tic poli
cies, click fab
ric > fab
ric poli
cies > Mon
i
tor
ing poli
cies > de
fault > di
ag
nos
tics pol
icy
In the work pane se
lect the fab
ric el
e
ment that you would like to view the di
ag
nos
tic
mon
i
tor
ing pol
icy for.
Viewing Diagnostic Monitoring Policies in the GUI
Test re
sults are view
able by click
ing on
Fab
ric > in
ven
tory > Pod-1 > Leaf-xx or Spine-xx > Chas
sis > Su
per
vi
sor mod
ules > Slot-1
AND
Fab
ric > in
ven
tory > Pod-1 > Leaf-xx or Spine-xx > Chas
sis > Line mod
ules > Slot-1
Once there, in the work pane se
lect the Trou
bleshoot
ing tab to view GOLD di
ag
nos
tic
re
sults for the su
per
vi
sor.
Viewing GOLD Diagnostics Information in the GUI

In a mod
u
lar chas
sis-based sys
tem such as the Cisco Nexus 9500 se
ries switch, di
ag
nos
tic re
sults are avail
able for all the su
per
vi
sor, mod
ules, sys
tem con
troller and the
fab
ric mod
ules in the sys
tem.
Create new on-demand diagnostic test

As a part of op
er
a
tional pro
ce
dures, it may be nec
es
sary to val
i
date a sys
tem is healthy
by run
ning non-dis
rup
tive tests on the sys
tem.
In order to do this, the APIC GUI al

lows cre
ation of an On-de
mand di
ag
nos
tic test.
To do this, nav
i
gate to Fab
ric > Fab
ric Poli
cies > Trou
bleshoot Poli
cies > On de
mand di
ag
nos
tics
Once there, right click on the test or test set you would like to run on de
mand, and
click on Cre
ate.
In the ex
am
ple below, an op
er
a
tor is cre
at
ing a fab
ric-port on-de
mand diag pol
icy that
al
lows the abil
ity to test the leaf up
link ports going to the spine.
Creating a On-Demand Diag Policy in the GUI

Once you click cre
ate, en
sure you check the box against "in
clude dis
rup
tive tests" if this
is the in
tent.
Enter a name for the test, se
lect the admin state, under the di
ag
nos
tic tests, se
lect a
test con
fig (de
fault is "no tests") and se
lect the fab
ric ports that the test would need to
be run on.
In this case, the op
er
a
tor se
lects the op
tions for a non-dis
rup
tive test to be done on
leaf-2, fab
ric-port 1/97.
Creating a Fabric Port On-Demand Diag Policy in the GUI

After ver
i
fy
ing the pa
ra
me
ters above, click "SUB
MIT" to sub
mit the pol
icy.
Once sub
mit
ted, you can kick off the di
ag
nos
tic test by click
ing on the test it
self and
click
ing sub
mit.
Note that APIC dis
plays a warn
ing mes
sage in cases where a non-dis
rup
tive tests are
se
lected to the ef
fect of below.
GUI Warning for Executing a Disruptive Diag Policy

Once you con
firm the test run, the di
ag
nos
tic re
sults can be ob
tained from the same
lo
ca
tion as the lo
ca
tion for di
ag
nos
tics that are run at bootup or are on
go
ing.
Fab
ric > In
ven
tory > Pod-1 > Leaf-2 > Line-mod
ules > Slot-1 > Fab
ric ports > 1/97
Viewing On-Demand Diag Policy Test Results in the GUI

Note the on-de
mand re
sults in the right hand cor
ner after the test has com
pleted its
run.
413
Appendix
Appendix 415
Classes
The Ap
pli
ca
tion Pol
icy In
fra
struc
ture Con
troller (APIC) classes are cru
cial from an op
er
at
ional per
spec
tive to un
der
stand how sys
tem events and faults re
late to ob
jects
within the ob
ject model. Each event and/or fault in the sys
tem is a unique ob
ject that
can be ac
cessed for con
fig
ur
a
tion, health, fault, and/or sta
tis
tics.
All the phys
ic
al and log
ic
al com
po
nents that com
prise the Ap
pli
ca
tion Cen
tric In
fra
struc
ture fab
ric are rep
re
sented in a hi
er
ar
chi
cal man
age
ment in
for
ma
tion tree (MIT).
Each node in the tree rep
re
sents a man
aged ob
ject (MO) or group of ob
jects that con
tains its ad
min
is
tra
tive state and its op
er
at
ional state.
The APIC REST API is a pro
gram
matic in
ter
face to the APIC that uses a REST ar
chi
tec
ture. The API ac
cepts and re
turns HTTP or HTTPS mes
sages that con
tain JSON or
XML doc
um
ents. You can use any pro
gram
ming lan
guage to gen
er
ate the mes
sages,
and the JSON or XML doc
um
ents that con
tain the API meth
ods or man
aged ob
ject (MO)
de
scrip
tions.
You can in
voke an API func
tion by send
ing an HTTP/1.1 or HTTPS POST, GET, or
DELETE mes
sage to the APIC. The HTML body of the POST mes
sage con
tains a JSON or
XML data struc
ture that de
scribes an MO or an API method. The HTML body of the re
sponse mes
sage con
tains a JSON or XML struc
ture that con
tains re
quested data, con
fir
ma
tion of a re
quested ac
tion, or error in
for
ma
tion.
The fol
low
ing sec
tion is a rep
re
sen
ta
tion of use
ful classes for es
tab
lish
ing a foun
da
tion
for mon
it
or
ing and man
age
ment of the fab
ric. The list below is a sub
set of the full list of
the avail
able classes.
To ac
cess the com
plete list of classes, point to the APIC and ref
er
ence
the doc/html di
rec
tory at the end of the URL:
https://apic_ip_address/doc/html/
416 Appendix
Fabric Monitoring
topSystem
Name:
top:Sys
tem
De
scrip
tion:
Pro
vides a list of all the de
vices within the fab
ric, in
clud
ing con
trollers,
leafs and spines.
Usage:
The top
Sys
tem class can be used to de
rive ob
ject prop
er
ties in
clud
ing
inb/oob man
age
ment de
tails, cur
rent time, sys
tem up
time and cur
rent state.
topSystem REST :: https://172.16.96.2/api/node/class/topSystem.json
fabricNode
Name:
fab
ric:Node
De
scrip
tion:
Pro
vides a list of all the nodes that are part of the fab
ric, in
clud
ing
con
trollers, leafs and spines.
Usage:
The fab
ric
N
ode class can be used to de
rive ob
ject prop
er
ties in
clud
ing
node se
ri
al num
bers, as
signed node ids, node model num
bers and de
vice roles.
fabricNode REST :: https://172.16.96.2/api/node/class/fabricNode.json
faultInst
Name:
fault:Inst
De
scrip
tion:
Con
tains de
tailed in
for
ma
tion of the fault. This ob
ject is at
tached as a
child of the ob
ject on which the fault con
di
tion oc
curred. One in
stance ob
ject is cre
ated for each fault con
di
tion of the par
ent ob
ject. A fault in
stance ob
ject is iden
ti
fied by
a fault code.
Usage:
The fault
Inst class can be used to de
rive all faults as
so
ci
ated with the
fab
ric, ten
ant or in
di
vid
ual man
aged ob
jects within the APIC.
Appendix 417
faultInst REST :: https://172.16.96.2/api/node/class/faultInst.json
fabricHealthTotal
Name:
De
scrip
tion:
Usage:
health.
fab
ric:Health
To
tal
The fab
ric total health score in
stance.
The fab
ricHealth
To
tal class can be used to de
rive the over
all sys
tem
fabricHealthTotal REST :: https://172.16.96.2/api/node/class/fabricHealthTotal.json
fvCEp
Name:
De
scrip
tion:
fv:CEp
A client end
point at
tach
ing to the net
work.
Usage:
The fvCEp class can be used to de
rive a list of end points at
tached to the
fab
ric and the as
so
ci
ated ip/mac ad
dress and en
cap
su
la
tion for each ob
ject.
fvCEp REST :: https://172.16.96.2/api/node/class/fvCEp.json
fvRsCEpToPathEp
Name:
De
scrip
tion:
fv:RsCEp
ToPathEp
This is an in
ter
nal ob
ject that pro
vides a re
la
tion to a path end
point.
Usage:
The fvRsCEp
ToPathEp class can be used to de
rive path fab
ric de
tails
such as the node and port as well as the ten
ant de
tails such as the ten
ant name, ap
pli
ca
tion pro
file and end point group.
fvRsCEpToPathEp REST :: https://172.16.96.2/api/node/class/fvRsCEpToPathEp.json
eqptFabP
418 Appendix
eqptFabP
Name:
De
scrip
tion:
eqpt:FabP
Fab
ric port, the fab
ric fac
ing ex
ter
nal IO port.
Usage:
The eqpt
FabP class can be used to de
rive a list of fab
ric port and the as
so
ci
ated de
tails such as the line card and chas
sis place
ment.
eqptFabP REST :: https://172.16.96.2/api/node/class/eqptFabP.json
eqptLeafP
Name:
De
scrip
tion:
eqpt:LeafP
Fab
ric port, the non-fab
ric fac
ing ex
ter
nal leaf IO port.
Usage:
The eqpt
FabP class can be used to de
rive a list of non-fab
ric port and
the as
so
ci
ated de
tails such as the line card and chas
sis place
ment.
eqptLeafP REST :: https://172.16.96.2/api/node/class/eqptLeafP.json
eqptCh
Name:
De
scrip
tion:
eqpt:ChA
The hard
ware chas
sis con
tainer.
Usage:
The eqptCh class can be used to de
rive a chas
sis list and the as
so
ci
ated
de
tails such as the op
er
at
ional state, se
ri
al num
ber and model num
ber.
eqptCh REST :: https://172.16.96.2/api/node/class/eqptCh.json
eqptLC
Name:
eqpt:LCA
Appendix 419
De
scrip
tion:
Usage:
The line card (IO card), con

tain
ing IO ports.
The eqptLC class can be used to de
rive a list of line cards de
ployed
within the fab

ric and the as
so
ci
ated de
tails such as the re
dun
dancy state, model, se
ri
al
num
bers and the num
ber of ports.
eqptLC REST :: https://172.16.96.2/api/node/class/eqptLC.json
eqptFt
Name:
De
scrip
tion:
eqpt:Ft
The in
ven
to
ried fan tray.
Usage:
The eqptFt class can be used to de
rive a list of fan trays and the as
so
ci
ated
de
tails such as the op
er
a
tional sta
tus, model num
ber, se
r
ial num
ber and hard
ware ver
sion.
eqptFt REST :: https://172.16.96.2/api/node/class/eqptFt.json
eqptPsu
Name:
De
scrip
tion:
eqpt:Psu
The power sup
ply unit.
Usage:
rive a list of power sup
plies within the
fab
ric and the as
so
ci
ated de
tails such as the model num
ber, se
ri
al num
ber, op
er
at
ional
sta
tus, and the volt
age source.
eqptPsu REST :: https://172.16.96.2/api/node/class/eqptPsu.json
eqptSupC
Name:
De
scrip
tion:
eqpt:SupC
The su
per
vi
sor card, which con
tains the CPU run
ning con
trol plane.
420 Appendix
Usage:
rive a list of su
per
vi
sor cards de
ployed within the fab
ric and the as
so
ci
ated de
tails such as the model num
ber, se
ri
al
num
ber, op
er
at
ional sta
tus and re
dun
dancy state.
eqptSupC REST :: https://172.16.96.2/api/node/class/eqptSupC.json
ethpmPhysIf
Name:
De
scrip
tion:
ethpm:PhysIf
The phys
ic
al in
ter
face in
for
ma
tion holder.
Usage:
The eth
pm
PhysIf class can be used to de
rive a list of phys
ic
al in
ter
faces
in the fab
ric and the as
so
ci
ated de
tails such as a the speed, du
plex, op
er
at
ional sta
tus,
and usage state.
ethpmPhysIf REST :: https://172.16.96.2/api/node/class/ethpmPhysIf.json
dbgAcTrail
Name:
De
scrip
tion:
dbg:Ac
Trail
The atomic counter trail.
Usage:
The db
gAc
Trail class can be used to de
rive a list of the atomic coun
ters
de
ric and the as
so
ci
ated de
tails such as dropped packet sta
tis
tics
and packet counts.
dbgAcTrail REST :: https://172.16.96.2/api/node/class/dbgAcTrail.json
dbgEpgToEpgRslt
Name:
De
scrip
tion:
entry.
dbg:Epg
ToEp
gRslt
The end
point group to end
point group atomic counter, on-de
mand,
Appendix 421
Usage:
The dbgEpg
ToEp
gR
sIt class can be used to de
rive a list of the EPG to
EPG atomic coun

ters de
ric, and the as
so
ci
ated de
tails such as
dropped packet sta
tis
tics and packet counts.
dbgEpgToEpgRsIt REST :: https://172.16.96.2/api/node/class/dbgEpgToEpgRslt.json
dbgEpToEpRslt
Name:
De
scrip
tion:
Usage:
dbg:Ep
ToEpRslt
The end
point to end
point atomic counter, On-de
mand, Entry.
The dbgEp
ToEpT
sIt class can be used to de
rive a list of the end
point to
end
point atomic coun
ters de
ric and the as
so
ci
ated de
tails such as
dropped packet sta
tis
tics and packet counts.
dbgEpToEpTsIt REST :: https://172.16.96.2/api/node/class/dbgEpToEpRslt.json
VMM Monitoring
compVm
Name:
De
scrip
tion:
comp:Vm
The Vir
tual ma
chine ob
ject.
Usage:
The com
pVm class can be used to de
rive a list of vir
tual ma
chines de
ric and the as
so
ci
ated de
tails such as the name and state.
compVm REST :: https://172.16.96.2/api/node/class/compVm.json
compHv
Name:
De
scrip
tion:
comp:Hv
An ob
ject rep
re
sent
ing the com
pute hy
per
vi
sor.
422 Appendix
Usage:
The com
pVm class can be used to de
rive a list of com
pute hy
per
vi
sor
de
ric and the as
so
ci
ated de
tails such as the name and sta
tus.
compHv REST :: https://172.16.96.2/api/node/class/compHv.json
fvRsVm
Name:
De
scrip
tion:
ter
nal ob
ject.
fv:RsVm
A re
la
tion to a vir
tual ma
chine con
nected to a hy
per
vi
sor. This is an in
-
Usage:
The fvRsVm class can be used to de
rive the re
la
tion
ship of the vir
tual
ma
chines con
nected to the hy
per
vi
sor.
vRsVm REST :: https://172.16.96.2/api/node/class/fvRsVm.json
fvRsHyper
Name:
De
scrip
tion:
fv:RsHy
per
A re
la
tion to the hy
per
vi
sor that con
trols and mon
it
ors the APIC VMs.
This is an in
ter
nal ob
ject.
Usage:
The fvR
sHy
per class can be used to de
rive the re
la
tion
ship of the hy
per
vi
sor that con
trols and mon
it
ors the APIC VMs.
fvRsHyper REST :: https://172.16.96.2/api/node/class/fvRsHyper.json
vmmCtrlrP
Name:
vmm:CtrlrP
De
scrip
tion:
The VMM con
troller pro
file, which spec
if
ies how to con
nect to a sin
gle
VM man
age
ment con
troller that is part of con
tain
ing pol
icy en
force
ment do
main. For
ex
am
ple, the VMM con
troller pro
file could be a pol
icy to con
nect a VMware vCen
ter
that is part a VMM do
main.
Appendix 423
Usage:
The vmm
C
trlrP class can be used to de
rive the ip ad
dress and the dat
a-
cen
ter name of the con
nected VM do
main.
vmmCtrlrP REST :: https://172.16.96.2/api/node/class/vmmCtrlrP.json
Layer 4 to Layer 7 Monitoring

vnsAbsGraph
Name:
vns
Ab
s
Graph
De
scrip
tion:
The ab
stract graph is made up of ab
stract nodes and used to de
fine
the traf
fic flow through a ser
vice func
tion such as load bal
anc
ing, SSL of
fload, or fire
wall. Ab
stract nodes are com
prised of ser
vice nodes such as a ser
vice node bal
ancer
(SLB) or fire
wall (FW), ab
stract term nodes (the nodes that are con
nected to end
point
groups), and con
nec
tions.
Usage:
The class vns

Ab
s
Graph can be used to de
rive a list of ser
vice graph tem
-
plates con
fig
ured on the APIC, along with their prop
er
ties.
vnsAbsGraph
REST :: https://172.16.96.2/api/node/class/vnsAbsGraph.json
vnsLDevVip
Name:
vnsLDevVip
De
scrip
tion:
An L4-L7 de
vice clus
ter, which is rep
re
sented by a sin
gle vir
tual IP
(VIP). The con
fig
ur
a
tion is pushed down to the VIP ad
dress.
Usage:
The class vnsLDevVip can be used to de
rive all the VIPs con
fig
ured for
the log
ic
al de
vice clus
ters in the fab
ric
vnsLDevVip
REST :: https://172.16.96.2/api/node/class/vnsLDevVip.json
424 Appendix
vnsCDev
Name:
vn
sCDev
De
scrip
tion:
The in
di
vid
ual ser
vice de
vice, which is used to de
fine a con
crete l4-l7
ser
vice de
vice.
Usage:
The class vn
sCDev can be used to de
rive a list of con
crete de
vices con
fig
ured as part of the L4-7 ser
vice in
te
gra
tion
vnsCDev
REST :: https://172.16.96.2/api/node/class/vnsCDev.json
vnsLif
Name:
vnsLif
De
scrip
tion:
The log
ic
al in
ter
face, which is as
so
ci
ated with a set of con
crete in
ter
-
faces from the L4-L7 de

vice clus
ter.
Usage:
The class vnsLif can be used to de
rive the con
nec
tion be
tween a ser
vice
graph and de
vice in
ter
faces.
vnsLif
REST :: https://172.16.96.2/api/node/class/vnsLIf.json
vnsLDevCtx
Name:
vnsLDe
vCtx
De
scrip
tion:
A de
vice clus
ter con
text, which points to the de
vice clus
ter used to
pick a spe
cific de
vice based on con
tract, sub
ject, and func
tion label or names. To spec
ify a wild card, set the name to Any.
Usage:
The class vnsLDe

vCtx can be used to de
rive the node and con
tract
name.
nsLDevCtx
REST :: https://172.16.96.2/api/node/class/vnsLDevCtx.json
vnsRsLDevCtxToLDev
Appendix 425
vnsRsLDevCtxToLDev
Name:
De
scrip
tion:
vn
sRsLDe
vC
tx
ToLDev
A source re
la
tion to the ab
strac
tion of a ser
vice de
vice clus
ter or of a
proxy ob
ject for a log
ic
al de
vice clus
ter in the ten
ant.
Usage:
The class vn
sRsLDe
vC
tx
ToLDev can be used to de
rive the re
la
tion
ship
be
tween vnsLDe
vCtx and vnsLDev.
vnsRsLDevCtxToLDev
REST :: https://172.16.96.2/api/node/class/vnsRsLDevCtxToLDev.json
Statistics
compHostStats1h
Name:
comp:Host
Stat
s1h
De
scrip
tion:
A class that rep
re
sents the most cur
rent sta
tis
tics for host in a 1 hour
sam
pling in
ter
val. This class up
dates every 15 min
utes.
Usage:
The com
pHost
Stat
s1h class can be used to de
rive the sta
tis
tics as
so
ci
ated with the com
pute hy
per
vi
sor.
compHostStats1h REST :: https://172.16.96.2/api/node/class/compHostStats1h.json
compRcvdErrPkts1h
Name:
comp:RcvdEr
rP
kt
s
1h
De
scrip
tion:
A class that rep
re
sents the most cur
rent sta
tis
tics for re
ceived error
pack
ets in a 1 hour sam
pling in
ter
val. This class up
dates every 15 min
utes.
Usage:
The com
pRcvdEr
rP
kt
s
1h class can be used to de
rive the most cur
rent
sta
tis
tics for re
ceived error pack
ets.
compRcvdErrPkts1h REST :: https://172.16.96.2/api/node/class/compRcvdErrPkts1h.json
426 Appendix
compTrnsmtdErrPkts1h
Name:
De
scrip
tion:
comp:Trnsmt
dEr
rP
kt
s
1h
A class that rep
re
sents the most cur
rent sta
tis
tics for trans
mit
ted
error pack
ets in a 1 hour sam
pling in
ter
val. This class up
dates every 15 min
utes.
Usage:
The compTrnsmt
dEr
rP
kt
s
1h class can be used to de
rive the most cur
rent sta
tis
tics for trans
mit
ted error pack
ets.
compTrnsmtdErrPkts1h REST ::
https://172.16.96.2/api/node/class/compTrnsmtdErrPkts1h.json
Authentication, Authorization, and Accounting

aaaModLR
Name:
aaa:ModLR
De
scrip
tion:
The AAA audit log record. A log record is au
to
mat
ic
ally gen
er
ated
when
ever a user mod
if
ies an ob
ject.
Usage:
The aaaModLR class can be used to de
rive a fab
ric based audit log for all
changes and events.
aaaModLR REST :: https://172.16.96.2/api/node/class/aaaModLR.json
aaaUser
Name:
De
scrip
tion:
aaa:User
A lo
cally-au
then
ti
cated user ac
count.
Usage:
The aaaUser class can be used to de
rive a list of user ac
counts de
ployed
within the fab
ric.
aaaUser REST :: https://172.16.96.2/api/node/class/aaaUser.json
Appendix 427
aaaRemoteUser
Name:
aaa:Re
mo
teUser
De
scrip
tion:
A re
mote user login ac
count.
Usage:
The aaaUser class can be used to de

rive a list of re
mote user ac
counts
de
ric.
aaaRemoteUser REST :: https://172.16.96.2/api/node/class/aaaRemoteUser.json
Fabric Capacity
Policy TCAM
Name:
eqpt
ca
pac
it
y
Po
lEn
try5min
De
scrip
tion:
Pol
icy CAM entry sta
tis
tics. A class that rep
re
sents the most cur
rent
sta
tis
tics for pol
icy entry in a 5 minute sam
pling in
ter
val. This class up
dates every 10
sec
onds.
Usage:
The eqpt
ca
pac
it
y
Po
lEn
try5min class can be used to de
rive the cur
rent
value as
so
ci
ated with the Pol
icy TCAM usage.
eqptcapacityPolEntry5min REST ::
http://172.16.96.2/api/class/eqptcapacityPolEntry5min.json
Prefix TCAM
Name:
eqpt
ca
pac
ityL3En
try5min
De
scrip
tion:
Lay
er3 entry sta
tis
tics. A class that rep
re
sents the most cur
rent sta
tis
tics for lay
er3 entry in a 5 minute sam
pling in
ter
val. This class up
dates every 10 sec
onds.
Usage:
The eqpt
ca
pac
ityL3En
try5min class can be used to de
rive the cur
rent
value as
so
ci
ated with the Pre
fix TCAM usage.
428 Appendix
eqptcapacityL3Entry5min
REST ::
https://172.16.96.2/api/class/eqptcapacityL3Entry5min.json
SNMP/SYSLOG
SNMP Trap Destination
Name:
sn
mpTrapDest
A des
ti
na
tion to which traps and in
forms are sent.
De
scrip
tion:
Usage:
The sn
mpTrapDest class can be used to de
rive the cur
rent list of snmp
trap des
ti
na
tions im
ple
mented within the fab
ric.
snmpTrapDest REST :: https://172.16.96.2/api/node/class/snmpTrapDest.json
Prefix TCAM
Name:
sys
lo
gRe
mot
eDest
De
scrip
tion:
The sys
log re
mote des
ti
na
tion host en
ables you to spec
ify sys
log
servers to which mes
sages from the APIC and fab
ric nodes should be for
warded.
Usage:
The sys
lo
gRe
mot
eDest class can be used to de
rive the cur
rent list of
sys
log re
mote des
ti
na
tions im
ple
mented within the fab
ric.
syslogRemoteDest REST :: https://172.16.96.2/api/node/class/syslogRemoteDest.json
Use Cases
The class fault
Inst used in Use Case #1 and Use Case #2 below can be re
placed with
any of the man
aged ob
ject classses dis
cussed above or spec
if
ied within the APIC doc
u-
men
ta
tion. The Cisco APIC Com
mand-Line In
ter
face User Guide may also be help
ful for
un
der
stand
ing the fol
low
ing sec
tions - see: http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
cli/
b_
APIC_
CLI_
User_
Guide.
html
Appendix 429
Case 1: Creating an application script to retrieve the current list of

faults in the fabric.
ic
al for en
vi
ron
min
is
tra
tor wishes to
ob
tain the list of cur
rent faults in the fab
ric. The user has the op
tion of col
lect
ing the
re
sults via CLI, Vi
sore, POST
MAN and/or Cobra. Please refer to the sec
tion above for
ap
pli
ca
tion spe
cific ac
cess and ex
plan
tions.
From a CLI per
spec
tive, use the fol
low
ing com
mand to per
form the query:
admin@apic1:~> moquery -c faultInst
From a Vi
sore per
spec
tive, use the fol
low
ing pa
ra
me
ters to per
form the query.:
Class or DN
:: faultInst
Property
:: n/a
Op
:: n/a
Value
:: n/a
From a POST
MAN per
spec
tive, user the fol
low
ing REST GET to per
form the query:
GET http://<your apic ip address>/api/node/class/faultInst.xml
From a Cobra per

spec
tive, use the fol
low
ing class query:
# Class Query
classQuery= ClassQuery('faultInst')
for fault in md.query(classQuery):
print fault.name
Sam
ple Cobra script to cap
ture faults within the fab
ric:
from cobra.mit.session import LoginSession
430 Appendix
from cobra.mit.request import ClassQuery
ls = cobra.mit.session.LoginSession('https://'<your apic ip address>, <username>,
<password>, secure=False)
md.login()
# Class Query
print fault.name

faults in the fabric that have been caused by a failed configuration.
ic
al for en
vi
ron
min
is
tra
tor wishes to
cob
tain the list of cur
rent faults in the fab
ric. The user has an op
tion of col
lect
ing the
re
sults via CLI, Vi
sore, POST
tion above for
ap
pli
ca
tion spe
cific ac
cess and ex
plan
tions.
From a CLI per
spec
tive, use the fol
low
ing com
mand to per
form the query:
admin@apic1:~> moquery -c faultInst -f 'fv.faultInst.cause=="config-failure"'
From a Vi
sore per
spec
tive, use the fol
low
ing pa
ra
me
ters to per
form the query:
Class or DN
:: faultInst
Property
:: cause
Op
:: ==
Value
:: config-failure
From a POST
MAN per
spec
tive, use the fol
low
ing REST GET to per
form the query:
GET http://<your apic ip address>/api/node/class/faultInst.xml?query-targetfilter=and(eq(faultInst.cause,"config-failure"))
Appendix 431
From a Cobra per

spec
tive, use the fol
low
ing class query:
# Class Query
classQuery.propFilter = 'wcard(faultInst.cause, "{0}")'.format('config-failure')
print fault.name
Cobra Script to cap

ture faults ca
sued by con
fig
ur
a
tion fail
ures.
md.login()
# Class Query
print fault.name
Case 3: Creating an application script to retrieve the properties for a

specific managed object, DN
ic
al for en
vi
ron
min
is
tra
tor wishes to
ob
tain the prop
er
ties of the ten
ant name Com
mon. The user has an op
tion of col
lect
ing
the re
sults via CLI, Vi
sore, POST
tion above
for ap
pli
ca
tion spe
cific ac
cess and ex
plan
tions.
From a CLI per
spec
tive, use the fol
low
ing com
mand to per
form the query:
admin@apic1:~> moquery -d uni/tn-common
432 Appendix
From a Vi
sore per
spec
tive, use the fol
low
ing pa
ra
me
ters to per
form the query:
Class or DN
:: uni/tn-common
Property
:: n/a
Op
:: n/a
Value
:: n/a
From a POST
MAN per
spec
tive, use the fol
low
ing REST GET to per
form the query:
GET http://<your apic ip address>/api/node/mo/uni/tn-common.xml?query-target=self
From a Cobra per

spec
tive, use the fol
low
ing class query:
# DN Query
dnQuery= DnQuery('uni/tn-common')
for results in md.query(dnQuery):
print results.dn
Cobra Script to cap

ture faults ca
sued by con
fig
ur
a
tion fail
ures.
from cobra.mit.request import DnQuery
md.login()
# DN Query
dnQuery= DnQuery('uni/tn-common')
for results in md.query(dnQuery):
print results.dn
Appendix 433

endpoints (mac-addresses) attached to the fabric
ic
al for en
vi
ron
min
is
tra
tor wishes to
cre
ate an ap
pli
ca
tion script to cap
ture the list of cur
rent end
points at
tached to the fab
ric along with the node de
tails per
tain
ing to each end
point.
Cobra Script to cap
ture faults ca
sued by con
fig
ur
a
tion fail
ures.
from cobra.mit.access import MoDirectory
lls = cobra.mit.session.LoginSession('https://'<your apic ip address>, <username>,
md = MoDirectory(ls)
md.login()
q = ClassQuery('fvCEp')
q.subtree = 'children'
q.subtreeClassFilter = 'fvRsCEpToPathEp'
mos = md.query(q)
for mo in mos:
for child in mo.rscEpToPathEp:
print child.dn
Appendix 435
Package Decoder
There are sev
eral ab
bre
vi
at
ions used in the names of classes in the ACI ob
ject model.
Here are some de
scrip
tions of com
monly used ab
bre
vi
at
ions, which may help when de
ci
pher
ing what class ob
jects are when using them with REST calls.
Package Decoder
Aaa: au
then
ti
ca
tion, au
tho
riza
tion, ac
count
ing
ac: atomic coun
ters
actrl: ac
cess con
trol
ac
trl
cap: ac
cess con
trol ca
pa
bil
ity
adcom: ap
pli
ance di
rec
tor com
mu
ni
ca
tion
aib: ad
ja
cency in
for
ma
tion base
arp: ad
dress res
o
lu
tion pro
to
col
bgp: bor
der gate
way pro
to
col
call
home: Cisco smart call home ser
vices
cap: ca
pa
bil
ity
cdp: Cisco dis
cov
ery pro
to
col
cnw: node clus
ter
comm: com
mu
ni
ca
tion pol
icy
comp: com
pute
436 Appendix
com
pat: com
pat
ib
il
ity
con
di
tion: health pol
icy
con
fig: con
fig
ur
a
tion pol
icy
coop: Coun
cil of Or
ac
les pro
to
col
copp: con
trol plane polic
ing pol
icy: con
tains set of rules de
scrib
ing po
licer rates
ctrlr: con
troller
ctx: con
text
date
time: date/time pol
icy
dbg: debug
dbgac: debug atomic coun
ters
dbg
exp: debug ex
port pol
icy
dhcp: dy
namic host con
fig
ur
a
tion pro
to
col
dhcptlv: dy
namic host con
fig
ur
a
tion pro
to
col type length value
dhcptlvpol: dy
namic host con
fig
ur
a
tion pro
to
col type length value pol
icy
dns: do
main name ser
vice
draw: graph vi
su
al
iza
tion for GUI
epm: end
point man
ager
eqpt: equip
ment
eqpt
cap: equip
ment ca
pa
bil
ity
eqpt
ca
pac
ity: equip
ment ca
pac
ity
Appendix 437
eqpt
diag: equip
ment di
ag
nos
tics
eqpt
di
agp: equip
ment di
ag
nos
tics pol
icy
ethpm: eth
er
net pol
icy man
ager
event: event pol
icy
extnw: ex
ter
nal net
work
fab
ric: fab
ric
fault: fault pol
icy, coun
ters
file: file path, con
fig im
port/ex
port pol
icy
firmware: firmware
fm
cast: fab
ric mul
ti
cast
fsm: fi
nite state ma
chine
fv: fab
ric vir
tu
al
iza
tion
fvns: fab
ric vir
tu
al
iza
tion name
space
fv
topo: fab
ric vir
tu
al
iza
tion topol
ogy
geo: ge
olo
ca
tion
glean: glean ad
ja
cency
ha: high avail
abil
ity
health: health score
hvs: hy
per
vi
sors vir
tual switch
icmp: in
ter
net con
trol pro
to
col
438 Appendix
icm
pv4: in
ter
net con
trol pro
to
col ver
sion 4
icm
pv6: in
ter
net con
trol pro
to
col ver
sion 6
ident: iden
tity
igmp: in
ter
net group man
age
ment pro
to
col
igmp
snoop: in
ter
net group man
age
ment pro
to
col snoop
ing
im: in
ter
face man
ager mod
ule
im
gin
stall: image in
stall
infra: in
fra
struc
ture
ip: in
ter
net pro
to
col
ipv4: in
ter
net pro
to
col ver
sion 4
ipv6: in
ter
net pro
to
col ver
sion 6
isis: in
ter
me
di
ate sys
tem to in
ter
me
di
ate sys
tem
isistlv: in
ter
me
di
ate sys
tem to in
ter
me
di
ate sys
tem type length value
l1: layer 1
l1cap: layer 1 ca
pa
bil
ity
l2: layer 2
l2cap: layer 2 ca
pa
bil
ity
l2ext: layer 2 ex
ter
nal
l3: layer 3
l3cap: layer 3 ca
pa
bil
ity
Appendix 439
l3ext: layer 3 ex
ter
nal
l3vm: Layer 3 Vir
tual Ma
chine
lacp: link ag
gre
ga
tion pro
to
col
lbp: load bal
anc
ing pol
icy
leqpt: loose equip
ment (un
man
aged nodes, not in the fab
ric)
lldp: link layer dis
cov
ery pro
to
col
lldptlv: link layer dis
cov
ery pro
to
col type length value
lldptlvpol: link layer dis
cov
ery pro
to
col type length value pol
icy
maint: main
te
nance
mcast: mul
ti
cast
mcp: mas
ter con
trol proces
sor
mem
ory: mem
ory sta
tis
tics
mgmt: man
age
ment
mo: man
aged ob
ject
mock: mock (ob
jects used on the sim
ul
a
tor mostly for show
ing stats/faults/etc)
mon: mon
it
or
ing
mon
i
tor: mon
it
or (SPAN)
nam
ing: ab
stract for ob
jects with names
nd: neigh
bor dis
cov
ery
nw: net
work
440 Appendix
oam: eth
er
net op
er
at
ions, ad
min
is
tra
tions and man
age
ment
ob
server: ob
server for sta
tis
tics, fault, state, health, logs/his
tory
opflex: OpFlex
os: op
er
at
ing sys
tem
ospf: open short
est path first
pc: port chan
nel
pcons: **gen
er
ated and used by in
ter
nal processes**
phys: phys
ic
al do
main pro
file
ping: ping ex
e
cu
tion and re
sults
pki: pub
lic key in
fra
struc
ture
pol: pol
icy de
f
in
i
t
ion
po
licer: traf
fic polic
ing (rate lim
it
ing)
pool: ob
ject pool
pres: **gen
er
ated and used by in
ter
nal processes**
proc: sys
tem load, cpu, and mem
ory uti
liza
tion sta
tis
tics
psu: power sup
ply unit pol
icy
qos: qual
ity of ser
vice pol
icy
qosm: qos sta
tis
tics
qosp: qos/ 802.1p
rbqm: de
bug
ging
Appendix 441
regress: re
gres
sion
reln: **gen
er
ated and used by in
ter
nal processes**
repl: **gen
er
ated and used by in
ter
nal processes**
res: **gen
er
ated and used by in
ter
nal processes**
rib: rout
ing in
for
ma
tion base
rmon: re
mote net
work mon
it
or
ing/ in
ter
face stats/coun
ters
rpm: route pol
icy map
rtcom: route con
trol com
mu
nity list
rtc
trl: route con
trol
rtextcom: router ex
tended com
mu
nity
rtflt: route fil
ter
rtleak: route leak
rtmap: RPM route map
rtpfx: route pre
fix list
rtreg
com: route reg
ul
ar com
mu
nity list
rtsum: route sum
ma
riza
tion ad
dress/pol
icy
satm: satel
lite man
ager
snmp: sim
ple net
work man
age
ment pro
to
col
span: switched port an
al
yzer
stats: sta
tis
tics col
lec
tion poli
cies
442 Appendix
stat
store: sta
tis
tics data hold
ers
storm
c
trl: storm con
trol (traf
fic sup
pres
sion) pol
icy
stp: span
ning tree pro
to
col de
f
in
i
t
ions and pol
icy
sts: Ser
vice Tag Switch
ing (used for ser
vices in
ser
tion)
svc
core: core pol
icy
svi: switched vir
tual in
ter
face/ routed VLAN in
ter
face
syn
thetic: syn
thetic ob
jects (for test
ing)
sys
de
bug: sys
tem debug
sys
file: sys
tem files
syshist: sys
tem cards reset records/his
tory
sys
log: sys
log pol
icy
sys
mgr: sys
tem man
ager (firmware, su
per
vi
sor, sys
tem states, etc)
sys
m
grp: con
tainer for cores pol
icy & ab
stract class for all qos pol
icy de
f
in
i
t
ions
tag: alias (use de
scrip
tive name for dn), tags (group mul
ti
ple ob
jects by a de
scrip
tive
name)
task: task ex
e
cu
tion, in
stance, and re
sult
test: ab
stract class for test rule, sub
ject, and re
sult
testin
fralab: test in
fra
struc
ture
tlv: type, length, value sys
tem struc
tures
top: sys
tem task man
ager for proces
sor ac
tiv
ity
Appendix 443
topoc
trl: topol
ogy con
trol pol
icy (shard
ing, fab
ric LB, fab
ric VxLan, etc)
tracer
oute: tracer
oute ex
e
cu
tion and re
sults
tracer
outep: tracer
oute end points
trig: trig
ger
ing pol
icy
tun
nel: tun
nel
ing
uribv4: ipv4 uni
cast rout
ing in
for
ma
tion base en
tity
vlan: vlan in
stances
vlan
mgr: vlan man
ager con
trol plane
vmm: vir
tual macine man
ager (con
troller, vmm pol
icy and de
f
in
i
t
ions)
vns: vir
tual net
work ser
vice (L4-L7 pol
icy and de
f
in
i
t
ions)
vpc: vir
tual port chan
nel (vpc pol
icy and de
f
in
i
t
ions)
vsvc: ser
vice la
bels (provider/con
sumer)
vtap: trans
lated ad
dress of ex
ter
nal node (NATed IP of ser
vice node)
vxlan: Vir
tu
ally ex
ten
si
ble LAN de
f
in
i
t
ions
vz: vir
tual zones (for
mer name of the pol
icy con
trols) i.e. Con
tracts
Model Naming schemes

Rs: Re
la
tion
ship source
Rt: Re
la
tion
ship tar
get
Ag: Ag
gre
gated stats
BrCP: Bi
nary Con
tract Pro
file
Appendix 445
Acronyms and Definitions

Overview
This sec
tion is de
signed to pro
vide a high level de
scrip
tion of terms and con
cepts that
get brought up in this book. While ACI does not change how pack
ets are trans
mit
ted on
a wire, there are some new terms and con
cepts em
ployed, and un
der
stand
ing those
new terms and con
cepts will help those work
ing on ACI com
mu
ni
cate with one an
other
about the con
structs used in ACI to trans
mit those bits. As
so
ci
ated new acronyms are
also pro
vided.
This is not meant to be an ex
haus
tive list nor a com
pletely de
tailed dic
tio
nary of all of
the terms and con
cepts, only the key ones that may not be a part of the com
mon ver
nac
ul
ar, or which would be rel
e
vant to the trou
bleshoot
ing ex
er
cises that were cov
ered
in the trou
bleshoot
ing sce
nar
ios dis
cussed.
A
AAA: acronym for Au
then
ti
ca
tion, Au
tho
riza
tion, and Ac
count
ing.
ACI Ex
ter
nal Con
nec
tiv
ity: Any con
nec
tiv
ity to and from the fab
ric that uses an ex
ter
nal routed or switched in
ter
me
di
ary sys
tem, where end
points fall out
side of the man
aged scope of the fab
ric.
ACID trans
ac
tions: ACID is an acronym for Atom
ic
ity, Con
sis
tency, Iso
la
tion, Dura
bil
ity
prop
er
ties of trans
ac
tions that en
sure con
sis
tency in data
base trans
ac
tions. Trans
ac
tions to APIC de
vices in an ACI clus
ter are con
sid
ered ACID, to en
sure that data
base
con
sis
tency is main
tained. This means that if one part of a trans
ac
tion fails the en
tire
trans
ac
tion fails.
AEP: At
tach
able En
tity Pro
file this is a con
fig
ur
a
tion pro
file of the in
ter
face that gets
ap
plied when an en
tity at
taches to the fab
ric. An AEP rep
re
sents a group of ex
ter
nal
en
ti
ties with sim
il
ar in
fra
struc
ture pol
icy re
quire
ments. AEPs are also the mech
an
ism
that ties the phys
ic
al port to the do
main (phys
ic
al or vir
tual) to a switch pol
icy.
ALE: Ap
pli
ca
tion Leaf En
gine, an ASIC on a leaf switch.
446 Appendix
APIC: Ap
pli
ca
tion Pol
icy In
fra
struc
ture Con
troller is a cen
tral
ized pol
icy man
age
ment
con
troller clus
ter. The APIC con
fig
ures the in
tended state of the pol
icy to the fab
ric.
API: Ap
pli
ca
tion Pro
gram
ming In
ter
face used for pro
gram
ma
ble ex
ten
si
bil
ity.
Ap
pli
ca
tion Pro
file: Term used to ref
er
ence an ap
pli
ca
tion pro
file-man
aged ob
ject ref
er
ence that mod
els the log
ic
al com
po
nents of an ap
pli
ca
tion and how those com
po
nents com
mu
ni
cate. The AP is the key ob
ject used to rep
re
sent an ap
pli
ca
tion and is
also the an
chor point for the au
to
mated in
fra
struc
ture man
age
ment in an ACI fab
ric.
ASE: Ap
pli
ca
tion Spine En
gine, an ASIC on a Spine switch.
B
BGP: Bor
der Gate
way Pro
to
col, on the ACI fab
ric Multi-Pro
to
col BGP is used to dis
trib
ute reach
ab
il
ity in
for
ma
tion within the fab
ric, and In
ter
nal BGP is used to peer the fab
ric with ex
ter
nal Layer 3 de
vices.
Bridge Do
main: An ACI con
struct that de
fines Layer 2 for
ward
ing be
hav
iors (Broad
cast,
ARP flood
ing, etc.) for each unique Layer 2 for
ward
ing do
main. Bridge Do
mains are also a
con
tainer for IP sub
nets and are where fab
ric Layer 3 gate
way func
tion
al
ity is con
fig
ured. BDs can em
u
late the be
hav
ior of a tra
di
tional VLAN but are not con
strained by for
ward
ing scale lim
i
ta
tions. In the ACI ob
ject model, a BD is a child of a Pri
vate Layer 3 or
con
text.
C
CLOS fab
ric: A multi-tier non
block
ing leaf-spine ar
chi
tec
ture net
work.
Clus
ter: Set of de
vices that work to
gether as a sin
gle sys
tem to pro
vide an iden
ti
cal or
sim
il
ar set of func
tions.
Con
tracts: A log
ic
al con
tainer for the sub
jects which re
late to the fil
ters that gov
ern
the rules for com
mu
ni
ca
tion be
tween end
point groups. ACI works on a white list pol
icy
model. With
out a con
tract, the de
fault for
ward
ing pol
icy is to not allow any com
mu
ni
ca
tion be
tween EPGs, but com
mu
ni
ca
tion within an EPG is al
lowed.
Appendix 447
Con
text: A Layer 3 for
ward
ing do
main, equiv
al
ent to a VRF, and in ACI ver
nac
ul
ar a Pri
vate Layer 3.
D
DLB: Dy
namic Load Bal
anc
ing a net
work traf
fic load-bal
anc
ing mech
an
ism in the ACI
fab
ric based on flowlet switch
ing.
DME: Data Man
age
ment En
gine, a ser
vice that runs on the APIC that man
ages data for
the data model.
dMIT: dis
trib
uted Man
age
ment In
for
ma
tion Tree, a rep
re
sen
ta
tion of the ACI ob
ject
model with the root of the tree at the top and the leaves of the tree at the bot
tom. The
tree con
tains all as
pects of the ob
ject model that rep
re
sent an ACI fab
ric.
Dn: Dis
tin
guished name a fully qual
if
ied name that rep
re
sents a spe
cific ob
ject within
the ACI man
age
ment in
for
ma
tion tree as well as the spe
cific lo
ca
tion in
for
ma
tion in the
tree. It is made up of a con
cate
na
tion of all of the rel
at
ive names from it
self back to the
root of the tree. As an ex
am
ple, if pol
icy ob
ject of type Ap
pli
ca
tion Pro
file is cre
ated,
named com
merce work
space within a Ten
ant named Prod, the dn would be ex
pressed
as uni/tn-Prod/ap-com
merce
work
space.
E
EP: End
point - Any log
ic
al or phys
ic
al de
vice con
nected di
rectly or in
di
rectly to a port
on a leaf switch that is not a fab
ric fac
ing port. End
points have spe
cific prop
er
ties like
an ad
dress, lo
ca
tion, or po
ten
tially some other at
tribute, which is used to iden
tify the
end
point. Ex
am
ples in
clude vir
tual-ma
chines, servers, stor
age de
vices, etc.
EPG: End Point Group. A col
lec
tion of end
points that can be grouped based on com
mon re
quire
ments for a com
mon pol
icy. End
point groups can be dy
namic or sta
tic.
F
Fault: When a fail
ure oc
curs or an alarm is raised, the sys
tem cre
ates a fault-man
aged
ob
ject for the fault. A fault con
tains the con
di
tions, in
for
ma
tion about the op
er
at
ional
state of the af
fected ob
ject and po
ten
tial res
ol
u
tions for the prob
lem.
448 Appendix
Fab
ric: The col
lec
tive end
points as
so
ci
ated with an ACI so
lu
tion (Leaf, Spine and Vir
tual
Switches plus APICs)
fines net
work man
age
ment tasks. FCAPS is n acronym for
FCAPS: The ISO model de
fault, con
fig
ur
a
tion, ac
count
ing, per
for
mance, se
cu
rity, the man
age
ment cat
e
gories
Fil
ters: Fil
ters de
fine the rules out
lin
ing the Layer 2 to layer 4 fields that will be
matched by a con
tract.
Flowlet switch
ing: An op
ti
mized, mul
ti
path, load-bal
anc
ing method
ol
ogy based on re
search from MIT in 2004. Flowlet Switch
ing is a way to use TCPs own bursty na
ture to
more ef
fi
ciently for
ward TCP flows by dy
nam
ic
ally split
ting flows into flowlets, and
split
ting traf
fic across mul
ti
ple par
al
lel paths with
out re
quir
ing packet re
order
ing.
G
GUI: Graph
ic
al User In
ter
face.
H
HTML: Hy
per
Text Markup Lan
guage, a markup lan
guage that fo
cuses on the for
mat
ting of web pages.
Hy
per
vi
sor: Soft
ware that ab
stracts the hard
ware on a host ma
chine and al
lows the
host ma
chine to run mul
ti
ple vir
tual ma
chines.
Hy
per
vi
sor in
te
gra
tion: Ex
ten
sion of ACI Fab
ric con
nec
tiv
ity to a vir
tual ma
chine man
ager to pro
vide the APIC with a mech
an
ism for vir
tual ma
chine vis
ib
il
ity and pol
icy en
force
ment.
I
IFM: In
tra-Fab
ric Mes
sages, Used for com
mu
ni
ca
tion be
tween dif
fer
ent de
vices on the
ACI fab
ric.
Appendix 449
In
band Man
age
ment (INB): In
band Man
age
ment. Con
nec
tiv
ity using an in
band man
age
ment con
fig
ur
a
tion. This uses a front panel (data plane) port of a leaf switch for ex
ter
nal man
age
ment con
nec
tiv
ity for the fab
ric and APICs.
IS-IS: Link local rout
ing pro
to
col lever
aged by the fab
ric for in
fra
struc
ture topol
ogy.
Loop
back and VTEP ad
dresses are in
ter
nally ad
ver
tised over IS-IS. IS-IS an
nounces the
cre
ation of tun
nels from leaf nodes to all other nodes in fab
ric.
J
JSON: JavaScript Ob
ject No
ta
tion, a data en
cap
su
la
tion for
mat that uses human read
able text to en
cap
su
late data ob
jects in at
tribute and value pairs.
L
Layer 2 Out (l2out): Layer 2 con
nec
tiv
ity to an ex
ter
nal net
work that ex
ists out
side of
the ACI fab
ric.
Layer 3 Out (l3out): Layer 3 con
nec
tiv
ity to an ex
ter
nal net
work that ex
ists out
side of
the ACI fab
ric.
L4-L7 Ser
vice In
ser
tion: The in
ser
tion and stitch
ing of VLANs/Layer 3 con
structs of
vir
tual or phys
ic
al ser
vice ap
pli
ances (Fire
wall, IDS/IPS, Load Bal
ancers, DLP,
etc....) into the flow of traf
fic. Ser
vice nodes op
er
ate be
tween Lay
ers 4 and Layer 7 of
the OSI model, where as net
work
ing el
e
ments (i.e. the fab
ric) op
er
ate at lay
ers 1-3).
La
bels: Used for clas
si
fy
ing which ob
jects can and can
not com
mu
ni
cate with each
other.
Leaf: Net
work node in fab
ric pro
vid
ing host and bor
der con
nec
tiv
ity. Leafs con
nect
only to hosts and spines. Leafs never con
nect to each other.
M
MO: Man
aged Ob
ject every con
fig
urable com
po
nent of the ACI pol
icy model man
aged in the MIT is called a MO.
450 Appendix
Model: A model is a con

cept which rep
re
sents en
ti
ties and the re
la
tion
ships that exist
be
tween them.
Multi-tier Ap
pli
ca
tion: Clientserver ar
chi
tec
ture in which pre
sen
ta
tion, ap
pli
ca
tion
logic, and data
base man
age
ment func
tions re
quire phys
ic
al or log
ic
al sep
ar
a
tion and
re
quire net
work
ing func
tions to com
mu
ni
cate with the other tiers for ap
pli
ca
tion func
tion
al
ity.
O
Ob
ject Model: A col
lec
tion of ob
jects and classes are used to ex
am
ine and ma
nip
ul
ate
the con
fig
ur
a
tion and run
ning state of the sys
tem that is ex
pos
ing that ob
ject model. In
ACI the ob
ject model is rep
re
sented as a tree known as the dis
trib
uted man
age
ment in
for
ma
tion tree (dMIT).
Out-of-Band man
age
ment (OOB man
age
ment): Ex
ter
nal con
nec
tiv
ity using a spe
cific
out-of-band man
age
ment in
ter
face on every switch and APIC.
P
Port Chan
nel: A Port link ag
gre
ga
tion tech
nol
ogy that binds mul
ti
ple phys
ic
al in
ter
faces into a sin
gle log
ic
al in
ter
face and pro
vides more ag
gre
gate band
width and link
fail
ure re
cov
ery with
out a topol
ogy change.
R
RBAC: Role Based Ac
cess Con
trol, which is a method of man
ag
ing se
cure ac
cess to in
fra
struc
ture by as
sign
ing roles to users, then using those roles in the process of grant
ing or deny
ing ac
cess to de
vices, ob
jects and priv
il
ege lev
els.
Rep
re
sen
ta
tional State Trans
fer (REST): a state
less pro
to
col usu
ally run over HTTP
that al
lows a client to ac
cess a server-side or cloud-based API with
out hav
ing to write a
local client for the host ac
cess
ing the API. The lo
ca
tion that the client ac
cesses usu
ally
de
fines the data the client is try
ing to ac
cess from the ser
vice. Data is usu
ally ac
cessed
and re
turned in ei
ther XML or JSON for
mat.
REST
ful: An API that uses REST, or Rep
re
sen
ta
tional State Trans
fer.
Appendix 451
Rn: Rel
at
ive name, a name of a spe
cific ob
ject within the ACI man
age
ment in
for
ma
tion
tree that is not fully qual
if
ied. A Rn is sig
nif
ic
ant to the in
di
vid
ual ob
ject, but with
out
con
text, its not very use
ful in nav
ig
a
tion. A Rn would need to be con
cate
nated with all
the rel
at
ive names from it
self back up to the root to make a dis
tin
guished name, which
then be
comes use
ful for nav
ig
a
tion. As an ex
am
ple, if an Ap
pli
ca
tion Pro
file ob
ject is
cre
ated named com
merce
work
space, the Rn would be ap-com
merce
work
space be
cause Ap
pli
ca
tion Pro
file rel
at
ive names are all pref
aced with the let
ters ap-. See also
the Dn de
f
in
i
t
ion.
S
Ser
vice graph: A mech
an
ism within ACI that au
to
mates redi
rec
tion of traf
fic and VLAN
stitch
ing based on de
fined pa
ra
me
ters. Any ser
vices that are re
quired are treated as a
ser
vice graph that is in
stan
ti
ated on the ACI fab
ric from the APIC. Ser
vice graphs iden
tify the set of net
work or ser
vice func
tions that are needed by the ap
pli
ca
tion, and rep
re
sent each func
tion as a node.
Spine: Net
work node in fab
ric car
ry
ing ag
gre
gate host traf
fic from leafs, con
nected
only to leafs in the fab
ric and no other de
vice types.
Spine/Leaf topol
ogy: A clos-based fab
ric topol
ogy in which spine nodes con
nect to
leaf nodes, leaf nodes con
nect to hosts and ex
ter
nal net
works.
Sub
nets: Con
tained by a bridge do
main or an EPG, a sub
net de
fines the IP ad
dress
range that can be used within the bridge do
main.
Sub
jects: Con
tained by con
tracts and cre
ate the re
la
tion
ship be
tween fil
ters and con
tracts.
Su
per
vi
sor: Switch mod
ule that pro
vides the con
trol plane for the 95xx switches.
T
Ten
ants: The log
ic
al con
tainer to group all poli
cies for ap
pli
ca
tion poli
cies.
452 Appendix
V
Vir
tu
al
iza
tion: Tech
nol
ogy used to ab
stract hard
ware re
sources into vir
tual rep
re
sen
ta
tions and al
low
ing soft
ware con
fig
ura
bil
ity.
vPC: vir
tual Port Chan
nel, in which a port chan
nel is cre
ated for link ag
gre
ga
tion, but is
spread across no more or less than 2 phys
ic
al switches.
VRF: Vir
tual Rout
ing and For
ward
ing - A Layer 3 name
space iso
la
tion method
ol
ogy to
allow for mul
ti
ple con
texts to be de
ployed on a sin
gle de
vice or in
fra
struc
ture.
VXLAN: VXLAN is a Layer 2 over
lay scheme trans
ported across a Layer 3 net
work. A 24bit VXLAN seg
ment ID (SID) or VXLAN net
work iden
ti
fier (VNID) is in
cluded in the en
cap
su
la
tion to pro
vide up to 16 mil
lion VXLAN seg
ments for traf
fic iso
la
tion or seg
men
ta
tion. Each seg
ment rep
re
sents a unique Layer 2 broad
cast do
main. An ACI VXLAN
header is used to iden
tify the pol
icy at
trib
utes if the ap
pli
ca
tion end
point within the
fab
ric, and every packet car
ries these pol
icy at
trib
utes.
X
XML: eX
ten
si
ble Markup Lan
guage, a markup lan
guage that fo
cuses on en
cod
ing data
for doc
um
ents rather than the for
mat
ting of the data for those doc
um
ents.
Appendix 453
Reference Material
Top
ics that are out
side of the scope of this op
er
at
ions guide may be doc
um
ented in
other places. This sec
tion in
cludes links to other help
ful ref
er
ence doc
um
en
ta
tion for
fur
ther read
ing and view
ing.
ACI Install and Upgrade Guides

http://
www.
cisco.
com/
c/
en/
us/
support/
applicationpolicy-infrastructure-controller-apic/
products-installation-guides-list.
html
ACI Get
ting Started - Fab
ric Ini
tial
iza
tion
http://
www.
cisco.
com/
c/
en/
us/
support/
products-installation-and-configurationguides-list.
html
ACI De
sign Guide
http://
www.
cisco.
com/
c/
en/
us/
solutions/
collateral/
html#_
Toc405844675
ACI Trou
bleshoot
ing Guides
http://
www.
cisco.
com/
c/
en/
us/
support/
products-troubleshooting-guides-list.
html
https://
datacenter.
github.
io/
ACI White Pa
pers
http://
www.
cisco.
com/
c/
en/
us/
solutions/
applicationcentric-infrastructure/
white-paper-listing.
html
454 Appendix
ACI Case Stud

ies
www.
cisco.
com/
c/
en/
us/
solutions/
customer-case-study-listing.
html
ACI Demos, Pre
sen
ta
tions and Train
ings
www.
cisco.
com/
c/
en/
us/
solutions/
sales-resources-list.
html
ACI Ecosys
tem Com
pata
bil
ity List
http://
www.
cisco.
com/
c/
en/
us/
solutions/
collateral/
solution-overview-c22-732445.
html
ACI Part
ners and Cus
tomers Pre
sen
ta
tions
http://
www.
cisco.
com/
c/
en/
us/
solutions/
presentations-listings.
html
ACI So
lu
tions Overview
http://
www.
cisco.
com/
c/
en/
us/
solutions/
solution-overview-listing.
html
ACI Toolkit
http://
datacenter.
github.
io/
acitoolkit/
ACI Com
pata
bil
ity Tool
http://
www.
cisco.
com/
web/
techdoc/
aci/
acimatrix/
matrix.
html
AVS Con
fig
u
ra
tion and Scal
a
bil
ity Guides
http://
www.
cisco.
com/
c/
en/
us/
support/
switches/
application-virtual-switch/
products-installation-and-configuration-guides-list.
html
Appendix 455
AVS Topolo
gies and So
lu
tion Guide
http://
www.
cisco.
com/
c/
en/
us/
support/
switches/
application-virtual-switch/
products-technical-reference-list.
html
APIC Com
mand-Line In
ter
face User Guide
http://
www.
cisco.
com/
c/
en/
us/
support/
products-command-reference-list.
html
APIC Layer 4 to Layer 7 Ser
vices De
ploy
ment Guide
http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
L4-L7_
Services_
Deployment/
guide/
b_
L4L7_
Deploy.
html
Cobra Docs
http://
cobra.
readthedocs.
org/
en/
latest/
Cobra GitHub
http://
github.
com/
datacenter/
cobra
Con
nect
ing ACI to Out
side Layer 2 and 3 Net
works
http://
www.
cisco.
com/
c/
en/
us/
solutions/
collateral/
html
Fab
ric Con
nec
tiv
ity Video
https://
www.
youtube.
com/
watch?
v=_
iQvoC9zQ_
A
Nexus CLI to Cisco APIC Map
ping
http://
www.
cisco.
com/
c/
en/
us/
support/
products-configuration-examples-list.
html
456 Appendix
POST
man
getpostman.
com
http://www.
Sup
ported SNMP MIB List
http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
mib/
list/
mib-support.
html

Cisco OperatingApplicationCentricInfrastructure

Uploaded by

Copyright:

Available Formats

Cisco OperatingApplicationCentricInfrastructure

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Cisco OperatingApplicationCentricInfrastructure

Uploaded by

Copyright:

Available Formats

Operating Cisco

Alejandra Sanchez, Andres Vega, Arvind Chari, Carly Stoughton,

Warning and Disclaimer

Book Sprint Facilitation

Book Writing Methodology

Hardware and Software Included in the Book

Cisco Application Policy Infrastructure Controller (APIC)

ACI Spine switches, including Cisco Nexus 9508 and 9336PQ

Cisco Application Virtual Switch (AVS)

Cisco UCS B and C series servers

A variety of hypervisors, including KVM, Microsoft Hyper-V, and VMware

IXIA IxVM product family traffic generators

This book was writ

The Story of ACME Inc.

The Why, Who, What, When and How

tion of many dif

How does a networking person go about configuring the network forwarding?

How do the application team track performance and usage metrics?

optimal workload placement decisions?

Virtual Machine Manager (VMM) integrations

Layer 4 to Layer 7 devices

Layer 2 or Layer 3 external configuration

Attachable Entity Profile (AEP) creation

Server or external network attachment

Changes to currently deployed contracts and filters that would materially

Application profile creation

Endpoint group (EPG) creation

Contracts scoped at a tenant level

Layer 4 to Layer 7 service graphs

Domain associations for EPGs

The items men

A more in-depth dis

ACI addresses IT requirements from across the organization

Configuring Management Protocols

Role-Based Access Control

Import and Export Policies

Single point of management controller-based architecture

Desired state-driven eventual consistency model

Configuring Management Protocols

Link Layer Discovery Protocol (LLDP) - Discovery protocols are no longer

Network Time Protocol (NTP) - Maintaining accurate time across the

Domain Name Services (DNS) - providing name resolution to fabric components

Cisco Discovery Protocol

On the menu bar, choose Fabric > Access Policies.

Your CDP pol

Link Layer Discovery Protocol

On the menu bar, choose Fabric > Access Policies.

a. In the Name field, enter "LLDP-TX-ON-RX-ON".

Your LLDP pol

Time Synchronization and NTP

Out-of-Band Management NTP

On the menu bar, choose Fabric > Fabric Policies.

In the Navigation pane, choose Pod Policies > Policies.

In-Band Management NTP

Verifying NTP Operation

On the menu bar, choose Fabric > Fabric Policies.

In the Work pane, verify the details of the server.

Verifying that the NTP Policy Deployed to Each Node

SSH to an APIC in the fabric.

admin@apic1:~> attach <Tab> <Tab>