Why You Shouldn’t Fear

Upgrading Your ACI Fabric

The Handbook!
Takuya Kishida and Joseph Ristaino
Technical Leaders, Marketing – Data Center Business Unit

BRKDCN-2910
Cisco Webex App

Questions?
Use Cisco Webex App to chat
with the speaker after the session

How
1 Find this session in the Cisco Live Mobile App
2 Click “Join the Discussion”
3 Install the Webex App or go directly to the Webex space Enter your personal notes here

4 Enter messages/questions in the Webex space

Webex spaces will be moderated

until February 24, 2023.

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 2
 • Upgrade Architecture
• ACI Firmware Upgrade Types
• Upgrade Architecture – APIC
• Upgrade Architecture – Switches

Agenda
• (Bonus) Upgrade Enhancements

• Best Practices
• Best Practices Workflow Review
• Best Practices Configurations
• “Pre-Upgrade Checklist” Review and Execution
• “Do’s and Don’ts”

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 3
ACI Firmware
Upgrade Types
ACI Firmware Upgrade Types

Regular Upgrade

Software Maintenance
Upgrade (SMU)

EPLD/FPGA Upgrade
(Only Switches)

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 5
ACI Firmware Upgrade Types (Regular)
Base OS firmware upgrade
In principle, all APICs and switches should be on the same version
Regular Upgrade
APIC Upgrade Switch Upgrade
(through APIC)

Software Maintenance
Upgrade (SMU)

EPLD/FPGA Upgrade
(Only Switches)

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 6
Different versions in the same fabric??
In principle, this should be avoided. Supported Operations
with different switch versions
What if I cannot finish upgrades in a
single upgrade window? Create, update and delete BDs, EPGs,
contracts, L3Outs, VMM domains,
Access Policies
• Available options
Collect configuration backups,
APIC firmware techsupports, or troubleshoot with
➢ All APICs must be on the same version SPAN
Switch firmware
Physical operations such as enabling
➢ Switches can be on different versions disabling interfaces, replacing a node
with limited operations.
See Upgrade Guide for the complete list:
https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/all/apic-installation-aci-upgrade-
downgrade/Cisco-APIC-Installation-ACI-Upgrade-Downgrade-Guide/m-operations-
allowed-during-mixed-versions-on-cisco-aci-switches.html
BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 7
ACI Firmware Upgrade Types (SMU) 5.2(1)

A patch for a specific defect

No need to upgrade the entire fabric. You can apply it
only to APICs or affected switch nodes
Regular Upgrade
SMU for SMU for
all APICs specific switches
(through APIC)

Software Maintenance
Upgrade (SMU)

EPLD/FPGA Upgrade
(Only Switches) No need to upgrade
other switches

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 8
ACI Firmware Upgrade Types (EPLD/FPGA)

Hardware related firmware

Each ACI switch version has the desired EPLD/FPGA version.
Regular Upgrade Automatically upgraded via Regular Upgrade through APIC.
➢ No user configurations

Software Maintenance
Upgrade (SMU)

EPLD/FPGA Upgrade
What if a switch is new and didn’t go
(Only Switches) through Regular Upgrade via APIC?
➢ 5.2(1) got you covered

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 9
APIC
Upgrade Architecture

Note: for 4.0 or newer APICs

 APIC Upgrade Architecture
• A user uploads the APIC image on one of APICs
APIC
Image Upload • After md5sum check, the image is copied to Image
other APICs
Auto Sync Auto Sync

Trigger

Install

Data Conversion
&
Reboot

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 11
 APIC Upgrade Architecture
Estimated Time
A few min.
Image Upload
Prepare all shards for upgrade

• Set the target version on all APICs

Trigger
• APIC1 informs shards on all APICs of upgrades

Install
No disruptive operations from this point.
(details in later slides)

Data Conversion Each shard has 3 replicas

& across APICs.
Reboot Prepare all replicas for
upgrade.

Shard – user configurations and data spread across APICs

Replica – back up for each shard
BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 12
 APIC Upgrade Architecture

Image Upload
Estimated Time
A few min.
Trigger

• Install APIC OS in a backup partition

Install
• All APICs perform this in parallel

Data Conversion
& Install APIC OS in parallel.
Reboot
No reboot, no impact yet.

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 13
 APIC Upgrade Architecture

Estimated Time
Image Upload
Depends on the size of data.
A fair estimation would be 40 min per APIC
(potentially more or less)
Trigger

Install

Data Conversion • Convert user configurations and data to the target Wait until lower
& version format Convert data starting from numbered
Reboot • Conversion happens one APIC at a time APIC 1, then reboot. APICs finish
data conversion
After reboot, APIC1’s upgrade and reboot.
is considered completed.

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 14
Auto Firmware Update for APIC 6.0(2)

New APIC is automatically

Use Case 1: APIC Replacement upgraded to the same
version as the rest of APICs
APIC Cluster
6.0(2a) 6.0(2a) 6.0(2a) Ex.) 5.2(1a)
Replace

Use Case 2: Cluster Expansion

APIC Cluster
6.0(2a) 6.0(2a) 6.0(2a) Ex.) 5.2(1a)
Add

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 15
ACI Switch
Upgrade Architecture
ACI Switch Upgrade Flow

Image • The switch downloads the image from APIC No Traffic Impact
Download • The download is via infra TEP

Download the image

Queuing

Preparation

Reboot

Boot Up

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 17
ACI Switch Upgrade Flow

Image No Traffic Impact

Download
Upgrade Token
• The switch receives approval from APIC (Approval)
Queuing
• Controls switches that are upgraded in parallel

• One leaf at a time in each vPC pair

Preparation
Since APIC 4.1(1) • Not all spines in each pod if
graceful option is used

Reboot

Boot Up

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 18
ACI Switch Upgrade Flow

Image No Traffic Impact

Download

Queuing

• The switch extracts the image.

Preparation Preparation
• The switch sets the boot var and so on.

Reboot

Boot Up

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 19
ACI Switch Upgrade Flow Depends on other conditions such as:
• Link failure detection time on external devices
• Routing protocol and so on

Image
< 100 msec Traffic Impact
Download
in the best case

Queuing ISIS detects

the tunnel down

Preparation
Reboot

• Wipe the config and reboot (i.e. clean reboot)

Reboot
• Traffic failover relies on link failure

Boot Up
Fail over with
the link down

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 20
ACI Switch Upgrade Flow

Image
Download

Queuing

Preparation Boot Up

Reboot

• Various traffic flow optimizations

Boot Up
• (Continue to next slides)

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 21
ACI Switch Upgrade Flow (Boot Up Sequence)
Boot Up • Various traffic flow optimizations

• Bring up fabric links

01 • Bring up APIC connected down links No Traffic Flow Change
• Admin down other down links

02

03
Bring up
fabric ports
04

05

06

07
BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 22
ACI Switch Upgrade Flow (Boot Up Sequence)
Boot Up • Various traffic flow optimizations

• Bring up fabric links

01 • Bring up APIC connected down links No Traffic Flow Change
• Admin down other down links

• An APIC discovers the switch via DHCP/LLDP

02 • The same TEP IP is assigned

03 TEP IP is
restored

04

05

06

07
BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 23
ACI Switch Upgrade Flow (Boot Up Sequence)
Boot Up • Various traffic flow optimizations

• Bring up fabric links

01 • Bring up APIC connected down links No Traffic Flow Change
• Admin down other down links

• An APIC discovers the switch via DHCP/LLDP

02 • The same TEP IP is assigned

• ISIS overload mode is activated

03 ✓ ISIS advertises the TEP IP with a large metric
✓ ISIS does not advertise BD mcast groups to join Infra reachability is restored

04

05

06

07
BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 24
ACI Switch Upgrade Flow (Boot Up Sequence)
Boot Up • Various traffic flow optimizations

• Bring up fabric links

01 • Bring up APIC connected down links No Traffic Flow Change
• Admin down other down links

• An APIC discovers the switch via DHCP/LLDP

02 • The same TEP IP is assigned

• ISIS overload mode is activated

03 ✓ ISIS advertises the TEP IP with a large metric
Config from APIC
✓ ISIS does not advertise BD mcast groups to join
(Takes several min)

04 • Starts downloading configurations from an APIC

05

06

07
BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 25
ACI Switch Upgrade Flow (Boot Up Sequence)
Boot Up • Various traffic flow optimizations

• Bring up fabric links

01 • Bring up APIC connected down links No Traffic Flow Change
• Admin down other down links

• An APIC discovers the switch via DHCP/LLDP

02 • The same TEP IP is assigned

• ISIS overload mode is activated

03 ✓ ISIS advertises the TEP IP with a large metric
Flood traffic starts coming
✓ ISIS does not advertise BD mcast groups to join
but no impact because
downlinks are admin-down
04 • Starts downloading configurations from an APIC

• ISIS multicast overload mode completes (i.e. flood)

05 • vPC peer is established at the same time

06 ISIS multicast overload timer

• Leaf nodes – Fixed 1min
• Spine nodes – When FTAG tree is created
07 (Fixed 1 min prior to Switch 14.2(1))
BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 26
 Spine ISIS multicast overload timer (CSCvp79708)
Why not a fixed 1 min?
IPN sends BD1 flood IPN
traffic to this spine
FTAG Tree Links

Rebooting
BD 1 mcast group join

BD 1 BD 1 BD 1 BD 1
ISIS mcast overload done
➢ It may be elected to be the Then
designated receiver for BD1
mcast group even though BD 1 mcast group join IPN
FTAG is not ready.
➢ Then, IPN would send BD1
flood traffic to this not-ready
spine.

Booted up but not part of

FTAG tree (the blue lines) yet

BD 1 BD 1 BRKDCN-2910
BD 1 BD 1
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 27
ACI Switch Upgrade Flow (Boot Up Sequence)
Boot Up • Various traffic flow optimizations Ready to receive traffic
• VLANs are deployed
• Bring up fabric links • For VMM, depends on Resolution Immediacy
01 • Bring up APIC connected down links • Contracts are deployed
• Admin down other down links • Depends on Deployment Immediacy
• Spine-Proxy is ready
• An APIC discovers the switch via DHCP/LLDP • Flood handling (FTAG) is ready
02 • The same TEP IP is assigned

• ISIS overload mode is activated

03 ✓ ISIS advertises the TEP IP with a large metric
✓ ISIS does not advertise BD mcast groups to join

04 • Starts downloading configurations from an APIC

• ISIS multicast overload mode completes (i.e. flood)

05 • vPC peer is established at the same time

• Full configuration has been downloaded

06 ✓ Bring up access links (downlinks)
✓ and vPC ports after vPC restore delay timer expires
• vPC restore delay timer is fixed to 120s since Switch 12.0(2)
• vPC restore delay timer starts when vPC peer is established.
07
BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 28
ACI Switch Upgrade Flow (Boot Up Sequence)
Boot Up • Various traffic flow optimizations

• Bring up fabric links

01 • Bring up APIC connected down links
• Admin down other down links

• An APIC discovers the switch via DHCP/LLDP

02 • The same TEP IP is assigned Traffic flow is back to the previous status

• ISIS overload mode is activated

03 ✓ ISIS advertises the TEP IP with a large metric
✓ ISIS does not advertise BD mcast groups to join

04 • Starts downloading configurations from an APIC

• ISIS multicast overload mode completes (i.e. flood)

05 • vPC peer is established at the same time

• Full configuration has been downloaded

06 ✓ Bring up access links (downlinks)
✓ and vPC ports after vPC restore delay timer expires

• ISIS unicast overload mode completes ISIS unicast overload timer - 10 min fixed for all nodes
07 ✓ The TEP IP is advertised with a normal metric
BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 29
ACI Switch Upgrade
with Graceful Option
(a.k.a. Graceful Upgrade)
 ACI Switch Upgrade with graceful option

Image
Download

Scheduler
Graceful Option is to gracefully
isolate the switch before the
switch goes down for the upgrade
Preparation

• Wipe the config and reboot (i.e. clean reboot)

Reboot • Traffic failover relies on link failure

Boot Up The rest is the same as

without graceful option.

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 31
 ACI Switch Upgrade with graceful option

Image Older APIC GUI

Download

Scheduler
Graceful Option is to gracefully
isolate the switch before the
switch goes down for the upgrade
Preparation

• Wipe the config and reboot (i.e. clean reboot)

Reboot • Traffic failover relies on link failure

Boot Up The rest is the same as

without graceful option.

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 32
 Enhanced reboot sequence with graceful option
• Graceful option disabled
1. Wipe the config and reboot (i.e. clean reboot)
Reboot 2. Traffic failover relies on user configured link failure mechanism

• Graceful option enabled

1. Put the switch into MMode (Maintenance Mode)
1. ISIS Overload Mode enabled
2. Graceful Shutdown on Routing Protocols
✓ Leaf - BGP, EIGRP, OSPF for L3Out
✓ Spine – BGP, OSPF for IPN, GOLF
3. vPC informs its peer that this switch is going down
Reboot 4. LACP sends PDUs with aggregation bit zero (starting from 3.1(2))
➢ External devices can exclude the link from the port-channel before the link physically goes down.
5. Shutdown front panel ports
✓ Leaf – all down links including APIC connected links
✓ Spine – all IPN links
2. Wipe the config and reboot (i.e. clean reboot)

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 33
 Traffic Disruption without Graceful Upgrade
OSPF DR reboot example
10.0.0.0/8 10.0.0.0/8 10.0.0.0/8 10.0.0.0/8
→ L3 Switch → L3 Switch → L3 Switch SPF Recalculation
BDR DR BDR DR

DR leaf OSPF
OSPF upgrade OSPF hold timer OSPF
(broadcast) (reboot) (broadcast) expires (broadcast)

DROTHER L3 Switch (SVI) DROTHER L3 Switch (SVI) BDR L3 Switch (SVI)

10.0.0.0/8 10.0.0.0/8 10.0.0.0/8

No Traffic Impact < 100 msec Traffic Impact A few seconds loss
Due to failover with link failure Until the external router re-sends
OSPF LSA for 10.0.0.0/8

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 34
 With Graceful Upgrade
OSPF DR reboot example
10.0.0.0/8 10.0.0.0/8 10.0.0.0/8 10.0.0.0/8
→ L3 Switch → L3 Switch → L3 Switch Empty → L3 Switch
BDR DR BDR OSPF hello DR

Graceful
OSPF OSPF OSPF Reload OSPF
(broadcast) shutdown (broadcast) (broadcast)

DROTHER L3 Switch (SVI) DROTHER L3 Switch (SVI) BDR L3 Switch (SVI)

10.0.0.0/8 10.0.0.0/8 10.0.0.0/8

No Traffic Impact No Traffic Impact < 100 msec Traffic Impact

OSPF will _not_ lose existing routes Due to failover with link failure
due to SPF recalculation

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 35
 GIR and Graceful Upgrade in ACI
Both GIR (Graceful Insertion and Removal) and Graceful Upgrade put the switch in
MMode (Maintenance Mode) to isolate the switch from the fabric.
However, the use case for these two features are completely different.
GIR (Graceful Insertion and Removal) An upgrade with the graceful option

Use Case: Use Case:

• To isolate a switch for further debugging • To upgrade a switch after isolating the switch
• To quickly restore service by isolating a
malfunctioning switch
Difference:
Difference: • The switch will communicate to APIC and perform an
• It is not supported to upgrade a switch in MMode via upgrade immediately after the switch was put into
GIR MMode.

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 36
Upgrade
Enhancements
 ACI Upgrade Enhancement Quick Summary
Supported Switch version
4.1(1) 4.2(*) 4.2(5) 5.0/5.1 5.2(1) 5.2(3) 6.0(2)
APIC versions requirements
Switch Image Pre-download 14.1(1) or later
Upgrade
Time Multi-Pod Parallel Switch Upgrade No requirements
Optimization
Unlimited Parallel Switch Upgrade By Default No requirements

APIC Detailed Install Stage N/A

Visibility
Switch Image Download Progress 14.5(1) or later

Built-in Pre-Upgrade Validation No requirements

Unified Pre-Upgrade Validation * * No requirements

SMU Support 15.2(1) or later

Operation
Optimization Auto EPLD/FPGA upgrade 15.2(1) or later

NXOS to ACI auto conversion via POAP 15.2(3) or later

Auto Firmware Update for APIC N/A

Auto Firmware Update for switches No requirements

* Need to download pre-upgrade validator app from dcappcenter.cisco.com

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 38
 Upgrade Time Reduction

Switch Image Download Upgrade multiple

from APIC to switches pods/switches in parallel

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 39
 Switch Image Pre-Download with a scheduler 4.1(1)

New label in ACI 14.2(5).

The functionality of pre-download has been the same since ACI 4.1. Long time ahead
Prior to 14.2(5), it was labeled as “Schedule for Later” with the same
functionality..

1. Schedule for a long time ahead just to trigger pre-download of a switch image.
2. During the actual maintenance window, come back to this same window (maintenance group) and select “Now” to trigger
the upgrade on demand. Switches don’t need to re-download images and can proceed with the upgrade immediately.

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 40
 Switch Image Download Progress 4.2(5)

New in ACI 4.2(5), download progress

(switches need to be 14.2(5) for this functionality)

• All switches (regardless of pods or vPC) in the update group download the switch image from APICs in parallel. During this
period, the Upgrade Progress remains 0 %.
• With the new Download Progress bar, users can see whether switches finished the download and ready to upgrade.
• If it was triggered with a scheduler, all switches wait after they completed their download.
• If it was triggered with “Upgrade Now”, each switch proceed with the upgrade as soon as it has completed its download.

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 41
Switch Image Pre-Download (built-in) 5.1(1)

Pre-Download is built-in

Installation will not start until you manually

trigger installation after the download has
completed.

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 42
Switch Image Download Progress 5.1(1)

Progress of each step (download and install)

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 43
 Switch Image Download Progress (APIC 4.2(5), Switch 14.2(4))

Remain empty

Download Progress will not be displayed when switches are older than 14.2(5)
even if APIC is 4.2(5) or later BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 44
Upgrade multiple pods/switches in parallel OLD

One pod at a time

When the actual upgrade starts, APICs allow each switch to upgrade based on the following rules;
• One Pod at a time (14.2(5) has an update)
• When triggered with “Upgrade Now”, 20 switches at a time (14.2(5) has an update)
• When a vPC pair leaf nodes are in the same group, only one of the pair at a time
BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 45
 Unlimited Parallel Upgrade 4.2(5)

All pods at once

• From APIC 14.2(5) or later, any switches in any pods can be upgraded in parallel
• “Upgrade Now” is no longer limited to 20 switches at a time

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 46
ACI Upgrade
Best Practices
Agenda
• Best Practices Workflow Review
• Best Practice Configurations
• “Pre-Upgrade Checklist” Review and Execution.
• “Do’s and Don’ts”

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 48
Recommended Guides
!
Cisco ACI Upgrade Checklist – Important Starting Point

ACI Upgrade Checklist:

https://www.cisco.com/c/en/us/td/doc
s/switches/datacenter/aci/apic/sw/kb/C
isco-ACI-Upgrade-Checklist.html

Detailed Upgrade Guide (the basis for

this presentation)

https://www.cisco.com/c/en/us/td/doc
s/dcn/aci/apic/all/apic-installation-aci-
upgrade-downgrade/Cisco-APIC-
Installation-ACI-Upgrade-Downgrade-
Guide.html

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 49
ACI Firmware Upgrade Best Practice Checklist

Determine Desired Software and Check Support Matrix

Review and Implement Best Practice Configurations

Discover and Clear any issues raised from “pre-upgrade validations”

Review Upgrade Architecture and “do’s and don’ts”

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 50
ACI Firmware Upgrade Best Practice Checklist

Determine Desired Software and Check Support Matrix

Review and Implement Best Practice Configurations

Discover and Clear any issues raised from “pre-upgrade validations”

Review Upgrade Architecture and “do’s and don’ts”

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 51
 ACI Software Life Cycle
1 2 3
Cisco Recommended Software
Cisco ACI Release Notes
Releases Cisco ACI Upgrade/Downgrade
Support Matrix
https://www.cisco.com/c/en/us/suppor
https://www.cisco.com/c/en/us/td/doc
t/cloud-systems-
s/switches/datacenter/aci/apic/sw/reco https://www.cisco.com/c/dam/en/us/td
management/application-policy-
mmended- /docs/Website/datacenter/apicmatrix/in
infrastructure-controller-apic/tsd-
release/b_Recommended_Cisco_ACI_ dex.html
products-support-series-home.html
Releases.html

Determines if Multi-Step
Upgrade is Required

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 52
ACI Software Life Cycle
5
Review the ACI Upgrade/Downgrade Guide!

https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/all/apic-
installation-aci-upgrade-downgrade/Cisco-APIC-Installation-ACI-
Upgrade-Downgrade-Guide/m-aci-firmware-upgrade-
overview.html#id_48185

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 53
 New Release Cadence

Key objectives Predictable software release cadence | Reach maintenance mode quickly

7.0.1 12 months 15 months 15 months 6 months

6.1.1 12 months 15 months 15 months 6 months

6.0.1 12 months 15 months 15 months 6 months

Day 0 1Y 2Y 3Y 4Y 5Y

Legend Development cycle Maintenance cycle Extended support with PSIRT fixes TAC support

Three feature releases Fourth release is a Hardware lifecycle is defined by

No short-lived and Total release lifecycle
from FCS date, including maintenance release (MR), multiple release and not tied
long-lived release tags of four years
FCS release target for golden star to a single release

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 54
ACI Firmware Upgrade Best Practice Checklist

Determine Desired Software and Check Support Matrix

Review and Implement Best Practice Configurations

Discover and Clear any issues raised from “pre-upgrade validations”

Review Upgrade Architecture and “do’s and don’ts”

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 55
Best Practices
Configuration
ACI Firmware Upgrade Configuration Checklist
Ensure there is a Valid Backup Exported with AES Encryption

Validate Switch Upgrade Groups provide redundancy, and have

desired settings
Enable Auto-Firmware Update for Switches

Enable Best Practice Settings for Multi-Pod/Site Deployments.

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 57
Back Up Configuration with AES File Encryption !
• The AES passphrase that generates the Setting Global AES Encryption allows all
encryption keys cannot be recovered or read by the secure properties of the configuration
an ACI administrator or any other user. The AES (like credentials) to be successfully
passphrase is not stored. Copy your passphrase imported when restoring the fabric
somewhere safe!

• Setup automatic backups on a scheduler to

maintain a consist and up to date backup at all
times. Always export it to a remote location.

• In case of upgrade failure, AES backup can be

used to recover the system non-disruptively as
worst case scenario.

Pre ACI v4.0.1 Setting Location: ACI v4.0.1 and later Location:

Admin > AAA > AES Encryption Passphrase and Keys System > System Settings > Global AES Passphrase
for Config Export (and Import) Encryption Settings
Technote For Import/Export:
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_KB_Using_Import_Export_to_Recover_Config_States.html

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 58
Upgrade Group
Configuration Options
 ACI Firmware Upgrade Best Practice 101
Consider the fabric as
one modular switch
Spine
(Fabric Card)

Leaf (Line Card)

APIC (Supervisor)

ACI is a solution to manage multiple switches as if it’s one huge switch

➢ APIC (i.e. SUP of the fabric) can be upgraded non-disruptively.
➢ Each switch (i.e. modules of the fabric) can intelligently choose appropriate switch nodes
for non-disruptive traffic flow
Always keep hardware redundancy to achieve zero-to-minimum traffic disruption
1. Upgrade Green switch groups
2. Upgrade Blue switch groups
BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 60
Switch Upgrade Advanced Options
Rule of Thumb
Change defaults only when you must.

Upgrade Group • Ignore Compatibility Check (default: disabled)

Enable only in a lab where you would like to ignore the supported upgrade path.
• Name
• Node ID List
• Target Firmware Version • Graceful option (default: disabled)
Only used when sub-100ms routing protocol convergence is required.
• Scheduler
Never enable this when hardware redundancy is not ensured. (single spine/leaf pod)
• Ignore Compatibility Check
• Graceful option • Run Mode (default < 5.1: pause upon upgrade failure
• Run Mode
(default >= 5.1: don’t pause upon upgrade failure)
By default, APIC scheduler will stop putting new switches into queue if
a) APIC cluster is not fully-fit
b) The upgrade of previous switches in the same upgrade group failed.
Advanced Options Ex.) You have 20 leafs in a group. If 1 fails, it will pause all remaining switches that
are queued.

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 61
Auto Firmware
Update for Switches
Auto Firmware Update for Switches
Enforcing Version Consistency

Fabric > Inventory > Fabric Membership > Auto Firmware Update >=5.1(1)

Admin > Firmware > Infrastructure > Nodes > Enforce Bootscript Version Validation < 5.1(1)

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 63
 Auto Firmware Update for Switches
Caveats

1 If the node is part of a Firmware Group, the Firmware Group version will
take precedence
2 For EPLD Upgrade (Replacement), it was always recommended to install the
switch on a lower version, and than upgrade it. When doing this:

Prior to 5.2: Recommendation is to set the Default to “any”.

5.2 and above: EPLD Upgrade is handled automatically. Set desired version.

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 64
IS-IS Metric Policy for
Multi-Pod and Multi-Site
 Helpful Tips for Multi-Pod / Multi-Site
ISIS Metric Policy Configuration
IPN

ISIS Overload in process ISIS Overload in process

Node Upgrade Node Upgrade

Group 1 (in hold Group 1 (in hold
down) down)

• Default fabric wide IS-IS metric is set at 63 (max value)

• During upgrade, spines set the overload mode while policy is being
downloaded.
• If fabric-wide value is already at max, the overload functionality is
ineffective.
• This can create unexpected traffic interruption if leaf sends traffic to a
spine which is not fully upgraded. Settings > ISIS Policy (Default Config)

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 66
 Helpful Tips for Multi-Pod / Multi-Site
ISIS Metric Policy Configuration
IPN

ISIS Overload in process ISIS Overload in process

Node Upgrade Node Upgrade

Group 1 (in hold Group 1 (in hold
down) down)

Set this value to < 63

• By Lowering the Value, Remote POD TEP Routes will be preferred before any upgrade
through the remaining spines in each POD.
• Once Overload is completed, the spine which was upgraded will
advertise these routes using the metric configured.
• This results in ECMP between all spines after the upgrade has
completed.
Settings > ISIS Policy

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 67
 Helpful Tips for Multi-Pod / Multi-Site
Verify Spines are Exchanging Routes to the IPN after upgrade
IPN
Node Upgrade Group 1 Node Upgrade Group 1
Spine 1,4 Spine 1,4

Node Upgrade Node Upgrade

Group 1 Remaining spines Group 1
(Rebooting) sending pod TEP routes to IPN (Rebooting)

Node Upgrade Group 2

Spine 2,3

• When Node Upgrade Group 1 finishes, Spines may show as “completed” in upgrade
UI but routes towards IPN/ISN may still be in hold down period (up to 10 min)

• Before starting Spine Node Upgrade Group 2, verify that TEP routes of pods / sites
are being sent / received from newly upgraded spines in Group 1

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 68
Helpful Tips for Multi-Pod / Multi-Site
Verify Spines are Exchanging Routes to the IPN after upgrade

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 69
ACI Firmware Upgrade Best Practice Checklist

Determine Desired Software and Check Support Matrix

Review and Implement Best Practice Configurations

Discover and Clear any issues raised from “pre-upgrade validations”

Review Upgrade Architecture and “do’s and don’ts”

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 70
Pre-Upgrade Validation
Faults, and the Impact on Upgrades

• Faults can be raised if there is an

overlap, or invalid config.
• After an upgrade the switch
requests it’s configuration “fresh”
from APIC. This is the “stateless”
Faults raised but After upgrade, previous
behavior of ACI. functioning normally. working config can be
changed to “faulted” config.
• If Logical Config (APIC) has
conflicts, the “faulted” config can L2 Port Config (F0467 port-configured-as-l3)
get pushed before the previously L3 Port Config (F0467 port-configured-as-l2)
working config. Config On APIC Connected Port (F0467 port-
configured-for-apic)
etc . . .

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 72
 Pre-Upgrade Validation 3.2 - continuing
APIC 3.2, 4.0, 4.1

• Prior to 4.2, the APIC upgrade simply warned about the

number of all critical and major faults

APIC 4.2(1) – 4.2(3) • On 4.2(1) – 4.2(3), the APIC upgrade warned about
✓ config related critical faults
✓ some specific faults that are known to cause
issues during upgrades.

• On 4.2(4), the APIC upgrade warns about

✓ config related critical faults
✓ some specific faults that are known to cause
issues during upgrades
APIC 4.2(4) ✓ A few nonoptimal configurations that may disrupt
traffic during the upgrade.

• Additional validation items are being added on each

release.

For older APIC versions to run some of the validations added in later
release: https://dcappcenter.cisco.com/pre-upgrade-validator.html

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 73
 Pre-Upgrade Validation (AppCenter App)

The goal of the app

https://dcappcenter.cisco.com/pre-upgrade-validator.html To be able to apply the latest validations on any
APIC versions via AppCenter app

What happens if Cisco adds

additional checks? What if I don’t
allow apps?
BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 74
Pre-Upgrade Validation – Script (Preferred)
https://github.com/datacenter/ACI-Pre-Upgrade-Validation-Script

The goal of the script

To be able to apply the latest validations on any
APIC versions via a script

Both app and script are fully supported by TAC

Why the script may be a better choice?:

• Github script is updated more frequently

• Supports older versions
• With Github account, you can submit issues
or features directly

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 75
 Pre-Upgrade Validation – Script (Preferred)
admin@apic1:pre-upgrade> python aci-preupgrade-validation-script.py
==== 2021-11-16T08-45-58-0500 ====
User Enters Credentials
Enter username for APIC login : admin Checks that require login
Enter password for corresponding User : leverage this input
Checking current APIC version (switch nodes are assumed to be on the same version)...3.2(10e)

Gathering APIC Versions from Firmware Repository...

User Selects Target Version
[1]: aci-apic-dk9.4.2.7f.bin
Checks that require target version
What is the Target Version? : 1
leverage this input.
You have chosen version "aci-apic-dk9.4.2.7f.bin”
[Check 1/37] APIC Target version image and MD5 hash...
Checking fab3-apic1...... DONE
PASS
[Check 2/37] Target version compatibility... Failure Details are Provided PASS
[Check 3/37] Gen 1 switch compatibility... PASS
. . .
. . . Issue should be corrected (Script Re-Run to
. . . validate) before performing upgrade.
. . .
. . .
[Check 19/37] L2 Port Config (F0467 port-configured-as-l3)... FAIL - OUTAGE WARNING!!
Fault Pod Node Tenant AP EPG Port Recommended Action
----- --- ---- ------ -- --- ---- ------------------
F0467 pod-1 node-101 jr ap1 epg1 eth1/6 Resolve the conflict by removing this config or other configs using this port as L3

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 76
 Pre-Upgrade Validation – Script (Preferred)
[Check 32/37] BGP Peer Profile at node level without Loopback... PASS
[Check 33/37] L3Out Route Map import/export direction... PASS
[Check 34/37] Intersight Device Connector upgrade status... Connector reporting InternalServerError, Non-Upgrade issue PASS
[Check 35/37] EP Announce Compatibility... PASS
[Check 36/37] Eventmgr DB size defect susceptibility... PASS
[Check 37/37] Contract Port 22 Defect Check... PASS

=== Summary Result === Summary is Provided

PASS : 28
FAIL - OUTAGE WARNING!! : 4 All “FAIL” Categories need remediation.
FAIL - UPGRADE FAILURE!! : 2 Detailed Recommendations to Remediate are
MANUAL CHECK REQUIRED : 1 in the Upgrade Guide!
N/A : 2
ERROR !! : 0
TOTAL : 37

Pre-Upgrade Check Complete.

Log Bundle is Created
Next Steps: Address all checks flagged as FAIL, ERROR or MANUAL CHECK REQUIRED
Upload this to any TAC Case if Necessary.
Result output and debug info saved to below bundle for later reference.
Attach this bundle to Cisco TAC SRs opened to address the flagged checks.

Result Bundle: /data/techsupport/Scripts/pre-upgrade/preupgrade_validator_2021-11-16T08-45-58-0500.tgz

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 77
 Nexus Dashboard Insights (Optional)
Benefit of Nexus Insights
Does both a pre-check and a post-check to alert
on effects and changes in the upgrade window

• Pre-Update Verifications and Alerting

• Detailed list of bugs addressed in the
upgrade
• Post-upgrade Delta analysis of Anomalies,
Edits and Operations changes in the upgrade
process

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 78
 Future* Pre-Upgrade Validation Workflow

1 Use Nexus Dashboard Insights, or Pre-Upgrade Validator App on APIC

Connect Nexus Dashboard/APIC to Cisco Cloud

2 or
Use Local File Upload of Metadata Directly to ND or APIC

3 Cisco Releases New Checks and Metadata is Updated

MetaData is Automatically Pulled from Cisco Cloud and Ready to Use.

4 or
Download Latest Metadata and Upload Locally to ND to APIC

* Roadmap Item
BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 79
Future* Pre-Upgrade Validation Workflow

User Selects Target Version

Checks that require target version
leverage this input.

Pre-Upgrade Checklist Updates

Can be automatic (Intersight) or manual
(Air-Gapped)

App Supported on 5.2. Pre-Packaged in 6.0(2)*

* Roadmap Item
BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 80
Future* Pre-Upgrade Validation Workflow

Checks are Logged as Pass/Fail

Details of Each Failure are

identical to Script

App Supported on 5.2. Pre-Packaged in 6.0(2)*

* Roadmap Item
BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 81
ACI Firmware Upgrade Best Practice Checklist

Determine Desired Software and Check Support Matrix

Review and Implement Best Practice Configurations

Discover and Clear any issues raised from “pre-upgrade validations”

Review Upgrade Architecture and “Do’s and Don’ts”

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 82
Remember this guy?

ACI Upgrade Architecture

https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/all
/apic-installation-aci-upgrade-downgrade/Cisco-APIC-
Installation-ACI-Upgrade-Downgrade-Guide/m-aci-
upgrade-architecture.html

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 83
 Do’s and Don’ts
If at any point in time you believe the upgrade/downgrade has either stalled or failed,
follow the guidelines below:
Do View the APIC Faults and Installer Logs.
Do Collect the Tech Support Files.
Do Contact Cisco TAC if Needed.

admin@apic1:logs> pwd
/firmware/logs
admin@apic1:logs> ls -l
2021-04-15T07:42:57-50
2021-05-28T10:18:33-50
admin@apic1:logs> ls -l ./2021-05-28T10:18:33-50
atom_installer.log
insieme_4x_installer.log

leaf101# pwd
/mnt/pss
leaf102# ls installer_detail.log
installer_detail.log

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 84
Do’s and Don’ts
If at any point in time you believe the upgrade/downgrade has either stalled or failed, it
is critical that you do not take any of the actions listed below:
Don't reload any APIC in the cluster manually.
Don't decommission any APIC in the cluster.
Don't change the firmware target version back to the original version.

Version X Version Y

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 85
 Final Tip
You’ve read the “Do’s and Don’ts”…

When in Doubt,
Contact Cisco Support

With Proper Backups, Recovery is Always an Option

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 86
ACI Firmware Upgrade Best Practice Checklist

Determine Desired Software and Check Support Matrix

Review and Implement Best Practice Configurations

Discover and Clear any issues raised from “pre-upgrade validations”

Review Upgrade Architecture and “do’s and don’ts”

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 87
 • Always make sure you are
performing a supported upgrade.
• Best Practice Configuration and
Backups are Critical to Success

Key points to • ACI Pre-Upgrade Checklist will

prevent known issues from
remember impacting the upgrade.
• Never perform a disruptive
procedure during an upgrade
without help from Cisco.

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 88
 • Cisco APIC Installation and ACI
Upgrade and Downgrade Guide
https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/all/apic
-installation-aci-upgrade-downgrade/Cisco-APIC-
Installation-ACI-Upgrade-Downgrade-Guide.html

• Cisco ACI Upgrade Checklist

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/
aci/apic/sw/kb/Cisco-ACI-Upgrade-Checklist.html

• Cisco APIC Release Notes

Reference
https://www.cisco.com/c/en/us/support/cloud-systems-
management/application-policy-infrastructure-controller-
apic/tsd-products-support-series-home.html

• Release Notes for Cisco Nexus 9000

Series Switches in ACI Mode
https://www.cisco.com/c/en/us/support/switches/nexus-
9000-series-switches/products-release-notes-list.html

• Getting Started Guide (NX-OS to ACI

POAP Auto-conversion)
https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/5x/getti
ng-started/cisco-apic-getting-started-guide-52x/fabric-
initialization-52x.html#d5018e3247a1635

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 89
 • Cisco APIC Installation and ACI Upgrade /
Downgrade Guide
https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/all/apic-
installation-aci-upgrade-downgrade/Cisco-APIC-Installation-ACI-
Upgrade-Downgrade-Guide.html

• Cisco ACI Upgrade Checklist

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/
sw/kb/Cisco-ACI-Upgrade-Checklist.html

• Cisco APIC Release Notes

https://www.cisco.com/c/en/us/support/cloud-systems-

Reference
management/application-policy-infrastructure-controller-apic/tsd-
products-support-series-home.html

• Release Notes for Cisco Nexus 9000 Series

Switches in ACI Mode
https://www.cisco.com/c/en/us/support/switches/nexus-9000-
series-switches/products-release-notes-list.html

• Cisco ACI Upgrade Matrix

https://www.cisco.com/c/dam/en/us/td/docs/Website/datacenter/apic
matrix/index.html

• Pre-Upgrade Validation Script

https://github.com/datacenter/ACI-Pre-Upgrade-Validation-Script

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 90
Complete your Session Survey
• Please complete your session survey
after each session. Your feedback
is important.
• Complete a minimum of 4 session
surveys and the Overall Conference
survey (open from Thursday) to
receive your Cisco Live t-shirt.
• All surveys can be taken in the Cisco Events Mobile App or
by logging in to the Session Catalog and clicking the
"Attendee Dashboard” at
https://www.ciscolive.com/emea/learn/sessions/session-catalog.html

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 91
Continue
Agenda Your Education

Visit the Cisco Showcase for related demos.

Book your one-on-one Meet the Engineer meeting.

Attend any of the related sessions at the DevNet,

Capture the Flag, and Walk-in Labs zones.

Visit the On-Demand Library for more sessions

at ciscolive.com/on-demand.

BRKDCN-2910 © 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public 92
Thank you