Upgrade Enhancements
ACI Release 5.0 (Jordan)
Takuya Kishida
Technical Marketing Engineer
Last Update: 05/26/2020
What’s new?
• UI enhancements
• APIC “Install stage” indicator for upgrade/downgrade
• Prior to 5.0, APIC didn’t provide enough details even with the APIC upgrade progress bar.
• Switch image download progress bar
• Prior to 5.0, there were only upgrade progress bars (for both APIC and Switch)
• Backend improvements to reduce time required for switch upgrades.
• Unlimited number of switch nodes are upgraded at the same time by default
• Prior to 5.0, the default was 20. To change this, upgrade scheduler was required.
• Parallel Upgrade of multiple pods
• Prior to 5.0, when switches in different pods are upgraded at the same time, only one pod at a time was allowed
to upgrade.
• Improved pre-upgrade validation check
• Some configuration validation on top of faults
• Prior to 5.0, only faults were validated.
• The validation for switch upgrade
• Prior to 5.0, the validation was performed only for APIC upgrade.
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Recap:
APIC Upgrade
on previous versions
Go to slide 11 for 5.0 new information
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
1. pre-upgrade validation
APIC 3.2, 4.0, 4.1
• Prior to 4.2, the APIC upgrade simply warned about the
number of all critical and major faults
• On 4.2(1) – 4.2(3), the APIC upgrade warned about
APIC 4.2(1) – 4.2(3)
config related critical faults
some specific faults that are known to cause issues
during upgrades.
• On 4.2(4), the APIC upgrade warns about
config related critical faults
some specific faults that are known to cause issues
during upgrades
A few nonoptimal configurations that may disrupt
APIC 4.2(4)
traffic during the upgrade.
(5.0 has an update)
• Prior to 5.0, Switch upgrade didn’t show any warnings. (5.0
has an update)
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
2. APIC Upgrade Status (APIC Install Stage)
• APIC GUI provided only the percentage (5.0 has an update)
• The percentage didn’t provide much information even in 4.2(4).
• First APIC : 0% 5% 100%
• Remaining APICs: 0% 100%
(5.0 has an update)
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Recap:
Switch Upgrade
on previous versions
Go to slide 11 for 5.0 new information
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
1. Download Images on switches
“Upgrade Now” allows switches to download the
target image from APICs immediately and
proceed with the upgrade as soon as the download
is finished.
“Schedule for Later” allows switches running
14.1 or later version to download the target image
from APICs immediately, but proceed with the
upgrade later (pre-download). Users can change
the Upgrade Start Time to “Now” to trigger the
upgrade before the scheduler kicks in.
This allows administrators to finish the image
download prior to the maintenance window to
save time required for the window.
If switches are running 14.0 or older version,
switches will start both download and upgrade
later at a time when the scheduler specified.
In either case, download happens on all switches in the
group in parallel across pods.
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
2. Image Pre-Download with a scheduler finished
(How can we check if it’s finished?)
This will be empty or
0 % until the
upgrade starts.
Download Progress is
not displayed.
When a scheduler is used on switches running 14.1 or later (i.e. pre-download), users have to leave the fabric with the above status till the
maintenance window by assuming the pre-download has completed successfully.
F3-P1-Leaf-101# ls -l /bootflash During the download, the filename has “.tmp”
CLI was the only way to total 3489748
-rw-rw-rw- 1 root root 2597345 May 12 23:10 CpuUsage.Log
check the download -rw-rw-rw- 1 root root 1777998029 May 12 11:38 aci-n9000-dk9.14.1.2g.bin
progress. -rw-rw-rw- 1 root root 315314009 May 13 03:35 aci-n9000-dk9.14.1.2x.bin.tmp
-rw-r--r-- 1 root root 1472828621 May 12 23:06 auto-s
(5.0 has an update) -rw-rw-rw- 1 root root 2 May 12 23:11 diag_bootup
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
......
3. Proceed with the upgrade
(how many switches in parallel?)
One pod at a time
In the maintenance window, users trigger the upgrade with “Upgrade Now” or wait until the scheduler kicks in. Once the
actual upgrade started, APICs allow each switch to upgrade based on the following rules;
• One Pod at a time (5.0 has an update)
• When triggered with “Upgrade Now”, 20 switches at a time (5.0 has an update)
• When a vPC pair leaf nodes are in the same group, only one of the pair at a time
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
How to check the reason why some switches are not
upgraded
“Firmware > History > Schedulers” provides details of overall upgrade status. This tab is available since 4.1 release.
This example is regarding the pod 2 switches not being upgraded yet due to one-pod-at-a-time rule (5.0 has an update).
Alternatively, MO “maintUpgStatusCont” can be used via API or CLI
admin@f1-apic1:~> moquery -c maintUpgStatusCont | grep Qual
schedulerOperQualStr : Node: 104, Policy: ALL, Check constraint: Is VPC peer
upgrading?, Result: fail, Details: Rejecting upgrade request from node: 104.
VPC peer upgrading, node to retry periodically. Peer node: 10.0.80.67/32
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Upgrade Enhancement
Summary
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Upgrade Enhancement Quick Summary
Supported APIC
3.2 4.1(1) 4.2(*) 4.2(4) 5.0(1) Switch version requirements
Version
APIC Detailed Install Stage N/A
Switch Image Pre-download
Switches also need to be on 14.1(1) or later
(with a scheduler)
Switch Image
Switches also need to be on 15.0(1) or later
Download Progress
Multi-Pod
No requirements*
Parallel Switch Upgrade
Unlimited
Parallel Switch Upgrade No requirements*
By Default
Pre-Upgrade Validation For APIC upgrades, available on 4.2(1) or later
(Specific Faults) For Switch upgrades, available only on 15.0(1) or later
Pre-Upgrade Validation For APIC upgrades, available on 4.2(4) or later **
(Config) For Switch upgrades, available on 15.0(1) or later
* APIC is what enforces those rules
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential ** APIC 5.0 introduced more configs to validate
5.0 UI Enhancements
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
APIC “Install Stage”
• This is available when your APIC is already on APIC 5.0(1) or higher.
New in ACI 5.0
1. Ready for next upgrade • When upgrading from an older version such as 4.2(4) to 5.0, you will not see
2. Checking compatibility this. The value of this feature is seen after APICs are upgraded to 5.0(1) or
3. Checking controller health later.
4. Performing upgrade (i.e. pre-loading the image)
5. Waiting for other controllers to upgrade • If you are downgrading from 5.0(1) to an older version then you will see
6. Migrating configuration this, however after the downgrade is complete you will be back to the
(i.e. data conversion from old object model to the new one) previous version’s UI functionality.
• In the future, if you downgrade from 5.0(2) to 5.0(1) this UI enhancement is
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential still available.
APIC “Install Stage” under the hood
admin@F3-APIC1:~> moquery -d topology/pod-1/node-1/sys/ctrlrfwstatuscont/upgjob
Total Objects shown: 1
# maint.UpgJob
...
installStage : ctrlrPreUpgradePhase
instlProgPct : 5
...
upgradeStatus : inprogress
upgradeStatusStr : Stage: Preupgrade configuration
The MO “maintUpgJob” can be used to see the same information via API or CLI.
These attributes “installStage”, “UpgradeStatus” etc. from the MO “maintUpgJob” are available also on older versions, however, it had
only one or two steps (0% 5% 100%), which was not useful.
In 5.0, these attributes are reflecting more detailed status of the upgrade. Details of each step is explained in this document.
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/all/apic-installation-upgrade-downgrade/Cisco-API
C-Installation-Upgrade-Downgrade-Guide/Cisco-APIC-Installation-Upgrade-Downgrade-Guide_chapter_011.html
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Switch Image Download Progress
New in ACI 5.0, download progress
(switches need to be 15.0 for this functionality)
• All switches (regardless of pods or vPC) in the update group download the switch image from APICs in parallel. During this period, the
Upgrade Progress remains 0 %.
• With the new Download Progress bar, users can see if switches finished the download and ready to upgrade.
• If it was triggered with a scheduler, all switches wait after they completed their download.
See the slide “Switch Image Pre-Download with a scheduler” for details. (not new, available since switch 14.1)
• If it was not triggered with a scheduler, each switch proceed with the upgrade as soon as it has completed its download.
See the slides “Parallel Upgrade Improvements” about how APIC 5.0 handles this. (new)
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Switch Image Download Progress (APIC 5.0, Switch 14.2)
Remain empty
Download Progress will not be displayed when switches are prior to 15.0 even if APIC is 5.0 or later
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Switch Image Pre-Download with a scheduler (Available since 4.1)
New label in ACI 5.0.
Long time ahead
The functionality of pre-download has been the same since ACI 4.1.
Prior to 5.0, it was labeled as “Schedule for Later” with the same functionality..
1. Schedule for a long time ahead just to trigger pre-download of a switch image.
2. During the actual maintenance window, come back to this same window (maintenance group) and select “Now” to trigger the upgrade on
demand. Switches don’t need to re-download images and can proceed with the upgrade immediately.
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Backend Enhancements
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Unlimited Parallel Upgrade
All pods at once
• From APIC 5.0 or later, any switches in any pods can be upgraded in parallel
• “Upgrade Now” is no longer limited to 20 switches at a time
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-upgrade validations (new on switch upgrade)
• In 5.0, the pre-upgrade validation is performed for
switch upgrade on top of APIC upgrade.
• For switch upgrade, you need to click “Submit” in
“Schedule Node Upgrade” once, then the validation is
performed. You need to click “Submit” again to actually
submit or schedule the upgrade.
• For APIC upgrade, the validation is performed when
“Schedule Controller Upgrade” is opened.
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-upgrade validations (new validation items)
Validation Items for APIC Upgrade
Reason Recommended Action Introduced
Version
The fabric has <fault counts> active critical Go to System > Faults to find details of any critical faults or the listed faults. 4.2(1)
config faults and one or more of the following
faults: F0101, F0130, F1528, F1529, F2732,
F3073, F3074, F606391.
Following nodes do not have out-of-band Configure out-of-band management IP for the listed nodes to ensure that you always have access to all 4.2(4)
management IP: <node IDs> nodes. During the upgrade/downgrade, nodes may not be reachable via ACI infra. Static out-of-band
management IP can be configured from APIC GUI under Tenants > ALL TENANTS > mgmt > Node
Management Addresses > Static Node Management Addresses.
Following nodes are not in vPC <node IDs>. Configure vPC for the listed leaf nodes to avoid traffic loss during the reboot of leaf nodes. 4.2(4)
Pod(s) <pod IDs> have fewer than two route Configure spine nodes as route reflector for infra MP-BGP. Make sure that at least one route reflector 4.2(4)
reflectors for infra MP-BGP. spine is always up by upgrading/downgrading them in separate groups.
NTP is not configured. Configure NTP via System > QuickStart > First time setup of the ACI fabric > NTP. This is 5.0(1)
recommended to avoid any issues in database synchronization between nodes, SSL certificate check,
etc.
APICs in cluster have different infra VLAN IDs. Check the configured infra VLAN ID via “cat /data/data_admin/sam_exported.config” on each APIC. 4.2(4)
Decommission the APIC with incorrect infra VLAN ID from the APIC cluster and initialize it.
Initialization is required to re-configure infra VLAN ID.
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-upgrade validations (new validation items)
Validation Items for APIC Upgrade (cont.)
Reason Recommended Action Introduced
Version
The target version is the same as the current Select a different target version. 4.2(4)
running version.
APICs are not running recommended CIMC Upgrade to the recommended CIMC version. APICs have recommended CIMC versions based on its 4.2(4)
versions : <recommended CIMC versions> hardware model and APIC firmware version.
The target version is not compatible with the Select a compatible target version. If the final target version is not compatible with the current running 4.2(4)
current running version. version, upgrade/downgrade to another version that is compatible with both target and current version
first. Then upgrade/downgrade to the final target version. Alternatively, you can enable "ignore
Compatibility Check" option. However, it is not recommended because this has a potential risk
such as loosing configurations.
NOTE:
The wordings in these tables are the updated ones from 5.0(1). The validation items that were introduced from 4.2(4) may have different
wordings for the same validation.
The following enhancement will back port all the 5.0 updates (both APIC and switch) for pre-upgrade validation into 4.2 train.
CSCvu33163 Port APIC and switch pre-validation fixes to IMR
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-upgrade validations (new validation items)
Validation Items for Switch Upgrade
Reason Recommended Action Introduced
Version
Following nodes do not have space in the Remove unnecessary files from the bootflash folder on the listed switch nodes so that the nodes can 5.0(1)
bootflash folder to download the image: download the target firmware image from APICs. You can SSH to the listed nodes and use “ls -l
/bootflash” to check the existing files, then “rm /bootflash/<filename>” to remove unnecessary files.
Make sure that you delete files you know that are unnecessary such as unused switch firmware images.
Each pod should upgrade/downgrade spine Remove some spine nodes from the current group to upgrade/downgrade spine nodes in two separate 5.0(1)
nodes with at least two separate groups to avoid groups. Make sure that one group do not have all spine nodes with IPN or ISN connectivity, or all route
traffic loss. All spines in the following pod are reflector spine nodes.
part of same maintenance group:
Following nodes are not compatible with the Remove the listed nodes from the group to avoid the upgrade/downgrade of such nodes. However, it is 5.0(1)
target version: recommended to replace those nodes with the compatible hardware prior to any upgrade/downgrade to
run a consistent firmware version on all nodes.
Remote leaf is not supported in the target Decommission all remote leaf nodes from the fabric. Remote leaf nodes will not function in the fabric 5.0(1)
version. This fabric has the following nodes as with the target firmware version.
remote leaf:
Tier 2 leaf is not supported in the target version. Decommission all Tier-2 leaf nodes from the fabric. Multi-Tier topology with Tier-2 leaf nodes will not 5.0(1)
The fabric has the following nodes as Tier 2 leaf: function in the fabric with the target firmware version.
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
Pre-upgrade validations (APIC and Switch inconsistency)
APIC Upgrade
The pre-upgrade validation view for APIC upgrade
still uses an old format without the recommended
action. This is planned to be fixed in JMR (5.0(2))
via CSCvu26641.
Switch Upgrade
Reason and
Recommended Action
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Confidential
