Hyperconverged Infrastructure - VxRail

Upgrade Troubleshooting Guide

Version 1.0
07/08/2019

Dell Internal Communication - Confidential

 Dell EMC- VxRail Upgrade Troubleshooting Guide Table of Contents

Date Description Created by

July 2019 Draft – version 1.0 Ahmed Abouelella and Amr
Elbehairy
August - Included the pre-checks needed prior Ahmed Abouelella and Amr
2019 starting the upgrade Elbehairy
- Mention the KBs of the most known
issues
- Attached the upgrade paths and
release notes
- Attached the upgrade stages and
their common issues
- Health checks and the password
verifications are included

UM Version 1.0 ii <Project and release name>

Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide Table of Contents

UM Version 1.0 iii <Project and release name>

Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide Table of Contents

Table of Contents
1. Introduction .............................................................................................................. 5
2. Upgrade Stages........................................................................................................ 6
2.1 LCM Introduction............................................................................................... 6
2.2 Uploading and Unpacking Composite Bundle ................................................... 6
2.2.1 Upload and Unpack common Issues .......................................................... 6
2.3 Upgrade VxRail Manager .................................................................................. 9
2.3.1 Upgrade VxRail Manager Common issues ................................................. 9
2.4 Upgrade Platform Service Controller and vCenter .......................................... 10
2.4.1 Platform Service Controller and vCenter common issues ......................... 10
2.5 Upgrade ESXi ................................................................................................. 13
2.5.1 Service Module (iSM) ............................................................................... 13
2.5.2 DellPTagent .............................................................................................. 14
2.5.3 Platform Service ....................................................................................... 14
2.5.4 Firmware ................................................................................................... 15
2.5.5 ESXi and vibs ........................................................................................... 15
2.5.6 ESXi and vibs Common issues, ................................................................ 15
3. Upgrade pre-checks .............................................................................................. 17
3.1 Upgrade Paths ................................................................................................ 17
3.2 System compatibilities .................................................................................... 21
3.2.1 NSX Compatibilities .................................................................................. 21
3.2.2 VDP .......................................................................................................... 24
3.2.3 RP4VM (Recover Point for VM) ................................................................ 25
3.3 Unsupported CPUs ......................................................................................... 26
3.4 Resources utilization ....................................................................................... 28
3.4.1 Compute and space utilization .................................................................. 28
3.5 Health checks ................................................................................................. 30
3.6 Password verification checks .......................................................................... 33
3.6.1 vCenter / VCSA VM .................................................................................. 33
3.6.2 PSC VM .................................................................................................... 33
3.6.3 VxRail Manager ........................................................................................ 34
3.6.4 Host / Node ............................................................................................... 34
3.7 DNS Check ..................................................................................................... 34
3.8 Temporary IP check ........................................................................................ 35
3.9 Upgrade bundle Checksum ............................................................................ 35
3.10 BOSS firmware check ..................................................................................... 36
3.11 Scripts ............................................................................................................. 36

UM Version 1.0 iv <Project and release name>

Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

1. Introduction
Dell EMC- VxRail Upgrade Troubleshooting Guide is a document to assist in
troubleshooting the failures occurs while upgrading the VxRail cluster from a code
version to another. In this document, resolutions and workarounds for the upgrade
failures are proposed based on the official Dell EMC knowledge base articles and
documents, in addition to other resolutions that are used to resolve the similar issues
when happened in the VxRail systems previously.

VxRail versions on which the guide is applicable: v4.0.302, v4.0.402, v4.0.510,

v4.0.520, v4.0.521, v4.0.522, v4.0.523, v4.0.524, v4.5.225, v4.0.227, v4.5.229, v4.5.310,
v4.5.311, v4.5.314, v4.7.001, v4.7.110, v4.7.111, v4.7.211

Upgrade Guide v1.0 5 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

2. Upgrade Stages

2.1 LCM Introduction

VxRail LCM is the life cycle management module that provides the upgrade of VxRail system
components. It exposes a set of CLI commands as Python script for users to directly launch LCM
operations. It is delivered as a ZIP bundle.

2.2 Uploading and Unpacking Composite Bundle

In this stage VxRail LCM script checks for free space in VxRail Manager before upload starts, java code
LocalDiskChecker runs to check if there is a free space available . The VxRail Upload Bundle is uploaded
at /data/store2/lcm , VxRail Composite upload Bundle is also uploaded in the vSAN Datastore at
upgradeBundles , if there is not enough free space in the vSAN Datastore , lcm will through an IO Failure
, in addition , communication or permission issues will cause the same IO failure in VxRail GUI

2.2.1 Upload and Unpack common Issues

Article Number 000497025

VxRail: "Not enough free disk space for upgrade" during VxRail upgrade
https://support.emc.com/kb/497025

Article Number 000534769

VxRail: Upgrade Bundle upload failed with Error : - "Not enough free disk space for upgrade.
390 MB more disk space is required" when doing internet upgrade
https://support.emc.com/kb/534769

Article Number 000535499

Dell EMC VxRail: Upgrade failed to upload upgrade bundle: with exception Not enough free disk
space for upgrade, even with free space
https://support.emc.com/kb/535499

Article Number 000535456

Dell EMC VxRail: upload bundle failure at 60%
https://support.emc.com/kb/535456

Upgrade Guide v1.0 6 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

Article Number 000516865

VxRail: Trying to upgrade from 4.0.200 to 4.0.400 and when local upload just disappears at 0%
https://support.emc.com/kb/516865

Article Number 000498669

Upload of the file to perform the upgrade fails with error message: "Failed to upload bundle:
VXRAIL_COMPOSITE-4.0.132-5128179 .zip
https://support.emc.com/kb/498669

Article Number 000527894

Dell EMC VxRail: Upgrade upload fails with error:"There are some disk replacement operations
running, please try again later."
https://support.emc.com/kb/0527894

Article Number 000527745

VxRail: Composite Upload fails with "The object 'vim.HostSystem:host-xx' has already been
deleted or has not been completely created"
https://support.emc.com/kb/527745

Article Number 000521498

VxRail: Failed to upload bundle: I/O failure
https://support.emc.com/kb/521498

Article Number 000506099

VxRail: Failed to upload bundle: VXRAIL_COMPOSITE-XXX.zip with exception
https://support.emc.com/kb/506099

Article Number 000511718

VxRail: Failed to upload bundle VXRAIL_COMPOSITE-xxxxx.zip with exception error while
unzipping the composite bundle file
https://support.emc.com/kb/511718

Upgrade Guide v1.0 7 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

Article Number 000494514

VxRail: Upgrade hangs during uploading without any return message
https://support.emc.com/kb/494514

Article Number 000493138

VxRail: Uploading stuck during system upgrade retry
https://support.emc.com/kb/493138

Article Number 000535536

Dell EMC VxRail: Uploading the upgrade bundle is stuck and after a while it fails with no error
when proxy is enabled (Customer Correctable)
https://support.emc.com/kb/535536

Article Number 000513903

vxRail: Failed to upload bundle: VxRailManagerUpgrade-4.0.100-4907696.bin with exception:
failed to write tmp
https://support.emc.com/kb/513903

Article Number 000528359

VxRail: Upgrade Failed due to "Failed to upload bundle:VXRAIL_COMPOSITE-4.x.x-
x_4.x.zip:null."
https://support.emc.com/kb/513903

Article Number 000531054

Dell EMC VxRail: Upgrade Composite Bundle fails to upload with error "result returns more than
one elements"
https://support.emc.com/kb/531054

Article Number 000517613

Dell EMC VxRail: Upgrade to 4.0.401 or 4.5.150 hung at 80% of unpacking composite bundle
with: Enabling HTTP on host, then ERROR: VxRail Platform Service is not installed/ready yet

Upgrade Guide v1.0 8 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

https://support.emc.com/kb/517613

Article Number 000527493

VxRail: Uploading upgrade bundle stuck at 80% while getting firmware
https://support.emc.com/kb/527493

2.3 Upgrade VxRail Manager

Usually upgrading VxRail Manager component goes smooth, However, In some scenarios, after
retying upgrade with HAS_NEWER as a part of troubleshooting it fails, especially if the VxRail
Manager is already upgraded to the newer version, that’s why taking a snapshot of VxRail
Manger is a must prior initiating upgrade.

2.3.1 Upgrade VxRail Manager Common issues

Article Number 000530145

VxRail: Upgrade shows VxRail Manager component upgraded while it is not and might fail
subsequently with various errors
https://support.emc.com/kb/530145

Article Number 000534485

VxRail: Fail to upgrade VxRail Manager if the VM has gone through file-base backup and
restore
https://support.emc.com/kb/534485

Article Number 000502756

VxRail: The upgrade has failed in 80% with error: "Could not find internal representation of
VxRail Manager"
https://support.emc.com/kb/502756

Article Number 000502756

VxRail: The upgrade has failed in 80% with error: "Could not find internal representation of
VxRail Manager"
https://support.emc.com/kb/502756

Upgrade Guide v1.0 9 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

Article Number 000511738

VxRail: VxRail component's code levels are not matching the VxRail version.
https://support.emc.com/kb/511738

2.4 Upgrade Platform Service Controller and vCenter

Upgrading vCenter and PSC consist of two phases

Phase 1: The process will deploy a new PSC and VCSA and assign it with a temp IP address
that it is not used in the network click here for Temp IP prechecks
Phase 2 Once deployment completes it will migrate the data from the source system, then shut
down the old appliances, the IP address of the old VM will be configured on the newly deployed
appliance.

Notice: The above process will be observed during the upgrade if the target version of PSC and
VC are different generation.
Example: Upgrading from 6.0 to 6.5

2.4.1 Platform Service Controller and vCenter common issues

Article Number 000521538

VxRail: Upgrade to 4.5: Failed to pre-check virtual machine VMware vCenter Server Platform
Services Controller 6.5 meets upgrade
https://support.emc.com/kb/521538

Article Number 000525028

VxRail: PSC Deployment fails during upgrade with TimeoutException error on lcm.log
https://support.emc.com/kb/525028

Article Number 000521158

Dell EMC VxRail: Upgrade failed with an error " Failed to create management account on
VMware vCenter Server Platform Services Controller 6.5"
https://support.emc.com/kb/521158

Upgrade Guide v1.0 10 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

Article Number 000523146

VxRail: Upgrade fails when verifying root credential of PSC/VCSA after upgrade of PSC/VCSA
https://support.emc.com/kb/523146

Article Number 000534816

Dell EMC VxRail: LCM upgrade from 4.5.225 to 4.7.100 failed with error "Failed to verify
memory and disk resource requirements for VCSA and PSC deployment."
https://support.emc.com/kb/534816

Article Number 000519205

VxRail: Upgrade to 4.0.400/401 failed due to DNS service abnormal if internal vCenter applied.
https://support.emc.com/kb/519205

Article Number 000519205

VxRail: Composite bundle upgrade fails on validation while trying to log to vCenter and PSC
VMs using root account
https://support.emc.com/kb/519205

Article Number 000535417

Dell EMC VxRail: Failed to upgrade VxRail from version 4.5.x to version 4.7.x with error "Unable
to connect to VC"
https://support.emc.com/kb/535417

Article Number 000531875

Dell EMC VxRail: Upgrade Fails with "Host credential empty" Error When Upgrading vCenter or
PSC From 4.0 to 4.5 or 4.5 to 4.7
https://support.emc.com/kb/531875

Article Number 000533687

VxRail : Upgrade of PSC failing with error message "Make sure the patch iso is a valid one"
https://support.emc.com/kb/533687

Article Number 000525023

VxRail: Upgrade from 4.0.301 to 4.5.211 failed while deploying PSC 6.5 with Timeout exception

Upgrade Guide v1.0 11 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

https://support.emc.com/kb/525023

VxRail: Failed to pre-check virtual machine VMware vCenter Server Platform Services
Controller 6.5 meets upgrade requirements
Article Number 000520723
https://support.emc.com/kb/520723

Article Number 000531875

Dell EMC VxRail: Upgrade Fails with "Host credential empty" Error When Upgrading vCenter or
PSC From 4.0 to 4.5 or 4.5 to 4.7
https://support.emc.com/kb/531875

Article Number 00049637

VxRail: Upgrade fails with a message like: No Right permissions for vCenter Server Appliance
administrator account
https://support.emc.com/kb/496379

Article Number 000535056

VxRail: VxRail update fails with Failed to pre-check virtual machine VMware vCenter Server
Appliance meets upgrade requirements.
https://support.emc.com/kb/535056

Article Number 000530895

Dell EMC VxRail: vCenter upgrade got stuck with error : Failed-to-pre-check-virtual-machine-
VMware-vCenter-Server-Appliance-OOSMNKy-meets-upgrade-requirements (bypass firewall if
exiting)
https://support.emc.com/kb/530895

Article Number 000497660

VxRail: Upgrade fails around 17% with Pre-upgradehook for vCenter
https://support.emc.com/kb/497660

Upgrade Guide v1.0 12 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

Article Number 000530546

Dell EMC VxRail: Upgrade fails with "Incorrect permissions for VMware vCenter Server
Appliance administrator account"
https://support.emc.com/kb/530546

2.5 Upgrade ESXi

There are three main components that contribute in upgrading ESXi , any communication issues
on those components will cause an upgrade failure.

2.5.1 Service Module (iSM)

The iDRAC Service Module (iSM) is a small OS-resident process that expands iDRAC
management into supported host operating systems. Specifically, iSM adds the following
services:
OS information
Lifecycle Controller Log replication into operating system
Automatic system recovery
Windows Management Instrumentation (WMI) providers inclusive of storage data
SupportAssist collection (formerly known as Automatic Technical Support Report Collection)
NVMe PCIe SSD device removal support

iSM is the primary connection between iDRAC and ESXi , failure in communication will affect
the functionality of Dell PTagent

Upgrade Guide v1.0 13 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

2.5.2 DellPTagent

Article Number 000498849

VxRail: Troubleshooting guide for the Dell PTAgent
https://support.emc.com/kb/498849

2.5.3 Platform Service

Article Number 000517433

VxRail: Troubleshooting guide for Platform Service
https://support.emc.com/kb/517433

2.5.3.1 (iSM, Platform Service , DellPTagent ) Common issues

Article Number 000535203
Dell EMC VxRaill : Upgrade stuck at ESXi checking platform service
https://support.emc.com/kb/535203

Article Number 000535513

Dell EMC VxRail: Failure occurred during upgrade with error message "VxRail Platform Service
on host is not installed/ready."
https://support.emc.com/kb/535513

Article Number 000526229

VxRail: Upgrade to 4.5.218 fails due to platform service that wasn't initialized
https://support.emc.com/kb/526229

Article Number 000535571

Dell EMC VxRail: Upgrade failed on host due to VxRail Agent initialized is false, Service Module
is not running (Customer Correctable)
https://support.emc.com/kb/535571

Upgrade Guide v1.0 14 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

2.5.4 Firmware

Upgrading Firmware hardware component will require the host to reboot, it is always a best
practice to monitor firmware upgrade process by logging to the iDRAC virtual console to verify
that the firmware upgrade is going smooth.

2.5.4.1 Firmware Upgrade issues

Article Number 000534271

VxRail: Upgrade to 4.7.200 may fail when upgrading Boss firmware on Dell 14G
https://support.emc.com/kb/534271

Article Number 000525871

VxRail: Upgrade Failure with error UEFI0305: Unable to update BIOS firmware in the previous
boot because of some issues
https://support.emc.com/kb/525871

2.5.5 ESXi and vibs

Failure in those components are more of a VMware issue, troubleshooting from VMware side
should be considered.
Logs: esxupdate.log

2.5.6 ESXi and vibs Common issues,

Article Number 000498717

VxRail: VxRail upgrade to 4.0.132 fails during VIB task
https://support.emc.com/kb/498717

Upgrade Guide v1.0 15 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

Article Number 000516950

VxRail: Trying to upgrade from 4.0.200 to 4.0.400, Failed to obtain Vxrail Vib version info for
host: ESXi-005
https://support.emc.com/kb/516950

Article Number 000511032

VxRail: VIB upgrade fails to remove '/tardisks/vmware_m.v00'
https://support.emc.com/kb/511032

Article Number 000525732

VxRail: Upgrade fail with error: Failure occurred while running an upgrade for bundle:
VXRAIL_COMPOSITE-4.5.218-9630256_for_4.5.x.zip. The error message: {"VIB": {"Message":
" [InstallationError] [Errno 32] Broken pipe vibs = VMware_locker_tools-light_6.5
https://support.emc.com/kb/511032
Article Number 000533761
Dell EMC VxRail: Upgrade fails with message "Failure occurred while running an upgrade for
bundle: ESXi. The error message: ESXi command failed with 2."
https://support.emc.com/kb/533761

Upgrade Guide v1.0 16 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

3. Upgrade pre-checks

3.1 VxRail Code Versions Release Notes

Release Note 4.0.x
Release Note 4.5.x
Release Note 4.7.x

3.2 Upgrade Paths

For VxRail 4.0.x upgrades:

Upgrade Guide v1.0 17 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

For VxRail 4.5.x upgrades:

Upgrade Guide v1.0 18 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

Upgrade Guide v1.0 19 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

For VxRail 4.7.x upgrades:

Upgrade Guide v1.0 20 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

3.3 System compatibilities

Be aware of special considerations when upgrading to 4.7 code family:

• NSX
• RP4VM's
• VDP (VMware Data Protection)

3.3.1 NSX Compatibilities

Does the customer have NSX (Network & Security) on the system?

Before you upgrade VxRail to 4.7 please make sure customer has
upgraded the NSX to the compatible version in case it causes
downtime.

This can be checked from vSphere Web Client by navigating to “Home”

If NSX plugin (as shown in the figure below) is exist, then the customer is having NSX.

Upgrade Guide v1.0 21 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

There are two types of NSX:

• NSX for vSphere (NSX-V)
• NSX-Transformer (NSX-T)

If the customer is having NSX-T and he wants to upgrade the VxRail cluster, then the
compatibility check is needed.

To check if the customer is having NSX-T, you have to run debug_vxrm-vc.sh script on VxRail
Manager CLI.

The below is the output for an environment have NSX

NSX extension found (com.vmware.vShieldManager)

NSX version: 6.2.7.5343628

No VDP extension registered

No vSphere Replication extension registered

No vCloud Director extension registered

RPVM extension found (com.emc.recoverpoint)

RPVM version: 5.0.1.2

Upgrade Guide v1.0 22 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

No VxRail Manager plugin registered

If the customer is having NSX-T please check the compatibility of vCenter and NSX. Click
here to check the compatibility.

If Customer’s VxRail cluster has been installed with NSX

(6.4.2 & 6.4.3 & 6.4.4), DON’T proceed the upgrade to 4.7.x
due to compatible issues

To check the current NSX version:

• From NSX Manager CLI

• From NSX Manager GUI

Upgrade Guide v1.0 23 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

• From vCenter Web Client

3.3.2 VDP
On April 5th, 2017, VMware announced the End of Availability (EOA) of the VMware vSphere
Data Protection (VDP) product. VMware vSphere 6.5 is the last release to include vSphere Data
Protection and future vSphere releases will no longer include this product. Therefore
vSphere/vCenter 6.7 do not include support for VDP .

Please use the debug_vxrm-vc.sh script to determine if the customer is using VDP and wants
to upgrade to VxRail 4.7.x code.

Upgrade Guide v1.0 24 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

Checking for known vCenter extensions....

No NSX-V extension registered
No NSX-T extension registered [manual check for NSX-T recommended as
VC is not required by NSX-T]
VDP extension found (com.vmware.vdp)
VDP version: 6.1.3.70
No vSphere Replication extension registered
No vCloud Director extension registered
No RPVM extension registered
No VxRail Manager plugin registered

Refer to KB https://kb.vmware.com/s/article/2149614
Support Matrix:
https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#interop&68=&2=
Dell EMC KB: https://support.emc.com/kb/536118

Customer will need to need to migrate their backup before

upgrading to VxRail 4.7/ vCenter 6.7.

The following link contains a VMware/DellEMC FAQ on transitioning.

https://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/products/vsphere/vmw-
vdp-eoa-faqs.pdf

Customers should contact their local team and/or Sales representative for the latest information.

3.3.3 RP4VM (Recover Point for VM)

In vCenter Web Client, select the Home section and verify if RecoverPoint for VM's is
configured. Alternatively you can ask the customer if RP4VM's is being used.
Before upgrading to VxRail 4.7.x the RP4VM's cluster(s) must be upgraded to 5.2 Patch 4 first

1. Navigate to Home on the vSphere web client

2. Make sure the ‘Home’ page is shown

3. Look for ‘RecoverPoint for VM’s’ under ‘Inventories’ section

Upgrade Guide v1.0 25 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

4. If RP4VM is present and customer wants 4.7.X – RP4VM's cluster(s) must be upgraded
to 5.2 Patch 4 first [5.2.0.4] [Customer Upgradeable not support]

5. If RP4VM is NOT present – proceed as normal.

3.4 Unsupported CPUs

Comparing the processors supported by vSphere 6.5, vSphere 6.7 no longer supports the
following processors:
• AMD Opteron 13xx Series
• AMD Opteron 23xx Series
• AMD Opteron 24xx Series
• AMD Opteron 41xx Series
• AMD Opteron 61xx Series
• AMD Opteron 83xx Series
• AMD Opteron 84xx Series
• Intel Core i7-620LE Processor

Upgrade Guide v1.0 26 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

• Intel i3/i5 Clarkdale Series

• Intel Xeon 31xx Series
• Intel Xeon 33xx Series
• Intel Xeon 34xx Clarkdale Series
• Intel Xeon 34xx Lynnfield Series
• Intel Xeon 35xx Series
• Intel Xeon 36xx Series
• Intel Xeon 52xx Series
• Intel Xeon 54xx Series
• Intel Xeon 55xx Series
• Intel Xeon 56xx Series
• Intel Xeon 65xx Series
• Intel Xeon 74xx Series
• Intel Xeon 75xx Series

Please reference the following page for more details: https://docs.vmware.com/en/VMware-

vSphere/6.7/rn/vsphere-esxi-vcenter-server-67-release-notes.html

If you are not able to access a vCenter UI and need to run the check of CPU type via CLI please
use the following command on each host or on all hosts via vxtool
vim-cmd hostsvc/hostsummary | grep cpuModel

Example output below:

Upgrade Guide v1.0 27 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

cpuModel = "Intel(R) Xeon(R) Bronze 3104 CPU @ 1.70GHz"

3.5 Resources utilization

This shows how much resources will be available when a host is rebooted during the upgrade.

Recommendations:

• Space usage in the time before upgrade with no failures should be 75% or less.
• Space usage after failure/re-protection should be 90% or less.

Exceeding these values increases the chances of a resync occurring during the upgrade and
will cause extended delays while waiting for the resyncs to complete.
Customers can try and free up space by deleting snapshots or moving VM's off the VxRail
before the upgrade otherwise a capacity upgrade [add more disks] would be required.

The utilization checks should be performed for both compute and space resources and this is
illustrated with the steps in the below sections.

3.5.1 Compute and space utilization

VCSA CLI What If Host Failure Check:

Command> rvc

Install the "ffi" gem for better tab completion.

Host to connect to (user@host): localhost

{if this fails try [email protected]@localhost}

Using default username "[email protected]".

password:

0 /

1 localhost/

> ls

0 /

Upgrade Guide v1.0 28 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

1 localhost/

> cd 1

/localhost> ls

0 MARVIN-Datacenter (datacenter)

/localhost> cd 0

/localhost/MARVIN-Datacenter> ls

0 storage/

1 computers [host]/

2 networks [network]/

3 datastores [datastore]/

4 vms [vm]/

/localhost/MARVIN-Datacenter> cd 1

/localhost/MARVIN-Datacenter/computers> ls

0 Production-Cluster

1 MARVIN-Virtual-SAN-Cluster-0df82f82-4860-4728-9aa0-6257f52a86d5 (cluster):
cpu 82 GHz, memory 307 GB

/localhost/MARVIN-Datacenter/computers> vsan.whatif_host_failures 1 -s

Note: Please use the number shown at the start of the line "*-Virtual-SAN-Cluster-UUID"
in the command.

In this case the value is a '1' but can vary between each VxRail setup and whether
customer has renamed the cluster.

Current utilization of hosts:

+----------------------+---------+--------------+------+----------+-----------------+--------------+
| | | HDD Capacity | | | Components | SSD Capacity |
| Host | NumHDDs | Total | Used | Reserved | Used | Reserved |
+----------------------+---------+--------------+------+----------+-----------------+--------------+
| g3node01.vspex.local | 2 | 3576.98 GB | 20 % | 15 % | 73/9000 (1 %) | 0 % |
| g3node03.vspex.local | 2 | 3576.98 GB | 20 % | 14 % | 73/9000 (1 %) | 0 % |
| g3node02.vspex.local | 2 | 3576.98 GB | 26 % | 23 % | 73/9000 (1 %) | 0 % |

Upgrade Guide v1.0 29 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

+----------------------+---------+--------------+------+----------+-----------------+--------------+

Simulating 1 host failures:

+-----------------+-----------------------------+-----------------------------------+
| Resource | Usage right now | Usage after failure/re-protection |
+-----------------+-----------------------------+-----------------------------------+
| HDD capacity | 22% used (8370.69 GB free) | 33% used (4793.71 GB free) |
| Components | 1% used (26781 available) | 1% used (17781 available) |
| RC reservations | 0% used (0.00 GB free) | 0% used (0.00 GB free) |
+-----------------+-----------------------------+-----------------------------------+

Compute (CPU and memory) resources:

In order to ensure successful upgrade process, the available compute resources in the cluster
should tolerate placing at least one host in maintenance mode.
As recommended by VMware, there should be 30% or more free resources available for cluster
management, maintenance activities and High Availability fail-over.
The below steps can help in examining the cluster if it will allow the ESXi host to be placed in
maintenance mode or not.

1- Change the ClomDelay value of one host to 360 instead of 60 to avoid triggering a
resync.

2- Place one host in maintenance mode with “Ensure accessibility” option.

3- If the host is placed successfully in maintenance mode, this means that there are
sufficient compute resources to place hosts in maintenance mode.

4- Take the host back out of maintenance mode.

5- In case of failures while placing the host in maintenance mode because of insufficient
resources, a downtime is needed to power off some VMs to proceed with the upgrade.

3.6 Health checks

The data health should be checked both from vCenter CLI and vSphere web client.

1- SSH to vCenter CLI

2- Type “rvc” and follow the below steps :

/localhost/MARVIN-Datacenter/computers> ls

Upgrade Guide v1.0 30 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

0 Production-Cluster

1 MARVIN-Virtual-SAN-Cluster-0df82f82-4860-4728-9aa0-6257f52a86d5 (cluster):
cpu 82 GHz, memory 307 GB
/localhost/MARVIN-Datacenter/computers> vsan.health.health_summary 1

Overall health: yellow (vSAN VUM integration issue)

+------------------------------------------------------+---------+
| Health check | Result |
+------------------------------------------------------+---------+
| vSAN Build Recommendation | Warning |
| vSAN Build Recommendation Engine Health | Warning |
| vSAN build recommendation | Passed |
+------------------------------------------------------+---------+
| Network | Passed |
| Hosts disconnected from VC | Passed |
| Hosts with connectivity issues | Passed |
| vSAN cluster partition | Passed |
| All hosts have a vSAN vmknic configured | Passed |
| All hosts have matching subnets | Passed |
| vSAN: Basic (unicast) connectivity check | Passed |
| vSAN: MTU check (ping with large packet size) | Passed |
| vMotion: Basic (unicast) connectivity check | Passed |
| vMotion: MTU check (ping with large packet size) | Passed |
| Network latency check | Passed |
+------------------------------------------------------+---------+
| Physical disk | Passed |
| Overall disks health | Passed |
| Metadata health | Passed |
| Disk capacity | Passed |
| Software state health | Passed |
| Congestion | Passed |
| Component limit health | Passed |
| Component metadata health | Passed |
| Memory pools (heaps) | Passed |
| Memory pools (slabs) | Passed |
+------------------------------------------------------+---------+
| Data | Passed |
| vSAN object health | Passed |
+------------------------------------------------------+---------+
| Cluster | Passed |
| ESXi vSAN Health service installation | Passed |
| vSAN Health Service up-to-date | Passed |
| Advanced vSAN configuration in sync | Passed |
| vSAN CLOMD liveness | Passed |
| vSAN Disk Balance | Passed |
| Resync operations throttling | Passed |
| vCenter state is authoritative | Passed |
| vSAN cluster configuration consistency | Passed |
| Time is synchronized across hosts and VC | Passed |
| vSphere cluster members match vSAN cluster members | Passed |
| Software version compatibility | Passed |
| Disk format version | Passed |
+------------------------------------------------------+---------+
| Limits | Passed |
| Current cluster situation | Passed |

Upgrade Guide v1.0 31 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

| After 1 additional host failure | Passed |

| Host component limit | Passed |
+------------------------------------------------------+---------+
| Hardware compatibility | Passed |
| vSAN HCL DB up-to-date | Passed |
| vSAN HCL DB Auto Update | Passed |
| SCSI controller is VMware certified | Passed |
| Controller is VMware certified for ESXi release | Passed |
| Controller driver is VMware certified | Passed |
| Controller firmware is VMware certified | Passed |
| Controller disk group mode is VMware certified | Passed |
+------------------------------------------------------+---------+
| Performance service | Passed |
| Stats DB object | Passed |
| Stats master election | Passed |
| Performance data collection | Passed |
| All hosts contributing stats | Passed |
| Stats DB object conflicts | Passed |
+------------------------------------------------------+---------+
| Online health | skipped |
| Customer experience improvement program (CEIP) | skipped |
+------------------------------------------------------+---------+

Details about any failed test below ...

vSAN Build Recommendation - vSAN Build Recommendation Engine Health: yellow
+--------------------------------------------------------------------------
--------------------------------------------+
| Issue
Information
|
+--------------------------------------------------------------------------
--------------------------------------------+
| Internet access is unavailable. Please restore Internet connectivity for
vSAN to come up with build recommendations. |
+--------------------------------------------------------------------------
--------------------------------------------+

One more hralth check on vCenter CLI is the oject health check, to check the validation of
objects to proceed with the upgrade.

1- Open “rvc”
2- Follow the below steps

/localhost/MARVIN-Datacenter/computers> ls
0 Production-Cluster

Upgrade Guide v1.0 32 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

1 MARVIN-Virtual-SAN-Cluster-0df82f82-4860-4728-9aa0-6257f52a86d5 (cluster): cpu 82 GHz,

memory 307 GB
/localhost/MARVIN-Datacenter/computers> vsan.upgrade_status 1

2019-01-07 12:19:22 +0000: No upgrade in progress

2019-01-07 12:19:22 +0000: 108 objects are with latest version (v7)
2019-01-07 12:19:22 +0000: 0 objects are not upgraded to latest
version

3.7 Password verification checks

Please perform a password check to verify all passwords are working and are not expired
Use the following guide to test all the passwords

3.7.1 vCenter / VCSA VM

Take the IP / FQDN from the URL on the browser page for vSphere Web Client
Please verify that the administrator Single-Sign-On account is logging in to the web UI
successfully as well as the root account from the CLI.

If the root account is not logging in successfully, click here for the article to reset it.

If SSH service is not enabled, click here for the procedure to enable it

3.7.2 PSC VM

Select the PSC from the list of VM’s and navigate to summary, copy the IP from the details pane.
User Account required:
• Root
SSH to the node and try to login using the root account of PSC. If the root account is not logging
in successfully, click here for the article to reset it.

If SSH service is not enabled, click here for the procedure to enable it

Upgrade Guide v1.0 33 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

3.7.3 VxRail Manager

Take the IP / FQDN from the URL of the VXRM web page or retrieve using same instructions as for PSC
VM.
User Account required:
• mystic AND root

SSH to the VxRail Manager VM and try to login using the mystic account of VxRail Manager.
The default mystic password is VxRailManager@201602!

Note: The mystic password might be changed by the customer.

If the mystic account is locked due to multiple incorrect login attempts, click here for the
unlock steps
After logging in to the CLI with the mystic account, run the below command to test the root
password login.

su root

If the root account is not logging in successfully, click here for the article to reset it.

3.7.4 Host / Node

Select a host from the list given on the cluster, on the top of the details there should be an
FQDN for the host – select the output and copy (ctrl + c)
User Account required:
• root
SSH to the Host and try to login using the root account.
If the SSH is not enabled for the hosts, follow the below steps to enable SSH service

3.8 DNS Check

VxRail Manager v4.5 and higher does not support internal DNS. Please prepare for and supply
your external DNS prior to upgrading to v4.5.x.

• Run Vxtool to get the DNS IP address, compare this to the vCenter Server IP. If they
match then the customer is using the vCenter DNS Server.

• Please inform customer to supply the external DNS before upgrading to v4.5.x code.

Upgrade Guide v1.0 34 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

3.9 Temporary IP check

A temporary IP Address on the same subnet is needed to upgrade if Customer is using internal
vCenter server.

Please inform Customer to provide a temporary IP address before upgrade start.

(must be in the same network segment with VxRail Manager)

Try to ping the temporary IP provided by the customer to confirm that is not currently used and
please confirm with customer that this IP was not assigned to any decommissioned component.

3.10 Upgrade bundle Checksum

Checksum is used to verify that the upgrade bundle is not corrupted or wrongly downloaded.
Click here to access the knowledge base article for the procedure to check the bundle
validation.

Upgrade Guide v1.0 35 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential
 Dell EMC- VxRail Upgrade Troubleshooting Guide

3.11 BOSS firmware check

Use the vxtool [Option 3] and run the following command on all nodes to get the current
firmware level.

esxcli vxrail firmware list | grep boss

This should return the BOSS Firmware level for any 14G nodes. If firmware is below
2.5.13.3011 then it needs to be upgraded BEFORE the VxRail upgrade is completed.

Some code bundles do not include a BOSS update.

3.12 Scripts

• Checker.py

• debug_vxrm-vc.sh

• fixed_local_cert.sh

• preupgrade_check.sh

• supportool-0.3.2.py

• vsan_issue_check.sh

• VSANDTA525859.py

• vxtool_v3

Upgrade Guide v1.0 36 EMC Confidential Information- for internal use only
Dell Internal Communication - Confidential