PowerScale - Isilon - HD400-CTO Hardware Upgrade Checklist
PowerScale - Isilon - HD400-CTO Hardware Upgrade Checklist
Topic
HD400
Selections
HD400 - Select CTO Node Activity: CTO Hardware Upgrade Checklist
Serial Number(s) or Service Tag(s): xya
SR Number(s): 12342
Company Location: STCS
REPORT PROBLEMS
If you find any errors in this procedure or have comments regarding this application, send email to
[email protected]
Copyright © 2024 Dell Inc. or its subsidiaries. All Rights Reserved. Dell Technologies, Dell, EMC, Dell
EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be
trademarks of their respective owners.
The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of
any kind with respect to the information in this publication, and specifically disclaims implied warranties of
merchantability or fitness for a particular purpose.
Use, copying, and distribution of any software described in this publication requires an applicable
software license.
This document may contain certain words that are not consistent with Dell's current language guidelines.
Dell plans to update the document over subsequent future releases to revise these words accordingly.
This document may contain language from third party content that is not under Dell's control and is not
consistent with Dell's current guidelines for Dell's own content. When such third party content is updated
by the relevant third parties, this document will be revised accordingly.
Page 1 of 14
Contents
Preliminary Activity Tasks .......................................................................................................3
Read, understand, and perform these tasks.................................................................................................3
Known Issues........................................................................................................................12
Upgrade script and error handling output ...................................................................................................12
Empty node pools .......................................................................................................................................13
Document Change Management Version Control ......................................................................................13
Page 2 of 14
Preliminary Activity Tasks
This section may contain tasks that you must complete before performing this procedure.
Table 1 List of cautions, warnings, notes, and/or KB solutions related to this activity
Page 3 of 14
General Information for Removing and Installing FRUs
This section describes precautions you must take and general procedures you must follow when
removing, installing, or storing field-replaceable units (FRUs). The procedures in this section apply to FRU
handling during hardware upgrades as well as during general replacement.
FRUs are designed to be powered up at all times. This means you can accomplish FRU replacements
and most hardware upgrades while the cabinet is powered up. To maintain proper airflow for cooling and
to ensure EMI compliance, make sure all front bezels, filler panels, and filler modules are reinstalled after
the FRU replacement or hardware upgrade is completed.
IMPORTANT: These procedures are not a substitute for the use of an ESD kit. You should follow them
only in the event of an emergency.
• Before touching any FRU, touch a bare (unpainted) metal surface of the enclosure.
• Before removing any FRU from its antistatic bag, place one hand firmly on a bare metal surface of the
enclosure, and at the same time, pick up the FRU while it is still sealed in the antistatic bag. Once you
have done this, do not move around the room or contact other furnishings, personnel, or surfaces
until you have installed the FRU.
Page 4 of 14
• When you remove a FRU from the antistatic bag, avoid touching any electronic components and
circuits on it.
• If you must move around the room or touch other surfaces before installing a FRU, first place the
FRU back in the antistatic bag. When you are ready again to install the FRU, repeat these
procedures.
Page 5 of 14
This document presents the steps that project team members should complete before, during, and after a
CTO hardware upgrade. It is important to review this document in its entirety prior to performing any of
the steps.
Page 6 of 14
If the node is CTO-capable, output similar to the following is displayed:
Note: tool intended for use on CTO capable nodes and this node is CTO
capable.
For nodes that are not CTO capable (SKU-based nodes), you must install the CTO conversion
package. For instructions, see the "Non-CTO to CTO Node Conversion" procedure in the
SolVe Desktop.
9 � SA (Best Practice)
Gather the logs by running the isi_gather_info command. Review the as-built records in
the logs to verify the current CTO configuration for the nodes. These logs provide Support with
a baseline configuration of the cluster and will be necessary if issues arise.
Run the isi status -w command to view the status of the cluster and active jobs. Pause
jobs for the duration of the maintenance window.
Run the following commands to view the status and configuration information for the RAM,
drives, and network interfaces that are currently installed. The output should match the current
CTO configuration. Capture this pre-upgrade information to refer to after the upgrade is
complete.
isi_hw_status -iIc View the current hardware configuration, including
installed RAM.
isi_radish -q View the installed drives.
ifconfig | grep flags View the installed network interfaces
10 � SA (Best Practice)
Run the following command to display the upgrade receipt:
isi_inventory_tool --display --itemType=receipt
Compare the command output to the receipt file in the hardware upgrade package. The
receipt has a file name similar to <SO_number>_<Serial_Number>.xml. If the package is
already installed on the cluster, the receipt file can be found in
/var/crash/cto/hwPackages/isiHWPackage…/
You will see output similar to the following:
<receipt receiptFileName="30301230_SX4001013160081.xml"
receiptFormatVersion="1.2" date="2014-01-23T01:40:03Z">
<bom>
<sbb qty="1" desc="SBB, BASE, DDR, X400" part="610-0005-03"/>
<sbb qty="1" desc="SBB, 3.5" 66TB HDD / 1.2TB SSD EMLC" part="611-0075-02"/>
<sbb qty="1" desc="SBB, Memory, 48GB DDR3, 12x4G" part="612-0006-01"/>
<sbb qty="1" desc="SBB, 10GbE, Dual-port, SFP+, without Optics" part="613-
0002-01"/>
</bom>
</receipt>
Verify that the receipt file in the hardware upgrade package is consistent with the defined
upgrade path and hardware order. Verify the accuracy the Sales Building Blocks (SBBs),
including Base, Drive, RAM, and NIC.
If the receipt needs to be corrected, contact [email protected] to request a
change. For issues that are not related to the receipt, contact EMC Isilon Technical Support
instead.
Page 7 of 14
Prepare the Cluster for Upgrading
Complete these general preparatory steps before performing an upgrade. These steps will help you
determine if the cluster is in a healthy state beforehand, and if the hardware package is valid or needs to
be updated.
Upgrades
Review the General Guidelines prior to beginning the update process, then follow the checklist for the
particular type of upgrade that you are performing.
The hardware upgrade package should be installed prior to replacing the hardware. If the hardware is replaced first, it might not work correctly. For more information, see the "
Out-of-sequence Upgrades" section.
General Guidelines
Role Steps to complete
1 � All During the upgrade process, contact EMC Isilon Technical Support if any issues are
encountered.
2 � SA If new nodes need to be upgraded and are the same type as existing nodes, add them to
the cluster first, and then upgrade all the nodes together.
If new nodes match the post-configuration of the nodes to be upgraded, add the new
nodes after the CTO upgrade completes successfully and the cluster is healthy. This is to
prevent new node pools from being created before the CTO upgrade is finished.
3 � CE If the node has preexisting CTO configuration issues, they will be presented for review
during the pre-upgrade script execution. These configuration issues are frequently
cosmetic and relate to description changes and substitution rules in the CTO supporting
libraries. See Known Issues below for more information. You can accept the changes or
Page 8 of 14
engage EMC Isilon Remote Support for validation of the script output.
RAM Upgrades
Role Steps to complete
1 � CE RAM upgrades are "rip-and-replace." Remove all DIMMs and replace them with new
DIMMs to avoid hardware mismatches.
2 � SA Nodes with upgraded RAM that are running OneFS 7.0 and later will form their own pool.
SmartPools policies may need to be configured after the upgrade is completed.
NIC Upgrades
Role Steps to complete
1 � CE One NIC (ext-3/ext-4) is removed and replaced during the upgrade.
2 � SA Ensure ext-3/ext-4 1 GigE interfaces are removed from active pools before the upgrade
takes place.
Page 9 of 14
node is powered down.
Out-of-sequence Upgrades
Role Steps to complete
1 � CE In most cases, the hardware upgrade package should be installed prior to replacing the
hardware. If the hardware is replaced first, it might not work correctly until the hardware
upgrade package is installed.
For an out-of-sequence upgrade (when the hardware is already installed, but the
configuration is not yet updated), first review the hardware configuration to confirm that the
new hardware is installed and the configuration is not updated. To do this, run the following
commands to view the status and configuration information for the RAM, drives, and
network interfaces:
isi_hw_status -iIc View the current hardware configuration, including
installed RAM.
isi_radish -q View the installed drives.
ifconfig | grep flags View the installed network interfaces
2 � CE Run the pre-upgrade script. The output will warn about hardware mismatches compared to
the current receipt.
../isi_cto_upgrade --pre
When prompted to continue with the upgrade anyway, type y.
3 � CE Answer no to the prompt to shut down the node. Restart the node so that the new
configuration is recognized by the system.
4 � CE Run the post-upgrade script after the node restarts to complete the upgrade and generate
an ABR:
../isi_cto_upgrade --post
Post-Upgrade
Role Steps to complete
1 � SA After the upgrade is complete, review the cluster status and status of the node pools. This
can be done via the command-line or web administration interface.
SmartPools and file pool policies might need to be added or updated if a new tier was
created.
2 � CE Confirm that the configuration change was successful by running the following commands.
The expected hardware changes should be listed in the output.
isi_hw_status -iIc View the current hardware configuration,
including installed RAM.
isi_radish -q View the installed drives.
ifconfig | grep flags View the installed network interfaces
isi status -d View current storage pools
isi_inventory_tool --display -- View the hardware as-built record.
itemType=asbuilt
Page 10 of 14
isi_inventory_tool --display -- Confirm that the receipt was installed.
itemType=receipt
isi_inventory_tool --configCheck Perform a configuration check.
isi_cto_update --abr Verify the package integrity. Sends the
package report via SupportIQ.
3 � CE Run the isi_gather_info command to gather logs and send to EMC Isilon Technical
Support. This provides Support with the post-upgrade configuration information.
4 � CE Follow the standard EMC Installed Base (IB) debriefing process. Top-level assets need to
have updated SBBs to match the upgrade, and new part serial numbers need to be added
to the asset. As-Built Record contains the inventory of parts and serial numbers captured
after the upgrade. Supply As-Built Record xml to the IB, if necessary.
Installation Rollback
Complete these steps if you need to roll back the CTO hardware upgrade. In the event that a rollback is
necessary, you should contact EMC Isilon Technical Support for assistance.
When the new receipt is installed, a backup of the existing receipt is made on the boot drive partition. To
access and install one of the backup receipts, complete the following steps:
mount
mount /dev/mirror/mfg/mfg
cd /mfg/receipts
4. [ ] View receipts by date and determine which receipt you want to revert to:
ls -lan
cp <receipt.xml> /ifs/admin/receipts/
Page 11 of 14
isi_inventory_tool --import --fileType receipt --/ifs/admin/receipts/<receipt.xml>
Known Issues
Upgrade script and error handling output
The output for the upgrade script and subsequent error handling can be confusing. To help, a typical
script output is provided for reference. You can copy the hardware upgrade package anywhere on the
node (e.g. /ifs/admin). The script unpacks the content of the hardware package to a predetermined
location and runs an inventory check and comparison of the current configuration on the node and the
updated configuration as defined by the receipt.
Output similar to the following appears then the --pre command is run:
Verifying package integrity
Package integrity is good
Beginning pre-installation actions
Node CTO capability verified
Upgrade is for Order 30200245, Node S/N SX2003012123456
Chassis VPD verified
Node S/N verified
Node CTO configuration verified
Importing SoftAVL (./Soft-AVL_201304241555.xml)
Success
Importing local /etc/gconfig/probe_config.0.gc (/etc/gconfig/probe_config.0.gc)
Success
Importing local /etc/gconfig/probe_config.gc (/etc/gconfig/probe_config.gc)
Success
Importing package probe_config.0.gc (./probe_config.0.gc)
Success
Importing package probe_config.1.gc (./probe_config.1.gc)
"/usr/bin/isi_inventory_tool --importFile --fileType probeconfig --filePath
./probe_config.1.gc" failed
WARNING: "/mfg/probeconfigs/probe_config.1.gc" does not exist
cat: /etc/gconfig/probe_config.1.gc: No such file or directory
ERROR: Unable to cat "/etc/gconfig/probe_config.1.gc"
ERROR: Unable to get version of "/etc/gconfig/probe_config.1.gc"
Unable to import probe configuration file (./probe_config.1.gc)
Importing package probe_config.gc (./probe_config.gc)
Success
Importing receipt (./30200245_SX2003012123456.xml)
Success
CTO files imported
HAL re-initialized
"/usr/bin/isi_inventory_tool --configCheck" failed
ERROR: Configuration check FAILED!!!
Page 12 of 14
2) P/N="050-0025-01", Desc="PCA,PCI-E,GE DUAL PORT,82571EB", Qty="1", Vendor
ID="Intel(R) PRO/1000 Network Connection 6.9.6", Firmware="None"
Unable to run configuration check or configuration check failed
In the highlighted text in the example above, the current 1 GbE NIC configuration is listed as
"unexpected," and the new 10 GbE is listed as "missing." This configuration disparity is expected. You
can accept the changes and proceed with the upgrade.
You may see other hardware disparities for hardware components that are not related to the upgrade.
This happens when the hardware upgrade packages adds new SoftAVL and probe_config files to the
cluster; these files process current inventory differently. If the hardware descriptions and counts match,
you can accept these checks, particularly only the part revision numbers (that is, the last two digits) are
different. If you have questions, you should contact EMC Isilon Technical Support, and ask a hardware
coach or SME to validate the script output. For more on configuration change failures see Configuration
check fails when installing a FRU package on X200 or S200 CTO nodes, KB 88678.
The error message for probe_config.1.gc in the example above is typical for nodes that are running
OneFS 6.5. This format of probe_config file is designed for OneFS 7.0. This message is acceptable.
Here is another example of an acceptable probe_config check for a 10 GbE upgrade:
3 missing BOM item(s). Part breakdown is as follows:
1) P/N="415-0017-09", Desc="PC ASSY, PCIE,DUAL IB-NVRAM,DDR", Qty="1"
2) P/N="415-0039-05", Desc="PC ASSY,SEFC,2U 12 DRIVE", Qty="1"
3) P/N="050-0027-01", Desc="PCA,PCI-E,DUAL 10GE SFP+", Qty="1"
In the preceding example, the first highlighted item is the new card that is expected after the hardware
package is installed. The second highlighted item is the old card that is to be replaced.
Page 13 of 14
corrections.
Page 14 of 14