PowerScale OneFS

Upgrade Planning and Process Guide

8.0.0.0 - 9.0.0.0

June 2020
Rev. 14
Notes, cautions, and warnings

NOTE: A NOTE indicates important information that helps you make better use of your product.

CAUTION: A CAUTION indicates either potential damage to hardware or loss of data and tells you how to avoid the
problem.

WARNING: A WARNING indicates a potential for property damage, personal injury, or death.

© 2013 - 2020 Dell Inc. or its subsidiaries. All rights reserved. Dell, EMC, and other trademarks are trademarks of Dell Inc. or its
subsidiaries. Other trademarks may be trademarks of their respective owners.
 Contents

1 Introduction to this guide.............................................................................................................. 5

About this guide..................................................................................................................................................................... 5
Provide feedback about this document..............................................................................................................................5
Where to go for support.......................................................................................................................................................5

2 Planning an upgrade..................................................................................................................... 6
Checklist - Planning an Upgrade..........................................................................................................................................6
Upgrade planning overview .................................................................................................................................................6
Prerequisites for completing an upgrade............................................................................................................................6
Review required documentation.......................................................................................................................................... 7
Types of OneFS upgrades.................................................................................................................................................... 7
Parallel upgrades.............................................................................................................................................................. 7
Rolling upgrades............................................................................................................................................................... 8
Simultaneous upgrades................................................................................................................................................... 8
Verify the upgrade path........................................................................................................................................................8
Upgrade Paths from OneFS 8.x to OneFS 8.x and later............................................................................................ 8
Check supportability and compatibility requirements....................................................................................................... 9
Assess upgrade impact........................................................................................................................................................10
Consider upgrade limitations.........................................................................................................................................10
Review feature changes and known issues................................................................................................................ 10
Plan an upgrade schedule................................................................................................................................................... 10

3 Completing pre-upgrade tasks..................................................................................................... 12

Checklist - Pre-upgrade...................................................................................................................................................... 12
About pre-upgrade tasks.....................................................................................................................................................13
Collect cluster information.................................................................................................................................................. 13
Review and save information about cluster status ...................................................................................................13
Gathering cluster logs.................................................................................................................................................... 13
Check cluster readiness.......................................................................................................................................................13
Check hardware health..................................................................................................................................................14
Check the available free space.....................................................................................................................................15
Resolve outstanding events and errors.......................................................................................................................16
Verify configurations and settings......................................................................................................................................17
Upgrading in a Cloudpools environment...................................................................................................................... 17
Preserve the Kerberos keytab file................................................................................................................................ 17
Install a supported version of InsightIQ........................................................................................................................17
Download the OneFS installation file................................................................................................................................. 17
Upgrade compatibility check utility.................................................................................................................................... 18
Run the upgrade compatibility check utility................................................................................................................ 18
Reconfigure unsupported SMB settings..................................................................................................................... 18
Upgrade compatibility checks.......................................................................................................................................18
On-Cluster Analysis tool..................................................................................................................................................... 20
HealthCheck tool.................................................................................................................................................................20
Backup data......................................................................................................................................................................... 20

Contents 3
 SyncIQ backup............................................................................................................................................................... 20
NDMP backup................................................................................................................................................................20
Back up custom settings............................................................................................................................................... 21
Complete or stop jobs in progress.....................................................................................................................................22
Complete system jobs...................................................................................................................................................22
Update drive firmware........................................................................................................................................................ 22
Configure IPMI ports...........................................................................................................................................................22
Secure Remote Services (SRS)........................................................................................................................................ 22

4 Performing the OneFS upgrade................................................................................................... 23

Checklist - Upgrade.............................................................................................................................................................23
Upgrade process overview.................................................................................................................................................23
Completing a parallel upgrade of OneFS.......................................................................................................................... 23
Performing a parallel upgrade...................................................................................................................................... 23
Completing a rolling upgrade of OneFS............................................................................................................................24
Perform a rolling upgrade............................................................................................................................................. 24
Configuration changes during a rolling upgrade........................................................................................................ 26
Client connections during rolling upgrades.................................................................................................................26
Completing a simultaneous upgrade of OneFS............................................................................................................... 26
Perform a simultaneous upgrade................................................................................................................................. 27
Adding a node to the cluster while an upgrade is in progress........................................................................................27
Committing an upgrade of OneFS.................................................................................................................................... 28
Commit an upgrade....................................................................................................................................................... 28
Rolling back an upgrade of OneFS.............................................................................................................................. 28
Verify the OneFS installation............................................................................................................................................. 29

5 Completing post-upgrade tasks...................................................................................................30

Checklist - Post Upgrade................................................................................................................................................... 30
About post-upgrade tasks..................................................................................................................................................30
Allow the Upgrade job to run............................................................................................................................................. 30
Verify operational status......................................................................................................................................................31
Re-establish user privileges................................................................................................................................................32
Restore client connections, and test the workflow........................................................................................................ 32
Verify Kerberos migration...................................................................................................................................................32
......................................................................................................................................................................................... 33
Reapply custom settings.................................................................................................................................................... 33
Modify custom scripts........................................................................................................................................................ 33
Reinstall Aspera....................................................................................................................................................................33
Install the latest Patch........................................................................................................................................................ 33

6 Troubleshooting your upgrade..................................................................................................... 34

Troubleshooting overview.................................................................................................................................................. 34
Review the upgrade log files........................................................................................................................................ 34
Common issues.............................................................................................................................................................. 34

4 Contents
 1
Introduction to this guide
Topics:
• About this guide
• Provide feedback about this document
• Where to go for support

About this guide

This guide provides important information and steps that you must review and follow when upgrading your OneFS 8.x or 9.x cluster. Read
through this guide before upgrading, and complete the checklists for each stage of the upgrade process.
NOTE: For upgrades from OneFS 7.x to a later version of OneFS, see the OneFS 7.x Upgrade Planning and Process
Guide .

Provide feedback about this document

Your suggestions help to improve the accuracy, organization, and overall quality of the documentation. Send your feedback to http://
bit.ly/isilon-docfeedback. If you cannot provide feedback through the URL, send an email message to [email protected].

Where to go for support

This topic contains resources for getting answers to questions about PowerScale products.

Online support • Live Chat

• Create a Service Request
For questions about accessing online support, send an email to [email protected].

Telephone support • United States: 1-800-SVC-4EMC (1-800-782-4362)

• Canada: 1-800-543-4782
• Worldwide: 1-508-497-7901
• Local phone numbers for a specific country/region are available at Dell EMC Customer Support
Centers.

PowerScale OneFS You can find product documentation, troubleshooting guides, videos, and other resources about
Documentation Info Hubs PowerScale OneFS products and features on the PowerScale OneFS Info Hubs page.

Introduction to this guide 5

 2
Planning an upgrade
Topics:
• Checklist - Planning an Upgrade
• Upgrade planning overview
• Prerequisites for completing an upgrade
• Review required documentation
• Types of OneFS upgrades
• Verify the upgrade path
• Check supportability and compatibility requirements
• Assess upgrade impact
• Plan an upgrade schedule

Checklist - Planning an Upgrade

Use this checklist to help you track your progress as you plan your upgrade.

Review required documentation

Decide which type of upgrade you are going to perform

Verify the upgrade path

Check supportability and compatibility

Assess upgrade impact

Plan an upgrade schedule

(optional) Upgrade a test cluster

Upgrade planning overview

This chapter provides information to help you create an upgrade plan. Your upgrade plan helps you estimate the time it will take to
complete tasks and determine who needs to complete them.
Create an upgrade plan by evaluating how the upgrade will affect users and workflow, assessing the impact of the new version of OneFS
on your PowerScale cluster, and analyzing upgrade risks.
The services team is available to perform OneFS pre-upgrade health checks and will complete the OneFS upgrade remotely for some
customers. Contact your Account Team for more information.
NOTE: Clusters running a version of OneFS that has reached the End of Service Life milestone will also need a Time &
Materials engagement for upgrade. To confirm whether the version of OneFS you are upgrading from has reached the
End of Service Life, review the PowerScale Product Availability Guide. If your version of OneFS has reached End of
Service Life, contact your Account Team to discuss the available options and to develop an upgrade strategy.

Prerequisites for completing an upgrade

Permissions and interfaces
To complete the tasks described in this guide, you must be able to log in to the cluster as the root user through the following interfaces:
• OneFS command-line interface

6 Planning an upgrade
• OneFS web administration interface
NOTE: If you are not logged into the OneFS cluster with root privileges, you might not be able to run all of the
commands in this guide.

Review required documentation

Reviewing the documentation in this list helps you to understand the upgrade process and the impact the upgrade could have on your
workflow.

Required documentation
• OneFS Upgrade Process Flowchart
Review this step-by-step reference guide for OneFS upgrades.
• OneFS Release Notes and Maintenance Release Notes
Read the OneFS release notes for information about new features and changes, resolved issues, known issues, and supported upgrade
paths.
• Current PowerScale Software Releases
Confirm which current OneFS releases have reached Target Code status.
• Current PowerScale OneFS Patches
Review patches that have been released for the version of OneFS to which you are upgrading.
• PowerScale Supportability and Compatibility Guide
Confirm that your PowerScale software and PowerScale hardware is compatible with the version of OneFS to which you are
upgrading.
• PowerScale OneFS Technical Specifications Guide
Confirm the recommended settings and thresholds for the version of OneFS to which you are upgrading.
• OneFS Technical and Security Advisories
Determine whether any PowerScale Technical Advisories or Security Advisories have been issued for the version of OneFS to which
you are upgrading.

Types of OneFS upgrades

Depending on your version of OneFS, there are up to three options available for upgrading the OneFS operating system: a simultaneous
upgrade, a rolling upgrade, or a parallel upgrade.

Parallel upgrades
A parallel upgrade installs the new operating system on a subset of nodes and restarts that subset of nodes at the same time. Each subset
of nodes attempts to make a reservation for their turn to upgrade until all nodes are upgraded. Node subsets and reservations are based
on diskpool and node availability.
During a parallel upgrade, node subsets that are not being upgraded remain online and can continue serving clients. However, clients that
are connected to a restarting node are disconnected and reconnected. How the client connection behaves when a node is restarted
depends on several factors including client type, client configuration (mount type, timeout settings), IP allocation method, and how the
client connected to the cluster.
Parallel upgrades are recommended whenever possible, as they require a smaller maintenance window than rolling upgrades, and do not
require the interruption of service like simultaneous upgrades.
NOTE: The parallel upgrade feature is available in OneFS version 8.2.2 and newer. Only upgrades from OneFS 8.2.2 to
newer OneFS versions can take advantage of the parallel upgrade feature. Any upgrades where the starting cluster is on
OneFS 8.2.1 or older cannot take advantage of the parallel upgrade feature.

Planning an upgrade 7
Rolling upgrades
A rolling upgrade installs the new operating system and restarts each node individually in the OneFS cluster so that only one node is offline
at a time. A rolling upgrade takes longer to complete than a simultaneous upgrade. During a rolling upgrade, nodes that are not actively
being upgraded remain online and can continue serving clients. However, clients that are connected to a restarting node are disconnected
and reconnected. How the client connection behaves when a node is restarted depends on several factors including client type, client
configuration (mount type, timeout settings), IP allocation method, and how the client connected to the cluster.
You can specify the order in which nodes are upgraded by using the --node parameter of the isi upgrade cluster start
command. By default, nodes are upgraded in ascending order from the node with the lowest Array ID to the node with the highest Array
ID. For more information about Array IDs, see Understanding OneFS Group Changes on the Online Support site.
NOTE: See the Verify the upgrade path topic for information about which types of upgrades are supported between
OneFS versions.

Simultaneous upgrades
A simultaneous upgrade installs the new operating system and restarts all nodes in the OneFS cluster at the same time. Simultaneous
upgrades are faster than rolling upgrades but require a temporary interruption of service during the upgrade process. All client connections
to the cluster must be terminated prior to completing the upgrade and data is inaccessible until the installation of the new OneFS
operating system is complete and the cluster is back online.
NOTE: You cannot designate which nodes to upgrade when performing a simultaneous upgrade. You must upgrade all of
the nodes in the cluster when performing a simultaneous upgrade.

Verify the upgrade path

Verify that the current version of OneFS on your cluster can be upgraded to the target version. You might be required to upgrade to the
target version then add patches to reach bug fix and feature parity. For some upgrade paths, you might be required to upgrade to an
intermediate OneFS version before you can upgrade to the target OneFS version.
To view which version of OneFS is running on all nodes in the cluster, run the following command:

isi_for_array uname -r

Review the supported upgraded paths to confirm which version of OneFS you can upgrade your cluster to.
For additional information about upgrade paths compatible with your version of OneFS, please contact your account team.

Upgrade Paths from OneFS 8.x to OneFS 8.x and later

Upgrade Paths The following table can be used to determine the recommended upgrade path from the current version of OneFS
from OneFS 8.x to 8.x on your cluster to a target version of OneFS 8.x and later.
OneFS 8.x and
later The upgrade paths table displays the recommended upgrade paths from major OneFS version to major OneFS
version, but does not include patch information.
NOTE: These recommended upgrade paths do not guarantee bug fix and feature parity, but do
lead you through the path with the least amount of bug fix and feature parity loss. For more
information about using patches after an upgrade to achieve bug fix and feature parity, contact
your account team.
Key:
• "O" : Rolling and Simultaneous upgrades available
• "=" : Parallel, Rolling, and Simultaneous upgrades available
• Empty cells are not recommended upgrade paths

Table 1. OneFS Upgrade Paths

Upgrade to target version

8 Planning an upgrade
 Table 1. OneFS Upgrade Paths(continued)
Upgrade 8. 8. 8. 8. 8. 8. 8. 8. 8. 8. 8. 8. 8. 8. 8. 8. 8. 8. 8. 8. 8. 8. 8. 9.
from 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 1. 1. 1. 1. 1. 1. 1. 1. 2. 2. 2. 0.
current 0. 0. 0. 0. 0. 0. 0. 0. 1. 1. 1. 0. 0. 0. 0. 0. 1. 1. 2. 3. 0. 1. 2. 0.
version 0 1 2 3 4 5 6 7 0 1 2 0 1 2 3 4 0 1 0 0 0 0 0 0

8.0.0.0 O O O O O O O O O O O O O O O O O O O O O O

8.0.0.1 O O O O O O O O O O O O O O O O O O O O O

8.0.0.2 O O O O O O O O O O O O O O O O O O O

8.0.0.3 O O O O O O O O O O O O O O O O O O

8.0.0.4 O O O O O O O O O O O O O O O O O

8.0.0.5 O O O O O O O O O O O O O O

8.0.0.6 O O O O O O O O O O O

8.0.0.7 O O O O O

8.0.1.0 O O O O O O O O O O O O O O

8.0.1.1 O O O O O O O O O O O O O

8.0.1.2 O O O O O O O O O O

8.1.0.0 O O O O O O O O O O O O

8.1.0.1 O O O O O O O O O O O

8.1.0.2 O O O O O O O O O O

8.1.0.3 O O O O O O O

8.1.0.4 O O O O O O

8.1.1.0 O O O O O O O

8.1.1.1 O O O O O O

8.1.2.0 O O O O O

8.1.3.0 O O O

8.2.0.0 O O O

8.2.1.0 O O

8.2.2.0 =

9.0.0.0

Check supportability and compatibility

requirements
Review the PowerScale Supportability and Compatibility Guide to confirm that your current hardware components, software components,
and protocol versions are compatible with the version of OneFS to which you are upgrading.
For information about OneFS compatibility with Hadoop, see the Hadoop - Info Hub page on the Community Network website.

Planning an upgrade 9
Assess upgrade impact
Consider all the areas of your environment that might be affected by upgrading to a new version and plan a strategy for performing the
tasks that must be completed and addressing issues that might occur. Assessing the impact of the upgrade on your environment, clients,
and performance helps ensure that the upgrade does not disrupt your cluster, workflow, or users.
NOTE: If you have a performance-sensitive workload, PowerScale recommends that you consult your Sales Engineer for
assistance during the pre-upgrade planning process.

Consider upgrade limitations

If the upgrade cannot be completed for any reason—for example, if there is insufficient space on the cluster or if the upgrade process
detects a stalled drive—the system will revert to the current version and the upgrade will be cancelled. Preparing your cluster as
recommended in this guide will help you to avoid situations that might result in a cancelled upgrade.
NOTE: In OneFS 8.2.0 and newer, you have the ability to pause and resume the upgrade process in order to resolve
blocking issues.

Review feature changes and known issues

Familiarize yourself with new, modified, and deprecated features as well as known issues in the target version of OneFS to which you are
upgrading.
Functionality changes and new features in the target version might impact the performance, configuration, or functionality of your cluster
after upgrading. If the target version contains an issue that affects your environment, you should consider waiting to upgrade until after
the issue is resolved in a later version of OneFS.
See the OneFS Release Notes for a summary of new features, feature changes, and known issues in the target version of OneFS. See the
OneFS CLI Mappings guide for information about command name changes. You can find links to these documents, as well as other
resources, on the and by searching the Online Support site.

Plan an upgrade schedule

An upgrade schedule can help ensure that the upgrade goes smoothly. The schedule should consider all the factors that go into an
upgrade and estimate how long each stage of the upgrade process might take.
The upgrade process begins with ensuring that your OneFS cluster is ready to be upgraded. It is recommended that you set aside two
weeks to check the cluster health and resolve software and hardware issues before performing the upgrade itself.
The main factors that you should consider when estimating a schedule include:

Data back-up and Estimate the time that it takes to back up your data, considering cluster size, number of files, types of files, and
information file size. Also include time to collect information about the cluster such as status, logs, and settings.
collection
Cluster Schedule time for node and drive health checks and replacement of bad hardware. Include time to update
preparation and configurations and settings that are not supported in the new version.
readiness
Upgrade Estimate the time that it takes to run the upgrade considering cluster size and upgrade type (parallel, rolling, or
maintenance simultaneous). Schedule time to inform users when the upgrade will take place and that client connections might
window be slow, file access might be affected, and clients might be disconnected. A best practice is to upgrade the
cluster during an off-hours maintenance window.
If performing a parallel or rolling upgrade, consider whether you will configure client connection drain times, which
will extend the required maintenance window, but lower the impact on client connections.
Build in time to let the upgrade jobs run to completion and to reestablish permissions and connections. Schedule
time or extend the maintenance window to accommodate post-upgrade tasks such as reconfiguring custom
settings, updating scripts to reflect command and functionality changes in the upgrade version, and potential
troubleshooting.

(Optional) If available, upgrading a test cluster with the same current version of OneFS before you upgrade your production
Upgrade a test cluster can expose issues that could slow down or prevent the upgrade of your production system.
cluster

10 Planning an upgrade
After you upgrade a test cluster, verify that the cluster is operational and validate key workflows on the test
cluster by simulating how administrators, users, and applications interact with the system.

Planning an upgrade 11
 3
Completing pre-upgrade tasks
Topics:
• Checklist - Pre-upgrade
• About pre-upgrade tasks
• Collect cluster information
• Check cluster readiness
• Verify configurations and settings
• Download the OneFS installation file
• Upgrade compatibility check utility
• On-Cluster Analysis tool
• HealthCheck tool
• Backup data
• Complete or stop jobs in progress
• Update drive firmware
• Configure IPMI ports
• Secure Remote Services (SRS)

Checklist - Pre-upgrade
Use this checklist to help you track your progress as you perform the pre-upgrade tasks.

Collect cluster information

Collect cluster status

Gather cluster logs

Check hardware health

Check available space

Resolve outstanding events and errors

Preserve the Kerberos keytab file

Install supported version of InsightIQ

Download the OneFS installation file

Reconfigure unsupported SMB settings

Run the upgrade compatibility check utility

Run the On-Cluster Analysis tool

Run the Healthcheck tool

Back up data

Complete system jobs

Update drive firmware

Configure IPMI ports

Enable Secure Remote Support (SRS)

12 Completing pre-upgrade tasks

About pre-upgrade tasks
Performing the pre-upgrade tasks described in this guide helps to ensure that the OneFS cluster hardware, software modules,
configuration, features, and file system do not have preexisting issues that could adversely affect the upgrade process. Performing the
pre-upgrade tasks also ensures that important cluster data is collected, the current configuration settings are recorded, and the potential
for data loss is minimized.

Collect cluster information

Before you upgrade, collect and record key information about your OneFS cluster and how it is configured.

Review and save information about cluster status

Before you upgrade, run the isi status command to get the status of your cluster, events, and jobs then save the information to a
file.
Run the isi status > /ifs/data/isi_status_output command to save the output of the isi status command to a file
named isi_status_output in the /ifs/data directory.
NOTE: Run the isi status command to view the status of the cluster, events, and jobs without saving the output to a
file.

Gathering cluster logs

You can gather cluster logs and send the logs to Dell EMC PowerScale Technical Support for analysis. Cluster logs can be sent
automatically or manually through the cluster command-line and web administration interfaces.
NOTE: Your cluster must be connected to the internet to be able to send log files directly. If your cluster does not have
an internet connection or if your upload has failed, you can copy the log file from the cluster and upload the log file with
an FTP client to an FTP server. For more information about the isi_gather_info command, including the command
parameters for configuring how logs are uploaded to PowerScale Technical Support, see the OneFS CLI Administration
Guide or knowledge base article 304468.

Gather cluster logs

You must have root access to run the isi_gather_info command.
Command Line Interface instructions:
To gather the log files, run the following command:

isi_gather_info

Web Administration Interface instructions:

1. To gather the log files, navigate to Cluster Management > Diagnostics.
2. Click Start Gather.
After the log gathering process is complete, a link to the file that the process generates appears under Filename in the Archived Info
Manager section.

The files generated during the gathering process are stored on the cluster in the /ifs/data/Isilon_Support/pkg directory.

Check cluster readiness

Completing pre-upgrade tasks such as checking the health of the hardware components on the cluster, ensuring that storage space
requirements are met, and managing outstanding cluster events and errors are recommended for a successful upgrade.

Completing pre-upgrade tasks 13

Check hardware health
Run the following commands from the OneFS command-line interface as root user to evaluate the health of the cluster's hardware
components and the status of job engine jobs.
1. Run the following command to return information about cluster health and check for jobs or devices that report a status of
ATTENTION, SMARTFAIL, or DOWN.

isi status -v

2. Run the following command to check for drives that do not report a status of HEALTHY, L3, or JOURNAL.

isi_for_array -s 'isi devices drive list | egrep -v "HEALTHY|L3|JOURNAL"'

3. Run the following command to check the mirror status of the boot drives on each node.

isi_for_array -s 'gmirror status'

NOTE: If a drive is degraded, do not continue with the upgrade until the issue is resolved.

4. If the cluster has an InfiniBand network, run the following command to confirm whether a node has been assigned the OpenSM
(subnet manager) master role.

isi_for_array -s 'ps -auwwwx | grep master | grep opensm'

Confirm that the output displays only one node in the cluster with the phrase master (opensm). The output should be similar to
the following:

node-2: root 1610 0.0 2.3 436292 384672 ?? Ss 19May15 97:31.63

opensm: 0x00151b00007a671b master (opensm)

NOTE:

If the command does not return output that includes master (opensm) for any node, then another device has
assumed the OpenSM master role. In that case, confirm that cables from another cluster are not connected to this
cluster's primary or secondary InfiniBand switch, and then run the command again. For more information, contact
PowerScale Technical Support
5. It is recommended that you enable the Virtual Hot Spare (VHS) feature. VHS ensures that the cluster has enough free space available
to smartfail a drive and reprotect the drive's data. See article 471814, How to enable and configure Virtual Hot Spare (VHS) for
instructions.
6. In the OneFS web administration interface, perform the following steps for each node in the cluster.
a. Click Dashboard > Cluster Overview > Cluster Status.
b. In the Status area, click the ID of a node.
c. In the Chassis and drive status area, click Platform .
d. If any correctable or fatal errors are reported, do not continue with the upgrade. If the errors cannot be resolved, contact
PowerScale Technical Support
NOTE: For more information about these commands and checking hardware health, see the following articles:

• See article 317462, How to determine why a node is in an attention state.

• See article 456690, Cannot perform upgrade with degraded boot drive.
• Article 424865, ECC error policy violation alert does not clear after replacement
• Article 424324, Physical memory low
• Article 471897, Temperature sensitivity in the DIMM module used in the IQ 10000X-SSD, IQ 5000S-SSD, and IQ
• Article 471888, DIMM replacement policy for nodes

14 Completing pre-upgrade tasks

Check the available free space
Ensure that the minimum available-space requirements for the cluster, nodes, node pools, and critical directories are met before you
upgrade. Do not continue with the upgrade if the minimum available-space requirements are not met.

Available space requirements for clusters, nodes, node pools, and

directories
The cluster, nodes, node pools, and several critical directories in the file system must meet available space requirements prior to upgrading
OneFS. If the available space for any of these items falls below the minimum requirement, make more space available before upgrading.
Otherwise, the upgrade process will fail and might not return an error indicating available space as the cause.

Item Minimum Requirement Task

Cluster The cluster cannot be more than 90 Make more available space.
percent capacity.

Node Each node cannot be more than 92 Make more available space.
percent capacity.

Node pool Each node pool cannot be more than Make more available space.
90 percent capacity.

root partition (/) The root partition cannot be more If this directory is at or near the minimum available-space
than 97 percent capacity. requirement, see the following resources for steps to
address the issue:
• Knowledge base article 464118, Node reached 95%
used capacity on the root file system.
• Event ID 100010003, The /(root) partition is near
capacity, is the OneFS event reference.

/ifs The /ifs directory cannot be more If this directory is at or near the minimum available-space
than 90 percent capacity. requirement, see the following resources for steps to
address the issue:
• Knowledge base article 471816, "There is at least one
SmartPool at or over capacity " or "The SmartPool
'[name]' is near or over capacity".
• Event ID 100010004, The cluster's /ifs partition is
near capacity, is the OneFS event reference.

/var The /var partition cannot be more If this directory is at or near the minimum available-space
than 90 percent capacity. requirement, see the following resources for steps to
address the issue:
• Knowledge base article 471789, The /var partition is
near capacity (95% used).
• Event ID 100010001, The /var partition is near
capacity, is the OneFS event reference.

/var/crash The /var/crash directory cannot If this directory is at or near the minimum available-space
be more than 90 percent capacity. requirement, see the following resources for steps to
address the issue:
• Knowledge base article 458364, The crash partition
of a node in the cluster has reached 90% capacity
alert.
• Event ID 100010002, The /var/crash partition is near
capacity, is the OneFS event reference.

For more information, see the Best Practices Guide for Maintaining Enough Free Space on Clusters and Pools.

Completing pre-upgrade tasks 15

Confirm used and available space
• To confirm how much space is being used on the cluster and on each node, run the following command:

isi stat

• To confirm how much space is being used in each node pool, run the following command:

isi stat -p

• To confirm how much space is being used by critical directories on the cluster, run the following command:

isi_for_array -s 'df -h'

The isi_for_array output is similar to the following for each node in the cluster:
virtual-demo-3: Filesystem Size Used Avail Capacity Mounted on
--------------------------------------------------------------------
virtual-demo-3: /dev/imdd0a 495M 418M 38M 92% /
virtual-demo-3: devfs 1.0K 1.0K 0B 100% /dev
virtual-demo-3: /dev/imdd1a 495M 6.7M 449M 1% /var
virtual-demo-3: /dev/imdd2a 496M 5.9M 451M 1% /var/crash
virtual-demo-3: OneFS 40G 283M 20G 1% /ifs

virtual-demo-2: Filesystem Size Used Avail Capacity Mounted on

--------------------------------------------------------------------
virtual-demo-2: /dev/imdd0a 495M 417M 38M 92% /
virtual-demo-2: devfs 1.0K 1.0K 0B 100% /dev
virtual-demo-2: /dev/imdd1a 495M 6.1M 449M 1% /var
virtual-demo-2: /dev/imdd2a 496M 5.9M 451M 1% /var/crash
virtual-demo-2: OneFS 40G 283M 20G 1% /ifs

Resolve outstanding events and errors

Before you begin your upgrade, resolve any outstanding critical events, errors, and failures, as these issues can disrupt the OneFS upgrade
process.
1. Run the following command to view all events on the cluster:

isi event groups list --sort=severity

• If the command returns any critical errors, check the log files in the following directories for more information:
○ /var/log
○ /var/log/messages
○ /var/crash
• NOTE: If any log files contain messages about a dynamic sector recovery (DSR) failure or a Data Integrity (IDI)
failure, contact PowerScale before you upgrade.
2. Cancel non-critical events before upgrading to prevent a recurrence of notifications that you know to be harmless.
See the following content for more information:
• Article 317661, How to quiet or cancel an event (alert)
• Article 304312, How to reset the CELOG database and clear all historical alerts
• Article 471816, There is at least one smartpool at or over capacity
• Article 454806, Detected IDI failure, attempting DSR
• Article 373706, DSR - Dynamic Sector Recovery Failure
• Article 373712, IDI error. Shallow verification failure in block
• Article 447864, Detected IDI network checksum error on path
• Article 454399, Cluster needs to be restriped but FlexProtect is not running
• OneFS Event Reference, information about specific events for your current version of OneFS

16 Completing pre-upgrade tasks

Verify configurations and settings
Confirm that your OneFS configurations for CloudPools, NIC aggregation, Kerberos, InsightIQ, and SMB are supported by the target
version of OneFS to which your are upgrading.

Upgrading in a Cloudpools environment

If you're upgrading a OneFS 8.x cluster to OneFS 8.2.x. with a Cloudpools environment, there are additional items to consider. For more
information on pre-upgrade and upgrade steps, see KB article 533740

Preserve the Kerberos keytab file

If Kerberos authentication is configured on the cluster, upgrading OneFS might delete the local /etc/<name>.keytab file that is on
each node. Choose one of the following methods to preserve the keytab file during the upgrade:
• Copy the keytab file to the /ifs directory where it can be accessed by all nodes on the cluster instead of saved locally on each node.
• Add the following text to the /etc/mcp/override/user_preserve_files.xml file of each node on the cluster where
<name> is the name of the keytab file:

<?xml version="1.0" encoding="UTF-8"?>

<user_preserve>
<files>
<file name="etc/allow_unsupported_boot" recursive="no">
</file>
...
...
<file name="etc/<name>.keytab" recursive="no" />

</files>
</user_preserve>

For more information, see article 304460, How to configure a cluster to use Kerberos with NFS in a non-Active Directory environment.

Install a supported version of InsightIQ

Before you upgrade OneFS, confirm the version of InsightIQ you are running is compatible with the target version of OneFS to which you
are upgrading. If the versions are not compatible, upgrade InsightIQ before upgrading OneFS.
See the PowerScale Supportability and Compatibility Guide for OneFS and InsightIQ compatibility information.
See the InsightIQ - Info hub for documents and content that is related to InsightIQ, including release notes, installation guides, user guides,
and troubleshooting guides.

NOTE: See the InsightIQ 4.1.3 guides for additional information about monitoring OneFS 8.2.0 clusters.

NOTE: If you are upgrading from OneFS 8.1.x or older to OneFS 8.2.x or newer, a maximum of 99 FSA reports will be
retained after the upgrade. For more information, see KB article 540000.

Download the OneFS installation file

1. From the OneFS Downloads page on the Online Support site, under Browse Product, select your target version of OneFS.
2. Download the installation file for the target version of OneFS to which you are upgrading.
a. Optional: To validate the integrity of the installation file after downloading, click Checksum and record the MD5 or SHA-256
checksum value displayed.
3. Open a secure shell (SSH) connection to any node in the cluster and log in using the root account.
4. Move the installation file that you downloaded into the /ifs/data directory on the cluster you want to upgrade.
a. Optional: To validate the integrity of the downloaded installation file, run the following command for your checksum type where
<installation-file-name> is the name of the downloaded installation file:
• md5 /ifs/data/<installation-file-name>

Completing pre-upgrade tasks 17

 • sha256 /ifs/data/<installation-file-name>

b. Optional: To validate the integrity of the downloaded installation file, compare the MD5 or SHA-256 checksum value that you
recorded from the downloads page on the Online Support site to the checksum value returned from the MD5 or SHA-256
command. If the values do not match, re-download the installation file.

Upgrade compatibility check utility

The upgrade compatibility check utility examines cluster settings, activities, and statuses to verify whether your cluster is compatible with
the target version of OneFS to which you are upgrading.
The upgrade compatibility check utility is included in each OneFS installation package. You can run the utility alone or as part of the
upgrade process.

Run the upgrade compatibility check utility

To confirm that your OneFS cluster is compatible with the target version that you are upgrading to, run the upgrade compatibility check
utility.
NOTE: The upgrade compatibility check utility is included in each OneFS installation package, which must be
downloaded and accessible in an /ifs directory.

1. Open a secure shell (SSH) connection to any node in the cluster and log in to the cluster with the root account.
2. Start the upgrade compatibility check utility by running the following command, where <install-image-path> is the file path of the
upgrade installation file.
isi upgrade cluster assess <install-image-path>
NOTE: The upgrade compatibility check utility might take several minutes to run. If the utility returns errors, resolve
the errors before continuing with the upgrade. Warnings are informational and do not prevent an upgrade.

Reconfigure unsupported SMB settings

If the SMB settings on the cluster are not supported by the target version of OneFS to which you are upgrading, the upgrade might fail.
Run the upgrade compatibility check utility to confirm whether your current settings are supported.
If the upgrade compatibility check utility detects unsupported SMB settings, remove or modify the unsupported SMB settings through the
command-line interface or web administration interface before you upgrade.

Upgrade compatibility checks

The upgrade compatibility check utility examines the following areas of the cluster and returns warnings or errors if an area is not
compatible with the version of OneFS to which you are upgrading.

Check Description Recommendation

Disk load Checks the cluster usage level and PowerScale recommends that you disconnect all client connections and stop
returns a warning if the disk load is all system jobs before upgrading. A disk load that is greater than 50 percent
greater than 50 percent. might indicate that multiple clients are connected to the cluster or that
system jobs are running.

Free space Checks cluster free space and Do not continue with the upgrade if the utility reports a capacity-related
returns a warning or an error if error. See the Check the available free space section of the Upgrade
capacity thresholds are exceeded for Planning and Process Guide for more information.
the following partitions and node
pools:

/ Returns a
warning if
the root
partition
exceeds 97
percent of

18 Completing pre-upgrade tasks

Check Description Recommendation

its capacity.
Returns an
error if the
partition
reaches 100
percent
capacity.
/var Returns a
warning if
the /var
partition is
more than
90 percent
full. Returns
an error if
the partition
reaches 100
percent full.
/ifs Returns a
warning if
the /ifs
partition
exceeds 95
percent of
its capacity.
Returns an
error if the
partition
reaches 100
percent
capacity.
Node pools Returns a
warning if
one or more
node pools
exceed 90
percent of
its capacity.
Returns an
error if all the
node pools
reach 90
percent
capacity.

Drive stalls Checks the health of the drives in If the utility reports a stalled drive, do not continue with the upgrade until you
the cluster and returns a warning if smartfail and replace the stalled drive or resolve the stall. See article article
the cluster contains stalled drives. 466391, Introduction to drive stalls, for information about resolving drive
stalls.

Smartfail Determines whether a smartfail If a smartfail operation is running, the utility returns an error. Wait for the
operation status operation is running on any drives or smartfail operation to complete before continuing with the upgrade.
nodes in the cluster.

IntegrityScan job Determines whether the If the integrity scan job is running, wait for the job to complete before
status integrityScan job is running. continuing with the upgrade.

Unresolved Checks for unresolved, critical If the utility reports that there are unresolved critical events, do not continue
critical events events. with the upgrade until you resolve the issues. See the OneFS Event
Reference for more information.

Completing pre-upgrade tasks 19

 Check Description Recommendation
Unsupported Checks for unsupported SMB If the utility reports that the SMB configuration is not supported, do not
SMB settings. continue with the upgrade until you remove the unsupported settings and
configuration reconfigure SMB. See the Reconfigure unsupported SMB settings section
for more information.

SMB access Checks if one or more SMB shares If the utility reports an error, the upgrade will succeed. However, you cannot
zone association are associated with multiple access create access zones until the path overlap is fixed.
zones where the share paths overlap.

Kerberos keytab Checks whether Kerberos settings If the utility reports an error, you must configure Kerberos settings before
have been configured to preserve upgrading OneFS. See the Preserve the Kerberos keytab file section for
the keytab file. more information. The keytab file is used to migrate Kerberos settings into
the OneFS web administration interface and command-line interface.

HDFS Kerberos Checks whether HDFS Kerberos If the utility reports an error, you must configure Kerberos settings before
keytab settings have been configured to upgrading OneFS.See the Preserve the Kerberos keytab file section for more
preserve the keytab file. information. The keytab file is used to migrate Kerberos settings into the
OneFS web administration interface and command-line interface.

On-Cluster Analysis tool

The PowerScaleOn-Cluster Analysis (OCA) tool analyzes the health status of a running cluster and can help you prepare for an upgrade.
The IOCA tool analyzes your cluster and displays any issues that it encounters, a summary of the issue, and any associated KB articles to
help you resolve the issue. The IOCA tool also has a feature that provides you with an estimated upgrade plan based on the analysis.
For more information about the IOCA, please see the KB article: How to run the On-Cluster Analysis tool

HealthCheck tool
The HealthCheck tool enables you to evaluate the status of specific software and hardware components of your cluster and your cluster
environment. For more information about the HealthCheck tool, see this site: HealthCheck Info Hub

Backup data
It is recommended to backup your cluster data immediately before you upgrade. Schedule sufficient time for the back up to complete
before the upgrade window.

SyncIQ backup
SyncIQ is one option you can use to backup your OneFS cluster. SyncIQ creates and references snapshots to replicate a consistent point-
in-time image of a root directory.
NOTE: If you are upgrading your cluster from OneFS 8.1.0.x or earlier to OneFS 8.1.1.x, 8.1.2.x, or 8.1.3.x, and your
cluster is in Compliance mode, you must ensure that all SyncIQ partners are on the same code and patch level OneFS
before restarting SyncIQ backups, or the backups fail. This issue is resolved in OneFS 8.2.0 and later.
For more information about backing up your OneFS cluster, see the OneFS CLI Administration Guide or the OneFS Web Administration
Guide for your version of OneFS.

NDMP backup
Other OneFS cluster backup options include using the Network Data Management Protocol (NDMP).
From a backup server, you can perform both NDMP three-way backup and NDMP two-way backup processes between a cluster and
backup devices such as tape devices, media servers, and virtual tape libraries (VTLs).
See the OneFS Web Administration Guide or the OneFS CLI Administration Guide for information about backing up data using NDMP.

20 Completing pre-upgrade tasks

Complete NDMP backups
Before you upgrade, you must wait for Network Data Management Protocol (NDMP) backups to finish so you have saved copies of your
data.
NOTE: If you cannot wait for NDMP backups to finish, stop the active NDMP backups 30– 60 minutes before the
upgrade, as the NDMP backup process requires this additional time to come to a stop.
For instructions on how to stop backup jobs, see the documentation for your backup application.
NOTE: If you are upgrading from OneFS version 8.0.x or 8.1.x to OneFS 8.2.x, you must disable the NDMP service before
to performing the upgrade.

Back up custom settings

Most settings are preserved during a OneFS upgrade. However, documenting and backing up custom settings enables you to reapply any
settings that are not preserved during the upgrade process.
Document and back up the following custom settings:

Setting Description Recommendation

SMB audit If you have an audit log directory in / After upgrading, you must reconfigure SMB audit logging. See File System
logging ifs, you might have custom SMB Auditing with the Common Event Enabler (CEE) for more information about
logging settings configured. SMB audit logging.

Passwords for After you upgrade, you might have to Make a list of the local accounts and their passwords before you upgrade.
local user reset the passwords of the local user
accounts accounts that you configured on the
cluster. Other users should be
prepared to reset the passwords of
their local accounts after the
upgrade.
sysctl If you changed the default value PowerScale does not recommend modifying sysctl parameters unless you
parameters assigned to one or more sysctl are instructed to do so by PowerScale Technical Support. If you must modify
parameters by editing either a sysctl parameter, configure the parameter in the /etc/mcp/
the /etc/mcp/override/ override/sysctl.conf file to ensure that the change is preserved
sysctl.conf file or the /etc/ when you upgrade a node or a cluster.
local/sysctl.conf file, you
Before you upgrade, document your custom sysctl parameters and back
might need to reset the parameter
up the /etc/mcp/override/sysctl.conf and /etc/local/
after you upgrade.
sysctl.conf files.
If you modified asysctl parameter
For more information, see article 462759, Configuring sysctls and making
by editing another file—for example,
sysctl changes persist through node and cluster reboots and upgrades.
the /etc/sysctl.conf file—the
change will not be preserved during
the upgrade.

Aspera You must reinstall and reconfigure Before you upgrade, back up the Aspera configuration files in the following
Aspera after you upgrade. directories:
NOTE: Aspera is no longer
• /ifs/.ifsvar/aspera/etc/
supported In OneFS 8.2.0 and
• /ifs/.ifsvar/aspera/www/
newer.
• /usr/local/aspera/var/aspera-prepost
To determine which version of Aspera is compatible with the version of
OneFS to which you are upgrading, see the Supportability and Compatibility
Guide, and then visit the Aspera website to download the Aspera install files.
For more information, see article 493022, How to download Aspera for
OneFS.

Cron jobs Cron jobs settings that were not Document and back up custom cron job settings or configure them in
configured in the /etc/mcp/ the /etc/mcp/override/crontab.smbtime file before you upgrade.
override/crontab.smbtime

Completing pre-upgrade tasks 21

Setting Description Recommendation
file are not preserved during an After you upgrade, you might have to modify a cron job to accommodate
upgrade. changes to OneFS commands. Check the OneFS CLI Mapping Guide for
command syntax changes between versions of OneFS.

Complete or stop jobs in progress

You should ensure that there are no jobs running on your OneFS cluster before beginning the upgrade. Wait for jobs to fully complete or
stop jobs before upgrading.

Complete system jobs

Ensure that no system jobs are running during the upgrade by allowing system jobs to finish before the upgrade starts, or by cancelling
them. OneFS performs system jobs through a service that runs in the background, and if any system jobs are running during the upgrade,
the upgrade process might fail.
1. To check for running system jobs, run the following command, and make a note of the job ID for any jobs that you want to cancel:

isi job status

2. To cancel a job, run the following command where <job_id> is the ID of the job you want to cancel:

isi job jobs cancel <job_id>

NOTE: Do not cancel the Upgrade, FlexProtect, FlexProtectLin, or IntegrityScan jobs. If any of these four system
jobs are running, you cannot continue with the upgrade. If an Upgrade, FlexProtect, FlexProtectLin, or IntegrityScan
system job takes longer than expected to complete, contact PowerScale Technical Support.

NOTE: Sync polices and jobs must be canceled or paused in order for the upgrade to complete successfully.

Update drive firmware

Before upgrading your OneFS cluster, ensure that your cluster is running the most recent version of firmware found in the Drive Support
Package.
For information about the Drive Support Package, see OneFS Drive Support Package, available on the Online Support site.

Configure IPMI ports

If you have enabled IPMI ports, it is recommended that you change the IPMI port IP configuration to static in the BIOS settings for each
affected node before the upgrade. If you change the IPMI port configuration during the upgrade process, and your workflow requires the
IPMI ports be enabled, you must manually re-enable the ports after the upgrade process is complete.

NOTE: Use of IPMI ports is supported in OneFS version 8.2.2 and later.

Secure Remote Services (SRS)

SRS can enable remote access, which allows PowerScale Technical Support personnel to troubleshoot the cluster remotely, run scripts to
request log files on the cluster, and upload logs to a secure location.
It is recommended that you disable SRS before you upgrade your OneFS cluster.
NOTE: If you are upgrading to OneFS 9.0.0.0 with PowerScale F200 or PowerScale F600 nodes, and have SRS v2
installed, you must upgrade to SRS v3 before upgrading OneFS.
See the SRS product page for documentation on how to enable, disable, upgrade, and configure SRS.

22 Completing pre-upgrade tasks

 4
Performing the OneFS upgrade
Topics:
• Checklist - Upgrade
• Upgrade process overview
• Completing a parallel upgrade of OneFS
• Completing a rolling upgrade of OneFS
• Completing a simultaneous upgrade of OneFS
• Adding a node to the cluster while an upgrade is in progress
• Committing an upgrade of OneFS
• Verify the OneFS installation

Checklist - Upgrade
Use this checklist to help you track your progress as you perform the upgrade tasks.

Perform the OneFS Upgrade

Commit the OneFS Upgrade

Verify the OneFS Upgrade

Upgrade process overview

The OneFS upgrade process consists of downloading the OneFS installation image, starting the upgrade, and verifying that the upgrade is
complete.
The following upgrade types are available:
• Parallel upgrade (for OneFS 8.2.2 and later)
• Rolling upgrade
• Simultaneous upgrade
A OneFS installation image is required to upgrade your cluster. See the Download the OneFS image section for more information.

NOTE: You can upgrade OneFS using the command-line interface or the web administration interface.

Completing a parallel upgrade of OneFS

If you perform a parallel upgrade, subsets of nodes within the OneFS cluster are restarted in an undetermined order. During a parallel
upgrade, client connections to the restarting subset of nodes are disconnected, but other subsets of nodes remain available for client
connection.
NOTE: The parallel upgrade feature is available in OneFS version 8.2.2 and later. Upgrades where the source cluster is
on OneFS 8.2.1 or older must use the rolling or simultaneous option.

Performing a parallel upgrade

You can use the command-line interface or the web administration interface to upgrade the version of OneFS that is running on your
cluster.
Perform the pre-upgrade steps in this guide, confirm cluster health, and resolve any compatibility issues before upgrading to the new
version of OneFS.
Command-line Interface instructions:

Performing the OneFS upgrade 23

1. Open a secure shell (SSH) connection on any node in the cluster and log in with the root account.
2. To perform a parallel upgrade, run the following command, where <install-image-path> is the file path of the upgrade install image.
The file path must be accessible in an /ifs directory.

isi upgrade cluster start --parallel <install-image-path>

NOTE: The isi upgrade cluster command runs asynchronously. The command does not run the entire upgrade
process; instead, it sets up the upgrade process, which nodes take turns controlling. For this reason, the command
returns quickly. To view the progress of the upgrade, use the isi upgrade view command or the web
administration interface.
After the upgrade, a number of upgrade-related jobs may continue to run on the cluster for some time. During this time, the cluster is
accessible, but you might experience a decrease in cluster performance. After the jobs complete, performance will return to normal. At this
stage, the upgrade is complete, but is not committed. You can still roll back to the previous version of OneFS. Some new features in the
upgrade might not be available until the upgrade is committed.

Web Administration Interface instructions:

1. Log in to any node in the cluster through the web administration interface with the root account.
2. Click Cluster Management > Upgrade.
3. Click Upgrade OneFS.
4. Browse to the location of the installation image that you want to install. The file path must be accessible in an /ifs directory. Once
the file path is chosen, click Select.
5. In the Upgrade Type dropdown list, select Parallel Upgrade.
6. Optional: (Optional) To skip the pre-upgrade checks for compatibility issues, select Skip optional pre-upgrade checks.
7. Click Start Upgrade.
The cluster might display several confirmation messages. Confirm each message to continue the upgrade process.
After the upgrade, a number of upgrade-related jobs may continue to run on the cluster for some time. During this time, the cluster is
accessible, but you might experience a decrease in cluster performance. After the jobs complete, performance will return to normal. At this
stage, the upgrade is complete, but is not committed. You can still roll back to the previous version of OneFS. Some new features in the
upgrade might not be available until the upgrade is committed.

Completing a rolling upgrade of OneFS

If you perform a rolling upgrade, the OneFS cluster nodes are restarted sequentially. During a rolling upgrade, client connections to the
restarting node will be disconnected, but other nodes will remain available for client connection.
Rolling upgrades are not available between all OneFS versions. See the Verify the upgrade path section of this document to determine
whether your upgrade path supports the rolling upgrade option.
NOTE: When you perform a rolling upgrade between OneFS releases, you can upgrade all the nodes in the cluster, or you
can select specific nodes to upgrade. Also, you can specify the upgrade order of the nodes. If you upgrade only some of
the nodes, the remaining nodes in the cluster are not upgraded but can be upgraded later in the upgrade process. Only
one upgrade can be in progress at a particular time. Also, you cannot upgrade some nodes to one version of OneFS and
then upgrade another group of nodes to a different version of OneFS.

Perform a rolling upgrade

You can use the command-line interface or the web administration interface to upgrade the version of OneFS that is running on the
cluster.
Review the pre-upgrade steps in this guide, confirm cluster health, and resolve any compatibility issues before upgrading to the new
version of OneFS.
Command-line Interface instructions:
1. Open a secure shell (SSH) connection on any node in the cluster and log in with the root account.
2. To perform a rolling upgrade, run the following command, where <install-image-path> is the file path of the upgrade install image. The
file path must be accessible in an /ifs directory.

24 Performing the OneFS upgrade

 For OneFS 8.2.2 and later:

isi upgrade cluster start --rolling <install-image-path>

For OneFS 8.2.1 and older:

isi upgrade cluster start <install-image-path>

NOTE: The isi upgrade cluster command runs asynchronously. The command does not run the entire upgrade
process; instead, it sets up the upgrade process, which nodes take turns controlling. For this reason, the command
returns quickly. To view the progress of the upgrade, use the isi upgrade view command or the web
administration interface.
3. Optional: (Optional) You can specify the following rolling upgrade options:

Options Description
Nodes to Upgrade specific nodes with the --nodes <integer_range_list> option.
select for
Specify the nodes in their upgrade order as a comma-separated list (for example, --nodes 7,3,2,5) or as a dash-
upgrade
separated range (for example, --nodes 1-7) of logical node numbers (LNNs).
NOTE: We recommend that you upgrade all the nodes. If you upgrade some nodes, a weekly alert is
sent to confirm that the upgrade is making progress. Do not leave the cluster in a partially upgraded
state for a prolonged period. Some new features in the upgrade might not be available until all the
nodes in the cluster have been upgraded and the upgrade is committed. Refer to the release notes for
the OneFS version that you are upgrading to for information about features that require all the nodes
to be upgraded.

The following example for OneFS 8.2.2 and later starts a rolling upgrade on nodes 7,3,2,5, in that order:

isi upgrade cluster start --rolling <install-image-path> --nodes 7,3,2,5

After the upgrade, a number of upgrade-related jobs may continue to run on the cluster for some time. During this time, the cluster is
accessible, but you might experience a decrease in cluster performance. After the jobs complete, performance will return to normal. At this
stage, the upgrade is complete, but is not committed. You can still roll back to the previous version of OneFS. Some new features in the
upgrade might not be available until the upgrade is committed.

Web Administration Interface instructions:

1. Log in to any node in the cluster through the web administration interface with the root account.
2. Click Cluster Management > Upgrade.
3. Click Upgrade OneFS.
4. Browse to the location of the installation image that you want to install. The file path must be accessible in an /ifs directory. Then
click Select.
5. In the Upgrade Type list, select Rolling upgrade.
6. Under Nodes to Upgrade, specify an upgrade option:
• To upgrade all the nodes in the cluster, click All nodes.
• To upgrade specific nodes, click Specify nodes and reboot order.
NOTE: We recommend that you upgrade all the nodes. If you upgrade some nodes, a weekly alert is sent to
confirm that the upgrade is making progress. Do not leave the cluster in a partially upgraded state for a
prolonged period. Some new features in the upgrade might not be available until all the nodes in the cluster have
been upgraded and the upgrade is committed. Refer to the release notes for the OneFS version that you are
upgrading to for information about features that require all the nodes to be upgraded.
7. If you are upgrading specific nodes, select the node or nodes in the Nodes Available for Upgrade list, and then click Add.
The selected node or nodes are added to the Nodes Selected for Upgrade section. To arrange the nodes in the preferred restart
order, click the arrows next to the nodes in the Nodes Selected for Upgrade section.
8. Optional: (Optional) To skip the pre-upgrade checks for compatibility issues, select Skip optional pre-upgrade checks.
NOTE: We recommend that you run the optional pre-upgrade checks. Before starting an upgrade, OneFS checks
that your cluster is healthy enough to complete the upgrade process. Some of the pre-upgrade checks are

Performing the OneFS upgrade 25

 mandatory, and will be performed even if you choose to skip the optional checks. All pre-upgrade checks contribute
to a safer upgrade.

9. Click Start Upgrade.

The cluster might display several confirmation messages. Confirm each message to continue the upgrade process.
NOTE: The isi upgrade cluster command runs asynchronously. The command does not run the entire upgrade
process; instead, it sets up the upgrade process, which nodes take turns controlling. For this reason, the command
returns quickly. To view the progress of the upgrade, use the isi upgrade view command or the web
administration interface.
After the upgrade, a number of upgrade-related jobs may continue to run on the cluster for some time. During this time, the cluster is
accessible, but you might experience a decrease in cluster performance. After the jobs complete, performance will return to normal. At this
stage, the upgrade is complete, but is not committed. You can still roll back to the previous version of OneFS. Some new features in the
upgrade might not be available until the upgrade is committed.

Configuration changes during a rolling upgrade

You can continue to manage data and modify some cluster configurations during a rolling upgrade. For example, you can modify SMB
shares and NFS exports. Dell EMC recommends that you make configuration changes from the node with the highest devid. The node
with the highest devid will be the last node to be upgraded and restarted and will help you avoid being disconnected during the upgrade
process.
Attempts to enable or configure settings on a node that has already been upgraded will fail until the upgrade is complete. New OneFS
features and software modules are not functional until all of the nodes in the cluster have been updated.

Client connections during rolling upgrades

Rolling upgrades allow users to access data before, during, and after the upgrade. However, as nodes are upgraded and restarted, users
may experience brief pauses in the time it takes to complete a read or write operation.
If the cluster is configured for dynamic IP allocation, client connections to restarted nodes are automatically reconnected. If a client is
reconnected to a node that has not yet been upgraded and restarted, the client might be required to re-establish a connection to the
cluster more than once.
The following table describes the expected behavior when a client is connected to a node that is restarted on a cluster that is configured
for dynamic IP allocation:

Client Expected client behavior

SMB2 Client quickly re-establishes a connection to a new node after the connection to the restarting node is
disrupted.

SMB3 Client transitions from the restarted node to a new node without disruption.

NFSv2 and NFSv3 Client transitions from the restarting node to a new node without disruption.

NFSv4 Clients that are connected to the cluster using static IP addresses wait for NFS serviceability to resume on the
nodes that they are connected to.
Clients that are connected to the cluster using dynamic IP addresses use NFSv4 failover support. The clients
transition from the restarting node to a new node without disruption.

NOTE: For more information, see article 457328, Best practices for NFS client settings.

Completing a simultaneous upgrade of OneFS

If you run a simultaneous upgrade, the OneFS cluster is upgraded simultaneously and then all of the nodes in the cluster are restarted
simultaneously. During a simultaneous upgrade, the entire cluster is down and unavailable for client connections.

26 Performing the OneFS upgrade

Perform a simultaneous upgrade
You can use the command-line interface or the web administration interface to upgrade the OneFS version that is running on a cluster.
Review the pre-upgrade steps in this guide, confirm cluster health, and resolve any compatibility issues that are discovered by the checks
before upgrading to the new version of OneFS.
Command Line Interface instructions:
1. Open a secure shell (SSH) connection on the lowest-numbered node in the cluster and log in with the root account.
2. To perform a simultaneous upgrade, run the following command, where <install-image-path> is the file path of the upgrade install
image. The file path must be accessible in an /ifs directory.

isi upgrade cluster start --simultaneous <install-image-path>

NOTE: The isi upgrade cluster command runs asynchronously. The command does not run the entire upgrade
process; instead, it sets up the upgrade process, which nodes take turns controlling. For this reason, the command
returns quickly. To view the progress of the upgrade, use the isi upgrade view command or the web
administration interface.
After the upgrade, a number of upgrade-related jobs may continue to run on the cluster for some time. During this time, the cluster is
accessible, but you might experience a decrease in cluster performance. After the jobs complete, performance will return to normal. At this
stage, the upgrade is complete, but is not committed. You can still roll back to the previous version of OneFS. Some new features in the
upgrade might not be available until the upgrade is committed.

Web Administration Interface instructions:

1. Log in to any node in the cluster through the web administration interface with the root account.
2. Click Cluster Management > Upgrade.
3. Click Upgrade OneFS.
4. Browse to the location of the installation image that you want to install. The file path must be accessible in an /ifs directory. Then
click Select.
5. In the Upgrade Type list, select Simultaneous Upgrade.
6. (Optional) To skip the pre-upgrade checks for compatibility issues, select Skip optional pre-upgrade checks.
NOTE: We recommend that you run the optional pre-upgrade checks. Before starting an upgrade, OneFS checks
that your cluster is healthy enough to complete the upgrade process. Some of the pre-upgrade checks are
mandatory, and will be performed even if you choose to skip the optional checks. All pre-upgrade checks contribute
to a safer upgrade.

7. Optional: Click Start Upgrade.

The cluster might display several confirmation messages. Confirm each message to continue the upgrade process.
NOTE: The isi upgrade cluster command runs asynchronously. The command does not run the entire upgrade
process; instead, it sets up the upgrade process, which nodes take turns controlling. For this reason, the command
returns quickly. To view the progress of the upgrade, use the isi upgrade view command or the web
administration interface.
After the upgrade, a number of upgrade-related jobs may continue to run on the cluster for some time. During this time, the cluster is
accessible, but you might experience a decrease in cluster performance. After the jobs complete, performance will return to normal. At this
stage, the upgrade is complete, but is not committed. You can still roll back to the previous version of OneFS. Some new features in the
upgrade might not be available until the upgrade is committed.

Adding a node to the cluster while an upgrade is in

progress
In some circumstances, you might need to add a node to the cluster while an upgrade is in progress and before the upgrade has been
committed.
To add a node during an upgrade, at least one node in the cluster must have been successfully upgraded to the target version of OneFS.
When a new node is added to the cluster during an upgrade, the system images the new node with the previously committed version of

Performing the OneFS upgrade 27

OneFS. Then as the cluster upgrade continues, the new node is upgraded to the target version of OneFS. If the cluster upgrade is
stopped and rolled back, all the nodes in the cluster are returned to the previously committed version of OneFS.
NOTE: Before adding new hardware to the cluster, first confirm that the hardware is compatible with the current
version of OneFS that is installed and the target version of OneFS that you are upgrading to. See the PowerScale
Supportability and Compatibility Guide for more information.

Committing an upgrade of OneFS

You must commit an upgrade to complete the upgrade process. Once you commit the upgrade, you cannot roll back to the previous
version of OneFS.
New features in the target version of OneFS are not available until the upgrade has been committed.

Commit an upgrade
You can commit an upgrade through the web administration interface or the command-line interface. Once you commit the upgrade, you
cannot roll back to the previous version of OneFS.
Command-Line Interface instructions:
1. Open a secure shell (SSH) connection on the lowest-numbered node in the cluster and log in with the root account.
2. To commit the upgrade, run the following command:

isi upgrade cluster commit

NOTE: Once you commit the upgrade, you cannot roll back to the previous version of OneFS. After the upgrade is
committed, all the new features are available.
Web Administration Interface instructions:
1. Log in to any node in the cluster through the web administration interface with the root account.
2. Click Cluster Management > Upgrade.
3. Click the Commit Upgrade to OneFS <version> button.
The cluster might display several confirmation messages. Confirm each message to continue the commit process.

Rolling back an upgrade of OneFS

The upgrade rollback feature allows you to stop the upgrade that is in progress and restore all the upgraded nodes to the previous
committed version of OneFS.
NOTE: You can roll back an upgrade only if the upgrade process is still active and the upgrade has not been committed.
A committed upgrade cannot be rolled back.

Roll back an upgrade

Rolling back will stop an upgrade and return to the previous committed version of OneFS.
NOTE: The rollback process must restart all the upgraded nodes simultaneously. This will temporarily disrupt cluster
services and data availability.
Command-Line Interface instructions:
1. Open a secure shell (SSH) connection on the lowest-numbered node in the cluster and log in with the root account.
2. To roll back the upgrade, run the following command:

isi upgrade cluster rollback

The rollback process must restart all the upgraded nodes simultaneously. This will temporarily disrupt cluster services and data availability.
After the OneFS image is updated on the final node and the node restarts, the web administration interface login page appears. If the login
page does not appear, clear the web browser cache and reload the page. A number of upgrade-related jobs may continue to run on the

28 Performing the OneFS upgrade

cluster for some time. During this time, the cluster is accessible, but you might experience a decrease in cluster performance. After the
jobs complete, performance will return to normal.

NOTE: The rollback process must restart all the upgraded nodes simultaneously. This will temporarily disrupt cluster
services and data availability.
Web Administration Interface instructions:
1. Log in to any node in the cluster through the web administration interface with the root account.
2. Click Cluster Management > Upgrade.
3. Click the Roll Back to OneFS <version> button.
The cluster might display several confirmation messages. Confirm each message to continue the rollback process.
The cluster displays the rollback progress.

Verify the OneFS installation

After you install OneFS, verify that the installation was successful.
1. Confirm that the health of all the nodes in the cluster is OK by running the following command:

isi stat

2. Remove the installation files from the /ifs/data directory by running the following command where <installation_file_name> is the
name of the installation file:

rm /ifs/data/<installation_file_name>

3. Collect information about the cluster by running the following command:

isi_gather_info

Performing the OneFS upgrade 29

 5
Completing post-upgrade tasks
Topics:
• Checklist - Post Upgrade
• About post-upgrade tasks
• Allow the Upgrade job to run
• Verify operational status
• Re-establish user privileges
• Restore client connections, and test the workflow
• Verify Kerberos migration
• Reapply custom settings
• Modify custom scripts
• Reinstall Aspera
• Install the latest Patch

Checklist - Post Upgrade

Use this checklist to help you track your progress as you perform the post-upgrade tasks.

Allow upgrade-related jobs to run

Verify operational status

Re-establish user privileges

Restore client connections and test your workflow

Verify Kerberos migration

Restore custom settings

Modify custom scripts

Install the latest patch

About post-upgrade tasks

After an upgrade, you should perform a number of restoration and change management tasks to ensure that your cluster performs and
behaves as expected. It is important that you build time into your upgrade plan to re-establish custom settings and privileges, and re-
enable connections and features. You should also make time to modify settings for new and changed features.
NOTE: Once the cluster has been upgraded, the system might run an Upgrade job that must be allowed to finish.

Allow the Upgrade job to run

After an upgrade to OneFS 8.x, an important job titled Upgrade that upgrades on-disk data structures, might continue to run for a while.
The Upgrade job must be allowed to run to completion.
NOTE: If the Upgrade job fails, OneFS will retry the job. If the Upgrade job is canceled, OneFS will queue the job for the
next time a device joins or rejoins the cluster.
Although the cluster remains accessible while the Upgrade job is running, the job might temporarily decrease the cluster's performance.

30 Completing post-upgrade tasks

• To check for running system jobs, run the following command:

isi job status

Verify operational status

You can run a series of commands as root to help verify that the PowerScale cluster is working correctly after an upgrade.
If you find an unresolvable issue, contact PowerScale Technical Support.
1. Check the new version number of the cluster:

isi_for_array -s uname -a

2. View the status of the cluster and ensure all the nodes are operational:

isi status

3. Check the devices in the nodes to validate the status of the drives:

isi_for_array -s "isi devices list"

4. Check the status of jobs and resume the jobs that you paused for the upgrade:

isi job status

5. Review the list of events, and address any critical events:

isi event groups list --sort=severity

6. Ping all the cluster's internal and external interfaces to verify network connectivity and to help verify that SmartConnect works
correctly.
7. Verify the network interfaces:

isi network interfaces list

8. Verify the subnets:

isi network subnets list --verbose

9. Verify the pools:

isi network pools list --verbose

10. Review the cluster's other log files to check for stray problems:

cat /var/log/messages

11. Check the cluster's input and output:

isi statistics system

12. Check the global SMB settings:

isi smb settings global view

13. Check the status of the firmware to ensure that the firmware is consistent across nodes:

isi upgrade firmware devices

Completing post-upgrade tasks 31

14. Ensure that all the licenses carried over and remain up to date:

isi license licenses list

NOTE: Following an upgrade to OneFS 8.1.0.0 or later, you must update your OneFS license by generating a license
activation file and submitting it to Software Licensing Central (SLC). For instructions on how to create a new license
file for your OneFS cluster, see the Licensing section of the OneFS Administration Guide.

15. Check the status of the authentication providers to ensure that they remain active:

isi auth status --verbose

16. Review the list of SyncIQ jobs:

isi sync jobs list

17. Check the SyncIQ job reports:

isi sync reports list

18. Review the list of the scheduled snapshots:

isi snapshot schedules list

19. (Optional) If you use SRS, confirm that it is re-enabled.

If you upgraded to OneFS 8.0.x or 8.1.x, use the following command:

isi_phone_home --enable

If you upgraded to OneFS 8.2.x, use the following command:

isi esrs telemetry modify --enabled yes

Re-establish user privileges

After you upgrade, re-establish user privileges and roles. You can log in to the cluster through SSH as root or though an administrator
account if that role has been assigned to any users.
NOTE: If the system administered the user roles through RBAC before the upgrade, any custom roles that existed are
still in place. However, if the privileges assigned to built-in roles have changed in the new version, the users who are
assigned to those built-in roles will have those new privileges.
See the OneFS CLI Administration Guide or the OneFS Web Administration Guide for more information.

Restore client connections, and test the workflow

After the OneFS upgrade, restore client connections and confirm that all users, clients, and applications can access the cluster.
Test your workflows to ensure that they function correctly.

NOTE: If you are using NDMP backups on your cluster, re-enable the NDMP service and test that it's working correctly.

Verify Kerberos migration

If you are using Kerberos authentication, you must verify that the Kerberos providers and settings have been migrated successfully.
Command-Line Interface instructions:
Verify that Kerberos authentication providers and settings are correctly represented in the output of each of these commands:
• ○ isi auth krb5 realm list

32 Completing post-upgrade tasks

 ○ isi auth krb5 domain list
○ isi auth krb5 spn list
○ isi auth settings krb5 view

Web Administration Interface instructions:

1. Click Access > Authentication Providers > Kerberos Provider.
2. In the Kerberos Realms, Kerberos Domains, and Kerberos Providers tables, verify the Kerberos providers are correct.
3. In the Kerberos Settings area, verify the Kerberos authentication settings are correct.

Reapply custom settings

Some custom settings might not have been preserved during the upgrade. Reapply the custom settings that you backed up and recorded
when you performed the pre-upgrade tasks.
The custom settings include:
• SMB audit logging
• Passwords for local user accounts
• Changes to system controls
• Aspera
• Cron jobs
• Certificates
• Static routes

Modify custom scripts

Review OneFS CLI Mappings to confirm whether any command syntax changes were implemented in the target version of OneFS to
which you upgraded. If command syntax changes were implemented that affect your custom scripts, you must update and test your
customer scripts.
NOTE: Custom scripts might not be compatible between OneFS versions. Ensure that your scripts work on a test
cluster before implementing them on your production systems.

Reinstall Aspera
If you were running Aspera for PowerScale before you upgraded your cluster, you must reinstall Aspera after you upgrade. Aspera is not
supported in OneFS 8.2.0 and later.
1. Run the following command to verify that the upgrade process enabled the Aspera Central service and the Aspera node daemon:

isi services

2. If the Aspera service is not running properly, you must download and re-install a compatible version of Aspera from the Aspera
Enterprise Server website and verify that the Aspera Central service is enabled.
• For information about downloading and installing Aspera for OneFS, see article 493022, How to download Aspera for OneFS.
• For information about which versions of Aspera are compatible with your version of OneFS, see the PowerScale Supportability and
Compatibility Guide.

Install the latest Patch

Install the latest patch for the target version of OneFS to which you upgraded.
1. See Current PowerScale OneFS Patches to view a list of patches that are available for your version of OneFS.
2. Install the latest roll-up patch.

Completing post-upgrade tasks 33

 6
Troubleshooting your upgrade
Topics:
• Troubleshooting overview

Troubleshooting overview
If you experience problems with your upgrade, check the upgrade logs and review common upgrade issues.
You can search for OneFS documentation, troubleshooting guides, and knowledge base articles on the Online Support site.
You can go through the Customer Troubleshooting Guide: Upgrades From OneFS 8.x to a later version for troubleshooting information
that is related to upgrade failures and upgrade error messages.
If you need additional help with troubleshooting, contact PowerScale Technical Support.

Review the upgrade log files

After an upgrade, you can run the isi_upgrade_logs command to retrieve and review errors that were logged during the upgrade.
NOTE: If no errors were logged, the following message appears on the console:
No Upgrade Process Errors Found, if you ran an assessment please use the -a option

Common issues
Some problems with cluster upgrades might be caused by the following common upgrade issues:

Dropped node
If a node does not successfully reboot after a OneFS upgrade, you will not be allowed to commit the upgrade. If the upgrade is not
committed, then you will not be allowed to make changes to certain features, such as SMB shares and NFS exports.
Follow these steps for any node that didn't successfully reboot:
1. Smartfail the node that did not come back after rebooting.
2. Check to ensure that the name of the smartfailed node is removed from the cluster node list.
This operation might take some time depending on the amount of data that is migrated off of the node.
3. Reboot any node in the cluster.

SMB client connections

If any of the following SMB2 client connection errors occur, you might need to adjust the SMB client-credit minimum:
• The SMB clients show system "error 51" or "error 0x80070033."
• The Microsoft Windows event log contains the following entry:
Operating system error 64 (The specified network name is no longer available.)
• An error similar to the following appears in the packet captures on the servers:
58031 9.752678000 10.232.21.62 146.168.81.131 SMB2 131 Write Response, Error:
STATUS_INSUFF_SERVER_RESOURCES
For information on how to adjust the SMB client-credit minimum, see article 460492, SMB2 connections stop responding because client
credit minimum value is too high.

34 Troubleshooting your upgrade

NFS performance
If you are experiencing lower than expected NFS performance after the upgrade, it might be due to the min and max number of NFS
server threads.
It is recommended that you set threads_min and threads_max to the same value. Increasing the number of threads can improve
performance at the expense of stability. Before you change the number of threads, contact support to determine the values that work
best for your cluster; the values vary by CPUs, memory, the number of nodes, and other factors.
You can adjust the number of NFS threads by running the following commands as root, where x is the number of threads.
1. To modify the minimum number of threads, run the following command:

isi_sysctl_cluster vfs.nfsrv.rpc.threads_min=x

2. To modify the maximum number of threads, run the following command:

isi_sysctl_cluster vfs.nfsrv.rpc.threads_max=x

Custom 32-bit applications

The OneFS system user space is 64-bit. If you are running any custom tools that require a 32-bit user space, you must recompile them
after the upgrade.
NOTE: Running your own 32-bit applications on an PowerScale cluster is neither recommended nor supported.

Troubleshooting your upgrade 35