Replacing a disk in GPFS cluster Ⓒ

===================================

This is a real scenario on how to change/replace a disk inside the GPFS cluster.
November 2017

1) We have the following disks in the AIX server:

# lspv
hdisk0 00f79651f3734b2e rootvg active
hdisk1 00f79651f9fba3d7 nsd1

# mmlsnsd -L
File system Disk name NSD volume ID NSD servers
-----------------------------------------------------------------
gpfs1 nsd1 AC1046135A01C1D8 (directly attached)

2) We are in need to replace hdisk1 (which is nsd1) by another new disk.

- We have managed to map a new LUN to both AIX nodes:
# lspv
hdisk0 00f79651f3734b2e rootvg active
hdisk1 00f79651f9fba3d7 nsd1
hdisk2 00f79651f9fbba94 None

3) We will create nsd2 on hdisk2, so firstly we will create the stanza file,
eg: /var/mmfs/gen/nsd.conf which will contain the line:
hdisk2:::dataAndMetadata::nsd2:

# mmcrnsd -F /var/mmfs/gen/nsd.conf
# cat /var/mmfs/gen/nsdpvol
hdisk1 nsd1
hdisk2 nsd2

# cat /var/mmfs/gen/nsdmap
AC1046135A01C1D8 /dev/rhdisk1 hdisk
AC1046135A01C1D9 /dev/rhdisk2 hdisk

4) Now we have two NSD's under the GPFS cluster, one is holding the data,
And the other is free:
# mmlsnsd -L
File system Disk name NSD volume ID NSD servers
-----------------------------------------------------------------
gpfs1 nsd1 AC1046135A01C1D8 (directly attached)
(free disk) nsd2 AC1046135A01C1D9 (directly attached)

- The free one is nsd2 -> hdisk2:

# mmlsnsd -F
File system Disk name NSD servers
----------------------------------------------
(free disk) nsd2 (directly attached)

5) We will be replacing hdisk1 (nsd1 of gpfs1) with the free hdisk2 (nsd2):
- We have three methods to perform this activity:

I. We can use mmrpldisk command and put the filesystem name, old NSD and the
New NSD name - But this will not work in our case, as we have only 1 NSD
In file system gpfs1. So if we tried it as:
# mmrpldisk <FSName> <oldNSD> <newNSD>
# mmrpldisk gpfs1 nsd1 nsd2
We will end up with an error like the below:
"You cannot replace a disk when it is the only remaining disk in the file system"

II. Doing it in two steps by adding the new NSD with option "-r" to rebalance all existing
Files in the file system to make use of the new disks, then delete the old NSD:
# mmadddisk <FSname> <newNSD> -r
# mmdeldisk <FSName> <oldNSD>
Note: Rebalancing of files is an I/O intensive and time consuming operation, and is important
Only for filesystems with large files that are mostly invariant

In our case we will run:

# mmadddisk gpfs1 nsd2 -r
The following disks of gpfs1 will be formatted on node gpfs_node1:
nsd2: size 102400 MB
Extending Allocation Map
Checking Allocation Map for storage pool system
62 % complete on Thu Nov 9 17:20:00 2017
100 % complete on Thu Nov 9 17:20:03 2017
Completed adding disks to file system gpfs1.
mmadddisk: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
Restriping gpfs1 ...
Scanning file system metadata, phase 1 ...
31 % complete on Thu Nov 9 17:20:26 2017
74 % complete on Thu Nov 9 17:20:33 2017
99 % complete on Thu Nov 9 17:20:37 2017
100 % complete on Thu Nov 9 17:20:37 2017
Scan completed successfully.
Scanning file system metadata, phase 2 ...
Scan completed successfully.
Scanning file system metadata, phase 3 ...
Scan completed successfully.
Scanning file system metadata, phase 4 ...
Scan completed successfully.
Scanning user file metadata ...
100.00 % complete on Thu Nov 9 17:20:40 2017 (102464 inodes with total 557 MB data processed)
Scan completed successfully.
Done

# mmlsnsd -L
File system Disk name NSD volume ID NSD servers
-----------------------------------------------------------------
gpfs1 nsd1 AC1046135A01C1D8 (directly attached)
gpfs1 nsd2 AC1046135A01C1D9 (directly attached)

# mmdeldisk gpfs1 nsd1

Deleting disks ...
Scanning file system metadata, phase 1 ...
74 % complete on Thu Nov 9 17:23:11 2017
99 % complete on Thu Nov 9 17:23:14 2017
100 % complete on Thu Nov 9 17:23:14 2017
Scan completed successfully.
Scanning file system metadata, phase 2 ...
Scan completed successfully.
Scanning file system metadata, phase 3 ...
Scan completed successfully.
Scanning file system metadata, phase 4 ...
Scan completed successfully.
Scanning user file metadata ...
100.00 % complete on Thu Nov 9 17:23:18 2017 (102464 inodes with total 557 MB data processed)
Scan completed successfully.
Checking Allocation Map for storage pool system
61 % complete on Thu Nov 9 17:23:24 2017
100 % complete on Thu Nov 9 17:23:27 2017
Attention: A disk being removed reduces the number of failure groups to 1,
which is below the number required for replication: 2.
New blocks will be allocated from the remaining disks,
but files will be unreplicated and hence at risk.
tsdeldisk completed.
mmdeldisk: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

# mmlsnsd -L
File system Disk name NSD volume ID NSD servers
-----------------------------------------------------------------
gpfs1 nsd2 AC1046135A01C1D9 (directly attached)
(free disk) nsd1 AC1046135A01C1D8 (directly attached)
 # mmlsnsd -F
File system Disk name NSD servers
----------------------------------------------
(free disk) nsd1 (directly attached)

# mmdelnsd nsd1
mmdelnsd: Processing disk nsd1
mmdelnsd: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

# mmlsnsd
File system Disk name NSD servers
----------------------------------------------
gpfs1 nsd2 (directly attached)

# lspv
hdisk0 00f79651f3734b2e rootvg active
hdisk1 00f79651f9fba3d7 None
hdisk2 00f79651f9fbba94 nsd2

# cat /var/mmfs/gen/nsdpvol
hdisk2 nsd2

# cat /var/mmfs/gen/nsdmap
AC1046135A01C1D9 /dev/rhdisk2 hdisk

III. Using the mmrestripefs command to rebalance or restore the replication of all files in
Filesystem. The mmrestripefs will move existing filesystem data between different disks
In the filesystem based on changes to the disk state made by the mmchdisk command.

- So, the current state is:

# lspv
hdisk0 00f79651f3734b2e rootvg active
hdisk1 00f79651f9fba3d7 None
hdisk2 00f79651f9fbba94 nsd2
- Creating back nsd1 on hdisk1:
# mmcrnsd -F /var/mmfs/gen/nsd.conf
Content of /var/mmfs/gen/nsd.conf is: hdisk1:::dataAndMetadata::nsd1:

# mmlsnsd -L
File system Disk name NSD volume ID NSD servers
-----------------------------------------------------------------
gpfs1 nsd2 AC1046135A01C1D9 (directly attached)
(free disk) nsd1 AC1046145A047D49 (directly attached)

# mmchdisk gpfs1 suspend -d "nsd2"

# mmlsdisk gpfs1
disk driver sector failure holds holds storage
name type size group metadata data status availability pool
-----------------------------------------------------------------------------------------------
nsd2 nsd 512 -1 yes yes suspended up system

# mmadddisk gpfs1
The following disks of gpfs1 will be formatted on node gpfs_node1:
nsd1: size 102400 MB
Extending Allocation Map
Checking Allocation Map for storage pool system
70 % complete on Thu Nov 9 18:22:20 2017
100 % complete on Thu Nov 9 18:22:22 2017
Completed adding disks to file system gpfs1.
mmadddisk: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
# mmlsnsd -L
File system Disk name NSD volume ID NSD servers
--------------------------------------------------------------------------------------------
gpfs1 nsd2 AC1046135A01C1D9 (directly attached)
gpfs1 nsd1 AC1046145A047D49 (directly attached)

# mmlsdisk gpfs1
disk driver sector failure holds holds storage
name type size group metadata data status availability pool
------------ -------- ------ ----------- -------- ----- ------------- ------------ ---------
nsd1 nsd 512 -1 yes yes ready up system
nsd2 nsd 512 -1 yes yes suspended up system

# mmrestripefs gpfs1 -r
Scanning file system metadata, phase 1 ...
15 % complete on Thu Nov 9 18:29:40 2017
31 % complete on Thu Nov 9 18:29:45 2017
58 % complete on Thu Nov 9 18:29:49 2017
74 % complete on Thu Nov 9 18:29:53 2017
99 % complete on Thu Nov 9 18:29:58 2017
100 % complete on Thu Nov 9 18:29:58 2017
Scan completed successfully.
Scanning file system metadata, phase 2 ...
Scan completed successfully.
Scanning file system metadata, phase 3 ...
Scan completed successfully.
Scanning file system metadata, phase 4 ...
Scan completed successfully.
Scanning user file metadata ...
100.00 % complete on Thu Nov 9 18:30:03 2017 (102464 inodes with total 557 MB data processed)
Scan completed successfully.

# mmdeldisk gpfs1 nsd2

Deleting disks ...
Scanning file system metadata, phase 1 ...
74 % complete on Thu Nov 9 17:23:11 2017
99 % complete on Thu Nov 9 17:23:14 2017
100 % complete on Thu Nov 9 17:23:14 2017
Scan completed successfully.
Scanning file system metadata, phase 2 ...
Scan completed successfully.
Scanning file system metadata, phase 3 ...
Scan completed successfully.
Scanning file system metadata, phase 4 ...
Scan completed successfully.
Scanning user file metadata ...
100.00 % complete on Thu Nov 9 17:23:18 2017 (102464 inodes with total 557 MB data processed)
Scan completed successfully.
Checking Allocation Map for storage pool system
61 % complete on Thu Nov 9 17:23:24 2017
100 % complete on Thu Nov 9 17:23:27 2017
Attention: A disk being removed reduces the number of failure groups to 1,
which is below the number required for replication: 2.
New blocks will be allocated from the remaining disks,
but files will be unreplicated and hence at risk.
tsdeldisk completed.
mmdeldisk: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

# mmlsnsd -L
File system Disk name NSD volume ID NSD servers
-----------------------------------------------------------------
gpfs1 nsd1 AC1046145A047D49 (directly attached)
(free disk) nsd2 AC1046135A01C1D9 (directly attached)

# mmlsdisk gpfs1
disk driver sector failure holds holds storage
name type size group metadata data status availability pool
------------ -------- ------ ----------- -------- ----- ------------- ------------ --------
nsd1 nsd 512 -1 yes yes ready up system
 # mmdelnsd nsd2
mmdelnsd: Processing disk nsd2
mmdelnsd: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.

# lspv
hdisk0 00f79651f373504a rootvg active
hdisk1 00f79651f9fba3d7 nsd1
hdisk2 00f79651f9fbba94 None

Tested and written by:

Ahmed Mashhour
Global Subject Matter Expert,
EMEA TSS - AIX Technical Lead
[email protected]