NOTE: Don't "remove" drives, you replace them with servicemag.

REPLACING A FAILED PHYSICAL DISK WITH NEW ONE ON HP 3PAR STORAGE

cli % showpd -failed -degraded -//> Identify if there is a failed
disk on the system

cli % servicemag status -//> Check if there is an ongoing

servicemag operation

cli % servicemag start -pdid 125 -//> Start servicemag operation

Are you sure you want to run servicemag?

select q=quit y=yes n=no: y
servicemag start -pdid 125
... servicing disks in mag: 3 1
... normal disks: WWN [XXXXXXXXXXXX] Id [126] diskpos [2]
.................... WWN [XXXXXXXXXXXX] Id [127] diskpos [3]
.................... WWN [XXXXXXXXXXXX] Id [206] diskpos [0]
... not normal disks: WWN [XXXXXXXXXXXX] Id [125] diskpos [1]

The servicemag start operation will continue in the background.

NOTE: keep track of the status of the servicemag process and physical disk
state (Note that
magazine position is 3 1, we later use it after we brought it online):

cli% servicemag status

Cage 3, magazine 1:
The magazine is being brought offline due to a servicemag start.
The last status update was at Thu Sep 28 11:19:55 2017.
Unable to provide a relocation estimate
servicemag start -pdid 125 -- is in Progress

cli% showpd -state

Id CagePos Type -State-
---------------------Detailed_State---------------------
0 0:0:0 FC normal normal
1 0:0:1 FC normal normal
2 0:0:2 FC normal normal
3 0:0:3 FC normal normal
..
..
125 3:1:1 FC failed
vacated,invalid_media,smart_threshold_exceeded,servicing

cli% servicemag status

Cage 3, magazine 1:
The magazine was successfully brought offline by a servicemag start
command.
The command completed Thu Sep 28 13:27:54 2017.
servicemag start -pdid 125 -- Succeeded

NOTE: After the magazine brought offline successfully, we can insert a new one
by removing
the failed disk physically. We ensure new disk plugged into the right
place so we can
 bring the magazine online:

cli% servicemag resume 3 1

Are you sure you want to run servicemag?
select q=quit y=yes n=no: y
servicemag resume 3 1
... mag 3 1 already onlooped
... firmware is current on pd WWN [XXXXXXXXXXXX] Id [126]
... firmware is current on pd WWN [XXXXXXXXXXXX] Id [127]
... firmware is current on pd WWN [XXXXXXXXXXXX] Id [206]
... firmware is current on pd WWN [XXXXXXXXXXXX]
... firmware is current on pd WWN [XXXXXXXXXXXX] Id [125]
... checking for valid disks...
... checking for valid disks...
... disks in mag : 3 1
... normal disks: WWN [XXXXXXXXXXXX] Id [122] diskpos [1]
.................... WWN [XXXXXXXXXXXX] Id [126] diskpos [2]
.................... WWN [XXXXXXXXXXXX] Id [127] diskpos [3]
.................... WWN [XXXXXXXXXXXX] Id [206] diskpos [0]
... not normal disks: WWN [XXXXXXXXXXXX] Id [125]
... verifying spare space for disks 126 and 126
... verifying spare space for disks 127 and 127
... verifying spare space for disks 206 and 206
... verifying spare space for disks 125 and 122
... playback chunklets from pd WWN [XXXXXXXXXXXX] Id [122]
... playback chunklets from pd WWN [XXXXXXXXXXXX] Id [126]
... playback chunklets from pd WWN [XXXXXXXXXXXX] Id [127]
... playback chunklets from pd WWN [XXXXXXXXXXXX] Id [206]

NOTE: The servicemag resume operation will continue in the background.

NOTE: It will take a couple of hours to recover:

cli% servicemag status

Cage 3, magazine 1:
The magazine is being brought online due to a servicemag resume.
The last status update was at Thu Sep 28 15:58:25 2017.
Chunklets relocated: 6 in 2 minutes and 29 seconds
Chunklets remaining: 1442
Chunklets marked for moving: 1442
Estimated time for relocation completion based on 24 seconds per
chunklet is: 9 hours, 36 minutes and 48 seconds
servicemag resume 3 1 -- is in Progress

MOVING FILES/ REMOVING PD

NOTE: You would need to follow the process below, you can skip 1 as you have
already moved the data

1. Empty the drive (assuming plenty more drives above the cpg set-size *
rowsize) behind the same node) -> movepdtospare -vacate -nowait

2. Remove spares -> removespare :a

3. Dismiss pd from config -> dismisspd

4. Offloop drive -> controlmag offloop : (drive maintenance led will now be
lit)

5. Physically remove drive

OTHER COMMANDS
cli% checkhealth <//- Check the health
cli% showcage <//- Show cage Name Loop and other information
cli% showversion <//- Show release version

cli% showpdch -fail <//- Check failed chunklets

REMOVE A PD PERMANENTLY

RemovePD
Step Command Purpose
01 setpd ldalloc off <DiskID> To stop data to drive -
Repeat for all PDs
02 showpd -s Make sure disks are in
Degraded state
03 movepdtospare -f -vacate -nowait <DiskID> Vacate all data from PD
04 showpdch -mov See the Remaining chunklets to be
moved
05 movech -perm -ovrd X:Y OPTIONAL: If getting an
error, move Chunklet manually
(example X is the PDID and Y
is the Chunklet number) repeat till disk is empty
6 showpdch -spr <DiskID> Shows the Spare chunklets on
the PD
7 removespare PDID:a Removes the Spare chunklets
from the PD
8 dismisspd <DiskID> Dismisses the PD from the
3PAR
9 Redo Steps 3-7 for all PDs
10 Remove All PDs from Cage Remove the vacated PDs from
the 3PAR
11 tunesys -chunkpct 1 Tune System

HPE 3PAR Storage Systems - Drive Cage (DC) Midplane Replacement Steps

Environment /Questions/Symptoms /Cause

Answer/Solution

Environment /HPE 3PAR Storage Systems

Questions/Symptoms

Replacing a cage midplane

Drive Cage Midplane replacement

Cause
Answer/Solution

FIX:

When it becomes necessary to replace a cage midplane there are certain steps
to follow.
Steps for Replacing a cage midplane:
Add temporary spare space if necessary. (See Below Determining cage
vacate and Adding Temporary spare space)

Vacate all used chunklets from this cage

 ALL pdids in cage to be vacated

setpd ldalloc off pdid pdid pdid pdid

All dmags in the cage

servicemag start cageid dmag

NOTE: At 2.2.2 MU5 and above you can launch a servicemag start to each dmag
when the previous
servicemag start has been placed in the background. Below 2.2.2 MU5 you
must wait until
the previous servicemag start has completed before continuing on with
the next. On DC2s
and DC3s the last ESI dmag will cause an error and the servicemag start
will state that it failed but it will be vacated

Verify all disks are vacated. (showpdch or showpd -c for ALL pdids in cage
and verify ZERO used chunklets)

Power off all supplies (After confirming all disks have been 100% vacated)

Pull fcals and dmags out 1 inch.

Remove cage power supplies.

Remove cage sheetmetal securing screws (They are the screws with guide pins).

Replace midplane.

Install power supplies (Do not power on).

Reseat all fcals and dmags

Power on cage.

Verify cage is healthy (showcage -d new-cageid)

Upgradecage (If necessary update the cage firmware)

Repopulate chunklets to new cage

cageid will be different from the servicemag start: servicemag resume new-
cageid dmag

remove the old cage from the system configuration: servicecage remove old-
cageid

clear the servicemag start status on the old-cageid: servicemag clearstatus

old-cageid dmag

Verify all servicemag resumes have successfully completed (servicemag status

new-cageid dmag)

Determining if cage can be vacated:

Determine if system has sufficient sparing.

movepdtospare -dr pdid pdid pdid pdid ....

 ALL pdids in the cage. Look for this error:

Error: Could not find a relocation target for chunklet PD:CH (where PD = pdid
CH = chunklet number)

If you see this error then you will not be able to vacate the cage and
temporary spare space must be added.

Otherwise you will see extended output indicating source and destination are
valid for each chunklet to move.

Run chunktomove serial# cageid

NOTE: Please do step 1. movepdtospare -dr for confirmation!. The output will
let you know whether there is enough sparing on the system to vacate the cage.

Example: system 1000700 cage10

./chunk2move 1000700 10

Based on config.080519.055805.0001 for system 1000700.

Total used chunklets in cage10 = 12244 <--------------Cage has 12244 used

chunklets to relocate

System has 24996 available chunklets for relocation <--------------System

has 24996 available chunklets for sparing.

Be sure raid5 ha ch won't prevent the vacates! <--------------Please do

step A. movepdtospare -dr for confirmation!

Approximate servicemag(s) start completion = 612 minutes <--------------

Estimate only valid if
all dmags are vacating
concurrently

NOTE: The available chunklets for relocation include FC spare/free and NL

spare/free chunklets.
Servicemag will utilize all available chunklets for sparing including FC and
NL spare and free chunklets.

Adding Temporary spare space if necessary:

If necessary you may have to add a temporary cage to install the temporary
dmag space.

It is much easier if system has empty cage slots available but if not then it
will be necessary to add additional cage.

Adding a cage involves installing power supplies, fcals, fibre cables, cage
chassis and dmags

For DC2s populate in this order 0 9 1 8 2 3 4 5 6 7

For DC3s populate in this order 0 4 8 12, 3 7 11 15, 1 5 9 13, 2 6 10 14

For DC1s populate in this order 0 1 2 3 4

 Install dmag(s) into available slots. Select those cages on the same node
pair if possible,
otherwise any cage will do. Sometimes it may be necessary to add an
additional cage to accomplish this.
This can be direct connect to node or daisy chained with current cage.

Verify the TEMP disks have current firmware: (They will have a status of new)
showpd -i
(examine the TEMP disks) upgradepd -w WWN (upgrade all TEMP disk(s) requiring
current firmware BEFORE admitting)

admitpd -nold WWN (admit the TEMP disks with ldalloc off)

createspare pdid:a (Assign ALL chunklets as spares on ALL the TEMP disks)

setpd ldalloc on pdid pdid pdid pdid .... ALL the TEMP disks so they can be
used for sparing

Removing Temporary spare space if necessary:

Verify there are no used chunklets on the TEMP disks. (showpd -c pdid pdid
pdid pdid .... ALL TEMP disks)

setpd ldalloc off pdid pdid pdid pdid .... ALL pdids of TEMP disks

removespare pdid:a pdid:a pdid:a pdid:a .... ALL pdids of TEMP disks

dismisspd pdid pdid pdid pdid .... ALL pdids of TEMP disks

controlpd spindown WWN WWN WWN WWN .... ALL WWN of TEMP disks

controlmag offloop cageX Y .... For all TEMP dmags installed for sparing

Pull TEMP dmags and install blanking plates

If necessary you may have to deinstall temporary cage as well.

MANUAL DISK REPLACEMENT PROCEDURE IN CASE SERVICEMAG

PROCEDURE
In case servicemag command fail for some reasons,
then you are pushed to do it manually, using the whole sets of commands.

1. First thing to do is to stop disk from being use. To achieve it, 3PAR has
special command.

setpd ldalloc off <pd_id>

And to see detailed state of disk use command showpd -s.

3PAR-cluster cli% showpd -s <pd_id

2. Now you can initiate the movement process for data from specified physical
disk to location chosen by system, which is one of the main steps in terms
of disk replacement.

The suitable command is movepdtospare with -vacate option.

Vacate option makes moves pernament and removes source tags after
recolocation.
The -f parameter means that no confirmation is required. In case this command
fail,
you will be forced to do it manually, chunklet by chunklet.

3PAR-cluster cli% movech -perm -ovrd <pd_id>:<chunklet_location>

where: -perm: chunklet are moved pernamently and original location will be
forgot.
-ovrd: allows to move chunklet to some destination even if it will have
impact on
quality. Option is necessary with -perm parameter.

3. Time to see whether we have any spare chunklets on disk designated for
removal,
as previous step only moved data chunklets.

To display chunklets marked as spare use showpdch -spr command.

3PAR-cluster cli% showpdch -spr <pd_id>

4. Time to see whether we have any spare chunklets on disk designated for
removal,
as previous step only moved data chunklets.

Command designated for that kind of task is shown below. It will remove all
spare chunklets off the disk.
After execution check again whether any spares exist.

3PAR-cluster cli% removespare <pd_id>:a

5. After all previous steps you can safely remove physical disk definition
from system.
Hold on with physical disk replacement at this step.

3PAR-cluster cli% dismisspd <pd_id>

6. Check if dismissed disk shown us new. If yes, then it can be safely remove
from magazine.

7. In case you put in new disk and disk will not be automatically added to
the system, you have to do it manually

First thing is to determine the WWN of disk. Check this with showpd -i
command.

3PAR-cluster cli% showpd -i <pd_id>

After that use admitpd command to make new disk operational for system.

3PAR-cluster cli% admitpd <disk_wwn>

At the end, tunesys is necessary to make the proper layout of chunklets within
CPGs.

3PAR-cluster cli% tunesys