Live Partition Mobility: Viraf Patel
Live Partition Mobility: Viraf Patel
Live Partition Mobility: Viraf Patel
Agenda
Overview
Prerequisites Validation Migration Effects Demo Supplemental Material
4-May-12
Overview
Live Partition Mobility moves a running logical partition from one POWER6 server to another one without disrupting the operation of the operating system or applications Network applications may see a brief (~2 sec) suspension toward the end of the migration, but connectivity will not be lost
4-May-12
Overview
Live Partition Mobility is useful for
Server consolidation Workload balancing Preparing for planned maintenance e.g., planned hardware maintenance or upgrades In response to a warning of an impending hardware failure
4-May-12
Overview
Inactive partition migration moves a powered-off partition from one system to another
Less restrictive validation process because the migrated partition will boot on the target machine; no running state needs to be transferred
4-May-12
Overview
4-May-12
Prerequisites
From Fix Central website, Partition Mobility: http://www14.software.ibm.com/webapp/set2/sas/f/pm/component.html
4-May-12
Prerequisites
Two POWER6 systems managed by a single HMC or IVM on each server Advanced POWER Virtualization Enterprise Edition VIOS 1.5.1.1 (VIO 1.5.0.0, plus Fixpack 10.1) plus interim fixes IZ08861.071116.epkg.Z Partition Mobility fix 642758_vio.080208.epkg.Z VIO MPIO fix AX059907_3.080314.epkg.Z USB Optical Drive fix IZ16430.080327.epkg.Z various Qlogic Emulex FC fixes retrieve interim fixes, place in VIO at /home/padmin/interim_fix # emgr d e IZ16430.080327.epkg.Z v3 (as root, to see description) $ updateios dev /home/padmin/interim_fix install accept (install as padmin) VIOS 1.5.2.1 (VIO 1.5.0.0 plus Fixpack 11.1) rolls up all interim fixes Preferred Virtualized SAN Storage (rootvg and application vgs) Virtualized Ethernet (Shared Ethernet Adapter)
4-May-12
Prerequisites
All systems that will host a mobile partition must be on the same subnet and managed by a single HMC POWER6 Blades are managed by IVM instances All systems must be connected to shared physical disks (LUNs) in a SAN subsystem with no scsi reserve
SDDPCM, SVC, RDAC based LUN $ chdev dev hdisk8 attr reserve_policy=no_reserve
PowerPATH CLARiiON LUN $ chdev dev hdiskpower8 attr reserve_lock=no no LVM-based virtual disks no virtual disk logical volumes carved in VIO All resources must be shared or virtualized prior to migration (e.g., vscsi, virtual Ethernet)
4-May-12
Prerequisites
The pHypervisor will automatically manage migration of CPU and memory
Dedicated IO adapters must be de-allocated before migration cd0 in VIO may not be attached to mobile LPAR as virtual optical device The operating system and applications must be migration-aware or migration-enabled
10
4-May-12
Validation
System Properties support Partition Mobility Inactive and Active Partition Mobility Capable = True Mover Service Partitions on both Systems VIO Servers with VASI device defined, and MSP enabled
12
4-May-12
Migration
If validation passes, finish button starts migration
From this point, all state changes are rolled back if an error occurs
Mobile Partition
MSP
VASI
MSP
VASI
Mobile Partition
2
Source System
5
POWER Hypervisor Target System
POWER Hypervisor
13
4-May-12
Migration Steps
The HMC creates a shell partition on the destination system
The HMC configures the source and destination Mover Service Partitions (MSP) MSPs connect to PHYP thru the Virtual Asynchronous Services Interface (VASI) The MSPs set up a private, full-duplex channel to transfer partition state data The HMC sends a Resource Monitoring and Control (RMC) event to the mobile partition so it can prepare for migration The HMC creates the virtual target devices and virtual SCSI adapters in the destination MSP The MSP on the source system starts sending the partition state to the MSP on the destination server
14 4-May-12
Migration Steps
The source MSP keeps copying memory pages to the target in successive phases until modified pages have been reduced to near zero
The MSP on the source instructs the PHYP to suspend the mobile partition The mobile partition confirms the suspension by suspending threads The source MSP copies the latest modified memory pages and state data Execution is resumed on the destination server and the partition reestablishes the operating environment
The mobile partition recovers I/O on the destination server and retries all uncompleted I/O operations that were going on during the suspension
It also sends gratuitous ARP requests to all VLAN adapters (MAC address(es) are preserved)
15
4-May-12
Migration Steps
When the destination server receives the last modified pages, the migration is complete
In the final steps, all resources are returned to the source and destination systems and the mobile partition is restored to its fully functional state
16
4-May-12
Effects
Server properties The affinity characteristics of the logical memory blocks may change
The maximum number of potential and installed physical processors may change
The L1 and/or L2 cache size and association may change This is not a functional issue, but may affect performance characteristics Console Any active console sessions will be closed when the partition is migrated Console sessions must be re-opened on the target system by the user after migration LPAR uname will change. Partition ID may change. IP address, MAC address will not change.
17 4-May-12
Effects
Network A temporary network outage of seconds is expected to occur as part of suspending the partition Temporary network outages may be visible to application clients, but it is assumed that these are inherently recoverable VSCSI Server Adapters Adapters that are configured with the remote partition set to the migrating partition will be removed Adapters that are configured to allow any partition to connect will be left configured after the migration Any I/O operations that were in progress at time of the migration will be retried once the partition is resumed As long as unused virtual slots exist on the target VIO server, the necessary VSCSI controllers and target devices will be automatically created
18
4-May-12
Effects
Error logs When a partition migrates all of the error logs that the partition had received will appear on the target system All of the error logs contain the machine type, model, and serial number so it is possible to correlate the error with the system that detected it Partition time When a partition is migrated the Time of Day and timebase values of the partition are migrated. The Time of Day of the partition is recalculated ensuring partition timebase value increases monotonically and accounting for any delays in migration.
19
4-May-12
DEMO
20
4-May-12
Environment
Two POWER6 servers
8-way Mercury 01EM320_31 16-way Zeus 01EM320_31
21
4-May-12
Supplemental Material
22
4-May-12
IBM Training - 2008 Systems Technical Conference Client hdisk0, set hcheck_interval to 300 before reboot Client sees one hdisk with two MPIO paths lspath l hdisk0
Initial Configuration
No scsi reserve set on hdisk6, 7 in each VIO server. Also, with two fcs in a VIO server, change algorithm to round_robin for hdisk1. SDDPCM, RDAC, or PowerPATH driver installed in each VIO server
LUNs appears in each VIO server as hdisk6, 7 RAID5 LUNs carved in storage, zoned to 4 FC adapters in the two VIO servers
23 4-May-12
This LUN is zoned into another two VIO LPARs, on the other Power6 server also
2008 IBM Corporation
24
4-May-12
25
4-May-12
Have found that ieee_volname will not be visible up in the client LPAR
26 4-May-12
$ cat sk_clariion for d in `ioscli lspv | grep hdiskpower | awk '{print $1}'` do ioscli lsdev -dev $d -vpd | grep UI | awk '{print $1 $2} done
27
4-May-12
28
4-May-12
Target disk selection Option 77, alternative disk attributes >>> 1 hdisk0 00c23c9f291cc438
PVID from VIO shows up in client netboot No MPIO in network boot image, so disk only shows up on first vscsi client adapter ID 8
29
4-May-12
Sets MPIO to test failed and non-active paths every 5 minutes, bring them online if available. The newly Installed and booted LPAR has two vscsi client adapters
# lsdev -Cc adapter -F "name physloc" | grep vscsi vscsi0 U9117.MMA.1023C9F-V9-C8-T1 vscsi1 U9117.MMA.1023C9F-V9-C9-T1
The PVID we expected does come thru from VIO to the Client LPAR
# lspv hdisk0 00c23c9f291cc438 rootvg active
Starting Mobility
31
4-May-12
Starting Mobility
32
4-May-12
Starting Mobility
If you specify a new profile name, your initial profile will be saved. But do NOT assume it is bootable, or usable on return to source server. VIO mappings will change.
33
4-May-12
Starting Mobility
34
4-May-12
Starting Mobility
then
35
4-May-12
Starting Mobility
I selected the pair that were both SEA Failover primary, but any pair should do here
36
4-May-12
Starting Mobility
37
4-May-12
Starting Mobility
These are my client LPAR vscsi adapter IDs, matched to destination VIO LPARs
38
4-May-12
Starting Mobility
You may select from different shared pools on the destination server
39
4-May-12
Starting Mobility
Left to default
40
4-May-12
Starting Mobility
41
4-May-12
As migration starts, in the All Partitions view we see the LPAR residing on both Power6 servers
42
4-May-12
43
4-May-12
Migration Complete
Migrated LPAR resides solely on new server.
44
4-May-12
Migration Complete
Migration preserved my old profile, and created a new one
Same client adapter IDs, but different VIO server adapter IDs
45
4-May-12
When you migrate back, do not expect to be back on your original VIO Server adapter IDs. Your old client LPAR profile is historical, but will not likely be usable without some reconfig. Best to create a new profile on the way back over.
46
4-May-12
vt_ec04 client2lv vt_ec03 nimclientlv vt_ec05 client3lv vt_ec07 hdisk3 vt_bmark26 hdisk6 vtscsi0 hdisk6
48
4-May-12
49
4-May-12
50
4-May-12
Do the DLPAR of adapter into both source VIO LPARs, and both destination LPARs
51
4-May-12
Slightly different error mkvdev the new virtual onto the SEA
52
4-May-12
$ lsdev -type adapter -field name physloc | grep ent[0-9] ent0 U789D.001.DQDXYCW-P1-C10-T1 ent1 U9117.MMA.109A4AF-V2-C11-T1 ent2 U9117.MMA.109A4AF-V2-C12-T1 ent3 U9117.MMA.109A4AF-V2-C13-T1 ent4 $ cfgdev $ lsdev -type adapter -field name physloc | grep ent[0-9] ent0 U789D.001.DQDXYCW-P1-C10-T1 ent1 U9117.MMA.109A4AF-V2-C11-T1 ent2 U9117.MMA.109A4AF-V2-C12-T1 The newly ent3 U9117.MMA.109A4AF-V2-C13-T1 DLPARd in ent4 virtual ent5 U9117.MMA.109A4AF-V2-C18-T1 $ chdev dev ent4 attr virt_adapters=ent1,ent5 ent4 changed
adapter
Trunk priority on new virtual did not match the existing trunked virtual adapter
chgsea: Ioctl NDD_SEA_MODIFY returned error 22 for device ent4
54
4-May-12
55
4-May-12
Ready to Finish
56
4-May-12
Error configuring virtual adapter in slot 23 we had no vhost in slot 23 Virtual Optical device vtopt0 (cd0) cannot be attached to vhost adapter of migrating LPAR - not obvious. rmdev l cd0 d (in client LPAR) rmdev dev vtopt0 (in VIO server) Repeat validation
57 4-May-12
Reference
Live Partition Mobility Redbook
http://www.redbooks.ibm.com/redbooks/pdfs/sg247460.pdf
58
4-May-12
Trademarks
The following are trademarks of the International Business Machines Corporation in the United States, other countries, or both.
Not all common law marks used by IBM are listed on this page. Failure of a mark to appear does not mean that IBM does not use the mark nor does it mean that the product is not actively marketed or is not significant within its relevant market. Those trademarks followed by are registered trademarks of IBM in the United States; all others are trademarks or common law marks of IBM in the United States.
59
4-May-12