PowerHA

Advanced Features

© Copyright IBM Corporation 2010

 实施专家级课程 PowerHA

Agenda

1. PowerHA & Virtulization

2. Resouce Group Expand Capacity
3. Customizing Events
4. PowerHA & Storage
5. PowerHA & Network
6. PowerHA Cross-Site LVM mirroring
7. PowerHA/XD GLVM
8. PowerHA/XD And ESS/DS Metro Mirror
9. PowerHA/XD And SVC Metro Mirror/Global Mirror
10. PowerHA/XD Mirroring With SRDF Management

Page 2
 实施专家级课程 PowerHA

PowerHA & Virtulization

Page 3
 实施专家级课程 PowerHA

PowerHA 与虚拟化 – VIO Server

¾PowerHA 支持 VIO client partitions, 使用时有如下注意事项:

z PowerHA集群中的共享存储管理由PowerHA节点完成.VIO SERVER仅仅将
存储提供给PowerHA节点
z 注意在VIO SERVER上需要对PowerHA使用的共享盘设置reservation policy
为‘no_reserve’
z 在PowerHA节点创建VG，需要创建成”enhanced concurrent capable”模式，
不管该VG是否用于concurrent mode
z If any cluster nodes are accessing the shared disks by use of virtual SCSI
then all cluster nodes must use virtual SCSI, you cannot mix direct attached
and virtual SCSI in a cluster.
z Use of an HMC is only required if you want to utilize DLPAR with PowerHA.

z IVM contains a restriction on the number of VLANs a Virtual I/O Server can
have. The maximum is 4 VLANs.

Page 4
 实施专家级课程 PowerHA

Virtualization and PowerHA-VIO Server

¾建议采用双VIOS来避免单点故障

Page 5
 实施专家级课程 PowerHA

Virtualization and PowerHA-Virtual SCSI

¾在使用Virtual SCSI时的注意事项:
z PowerHA requires the use of enhanced concurrent volume groups when
using virtual SCSI.
z No hardware reserves are placed on the disks, and fast disk takeover is
utilized in the event that a volume group must be taken over by another
node.
z All volume group creation and maintenance is done using the C-SPOC
function of PowerHA and the bos.clvm.enh fileset must be installed.

Page 6
 实施专家级课程 PowerHA

Virtualization and PowerHA-Virtual Ethernet

¾在使用Virtual Ethernet时的注意事项:
z IP Address Takeover (IPAT) aliasing must be used. IPAT using
replacement and MAC Address Takeover are not supported with cluster
nodes using Virtual Ethernet.
z Virtual Ethernet interfaces defined to PowerHA should be treated as single-
adapter networks.
z Configure the file netmon.cf to monitor and detect failure of the network
interfaces.
z Recommend to use two Virtual Ethernets on the AIX client partition

Page 7
 实施专家级课程 PowerHA

Virtualization and PowerHA-NPIV

¾ N_Port ID Virtualization (NPIV) provides direct Fibre Channel connections from
client partitions to SAN resources , simplifying SAN management
z Fibre Channel Host Bus Adapter is owned by VIOS partition
z Each Fibre Channel Host Bus Adapter port (2 ports per card) can be used
by many client partitions
z Supported with PowerVM Express, Standard, and Enterprise Edition
z Supports AIX 5.3 and AIX 6.1 partitions
z Power 520, 550, 560, 570 and 595, with an 8 GB PCIe Fibre Channel
Adapter

VIOS

FC Adapter Virtual FC Adapter Virtual FC Adapter

Power Hypervisor

9Enables use of existing storage management tools

9Simplifies storage provisioning (i.e. zoning, LUN
masking)
9Enables access to SAN devices including tape libraries
.
Page 8
 实施专家级课程 PowerHA

Virtualization and PowerHA-NPIV

z N_Port ID Virtualization (NPIV) is a fibre channel industry standard

method for virtualizing a physical fibre channel port.

z NPIV allows one physical port to be associated with multiple N_Port IDs
(WWPNs), so a physical fibre channel HBA can be shared across
multiple guest operating systems in a virtual environment.

z On POWER 6, NPIV allows CLIENT logical partitions(LPARs) to have

dedicated N_Port IDs (WWPNs), giving the LPAR a unique identity
to the SAN, just as if it had a dedicated physical HBA(s).

z Client LPAR WWPNs become part of the partition definition and

move with the client from server to server (LPM)

Page 9
 实施专家级课程 PowerHA

Virtualization and PowerHA-NPIV

System 1
VIOS 1 Node 1
NPIV
HBA
hdisk0 } rootvg
vscsi0

HBA
vhost0

Hypervisor
hdisk MPIO
HBA vscsi1
hdisk1

hdisk2
} vscsi_vg
hdisk vhost0 fcs0

LUNS
VSCSI
VIOS 2
NPIV
HBA
fcs1
MPIO hdisk3

hdisk4
}npiv_vg

LUNS
NPIV
System 2
STORAGE
SUBSYSTEM VIOS 1 Node 2
NPIV
HBA
vscsi0
hdisk0 } rootvg
HBA

Hypervisor

hdisk vhost0 MPIO

vscsi1
hdisk1

hdisk2
} vscsi_vg
HBA

hdisk vhost0 fcs0

VIOS 2
NPIV
HBA
fcs1
MPIO hdisk3

hdisk4
}npiv_vg

Page 10
 实施专家级课程 PowerHA

Virtualization and PowerHA-DLpar Check Point

1. Which of the following IS supported using the configure HACMP for

Dynamic LPAR and CUod Resource menu?
A. Increasing memory in 32MB chunks
B. Resource can be taken from another running LPAR
C. Increasing CPU by whole CPUs
D. Node’s LPARname is not the same as node’s hostname and HACMP
nodename

Page 11
 实施专家级课程 PowerHA

Virtualization and PowerHA-DLpar

¾通过PowerHA实现DLpar的方法:
z 通过定义 Custom Event Scripts实现DLpar功能
z 通过PowerHA菜单配置RG的Application属性来实现DLpar和CUoD

Page 12
 实施专家级课程 PowerHA

Virtualization and PowerHA-DLpar

¾Integrate DLPAR using custom events

z Resource ( memory, CPU, and I/O ) can be added, moved, or removed, as
dictated by application movement between nodes
z Requires knowledge of HMC command-line interface and DLPAR operations

¾Which events to customize?

z Check /usr/es/adm/cluster.log to find events names and sequence
z To add resource to a node when an application starts, customize start_server
with a pre_event command
z To remove a resource when an application stops, customize stop_server with
a post-event command

Page 13
 实施专家级课程 PowerHA

Virtualization and PowerHA-DLpar

¾How to write the DLPAR Scripts

z Use ssh to execute commands remotely on the HMC from the node
executing the custom event
z Assumes automatic ssh authentication from each node to the HMC

z Have to hardcode system information (HMC hostname/IP, LPAR names,

etc)
z Execute lshwres and lssyscfg commands to gather information about the
current resource assignments for the target node
z Use chhwres to make any dynamic resouce changes

z Handle any non-zero return codes appropriately and exit with 0

Page 14
 实施专家级课程 PowerHA

Virtualization and PowerHA-DLpar

¾Add Custom DLPAR Events and Synchronize

z Use HACMP SMIT Extended Event Configuration menus
z Add the DLPAR scripts as new custome events

z Add the custom events as pre- and post-event commands for the
start_server and stop_server events
z Add DLPARadd as the pre-event command for start_server

z Add DLPARremove as the post-event command for stop_server

z Synchronize the cluster

z And finally:TEST

Page 15
 实施专家级课程 PowerHA

Virtualization and PowerHA-DLpar

¾Implement DLPAR and CUoD via configuring HACMP application in

RG
z Dynamic Logical Partitioning (DLPAR) allows you to dynamically allocate
additional resources (such as memory and CPUs) to each logical partition,
if needed, without stopping the application. These additional resources
must be physically present on the frame that uses logical partitions.
z Capacity Upgrade on Demand is one of the features of the DLPAR function
that lets you activate preinstalled but yet inactive (and unpaid for)
processors as resource requirements change.
z When it is necessary to run the application on the standby node, HACMP
ensures that the node has sufficient resources to successfully run the
application. The resources can be allocated from two sources:
ƒ The free pool.
ƒ CUoD provisioned resources

Page 16
 实施专家级课程 PowerHA

Virtualization and PowerHA-DLpar

¾Configure DLPAR to PowerHA

z Name resolution
ƒ /etc/hosts on all systems
ƒ /etc/netsvc.conf on all AIX nodes
ƒ /etc/host.conf, if applicable, on the HMC
z Installing SSH on PowerHA nodes
z Configuring HMC SSH access
ƒ HMC Management -> Remote Command Execution -> Enable remote command execution
using the ssh facility
z Defining HMC and managed system names to PowerHA
ƒ Node Name
ƒ HMC IP Address
ƒ Managed System Name

Page 17
 实施专家级课程 PowerHA

Virtualization and PowerHA-DLpar

¾Configure DLPAR to PowerHA

z select Extended Configuration →Extended Resource Configuration →
HACMP Extended Resources Configuration → Configure HACMP
Applications → Configure HACMP for Dynamic LPAR and CUoD
Resources → Configure Dynamic LPAR and CUoD Resources for
Applications
z Add Dynamic LPAR and CUoD Resource forApplications
ƒ Application Server Name
ƒ Minimum Number of CPUs
ƒ Number of CPU
ƒ Minimum Amount of Memory
ƒ Use CUoD if resource are insufficient
ƒ I agree to use CUoD resource

Page 18
 实施专家级课程 PowerHA

Virtualization and PowerHA- DLpar Limitations

¾Configure DLPAR to PowerHA

z Only whole CPUs can be allocated, no micro-partition or fractional CPU
support
z HACMP will try to acquire up to the desired amount of resources, if
available.
ƒ A resource group will not fallover if it does not meet the desired resource
configuration but does meet the minimum.
z If minimum on demand resource is not met on a DLPAR, the resource
groups will fallover.
z If a DLPAR command fails HACMP will not try other HMCs (if configured)
z When a resource group goes offline, HACMP turns off the trial period if
it's activated

Page 19
 实施专家级课程 PowerHA

Virtualization and PowerHA-LPM

¾Move running AIX and Linux operating system workloads from

one POWER6 processor-based server to another!

9 Improves Availability
Helps eliminate many planned outages

9 Balances Workloads
During peaks and to address
spikes in workload

9 Included with PowerVM

Enterprise Edition
Supports AIX and Linux partitions with
VIOS on Power servers

Virtualized
VirtualizedSAN
SANand
andNetwork
NetworkInfrastructure
Infrastructure

Page 20
 实施专家级课程 PowerHA

Virtualization and PowerHA-LPM

¾Live Partition Mobility 用于在计划内硬件维护中提供应用不间断的技

术，他并不能用于软件维护或非计划停机等场景
¾PowerHA主要是用于后两种场景中，尽量减少宕机时间，保证业务的连
续性
¾PowerHA can be used within a partition that is capable of being
moved with Live Partition Mobility. This does not mean that PowerHA
utilizes Live Partition Mobility in anyway. PowerHA is just treated as
another application within the partition. The following requirements are
for PowerHA to be supported with Live Partition Mobility:

Page 21
 实施专家级课程 PowerHA

Virtualization and PowerHA-LPM

¾PowerHA 5.5
– AIX 5.3 TL9 or AIX 6.1 TL2 SP1
– RSCT 2.4.10.0 or 2.5.2.0
¾ HACMP 5.4.1
– AIX 5.3.7.1, or AIX 6.1.0.1
– RSCT 2.4.5.4, or 2.5.0.0
– HACMP APAR #IZ02620
¾ HACMP 5.3
– AIX 5.3.7.1, or AIX 6.1.0.1
– RSCT 2.4.5.4, or 2.5.0.0
– HACMP APAR #IZ07791
¾ Firmware Level: 01EM320_40 (or higher)
¾ VIOS Level: 1.5.1.1-FP-10.1 (w/Interim Fix 071116)
¾ HMC Level: 7.3.2.0

Page 22
 实施专家级课程 PowerHA

Virtualization and PowerHA-LPM

¾With the requisite levels of software and firmware installed on the

source and destination POWER6 servers, the Live Partition Mobility
feature can be used to migrate an LPAR running as an HACMP node
without affecting the state or operation of the HACMP cluster, provided
the HACMP cluster is configured to use standard (that is, default)
heartbeat parameters.
¾In that case, the effect on the application servers running under
HACMP control is a brief suspension of operations during the
migration. Neither HACMP nor the application servers will have to
be restarted.

Page 23
 实施专家级课程 PowerHA

Virtualization and PowerHA-LPM

¾For those HACMP clusters configured to use fast heartbeating, and/or

application monitoring with short checking intervals, IBM recommends
customer testing to validate that the period of suspension during Live
Partition Mobility will not cause unwanted cluster events.
¾Another way to greatly minimize the chance of unwanted cluster
events is to simply stop the cluster utilizing the "Unmanage" option
on the cluster node in which an LPM is going to be performed on.
Then upon successful completion of the move, simply restart cluster
services again.

Page 24
 实施专家级课程 PowerHA

Extending RG capability

Page 25
 实施专家级课程 PowerHA

Extending RG capability

Page 26
 实施专家级课程 PowerHA

Extending RG capability-Settling time

¾Settling Time参数用于节点推迟获取资源组，在等待的这段时间内，如
果有优先级更高的节点加入集群中，则优先级高的节点将获得激活资源
组的权限；代替默认由第一个加入集群的节点获取资源组的方式

Check Point
1. An HACMP setting timer may used to allow:
A. The setting or fallback of a cluster custom resource group.
B. The cluster time to stabilize before starting a custom resource group.
C. A priority node time to acquire custom resource group at cluster start
D. A fallover node time to process custom resource states to be
monitored

Page 27
 实施专家级课程 PowerHA

Extending RG capability-Settling time characteristics

¾影响范围:If configured, settling time affects the startup behavior of all

offline resource groups in the cluster for which you selected the Online
on First Available Node startup policy.
¾个别情况没有影响:The only time that this attribute is ignored is when
the node joining the cluster is the first node in the node list for the
resource group. In this case the resource group is acquired
immediately.
¾资源组是ERROR情况:If a resource group is currently in the ERROR
state, PowerHA will wait for the settling time period before attempting
to bring the resource group online.
¾生效时间:The current settling time continues to be active until the
resource group moves to another node or goes offline.
¾DARE操作影响:A DARE operation might result in the release and re-
acquisition of a resource group, in which case the new settling time
values take effect immediately.

Page 28
 实施专家级课程 PowerHA

Extending RG capability-Settling time config

¾Enter the smitty hacmp fast path and select Select Extended
Configuration → Extended Resource Configuration → Configure
a Resource Group Run-Time Policies → Configure Settling Time for
Resource Group and press Enter.
¾Enter field values as follows:
Settling Time (sec.):

¾Remember that this is only valid for resource groups using the
startup policy, Online of First Available Node.

Page 29
 实施专家级课程 PowerHA

Extending RG capability-Node distribution policy

¾This policy causes resource groups having this startup policy to

spread across cluster nodes in such a way that only one resource
group is acquired by any node during startup. This could be used,
for instance, for distributing CPU-intensive applications on different
nodes.
¾If two or more resource groups are offline when a particular node joins
the cluster, this policy determines which resource group is brought
online based on the following criteria:
z The resource group with the least number of participating nodes will be
acquired.
z In case of tie, the resource group to be acquired is chosen using
alphabetical order.
z A parent resource group is preferred over a resource group that does not
have any child resource group.

Page 30
 实施专家级课程 PowerHA

Extending RG capability-Node distribution policy

¾使用Node Distribution Policy注意事项

z 资源组与节点数量:If the number of resource groups is larger than
the number of cluster nodes, HACMP issues a warning. It is
recommended that all resource groups that use node-based
distribution have potential nodes on which they could be brought
online during the cluster startup.
z Fallove配置:resource groups configured for distribution during
startup cannot have the fallover policy set to Bring Offline (on
Error Node Only). If you select this combination of policies,
HACMP issues an error.
z Fallback配置:resource groups configured for distribution during
startup must use the Never Fallback policy. This is the only
fallback policy HACMP allows for such resource groups.

Page 31
 实施专家级课程 PowerHA

Extending RG capability-Node distribution policy

¾To configure this type of startup policy, follow these steps:

¾Enter the smitty hacmp fast path and select Extended
Configuration →Extended Resource Configuration → HACMP
Extended Resource Group Configuration → Add a Resource Group
and press Enter.
¾Type in a resource group name.
¾Select a startup policy of Online Using Node Distribution Policy and
press Enter as shown in next slice.

Page 32
 实施专家级课程 PowerHA

Extending RG capability-Node distribution policy

Page 33
 实施专家级课程 PowerHA

Extending RG capability-Dynamic node priority

¾Implementing a dynamic node priority for a resource group allows you

to go beyond the default fallover policy behavior and influence the
destination of a resource group upon fallover based on the following
three RMC pre-configured attributes:
z cl_highest_free_mem - node with highest percentage of free memory
z cl_highest_idle_cpu - node with the most available processor time

z cl_lowest_disk_busy - node with the least busy disks

¾During a fallover event of a resource group with Dynamic Node

Priority configured, the most recently collected values are used in
the determination of the best node to acquire the resource group.

Page 34
 实施专家级课程 PowerHA

Extending RG capability-Dynamic node priority

¾In order for DNP to be effective, note these considerations:

z DNP is irrelevant for clusters comprising fewer than three nodes.
z DNP is irrelevant for concurrent resource groups.

z DNP is most useful in a cluster where all nodes have equal processing
power and memory.

¾Attention: The highest free memory calculation is performed based

on the amount of paging activity taking place. It does not take into
account whether one cluster node has less real physical memory than
another.

Page 35
 实施专家级课程 PowerHA

Extending RG capability-Dynamic node priority

¾Enter the smitty hacmp fast path and select Extended

Configuration →Extended Resource Configuration → Extended
Resource GroupConfiguration → Add a Resource Group and
press Enter. Set the Fallover Policy field to Fallover Using Dynamic
Node Priority.
¾Assign the resources to the resource group by selecting Extended
Configuration → Extended Resource Configuration → Extended
Resource Group Configuration → Change/Show Resources and
Attributes for a Resource Group and press Enter, Select one of the
three available policies from the pull-down lis:
z – cl_highest_free_mem
z – cl_highest_idle_cpu

z – cl_lowest_disk_busy

Page 36
 实施专家级课程 PowerHA

Extending RG capability-Dynamic node priority

¾How to query resource monitor result:

z lsrsrc -Ad IBM.Host | grep TotalPgSpFree
z lsrsrc -Ad IBM.Host | grep PctTotalTimeIdle

z lsrsrc -Ap IBM.PhysicalVolume

z lssrc -ls clstrmgrES

。。。
Current DNP values
DNP Values for NodeId - 0 NodeName - xdsvc1
PgSpFree = 0 PvPctBusy = 0 PctTotalTimeIdle = 0.000000
DNP Values for NodeId - 0 NodeName - xdsvc2
PgSpFree = 0 PvPctBusy = 0 PctTotalTimeIdle = 0.000000
DNP Values for NodeId - 0 NodeName - xdsvc3
PgSpFree = 0 PvPctBusy = 0 PctTotalTimeIdle = 0.000000

Page 37
 实施专家级课程 PowerHA

Extending RG capability-Delayed fallback timer

¾This feature allows you to

configure the fallback
behavior of a resource
group to occur at one of the
predefined recurring times:
daily, weekly, monthly, yearly.
Alternatively you can specify
a particular date and time.

¾This feature can be useful for

scheduling fallbacks to occur
during off-peak business
hours.

Page 38
 实施专家级课程 PowerHA

Extending RG capability-Delayed fallback timer

¾当使用”delayed fallback timers”时需要注意的内容:

z The delayed fallback timer applies only to resource groups having fallback
policy set to Fallback To Higher Priority Node In The List.
z If there is no higher priority node available when the timer expires, then the
resource group remains online on the current node. The timer is reset and
the fallback will be retried when the timer expires again.
z If a specific date is used for a fallback timer and at that moment there is
not any higher priority node, then the fallback will not be rescheduled.
z If a resource group being part of an Online on the Same Node
dependency relationship has a fallback timer, then the timer will apply to
all resource groups that are part of the Online on the Same Node
dependency relationship.
z When using an Online on the Same Site dependency relationship, if a
fallback timer is used for a resource group, then it must be identical for all
resource groups being part of the same dependency relationship.

Page 39
 实施专家级课程 PowerHA

Extending RG capability-Delayed fallback timer

¾To configure the delayed fallback timer, do the following steps:

z Define a fallback timer policy.
z Assign the fallback timer policy to a resource group.

z Select the desired fallback timer policy from the picklist and press Enter.

z Add any additional resources to the resource group and press Enter.

z Run a verification and synchronization on the cluster to propagate the

changes to all cluster nodes.

Page 40
 实施专家级课程 PowerHA

Extending RG capability-Delayed fallback timer

¾Displaying delayed fallback timers in a resource group:

z clshowres|grep -ip test_rg|egrep -i "resource group|timer“
root@ xdsvc1[] clshowres|grep -ip test_rg|egrep -i "resource group|timer"
Resource Group Name test_rg
Delayed Fallback Timer timer_policy1

z odmget HACMPtimer
HACMPtimer:
policy_name = "timer_policy2"
recurrence = "daily"
year = 0
month = 0
day_of_month = 1
week_day = 0
hour = 11
minutes = 11

Page 41
 实施专家级课程 PowerHA

Extending RG capability-Resource group dependency

¾PowerHA can include components used by an application in resource

groups and establish resource group dependencies that will accurately
reflect the logical relationships between application components.

¾Parent/child dependency
¾Resource group location dependency
z online on the same node
z online on the same site

z online on different nodes

z online on different site

¾Combining various dependency relationships

Page 42
 实施专家级课程 PowerHA

Extending RG capability-Resource group dependency

¾Planning for parent/child resource group dependencies

z Parent/child relationship can span up to three levels.
z There should be no circular dependencies between resource groups.

z A resource group can act as a parent for a resource group and as a child
for another resource group.

Page 43
 实施专家级课程 PowerHA

Extending RG capability-Resource group dependency

¾Configuring a resource group parent/child dependency

z Use the smitty hacmp fast path and select Extended Configuration
→HACMP Extended Resource Configuration → Configure Resource
Group Run-Time Policies → Configure Dependencies between
Resource Groups → Configure Parent/Child Dependency → Add
Parent/Child Dependency between Resource Groups
z Fill in the fields as follows: Parent Resource Group & Child Resource
Group.
z Use the Verify and Synchronize option to validate the dependencies and
propagate them on all cluster nodes.

Page 44
 实施专家级课程 PowerHA

Extending RG capability-Resource group dependency

¾Planning for Online on the Same Node Dependency dependencies

z All resource groups having an Online on the Same Node dependency
relationship must have the same node list and the participating nodes
must be listed in the same order.
z Both concurrent and non-concurrent resource groups are allowed.

z You can have more than one Online on the Same Node dependency
relationship in the cluster.
z All non-concurrent resource groups in the same Online on the Same Node
dependency relationship must have identical startup/fallover/fallback
policies.

Page 45
 实施专家级课程 PowerHA

Extending RG capability-Resource group dependency

¾Configure identical startup/fallover/fallback policies dependencies

z Online Using Node Distribution Policy is not allowed for startup policy.
z If Dynamic Node Priority policy is being used as the fallover policy, then all
resource group in the dependency must use the same DNP policy.
z If one resource group has a fallback timer configured, then the timer will also
apply to the resource groups that take part in the dependency relationship. All
resource groups must have identical fallback time setting
z If one or more resource groups in the Online on the Same Node dependency
relationship fail, then cluster services will try to place all resource groups on
the node that can accommodate all resource groups being currently online
plus one or more failed resource groups.

Page 46
 实施专家级课程 PowerHA

Extending RG capability-Resource group dependency

¾Configuring Online on the Same Node location dependency

z Use the smitty hacmp fast path and select Extended Configuration
→HACMP Extended Resource Configuration → Configure Resource
Group Run-Time Policies → Configure Dependencies between Resource
Groups → Configure Online on the same node Dependency → Add
Online on the Same Node Dependency Between Resource Groups
z In order to propagate the change across all cluster nodes remember to verify
and synchronize your cluster.

Page 47
 实施专家级课程 PowerHA

Extending RG capability-Resource group dependency

¾Planning for Online On Different Nodes Dependency

z Only one Online On Different Nodes dependency is allowed per cluster.
z Each resource group should have a different home node for startup.

z When using this policy, a higher priority resource group takes precedence over
a lower priority resource group during startup, fallover, and fallback.
ƒ If a resource group with High priority is online on a node, then no other resource group being part
of the Online On Different Nodes dependency can be put online on that node.
ƒ If a resource group being part of the Online On Different Nodes dependency is online on a cluster
node and a resource group being part of the Online On Different Nodes dependency and having
a higher priority falls over or falls back to the same cluster node, then the resource group having
a higher priority will be brought online, whereas the resource group having a lower priority
resource group will be taken offline or migrated to another cluster node if available.
ƒ Resource groups being part of the Online On Different Nodes dependency and having the same
priority cannot be brought online on the same cluster node.
ƒ Resource groups being part of the Online On Different Nodes dependency and having the same
priority do not cause each other to be moved from cluster node after a fallover or fallback.
ƒ If a parent/child dependency is being used, then the child resource group cannot have a priority
higher than its parent.

Page 48
 实施专家级课程 PowerHA

Extending RG capability-Resource group dependency

¾Configuring Online on Different Node location dependency

z Use the smitty hacmp fast path and select Extended Configuration
→HACMP Extended Resource Configuration → Configure Resource
Group Run-Time Policies → Configure Dependencies between Resource
Groups → Configure Online on the Same Node Dependency → Add
Online on Different Nodes Dependency between Resource Groups
z Fill in the following fields as explained
ƒ High Priority Resource Group
ƒ Intermediate Priority Resource Group
ƒ Low Priority Resource Group
z Continue configuring run-time policies for other resource groups or verify and
z synchronize the cluster.

Page 49
 实施专家级课程 PowerHA

Extending RG capability-Resource group dependency

¾Planning for Online on the Same Site Dependency

z All resource groups in an Online on the Same Site dependency relationship
must have the same Inter-Site Management policy. However, they might have
different startup/fallover/fallback policies. If fallback timers are used, these
must be identical for all resource groups part of the Online on the Same Site
dependency.
z The fallback timer does not apply to moving a resource group across site
boundaries.
z All resource groups in an Online on the Same Site dependency relationship
must be configured so that the nodes that can own the resource groups are
assigned to the same primary and secondary sites.
z Online Using Node Distribution policy is supported.

z Both concurrent and non-concurrent resource groups are allowed.

z You can have more than one Online on the Same Site dependency
relationship in the cluster.

Page 50
 实施专家级课程 PowerHA

Extending RG capability-Resource group dependency

¾Planning for Online on the Same Site Dependency

z All resource groups having an Online on the Same Site dependency
relationship are required to be on the same site, even though some of them
might be in OFFLINE or ERROR state.
z If you add a resource group being part of an Online on the Same Node
dependency to an Online on the Same Site dependency, then you must add all
other resource groups that are part of the Online on the Same Node
dependency to the Online on the Same Site dependency.

Page 51
 实施专家级课程 PowerHA

Extending RG capability-Resource group dependency

¾Combining various dependency relationships

z Only one resource group can belong to both an Online on the Same Node
dependency relationship and an Online on Different Nodes dependency
relationship.
z If a resource group belongs to both an Online on the Same Node dependency
relationship and an Online on Different Node dependency relationship, then all
other resource groups than are part of the Online of Same Node dependency
will have the same priority as the common resource group.
z Only resource groups having the same priority and being part of an Online on
Different Nodes dependency relationship can be part of an Online on the
Same Site dependency relationship.

Page 52
 实施专家级课程 PowerHA

Extending RG capability-Resource group dependency

¾Displaying resource group dependencies

z clrgdependency -t PARENT_CHILD -sl
z odmget HACMPrgdependency

Page 53
 实施专家级课程 PowerHA

Customizing Events

Page 54
 实施专家级课程 PowerHA

Check Point

1. The best way to customize HACMP event processing is to alter the

pre-defined event(for example node_down)
A. True
B. False

2.Which of the following is NOT true about custom event scripts:

A. Must be copied to each node in the cluster
B. Must have executable permission on all nodes
C. Are automatically copied to all nodes during synchronization
D. Should always exit with 0

Page 55
 实施专家级课程 PowerHA

Check Point

3.Which of the following runs if an HACMP event scripts fails?

A. Error methods
B. Recovery commands
C. Notify methods

Page 56
 实施专家级课程 PowerHA

Customizing Events-Overview

¾When a cluster event occurs, the Cluster Manager runs the event
script corresponding to that event.
¾PowerHA provides a script for each event and sub-event.
¾The default scripts are located in the /usr/es/sbin/cluster/events
directory.
¾By default, the Cluster Manager calls the corresponding event script
for a specific event.
¾You can specify additional actions to be performed when a particular
event occurs.

Page 57
 实施专家级课程 PowerHA

Customizing Events-Overview

Page 58
 实施专家级课程 PowerHA

Customizing Events-Overview

Page 59
 实施专家级课程 PowerHA

Customizing Events-Overview

Page 60
 实施专家级课程 PowerHA

Customizing Events-Overview

Page 61
 实施专家级课程 PowerHA

Customizing Events-Overview

¾You can customize the handling of a particular event for your cluster
using the following features:
z Pre-event and post-event processing
z Event notification

z Event recovery and retry

z Cluster automatic error notification

z Customizing event duration

z Defining new events

Page 62
 实施专家级课程 PowerHA

Customizing events-Overview

¾配置customizing cluster events时有如下注意事项:

z Test all possible input parameters.
z Provide correct return value: 0 for success, any other number for failure.

z Terminate within a reasonable amount of time.

z Make sure that a recovery program can recover from an event failure,
otherwise the cluster will fail.
z Keep in mind that synchronization does not copy pre-event and post-event
script content from one node to another.
z The name and location of scripts must be identical on all cluster no

Page 63
 实施专家级课程 PowerHA

Customizing events-Pre-event & Post-event

¾In order to define a pre-event or post-event script, you must create a

custom event and then associate the custom event with a cluster
event as follows:
z Write and test your event script carefully. Ensure that you copy the file to all
cluster nodes under the same path and name.
z Define the custom event using the smitty hacmp fast path and select
Extended Configuration →Extended Event Configuration → Configure
Pre/Post-Event Commands → Add a Custom Cluster Event
z Connect the custom event with pre/post-event cluster event using the
smitty hacmp fast path and select Extended Configuration →Extended
Event Configuration → Change/Show Pre-Defined HACMP Events.
z Verify and synchronize the cluster.

Page 64
 实施专家级课程 PowerHA

Customizing events-Automatic Error Notification

¾By default, PowerHA monitors only cluster nodes, networks, and

network adapters.
¾PowerHA provides a SMIT interface to the AIX Error Notification
facility. AIX Error Notification facility allows you to detect an event not
specifically monitored by the PowerHA
¾Automatic error notification applies to selected hard, non-recoverable
error types such as those related to disks or disk adapters. This utility
does not support media errors, recovered errors, or temporary errors.

Page 65
 实施专家级课程 PowerHA

Customizing events-Automatic error notification

¾Enabling automatic error notification assigns one of two error

notification methods for all error types as follows:
z The non-recoverable errors pertaining to resources that have been
determined to represent a single point of failure are assigned the
cl_failover
z All other non-critical errors are assigned the cl_logerror method and an
error entry will be logged against the hacmp.out file.
¾PowerHA configures automatically error notifications and recovery
actions for several resources and error types that include:
z All disks in the rootvg volume group
z All disks defined in a HACMP resource group

z SCSI adapters used by HACMP resources or rootvg

z Fibre Channel adapters used by HACMP resources

Page 66
 实施专家级课程 PowerHA

Customizing events-Automatic error notification

¾Disk monitoring consideration

z Additionally, PowerHA can monitor both mirrored and non-mirrored volume
groups regardless of the disk type. When the loss of quorum is detected, an
LVM_SA_QUORCLOSE entry is logged against the error log. PowerHA can
initiate a takeover for the resource group that contains the volume group.

z Note: Handling LVM_SA_QUORCLOSE for non-mirrored volume groups

has been added in APAR IZ21648 for AIX 5.3 and APAR IZ21631 for AIX
6.1, and PowerHA 5.5 APAR IZ41771.

Page 67
 实施专家级课程 PowerHA

Customizing events-Automatic error notification

¾Setting up automatic error notification

z Use the smitty hacmp fast path and select Problem Determination Tools
→ HACMP Error Notification → Configure Automatic Error Notification
→ Add Error Notify Methods for Cluster Resources.
z Note: You cannot configure automatic error notification while the cluster is
running.

¾Listing automatic error notification

z Use the smitty hacmp fast path and select Problem Determination Tools
→ HACMP Error Notification → Configure Automatic Error Notification.
z Select List Error Notify Methods for Cluster Resources

Page 68
 实施专家级课程 PowerHA

Customizing events-Automatic error notification

¾Add a Notify Method

z Use the smitty hacmp fast path
and select Problem Determination
Tools →HACMP Error
Notification → Add a Notify
Method.Note:
z Define the notification object
ƒ Notification Object Name:
ƒ Persist across system restart
ƒ Process ID for use by Notify Method
ƒ Select Error Class
ƒ Select Error Type
ƒ 。。。
z Press Enter to create the error
notification object

Page 69
 实施专家级课程 PowerHA

Customizing events-Automatic error notification

¾Emulate a notify method

z Use the smitty hacmp fast path and select Problem Determination
Tools → HACMP Error Notification → Emulate Error Log
z Select the error label or notify method name from the pop-up list. Only notify
methods that have an error label defined will be shown.
z SMIT shows the error label, notification object name, and the notify method.
Press Enter to confirm error log entry emulation

Page 70
 实施专家级课程 PowerHA

Customizing events- Customizing event duration

¾Customizing event duration

z For such events, you can customize the time period that cluster services
will wait for an event to complete before issuing the config_too_long
warning message.
z Two Cluster events
ƒ Fast events (Event Duration Time)
ƒ Slow events(Event-only Duration Time and Resource Group Processing Time)

Page 71
 实施专家级课程 PowerHA

Customizing events- Customizing event duration

¾Change Total Event Duration Time

z Use the smitty hacmp fast path and select HACMP Extended Configuration
→ Extended Event Configuration → Change/Show Time Until Warning
z Enter data in the fields as follows:
ƒ Max. Event-only Duration (in seconds):
ƒ Max. Resource Group Processing Time (in seconds)
ƒ Total time to process a Resource Group event before a warning is displayed
z Press Enter to create the error notification object.
z Verify and synchronize the cluster to propagate the changes

Page 72
 实施专家级课程 PowerHA

Customizing events-Define New events

¾With PowerHA you can define your own cluster events that would run
specified recovery programs.
¾The events you define can be related to RMC resources. An RMC
resource refers to an instance of a physical or logical entity that
provides services to some other component of the system. Resources
can refer to both hardware and software entities.
¾You can obtain information regarding resources and resource classes
using lsrsrcdef command.

Page 73
 实施专家级课程 PowerHA

Customizing events-Define New events

¾Recovery programs
¾A recovery program consists of a sequence of recovery command
specifications having the following format:
:node_set recovery_command expected_status NULL

¾Multiple recovery command specifications can be separated by the

barrier command. All recovery command specifications before a
barrier start in parallel. When a node encounters a barrier command,
all nodes must reach the same barrier before the recovery program
resumes.

Page 74
 实施专家级课程 PowerHA

Customizing events-Define New events

¾Config your new event,

¾Use the smitty hacmp fast path
and select HACMP Extended
Configuration → Extended
Event Configuration →
Configure User-Defined Events
→ Add Custom User-Defined
Events.
¾Enter data in the fields as follows:
¾Event name, Recovery
program path, Resource name,
Selection String, Expression,
Rearm expression

Page 75
 实施专家级课程 PowerHA

Customizing events-Define New events

¾The recovery program used was:

event "/usr/ha/trigger_user_defined_event.sh" 0 NULL

¾The /usr/ha/trigger_user_defined_event.sh script could perform any

form of notification, such as writing to a log file or sending an email,
sms or SNMP trap.

Page 76
 实施专家级课程 PowerHA

Storage Related Considerations

Page 77
 实施专家级课程 PowerHA

Storage related considerations-VG types

¾It is important to understand the different types of volume groups and

how PowerHA utilizes each type. We cover the following volume group
types:
z Enhanced concurrent

z Non-concurrent

z Concurrent

Page 78
 实施专家级课程 PowerHA

Storage related considerations-VG types

¾Enhanced concurrent
z Enhanced concurrent volume groups were first introduced in AIX 5.1. Unlike
concurrent (that is, for SSA only), it is supported for use on any disk
subsystem that is supported in a shared, AIX, POWER systems, PowerHA
configuration.
z In AIX 5.2 and above, the enhanced concurrent volume group type is the
only concurrent type available.
z Enhanced concurrent volume groups use the Group Services Concurrent
Logical Volume Manager (gsclvmd) daemon, which communicates over IP
to other cluster member nodes.
z Enhanced concurrent volume groups can be used in both concurrent and
non-concurrent environments. These features (such as fast disk takeover
and disk heartbeat) are dependent on it.
z Existing non-concurrent volume groups can be converted to enhanced
concurrent without losing any additional storage space.

Page 79
 实施专家级课程 PowerHA

Storage related considerations-VG types

¾Non-concurrent
z A non-concurrent volume group is the default when creating a new volume
group. It is also referred to as a standard volume group. The inherent
nature of non-concurrent volume groups is that the volume group will not be
accessed by more than one system at any time.

Page 80
 实施专家级课程 PowerHA

Storage related considerations-VG types

¾Concurrent
z Concurrent volume groups are no longer available starting in AIX 5.2 and
are now considered obsolete.
z This type of volume group, also referred to as Concurrent Capable, is
specific to SSA disks in a concurrent access configuration. This
combination provided the first true concurrent mode volume group.

Page 81
 实施专家级课程 PowerHA

Storage related considerations-Disk reservation

¾Disk reservations
z When a volume group is varied on in AIX, a disk reserve is placed against
each member disk. This is to ensure that no other systems can access
these drives to maintain data integrity.
z These reserves are often called SCSI reserves and they are based on
SCSI standards. Most of today’s newest FC disks are using FSCSI protocol
and still utilize a SCSI reserve.
z This reserve will normally stay in place until removed by a varying off the
volume group. Even if the AIX system is halted, if the disks maintain power,
the reserve normally stays set.
z Disk reserves are not used for concurrent, or enhanced concurrent
mode volume groups used when the volume groups are online in
concurrent access mode.
z This is also true when using enhanced concurrent volume groups in a fast
disk takeover configuration.

Page 82
 实施专家级课程 PowerHA

Storage related considerations-Forced varyon

¾Forced varyon of volume groups

z This ability is very important when using mirrored volume groups.
z It is an attribute of the varyonvg command is represented by the -f flag.

z Utilizing this flag enables a volume group to be brought online even when a
quorum of disks is not available. During the varyon process, each logical
volume is checked and at least one complete copy of each must be found
for the varyon to succeed.
z This setting is most commonly used when mirroring across storage
subsystems, and/or mirroring between locations via cross-site LVM
mirroring

z The difference between disable the quorum setting and forced varyonvg

Page 83
 实施专家级课程 PowerHA

Storage related considerations-Fast disk takeover

¾Prerequisites
z AIX 5.2 or higher
z Bos.clvm.enh 5.2.0.11 or higher

z Enhanced concurrent shared data volume groups

Page 84
 实施专家级课程 PowerHA

Storage related considerations-Fast disk takeover

¾How fast disk takeover works

z Historically, disk takeover has involved breaking a SCSI reserve on each
disk device in a serial fashion. The amount of time it takes to break the
reserve varies by disk type. In a large environment with hundreds of disks,
this can add significant amount of time to the fallover.
z Fast disk takeover reduces total fallover time by providing faster acquisition
of the disks without having to break SCSI reserves. It utilizes enhanced
concurrent volume groups, and additional LVM enhancements provided by
AIX 5.2.
z AIX 5.2 introduced the ability to varyon an enhanced concurrent volume
group in two different modes: Active Mode & Passive Mode

Page 85
 实施专家级课程 PowerHA

Storage related considerations-Fast disk takeover

¾Active Mode:
z Operations on file systems, such as file system mounts
z Operations on applications
z Operations on logical volumes, such as creating logical volumes
z Synchronizing volume groups

¾Passive Mode:
z LVM read-only access to the volume group’s special file
z LVM read-only access to the first 4 KB of all logical volumes that are owned
by the volume group
¾The following operations are not allowed when a volume group is
varied on in the passive state:
z Operations on file systems, such mount
z Any open or write operation on logical volumes
z Synchronizing volume groups

Page 86
 实施专家级课程 PowerHA

Storage related considerations-Fast disk takeover

¾Enabling fast disk takeover

z The shared volume groups must be enhanced concurrent volume groups.
z These volume groups are then added as resources to a non-concurrent
mode style resource group.

z PowerHA determines enhanced concurrent volume group or not by running

the lqueryvg –p devicename(PVname) –X.
z If the return output is 0, then it is a regular non-concurrent volume group. If
the return output is 32, then it is an enhanced concurrent volume group.

Page 87
 实施专家级课程 PowerHA

Storage related considerations-Fast disk takeover

¾Advantages
z Faster disk acquisition time
z LVM ODM synchronization

¾Special considerations
z Using C-SPOC to operate enhanced concurrent vg
z Never allow non-cluster nodes to have the shared data volume groups
visible to them
z LVM capabilities limitation
ƒ Commands: reorgvg, splivg
ƒ Volume group attributes:big vg, hot spare, sync(auto), factor size, dynamic
volume expansion, bad block relocation
ƒ This obviously means that these cannot be changed via C-SPOC either
z Forced down & Unmanage Resource Groups

Page 88
 实施专家级课程 PowerHA

Storage related considerations-Disk heartbeat

¾Overview
¾Traditional disk heartbeat
z A traditional disk heartbeat network is a point-to-point network. If more than
two nodes exist in your cluster, you will need a minimum of N number of
non-IP heartbeat networks.
¾Multi-node disk heartbeat
z Unlike traditional disk heartbeat networks, it is not a single point-to-point
network. But instead, as its name implies, it allows multiple nodes to use
the same disk. However, it does require configuring a logical volume on an
enhanced concurrent volume group.

z Important: Currently, multi-node disk heartbeating can only be utilized in a

resource group configured with a startup policy of “Online On All Available
Nodes”. Currently only Oracle RAC takes advantage of mndhb. For more
details consult Oracle Metalink note number 404474.1.

Page 89
 实施专家级课程 PowerHA

Storage related considerations-Disk heartbeat

¾Prerequisites
¾Traditional disk heartbeat
z HACMP 5.x, Bos.clvm.enh, Shared disks
¾Multi-node disk heartbeat
z HACMP
ƒ 5.3 PTF6 or
ƒ 5.4.1 or higher
z RSCT
ƒ 2.4.7.3 + IZ01838 or
ƒ 2.5.0.0
z AIX
ƒ bos.rte.lvm 5.3.0.60 or
ƒ bos.rte.lvm 6.1.0.0
ƒ A 32MB logical volume (The logical volume must be on a single physical disk & Mirror write
consistency (MWC) must be off)

Page 90
 实施专家级课程 PowerHA

Storage related considerations-Disk heartbeat

¾Performance considerations
z Recommend that you use disk that has fewer than 60 seeks per second.
The filemon tool can be used to monitor the seek activity on a disk.

Page 91
 实施专家级课程 PowerHA

Storage related considerations-Disk heartbeat

¾Configuring traditional disk heartbeat

z Create the diskhb network using the smitty hacmp fast path, select
Extended ConfigurationExtended Topology Configuration → Configure
HACMP Networks →Add a Network to the HACMP cluster → choose
diskhb.
z Enter the desired network name (defaults to net_diskhb_01)

z add two communication devices, one for each node, smitty hacmp →
Extended Configuration → Extended Topology Configuration →
Configure HACMP Communication Interfaces/Devices → Add
Communication Interfaces/Devices → Add Pre-Defined Communication
Interfaces and Devices → Communication Devices
z Choose the diskhb created in the previous step (net_diskhb_01), input
Device Name and Device Path parameters.

Page 92
 实施专家级课程 PowerHA

Storage related considerations-Disk heartbeat

¾Configuring multi-node disk heartbeat

z There are two different SMIT paths available to configure a multi-node disk
heartbeat device

z Using the smitty hacmp fast path, select Extended Configuration →

Extended Topology Configuration → Configure HACMP Networks →
Manage concurrent Volume Groups for Multi-Node Disk Heartbeat.
z Using the smitty hacmp fast path, select System Management (C-SPOC)
→HACMP Concurrent Logical Volume Management → Manage
concurrent Volume Groups for Multi-Node Disk Heartbeat.

Page 93
 实施专家级课程 PowerHA

Storage related considerations-Disk heartbeat

¾Testing disk heartbeat

connectivity
z Disk heartbeat testing can only
be run when PowerHA is not
running on the nodes.
z The devicename is the raw
device as designated with the
“r” proceeding the device name.
For hdisks, the dhb_read utility
automatically converts it to the
proper raw device name. For all
other devices (vpath,
hdiskpower, to name two), it is
required to specify it explicitly.

Page 94
 实施专家级课程 PowerHA

Storage related considerations-Disk heartbeat

¾Monitoring disk heartbeat

z cltopinfo -m
z The main field to monitor is the Current Missed Heartbeats. If the total
continues to grow, it is a good indication that there is a problem and possibly
that the disk is not optimal for a diskhb network. Either move the diskhb to
another disk, or change the failure detection rate of the diskhb network to
slow.
z Note: cltopinfo command was introduced in HACMP 5.4.1 and it is not multi-
node disk heartbeat aware. You can monitor multi-node disk heartbeat, and
all heartbeats, using the lssrc -l topscvs command.

Page 95
 实施专家级课程 PowerHA

Storage related considerations-Fast failure detection

¾This feature was added in HACMP 5.4. It provides faster response to

a system failing specifically due to AIX kernel panic. AIX provides a
kernel callout in the panic path. It sends a departing message to a
traditional disk heartbeat device within one heartbeat period. This
message is picked up by topology services and distributed throughout
the other cluster nodes about the node_down event. In turn, this
results in normal node fallover processing being initiated immediately
instead of waiting for missed heartbeats.
¾Note: Currently fast failure detection is not available for multi-node
disk heartbeat devices, only traditional disk heartbeat.

Page 96
 实施专家级课程 PowerHA

Storage related considerations-Fast failure detection

¾Prerequisites
z HACMP 5.4 or above is required.
z AIX 5.3 TL5 (5300-05) or above is required.

z RSCT 2.4.5.2 or above is required.

z Traditional disk heartbeat must be configured.

z Disk heartbeat network must be enabled for fast failure detection.

Page 97
 实施专家级课程 PowerHA

Storage related considerations-Fast failure detection

¾How to enable a traditional disk

heartbeat network for fast failure
detection, you must change the
custom parameters to the
corresponding diskhb network
module.
z smitty hacmp → Extended
Configuration → Extended
Topology Configuration →
Configure HACMP Network
Modules → Change a Network
Module using Custom Values
z Choose the diskhb network type and
press Enter.
z In the Parameters field, you must
manually enter “FFD_ON” because
that field will not generate a picklist to
choose from.
z Synchronize the cluster and restart
cluster services.

Page 98
 实施专家级课程 PowerHA

Network Related Considerations

Page 99
 实施专家级课程 PowerHA

Networking considerations -IPAT via aliasing with boot and service

on same subnet

¾When HACMP was first developed, the expected practice was that each
node would have at least two interfaces on each network.
¾ To get the routing correct, two interfaces on a node require that they
each be on a separate subnet
¾When IPAT via Aliasing was added, it was became necessary to have
the service IP address on a different subnet
¾With the advent of more modern networking technologies such as NIB,
etherchannel and VIO, the physical interface redundancy is pushed down
a layer below TCP/P and having only a single logical interface per
node per network is not necessarily a single point of failure.

¾To remove that burden, the cluster verification function has been
changed (via APAR IZ31674, IZ26020) PowerHA5.4.1 SP4 or later

Page 100
 实施专家级课程 PowerHA

Networking considerations -EtherChannel

¾EtherChannel is a port aggregation method whereby up to eight

ethernet adapters are defined as one EtherChannel. Remote systems
view the EtherChannel as one IP and MAC address, so up to eight
times network bandwidth is available in one network presence.
¾Traffic is distributed across the adapters in the standard way (address
algorithm) or on a round robin basis. If an adapter fails, traffic is
automatically sent to the next available adapter in the EtherChannel
without disrupting user connections
¾When only one link in the main EtherChannel is active, a failure test
triggers a rapid detection / fallover (in 2-4 seconds) to an optional
backup adapter with no disruption to user connections.

Page 101
 实施专家级课程 PowerHA

Networking considerations -EtherChannel

¾PowerHA officially supports the use of EtherChannel.

¾Integrating an EtherChannel solution into a cluster is relatively easy
and can actually simplify network addressing and subnet requirements.
z Higher bandwidth and load balancing
z Built in availability features

z A simple, flexible solution and growth path

z Various options for interoperability with network switch

z It is free

Page 102
 实施专家级课程 PowerHA

Networking considerations -EtherChannel

¾Configuration procedures
z Check the ethernet adapter configurations and adapter cabling
z Create the EtherChannel interface

z Configure IPs on new interface (en6) via TCP/IP

z Add boot and service IPs to PowerHA topology

z Create a resource group and assign it the service IP

z Synchronize the cluster

z Start cluster services

z Test redundancy of NICs and make sure that PowerHA does not detect it

Page 103
 实施专家级课程 PowerHA

Networking considerations -Distribution preference for service IP

¾PowerHA gives you added control over service IP’s placement and
allows you to define a distribution preference for your service IP label
aliases.
¾There are four different distribution policies available:
z Anti-Collocation
z Collocation

z Anti-Collocation with Persistent

z Collocation with Persistent

¾Anti-Collocation
z This is the default. PowerHA distributes all service IP aliases across all
base IP addresses using a “least loaded” selection process.

Page 104
 实施专家级课程 PowerHA

Networking considerations -Distribution preference for service IP

¾Collocation
z PowerHA allocates all service IP aliases on the same network interface card (NIC).
¾Anti-Collocation with Persistent
z PowerHA distributes all service IP aliases across all active physical interfaces that
are NOT hosting the persistent IP label. PowerHA will place the service IP alias on
the interface that is hosting the persistent label only if no other network interface is
available. If you did not configure persistent IP labels, PowerHA lets you select the
Anti-Collocation with Persistent distribution preference, but it issues a warning and
uses the regular anti-collocation preference by default.
¾Collocation with Persistent
z All service IP aliases are allocated on the same NIC that is hosting the persistent IP
label. This option can be useful in VPN firewall configurations where only one
interface is granted external connectivity and all IP labels (persistent and service)
must be allocated on the same interface card. If you did not configure persistent IP
addresses, PowerHA lets you select the Collocation with Persistent distribution
preference, but it issues a warning and uses the regular Collocation preference by
default.

Page 105
 实施专家级课程 PowerHA

Networking considerations -Distribution preference for service IP

¾Configuring service IP distribution policy

z In SMIT, select Extended Configuration → Extended Resource
Configuration → HACMP Extended Resources Configuration → Configure
Resource Distribution Preferences → Configure Service IP
labels/addresses Distribution Preferences
z Verification and Synchronization

¾Viewing the distribution preference for service IP label aliases

z cltopinfo -w
z cllsnw -c

Page 106
 实施专家级课程 PowerHA

Networking considerations -Site specific service IP

¾The site specific service IP label feature was added in HACMP 5.4. It
provides the ability to have unique service addresses at each site. This
helps solve the problem of having different subnets at each site. It can
be used for the IP network types of:
z Ether
z XD_data

z XD_ip

¾Important: In PowerHA/XD, configurations utilizing the site specific IP

labels with XD network types requires configuring a persistent IP label
on each node.

Page 107
 实施专家级课程 PowerHA

Networking considerations -Site specific service IP

¾Configure site specific service IP labels

z 1. Add sites.
z 2. Add network(s) (ether, XD_data, or XD_ip):

z – Enable IPAT via IP Alias

z 3. Add Service IP label(s):

z – Configurable on multiple nodes

z – Specify associated site

z 4. Add resource group(s)

z 5. Add service IP label(s) into the resource group(s)

z 6. Synchronize cluster

z 7. Test site specific IP fallover

Page 108
 实施专家级课程 PowerHA

Networking considerations - netmon.cf

¾Traditional netmon.cf
z In PowerHA you can create a netmon.cf configuration file with a list of
additional network addresses. These addresses will only be used by
topology services to send ICMP(互联网控制消息协议) ECHO requests in
order to help determine an adapter’s status under certain circumstances.
z The implementation of this file is therefore not required, but recommended
in cluster configurations with only a single network card on each node or in
a cluster where failures have left a single adapter network remaining.
z Another scenario is using a single logical EtherChannel interface made up
of multiple ethernet links(Etherchannel).
z The file must exist at cluster startup because RSCT topology services
scans the netmon.cf file during initialization.

Page 109
 实施专家级课程 PowerHA

Networking considerations - netmon.cf

¾Config traditional netmon.cf

z /usr/es/sbin/cluster directory
z The file must consist of one IP
address or IP label per line.
z The file should contain remote IP
labels/addresses that are not in the
cluster configuration and that can be
accessed from PowerHA interfaces.
We recommend the use of the router’s
IP address.
z A maximum of 30 IP addresses/labels
can be defined in netmon.cf
z Include each IP address and its
corresponding label in the /etc/hosts
file.

Page 110
 实施专家级课程 PowerHA

Networking considerations - netmon.cf

¾New netmon.cf format for VIO environments

z This new netmon functionality was added to support PowerHA on VIO.
z PowerHA customers using VIO within their clusters have experienced
problems with specific scenarios where an entire CEC is unplugged from
the network, but the PowerHA node within it does not detect a local adapter
down event.
z This is because traffic being passed between the VIO clients looks like
normal external traffic from the perspective of the LPAR's OS.

z Important: It is not recommended that any customers use this new function
unless they absolutely have to because of their VIO environment.

Page 111
 实施专家级课程 PowerHA

Networking considerations - netmon.cf

¾Config new netmon.cf format for VIO environments

/usr/es/sbin/cluster directory
#Example 1 #Example 2
9.12.4.11 !REQD host1.ibm 100.12.7.9
!REQD en2 100.12.7.9 !REQD host1.ibm host4.ibm
9.12.4.13 !REQD 100.12.7.20 100.12.7.10
!REQD en2 100.12.7.10 !REQD 100.12.7.20 host5.ibm

#Example 3
!REQD !ALL 100.12.7.9
!REQD !ALL 110.12.7.9
!REQD !ALL 111.100.1.10
!REQD en1 9.12.11.10
Page 112
 实施专家级课程 PowerHA

Networking considerations - netmon.cf

¾The netmon.cf format does not change heartbeating behavior itself in

any way. It only changes how the decision is made as to whether a
local adapter is up or down. So this new logic will be used upon:
z Startup, before heartbeating rings are formed during
z Heartbeat failure, when contact with a neighbor is initially lost

z During periods when heartbeating is not possible, such as when a node is

the only one up in the cluster

z Invoking the new format changes the definition of a good adapter from “Am
I able to receive any network traffic?” to “Can I successfully ping certain
addresses?” regardless of how much traffic is seen.

Page 113
 实施专家级课程 PowerHA

Networking considerations -clhosts

¾The clhosts file contains IP address information which helps to enable

communication among monitoring daemons on clients and within the
PowerHA cluster nodes. The tools that utilize this file include: clinfoES,
HAview, and clstat.
¾The file resides on all PowerHA cluster servers and clients in the
/usr/es/sbin/cluster/etc/ directory.
¾When a monitor daemon starts up, it reads the following file,
/usr/es/sbin/cluster/etc/clhosts, to determine which nodes are available
for communication.

Page 114
 实施专家级课程 PowerHA

Networking considerations -clhosts

¾/usr/es/sbin/cluster/etc/clhosts.client will be automatically created

when you perform a verification with the automatic corrective action
feature enabled in all cluster nodes in HACMP 5.3 or later.
¾On clients, clhosts must contain at least one address for each cluster
node(This file is usually configured by copying
/usr/es/sbin/cluster/etc/clhosts.client from a cluster node and
appending it to the clhosts file.
¾The clhosts file on a client should never contain 127.0.0.1, loopback,
or localhost.

Page 115
 实施专家级课程 PowerHA

Networking considerations - clinfo.rc

¾PowerHA can be configured to change the MAC address of a network

interface by the implementation of hardware address takeover (HWAT).
In a switched ethernet network environment, the switch might not
always get promptly informed of the new MAC. In turn, the switch will
not route the appropriate packets to the network interface.
¾The clinfo.rc script is used by PowerHA to flush the system’s ARP
cache in order to reflect changes to network IP addresses. It does not
update the cache until another address responds to a ping request.
¾The clinfoES program will calls the /usr/es/sbin/cluster/etc/clinfo.rc
script whenever a network or node event occurs.
¾When a cluster event occurs, clinfo.rc runs the following command for
each host specified in PING_CLIENT_LIST:
ping -c1 $host

Page 116
 实施专家级课程 PowerHA

Networking considerations - clinfo.rc

¾Config clinfo.rc
¾/usr/es/sbin/cluster/etc/clinfo.rc
PING_CLIENT_LIST=" "
¾Include on this line the names or IP addresses of at least one client on
each subnet on the switched Ethernet.
¾Run clinfoES on all nodes in the PowerHA cluster that are attached to
the switched Ethernet.

Page 117
 实施专家级课程 PowerHA

PowerHA & DR

Page 118
 实施专家级课程 PowerHA

PowerHA & DR

¾A DR solution must do at least three things

z Replicate data to a remote site
z Monitor for failure or recovery of systems in a cluster

z Manage the replicated volumes on fallover or fallback

¾Separate copies of data in each site

¾Data replication can be OS-based or storage device-based

Page 119
 实施专家级课程 PowerHA

PowerHA & DR

IBM
Intra-
Intra-Site
Communications
Link
IBM
!
pSeries
pSeries

Web Server

A
Redundant
Intra-
Intra-Site
Communications
B
Database
Link

Site A Site B

Page 120
 实施专家级课程 PowerHA

PowerHA & DR

¾Extension to PowerHA for AIX

z Providing automated fallover/fallback of applications over geographically
dispersed systems
¾It does the three things:
z Leverages data replication
ƒ GLVM in AIX (storage agnostic)
ƒ PPRC in IBM storage
ƒ SRDF in EMC Storage
z Monitors node status via base PowerHA for AIX
z Manages replicated data volumes during fallover and fallback (XD-specific
event processing)
ƒ GLVM
- Synchronous (HA/XD support in v5.2 and later)
- Asynchronous (HA/XD support in v5.5 and later)
ƒ PPRC
- Metro Mirror (HA/XD support in v5.1 and later)
- Global Copy (HA/XD support in v5.5 and later)
- SRDF (HA/XD support in v6.1 and later)
Page 121
ƒ GeoRM (legacy, no longer available, EOS 9/30/09)
 实施专家级课程 PowerHA

PowerHA & DR

PowerHA SystemMirror Standard Edition

GLVM Cluster Switched Disk Cluster
ƒ Geographic Logical ƒ Data Center HA
Geographic Mirroring Volume Manager ƒ Storage agnostic
ƒ Async mirroring over IP ƒ LVM mirrored copy of
ƒ Storage agnostic data

Metro Mirror/Global Mirror Cluster

EMC Symmetrix
Storage Volume
ƒ SRDF/S
Controller (SVC)
ƒ SRDF/A
ƒ TimeFinder ƒ Global Mirror
SVC & DS8000
Time Finder SRDF FlashCopy Metro Mirror ƒ Metro Mirror

PowerHA SystemMirror Enterprise Edition

Page 122
 实施专家级课程 PowerHA

PowerHA cross-site LVM mirroring

Page 123
 实施专家级课程 PowerHA

PowerHA SystemMirror cross-site LVM mirroring

¾This solution offers automation of AIX LVM mirroring within SAN disk
subsystems between different sites. It also provides automatic LVM
mirroring synchronization and disk device activation when, after a disk
or site failure, a node or disk becomes available.
Client
Legend:
Workstation write request
write complete response
1
End
User
8
Local
Network 2 7 Intra-Site TCP/IP
Network
p5 p5

Backup
Database Database
Server Server

6
3

Intra-Site
SAN
fabric
5
4
Backup
Storage
4 Storage
Subsystem 5
Subsystem

Page 124
 实施专家级课程 PowerHA

PowerHA/XD cross-site LVM mirroring

¾A storage area network (SAN) is a high-speed network that allows the

establishment of direct connections between storage devices and
servers. The maximum distance for the connections is defined by
Fibre Channel limitations. This allows for two or more servers, located
in different sites to access the same physical disks.
¾These remote disks can be combined into a volume group via the AIX
Logical Volume Manager (LVM) and this volume group can be
imported to the nodes located at different sites. You can create logical
volumes and set up a LVM mirror with a copy at each site. The
number of active sites in a cross-site LVM mirroring supported in
PowerHA is limited to two.

Page 125
 实施专家级课程 PowerHA

PowerHA/XD cross-site LVM mirroring

¾Requirements
z The force varyon
attribute for the resource
group must be set to true.
z The logical volumes
allocation policy must be
set to superstrict
z The LV mirroring copies
must be allocated on
separate volumes that
reside on different disk
subsystem

Page 126
 实施专家级课程 PowerHA

PowerHA/XD GLVM

Page 127
 实施专家级课程 PowerHA

PowerHA SystemMirror GLVM

PowerHA SystemMirror Enterprise Edition

GLVM
geographic logical volume manager

TCP/IP Network
¾ Provides long distance host based
asynchronous remote mirroring function
Unlimited distance between sites for disaster AIX LVM
recovery
¾ Does not wait for secondary I/O before RPV Client RPV server
completing host I/O Device Driver Kernel extension
¾ Designed to maintain a consistent secondary
copy
¾ Based on AIX LVM and RPV technology
Production Site DR Site
Page 128
 实施专家级课程 PowerHA

PowerHA/XD GLVM

¾PowerHA/XD for GLVM provides the following features for disaster recovery:
z Allows automatic detection and response to site and network failures in the
geographic cluster without user intervention
z Performs automatic site takeover and recovery and keeps mission-critical
applications highly available through application fallover and monitoring.
z Allows for simplified configuration of volume groups, logical volumes and resource
groups.
z Uses the TCP/IP network for remote mirroring over an unlimited distance
z Supports maximum sized logical volumes
¾PowerHA/XD GLVM is PowerHA extended distance using geographic logical
volumes to mirror data to the remote site

¾Important: HAGEO no longer exists starting in PowerHA/XD 5.5

Page 129
 实施专家级课程 PowerHA

PowerHA/XD GLVM

¾PowerHA/XD GLVM:
z Supports clusters with multiple nodes at 2 sites
z Mirrors data by providing a local representation of the remote physical
volume to the LVM
z The local and remote storage systems don’t have to be the same type of
equipment.

Page 130
 实施专家级课程 PowerHA

PowerHA/XD GLVM

¾Standard AIX LVM Concept:

¾To extend this, other concepts will be introduced

Page 131
 实施专家级课程 PowerHA

PowerHA/XD GLVM

¾Definitions and concepts

z Remote physical volume (RPV):
ƒ A pseudo device driver that provides access to the remote disks as though they were locally
attached.
ƒ The RPV consists of two parts:
ƒ RPV Client:
ƒ RPV Server:
z Geographically mirrored volume group (GMVG):
ƒ A volume group that consists of local and remote physical volumes.
z Network definitions
ƒ XD_data
ƒ XD_ip
ƒ XD_rs232
ƒ RSCT heartbeat packets will be sent over all the networks.
z Note: PowerHA/XD GLVM requires one XD_data network for data
replication and one of XD_rs232 or XD_ip to differentiate between a remote
node failure or XD_data network failure.

Page 132
 实施专家级课程 PowerHA

PowerHA/XD GLVM

¾A second site provides a second system to take mirrored copies

¾The second site contributes disks (LUNs) to the first site for mirroring
targets
z Disks are seen as local (hdiskY) but are only a representation of a physical
disk, that is in the remote site (hdiskZ)
z Data is replicated to remote disk via network

Page 133
 实施专家级课程 PowerHA

PowerHA/XD GLVM- Synchronous

¾Reads satisfied locally

z When possible, for performance
¾Write does not complete until data written at both sites
z D/R site always has latest data
z No data loss on site fallover

However

¾Remote writes slower – possibly much slower - due to network delay

z Latency – data transmission time – limits individual write performance
z Bandwidth – data throughput – limits overall application write speed

¾Application performance worse than local

z Tuning, network improvements, caching disk controllers can help

Page 134
 实施专家级课程 PowerHA

PowerHA/XD GLVM – Reversing the Mirroring

¾For fallover purposes, there must be a reverse relationship

¾Site A’s disks (LUNs) become mirroring targets for Site B
z Again, appearing to be local, but actually representing the physical disk from
the other site

Page 135
 实施专家级课程 PowerHA

PowerHA/XD GLVM- Asynchronous(New with PowerHA5.5)

¾Remote writes are cached locally and completed asynchronously

¾Built on synchronous GLVM and AIX LVM Mirror Pools
¾Application does not experience data transmission, remote write delay
z Can hide peeks in load (until the cache fills…)

However,
¾Increased risk of data loss on unplanned site fallover
z Cached but uncompleted writes
¾Possible data divergence after unplanned site fallover
z Running without latest data
¾More complex to configure than synchronous
z Enterprise must capable of defining mirror pools (AIX 6.1)
z Careful planning is necessary to decide how to handle data divergence

Page 136
 实施专家级课程 PowerHA

PowerHA/XD GLVM- Asynchronous (New Concept)

¾Mirror Pools – AIX 6.1 TL2 and up

z Collection of disks in a volume group containing logical volume copies
ƒ If used, logical volume does not cross mirror pool boundary
ƒ mirror pools do not cross volume groups
z Super strict mirror pools
ƒ Each mirror pool contains a complete mirror copy of each logical volume
ƒ Local physical volumes and remote physical volumes in separate mirror pools
ƒ Max 3 mirror pools in a volume group (As in 3 LVM copies per LV)
z Exploitable for synchronous mirroring and cross-site LVM mirroring
z Asynchronous mirroring configured by defining mirror pool as “asynchronous”

z Asynchronous and synchronous mirror pools allowed in same volume group

¾AIO Cache
z One logical volume per mirror pool, to cache asynchronous writes
z Are not themselves mirrored across sites
ƒ But must be defined as superstrict
z Can increase, but not decrease, LV size
Page 137
 实施专家级课程 PowerHA

PowerHA/XD GLVM - Asynchronous

¾Mirror Pool & AIO_Cache

Page 138
 实施专家级课程 PowerHA

PowerHA/XD GLVM

¾Configure synchronous GLVM

¾ Setting up the cluster:
z – Installing the filesets
z – Creating the base cluster
z – Creating a resource group
¾ Configuring and testing Synchronous GLVM:
z – Creating RPV servers
z – Creating RPV clients
z – Creating volume groups
z – Creating logical volumes
z – Creating file systems (if applicable)
z – Creating GLVM copies
z – Importing GMVGs to each node
z – Testing GMVGs
¾ Adding GMVGs into resource group(s)
z Starting the cluster

Page 139
 实施专家级课程 PowerHA

PowerHA/XD GLVM

¾GLVM status monitors commands

z Note: The gmvgstat and rpvstat commands were introduced with PowerHA/XD
GLVM 5.4.1

Page 140
 实施专家级课程 PowerHA

PowerHA/XD GLVM - Asynchronous

¾Factors to consider with asynchronous

z Network bandwidth
z Network latency

z Preventing data loss

z Data divergence

¾Attention: Up to now(2009.8), enhanced concurrent volume groups

are not supported with asynchronous GLVM. We must use scalable
volume groups (SVG) to configure Asynchronous GLVM.
z The reason for this requirement comes from the fact that asynchronous
GLVM depends on LVM Mirror Pool functionality, and AIX LVM currently
allows mirror pools only on Scalable volume groups.

Page 141
 实施专家级课程 PowerHA

PowerHA/XD GLVM – Requirement of VG

¾Only supported on:

z AIX V6.1 Technology Level 2 Service Pack 3 and up
¾Must be SVG (Scalable Volume Group) type
z Must be super strict mirror pool enabled
z Must have auto varyon and bad block relocation disabled

¾Must be non-concurrent volume group

z Not even ECM for fast disk takeover within a site
¾Asynchronous GMVG cannot contain paging type device
¾Cannot be snapshot volume group
¾rootvg cannot be configured for asynchronous mirroring

Page 142
 实施专家级课程 PowerHA

PowerHA/XD GLVM - Asynchronous GLVM Write Flow

Page 143
 实施专家级课程 PowerHA

PowerHA/XD GLVM - Asynchronous GLVM Write Flow

1. Application issues a write request to datalv1

2. LVM issues write requests to PV1 and PV4
PV1 is an ordinary local disk
PV4 is a remote physical volume, so the write goes to the RPV client
3. RPV client stores the write in aiocachelv1
RPV client passes write to aiocachelv1 to LVM
4. LVM writes the cache entry to logical volume aiocachelv1
5. LVM informs the RPV client that its cache entry has been written
6. The RPV client informs LVM that its write to PV4 has completed
The write really has not completed, but it has been recorded in the cache
7. LVM informs the application that its write to the logical volume has
completed
8. RPV client sends the write request to the RPV server
9. RPV server writes the data to disk PV4
10 RPV server informs the RPV client that its write to PV4 has completed
Page 144
 实施专家级课程 PowerHA

PowerHA/XD GLVM

¾ Configure asynchronous GLVM

¾ Setting up the cluster:
z – Installing the filesets
z – Creating the base cluster
z – Creating a resource group
¾ Configuring and testing Asynchronous GLVM:
z – Creating RPV servers
z – Creating RPV clients
z – Creating or concerting volume groups (scalable type & Disable bad block relocation)
z – Assign disks in the Scalable Volume Groups to mirror pools
z – Create mirrored logical volumes specifying First Copy and Second Copy mirror pools and
creating FS (Disable bad block relocation and set allocation policy to superstrict)
z – Create aio_cache logical volume for each asynchronous mirror pool (These are not
mirrored)
z – Change the mirror pool setting from synchronous to asynchronous (Only one
asynchronous mirror pool allowed per site per volume group)
z – Importing GMVGs to each node
z – Testing GMVGs
¾ Adding GMVGs into resource group (s) (some special parameters)
z Starting the cluster

Page 145
 实施专家级课程 PowerHA

PowerHA/XD GLVM

¾Migrate from synchronous to asynchronous GLVM

z Assign PV to Mirror Pool (local and remote)
z Update attribution of VG (disable auto varyon, turn on super strict mirror
pool, disable bad block relocation)
z Assign mirror pools to each LV

z Creating aio_cache lv for each Mirror Pool

z Convert to Asynchronous Mirroring for Mirror Pool with aio_cache LV

z Update attribution of Resource Group

ƒ Use forced varyon of volume groups
ƒ Default choice for data divergence recovery
ƒ Allow varyon with missing data updates
z Verification and Synchronization

Page 146
 实施专家级课程 PowerHA

PowerHA/XD GLVM

¾Commands to check GLVM status

z Check cluster and mirror pool status -- clifindres

z List mirror pool status – lsmp –A vgname

Page 147
 实施专家级课程 PowerHA

PowerHA/XD GLVM

¾Commands to check GLVM status

z GMVGs status – gmvgstat

z RPV status – rpvstat

Page 148
 实施专家级课程 PowerHA

PowerHA/XD GLVM

¾Commands to check GLVM status

z RPV status – rpvstat –A (-A Display the statistics for Asynchronous Mirroring)

z RPV status – rpvstat –C (-C Display the statistics for Asynchronous I/O
Cache)

Page 149
 实施专家级课程 PowerHA

PowerHA/XD GLVM – Production Site Failure & Recovery

¾If production site suddenly fails while it has volume group varied online
z Normal varyonvg at the D/R site will fail
ƒ Detects potential presence of non-mirrored data in cache at production site
z Risk of data divergence
¾Administrative/Enterprise choice to accept this
z Specify “Allow varyon with missing data updates” – choice for automatic
fallover and recovery
z Otherwise, manual intervention required to recover from an event error
z Run varyonvg with the new –d flag

¾On production site recovery, which version of data do you keep?

z Specify “Default choice for data divergence recovery” as the D/R site for latest
data
z Specify the production site to discard updates
z Any non-mirrored updates on non-selected site lost on the merge
z Selecting “ignore” indicates no choice made, and manual recovery required
z varyonvg –k from either site indicates which site (local or remote) is kept
ƒ “local” and “remote” relative to the site where the command is run
Page 150
 实施专家级课程 PowerHA

PowerHA/XD and ESS/DS Metro Mirror

Page 151
 实施专家级课程 PowerHA

PowerHA/XD and ES/DSS Metro Mirror

¾Peer-to-peer remote copy (PPRC) is a protocol to mirror a disk from

one storage system to another disk in a remote site.
¾PPRC can be used to provide fast data recovery after failure of the
primary site.
¾PPRC is classified as synchronous and asynchronous.
z Synchronous PPRC causes each write to the primary volume to be
performed to the secondary as well, and the I/O is only considered
complete when the update to both primary and secondary have completed.
(Metro Mirror)
z Asynchronous PPRC I/O is considered to be completed immediately after
performing the I/O on primary volume,and performs I/O to the secondary
volume asynchronously. (Global Mirror/Global Copy)
z The IBM System Storage DS8000, DS6000, and ESS800 devices support
PPRC.
¾Until now, PowerHA/XD support Metro Mirror for PPRC.

Page 152
 实施专家级课程 PowerHA

PowerHA/XD and ES/DSS Metro Mirror

Mirror type Hardware Type PPRC is managed How HACMP manages PPRC pairs
by

Synchronous ESS 800, etc Copy Services Directly manages the fallover and
Server (CSS, on resynchronization of the PPRC pairs by
storage controller) issuing commands directly to the ESS
systems. In this book, this is referred to as
Direct management. In general, it refers to
PPRC between ESS storage systems.
Synchronous ESS (800) or DS DSCLI management, Relies on the ESSNI server to manage
(8000, 6000) or via ESSNI Server PPRC pairs. Directly manages the fallover
intermix of any of on either storage and resynchronization of the PPRC pairs
these controller or by issuing commands directly to the
Hardware storage systems. In this book, this is
Management referred to as DSCLI management.
Console (HMC)
Synchronous vDisks as provided SVC management Relies on the Copy Services Server in order
by SAN Volume of PPRC services to manage PPRC function. Directly
Controller. on SVC-specific manages the fallover and
(Hardware as hardware resynchronization of the PPRC pairs by
supported by issuing commands directly to the Copy
SVC services) Services Server. In this book, this is
referred to as PPRC SVC management.

Page 153
 实施专家级课程 PowerHA

PowerHA/XD and ES/DSS Metro Mirror

PPRC Management Type Filesets to install

Direct management cluster.es.pprc.cmds

cluster.es.pprc.rte
cluster.msg.en_US.pprc
(and/or other appropriate language message sets)

DSCLI cluster.es.pprc.rte
cluster.es.spprc.cmds
cluster.es.spprc.rte cluster.msg.en_US.pprc
(and/or other appropriate language message sets)

SAN Volume Controller cluster.es.svcpprc.cmds

(SVC) cluster.es.svcpprc.rte
cluster.msg.en_US.svcpprc
(and/or other appropriate language message sets)

Page 154
 实施专家级课程 PowerHA

PowerHA/XD and ES/DSS Metro Mirror – Direct Management

¾Direct management is the oldest type of PPRC support PowerHA

provides. It has the simplest hardware configuration, but requires more
work on the administrator’s part to set up.
¾PowerHA provides support for direct management of ESS PPRC by
managing specified pair and path tasks as defined by the user on their
ESS storage systems Copy Services Server (CSS.)
¾Direct management can be used to support PPRC between ESS 800
storage systems.

Page 155
 实施专家级课程 PowerHA

PowerHA/XD and ES/DSS Metro Mirror – Direct Management

¾Planning for Direct management

z Plan for connections to the ESS
z Plan for ESS Copy Services

z Plan the HACMP/XD for Metro Mirror configuration:

ƒ Identify ESS systems to be included
ƒ Identify which resource groups will contain the PPRC replicated resources
z Identify PPRC replicated resources and provide information about the
associated volumes
z (Optional) Plan for any user-specific PPRC tasks.

¾Limitations
z You can only use non-concurrent volume groups with PPRC.

Page 156
 实施专家级课程 PowerHA

PowerHA/XD and ES/DSS Metro Mirror – DSCLI management

¾DSCLI management provides the ability to execute Copy Services

operations directly without relying on previously saved GUI tasks. The
software dynamically manages DSCLI PPRC-controlled disks, providing
a fully automated, highly available disaster recovery management
solution.
¾The PowerHA interface is designed to communicate with the DSCLI so
that once the basic PPRC environment is configured, PPRC relationships
are created automatically and there is no need for manual access to the
DSCLI.
¾It supports ESS and DS in different site.

Page 157
 实施专家级课程 PowerHA

PowerHA/XD and ES/DSS Metro Mirror – DSCLI Management

¾Planning for DSCLI management

z Identify the Copy Services Servers (CSS) to be used
z Identify the disk subsystems to be used in the cluster

z Identify the vpaths to be used in the configuration, including the volume IDs for
each that corresponds to the storage unit and LUN.
z Identify the PPRC Replicated Resources to be used

z Identify the Port Pairs to be used for PPRC Paths

z Identify Volume Pairs (LUNs)

z Identify the volume groups to be managed by PPRC Replicated Resources

z Plan which resource groups will contain the DSCLI-managed PPRC replicated
resources (if not already done in the General Planning section).
¾Limitations and restrictions for DSCLI management
z Volume group limitations
ƒ A volume group must have the same volume major number across all cluster nodes.

Page 158
 实施专家级课程 PowerHA

PowerHA/XD and ES/DSS Metro Mirror – DSCLI Management

¾Limitations and restrictions for DSCLI management

z Volume group limitations
ƒ A volume group must have the same volume major number across all cluster nodes.
ƒ Resource groups to be managed by HACMP cannot contain volume groups with both PPRC-
protected and non PPRC-protected disks.
z Managed resource limitations
ƒ Resource groups cannot manage both DSCLI and Direct management (ESS CLI)-managed
PPRC resources simultaneously
z Copy Services functions limitations
ƒ Only IBM TotalStorage Copy Services function Synchronous PPRC (Metro Mirror) is supported
z C-SPOC limitations
ƒ C-SPOC operations on nodes at the same site as the source volumes successfully perform all
tasks supported in HACMP.
ƒ C-SPOC operations will not succeed on nodes at the remote site :Creating a volume group ,
changing filesystem size, changing mount point, adding LVM mirrors
ƒ For other LVM operations, it is highly recommended that you perform all C-SPOC operations

Page 159
 实施专家级课程 PowerHA

PowerHA/XD and ES/DSS Metro Mirror

Page 160
 实施专家级课程 PowerHA

PowerHA/XD and ES/DSS Metro Mirror

Page 161
 实施专家级课程 PowerHA

PowerHA/XD and ES/DSS Metro Mirror

¾Use the configuration data and smit pprc_ds to configuration HACMP

smit pprc_ds
Copy Services Server Configuration
DS ESS Disk Subsystem Configuration
DSCLI-Managed PPRC Replicated Resource Configuration
PPRC Consistency Groups Configuration
Verify PPRC Configuration
z From the above smit menu:

z Choose the Copy Services Server Configuration option and configure CSS

z Choose DS ESS Disk Subsystem Configuration and configure DS ESS

subsystem
z Choose DSCLI-Managed PPRC Replicated Resource Configuration and
configure the PPRC-replicated resource

Page 162
 实施专家级课程 PowerHA

PowerHA with SAN Volume Controller

Page 163
 实施专家级课程 PowerHA

PowerHA with SAN Volume Controller

PowerHA cluster

PowerHA/XD PowerHA/XD

SAN SAN
SAN Volume SAN Volume
Global Mirror Controller
Controller
or
DS Metro Mirror DS
4000 4000
DS8000 DS8000

EMC EMC Migration

FlashCopy

Page 164
 实施专家级课程 PowerHA

PowerHA/XD and SVC

Page 165
 实施专家级课程 PowerHA

PowerHA/XD and SVC

¾Prerequisites
z AIX 5.3 TL 7 or higher
z openssh version 4.6.1 or higher (+ license), for access to the SVC interface

z Base PowerHA filesets (5.5 or higher is required for global copy support)

z Virtual I/O Server 1.5 (if applicable)

z 2145 SDD or SDDPCM 2.2.0.0 (for AIX client or VIOS)

z SVC copy services (metro and/or global)

z each node that can host SVC replicated resources must have access to its
own local SVC over an IP network via ssh
z cluster.es.svcpprc.cmds, cluster.es.svcpprc.rte, cluster.msg.en_US.svcpprc,
cluster.xd.license Metro Mirror/Global Mirror Cluster
Storage Volume
Controller (SVC)
ƒ Global Mirror
SVC & DS8000
FlashCopy Metro Mirror ƒ Metro Mirror

Page 166
 实施专家级课程 PowerHA

PowerHA/XD and SVC – Configure Steps

¾Add two sites.

¾Add XD_ip network(s).
¾Add SVC cluster definitions (smitty svcpprc_def → SVC Clusters
Definition to HACMP)
¾Add SVC PPRC relationships(smitty svcpprc_def → SVC PPRC
Relationships Definition → Add an SVC PPRC Relationship)
¾Add SVC PPRC-Replicated resources(smitty svcpprc_def → SVC
PPRC Relationships Definition → Add an SVC PPRC Resource)
¾Create volume group(s) locally
z Create logical volumes.
z Create file systems.

¾Tip: Before continuing to the next step, ensure that the file systems
are unmounted and the volume group is varied off.

Page 167
 实施专家级课程 PowerHA

PowerHA/XD and SVC – Configure Steps

¾Create temporary SVC PPRC relationship with SVC command

z This step allows the LVM related information previously created to be
replicated to the corresponding remote auxiliary Vdisks.
z ssh admin@<master_SVC_cluster IP> svctask mkrcrelationship –master
<master_Vdisk_name> -aux <aux_Vdisk_name> -cluster
<Aux_SVC_cluster> -name <relationship_name>
¾Upon completion, wait until the relationship moves from
inconsistent_copying to consistent_synchronised state.
z ssh admin@master<master_SVC_Cluster IP> svcinfo lsrcrelationship
<relationship name>
¾Import volume group(s) to remote site
z importvg -V <majornumber> -y <vgname> hdisk#
z Upon completion of verifying the data, unmount the file systems and varyoff
the volume groups on each node.

Page 168
 实施专家级课程 PowerHA

PowerHA/XD and SVC – Configure Steps

¾Create resource group(s)

z remember that mixing non-replicated volume groups and replicated volume
groups in the same resource group is not allowed.
z Add resource group

z Add VGs and replicated resources into resource group

z Synchronize cluster

¾Test

Page 169
 实施专家级课程 PowerHA

PowerHA/XD mirroring with SRDF

management

Page 170
 实施专家级课程 PowerHA

PowerHA/XD mirroring with SRDF management

¾New feature of PowerHA SystemMirror Enterprise Edition V6.1

¾Many IBM Power Systems clients deploy their storage via EMC
storage and have asked that IBM extend PowerHA SystemMirror
support to include their storage replication configurations.
¾Now with the PowerHA SystemMirror Enterprise Edition, clients using
EMC storage with SRDF can take advantage of the PowerHA
SystemMirror capabilities for HA/DR operations.

EMC Symmetrix
ƒ SRDF/S
ƒ SRDF/A
ƒ TimeFinder

Time Finder SRDF

Page 171
 实施专家级课程 PowerHA

PowerHA/XD mirroring with SRDF management

¾Configuration With SRDF

z Smitty hacmp->Extended Configuration->HACMP Extended Resources
Configuration->Configure EMC SRDF(R) Replicated Resources->Add EMC
SRDF(R) Replicated Resource->

Page 172
 Thank
You!
何兵 [email protected]

© Copyright IBM Corporation 2010