1

 PC or Laptop with internet Connection

 Red Hat Certified Engineer (RHCE) certification or equivalent experience or good Linux
knowledge is required

 CentOS 7, RHEL7 ISO Image, VMware workstation

2
Introduction – High Availability Clustering

• What is clustering & cluster types?

• Advantages of Clustering Servers
• Concepts and techniques
• Resource and resource groups
• Failover, Fencing, Shared Storage, Quorum
• Cluster Architecture
• Lab Setup using KVM Environment
• Configuring a Basic Cluster
• Configuring a fencing agent using KVM host machine
• Troubleshooting fencing device
3
Managing Cluster Nodes

• Starting & Stopping Cluster Services

• Enabling & Disabling the Cluster Services
• Adding & Removing A Cluster Node
• The Standby & unstandby Nodes
• Quorum Operations
• Lab Session on quorum
• Managing Quorum Calculations
• Cluster Setup Switches
1) wait_for_all
2) auto_tie_breaker
4
Creating and Configuring Resources

• Create and configure high-availability resources.

• Creating a clustered Apache service
• Managing Resources

5
Troubleshooting High-Availability Cluster

• Inspect and configure cluster logging

• Troubleshooting resource failures
• Troubleshooting cluster network issues

6
Troubleshooting High-Availability Cluster

• Inspect and configure cluster logging

• Troubleshooting resource failures
• Troubleshooting cluster network issues

7
Complex Resource Group

• Configuring an Active/Passive NFS Resource Group

• Lab Session

8
Managing Constraints

• Types of constraints:
• Order, Location & Colocation Constraint
• Practice Lab Session

9
Managing iSCSI Initiators

• iSCSI fundamentals
• Configuring an iSCSI Server
• Several types of backing Storage
• block, fileio, pscsi & ramdisk
• Creating iSCSI Target
• Lab Session to create a block backstore from the targetcli shell
10
Managing High Availability Logical Volumes

• Clustered LVM & HA-LVM

• Lab Session to shared a disk (lun) to all cluster nodes
• Practice Lab Session on HA-LVM

11
Managing Clustered Logical Volumes

• Active/Active configuration of logical volumes

• Distributed Lock Manager (DLM) for lock management
• clvmd daemon
• Practice Lab Session

12
Global File System 2 (GFS2)

• GFS2 concepts
• Creating a GFS2 formatted Cluster File Systems
• Managing a GFS2 File System
• Managing a GFS2 Resource in the cluster
• Growing & Repairing a GFS2 File System

13
14
Introduction

 Linux High Availability Clustering provides intensive, hands-on experience with the Pacemaker
component of the Red Hat Enterprise Linux /CentOS High-Availability Add-On, and cluster storage
components from the Resilient Storage Add-On, including Cluster Logical Volume Manager
(CLVM), Red Hat Global File System 2 (GFS2), and Device-Mapper Multipath.

 High-Availability cluster aka Failover-cluster (active-passive cluster) is one of the most widely used
cluster types in the production environment. This type of cluster provides you the continued
availability of services even one of the node from the group of computer fails. If the server running
application has failed for some reason (hardware failure), cluster software (pacemaker) will restart
the application on another node.

15
Introduction

 Computer cluster technology puts clusters of systems together to provide better system reliability and performance.
Cluster server systems connect a group of servers together in order to jointly provide processing service for the
clients in the network.

 High-Availability cluster aka Failover-cluster (active-passive cluster) is one of the most widely used cluster types
in the production environment. This type of cluster provides you the continued availability of services even one of
the node from the group of computer fails. If the server running application has failed for some reason (hardware
failure), cluster software (pacemaker) will restart the application on another node.

 Linux High Availability Clustering provides intensive, hands-on experience with the Pacemaker component of the
Red Hat Enterprise Linux /CentOS High-Availability Add-On, and cluster storage components from the Resilient
Storage Add-On, including Cluster Logical Volume Manager (CLVM), Red Hat Global File System 2 (GFS2), and
Device-Mapper Multipath.
16
What is Clustering?

Two different types of clusters:

1)High-availability Clusters: known as an HA Cluster or failover cluster

 High-availability clusters can be grouped into two subsets:

• Active-active high-availability clusters, where a service runs on multiple nodes, thus leading to
shorter failover times.
• Active-passive high-availability clusters, where a service only runs on one node at a time.
17
Advantages of Clustering Servers

 Clustering servers is completely a scalable solution. You can add resources to the cluster afterwards.

 If a server in the cluster needs any maintenance, you can do it by stopping it while handing the load over
to other servers.

 Among high availability options, clustering takes a special place since it is reliable and easy to configure.
In case of a server is having a problem providing the services furthermore, other servers in the cluster can
take the load.

18
 What is Clustering?

Two different types of clusters:

2)Storage Clusters:

 Storage clusters: In a storage cluster, all members provide a single cluster file system that can
be accessed by different server systems. The provided file system may be used to read and write
data simultaneously. This is useful for providing high availability of application data, like web
server content, without requiring multiple redundant copies of the same data. An example of a
cluster file system is GFS2
19
A high-availability cluster uses various concepts and techniques

 Resources and resource groups

 Failover
 Fencing
 Shared Storage
 Quorum

20
Architectural Overview

 Identify the components of a Linux High-Availability (HA) Cluster

 Hardware Components

21
22
 Architectural Overview

 Identify the components of a Linux High-Availability (HA) Cluster

 Software Components : In order to provide cluster services with the Linux High Availability Add-on,
multiple software components are required on the cluster nodes.

 Corosync is an open source Cluster Engine. It is actually a Communication System that enables two or
more Linux Cluster nodes to transfer information between them. Corosync is constantly listening on
configured port number where Linux Cluster nodes are sending information. Corosync Communication
System enables all of the nodes to know the exact state of each other at all time. In case one of the Linux
Cluster nodes fails this information will be immediately transferred to other still exsisting Linux Cluster
nodes.
23
Architectural Overview

 Identify the components of a Linux High-Availability (HA) Cluster

 Pacemaker is an open source high availability Resource Manager. This is the component
responsible for all cluster-related activities such as monitoring cluster membership, managing the
services and resources and fencing cluster members. In case one of the Linux Cluster nodes fails
Pacemaker will detect this and start configured Resources on one of the other available Linux
Cluster nodes.

24
Lab Setup

25
26
 Configure a machine for kvm

• yum install qemu* virt* libvirt* -y

• Create a filesystem for disk image for virtual machines
• Create a directory for storing the iso image of vm’s like CentOS, Red Hat Linux, Windows etc
• Networking for your virtual machines (Bridged Network)

 Create virtual machines as nodea, nodeb & nodec

 SSH Password less Login Using SSH Keygen

• ssh-keygen –t rsa
• ssh nodeb.example.com mkdir –p .ssh
• cat .ssh/id_rsa.pub | ssh root@nodeb 'cat >> .ssh/authorized_keys'

27
Configuring a Basic Cluster

28
Configuring Cluster Node Fencing

29
30
Managing Cluster Nodes

• Starting & Stopping Cluster Services

• Enabling & Disabling the Cluster Services
• Adding & Removing A Cluster Node
• The Standby & unstandby Nodes

31
Quorum Operations

• In order for a cluster to work as expected, the nodes must be in agreement on certain facts, such as which
machines are currently cluster members, where services are running, and which machines are using which
resources.

• The method in which this is implemented in the Red Hat High Availability Add-on is by the use of a
majority voting scheme. Every cluster node casts one vote if it successfully joins the corosync network
communication and is able to communicate with the other nodes that are already participating in the
cluster.

• The cluster is operational if more than half of all possible votes are successfully cast. The minimum
number of votes needed to achieve more than half of the votes is called the quorum. If quorum is
achieved, the cluster is considered quorate. A cluster loses quorum if half of the nodes or more cannot
communicate with each other.

32
Managing Quorum Calculations

• Quorum Calculation Options:

 The votequorum component allows a cluster administrator to build clusters with switches that alter the way
quorum gets calculated.
 When building a new cluster with pcs cluster setup command, we can add few switches to change the
behaviour of quorum handling in the cluster

• --wait_for_all
• --auto_tie_breaker
• --last_man_standing
33
34
Creating and Configuring Resources

• Resources : A resource can be a file system, an IP address or a service like httpd etc.
Clustered service consist of one or more resources.

• Resource Agents: All resources are monitored independently from each other and to
accomplish this, resources are controlled by resource agents. Pacemaker can manage
different kinds of resource agents.

• Commonly used resources : Filesystem, IPaddr2, apache, mysql etc

35
Creating and Configuring Resources

• Resources Groups: A service usually consists of more than one resource. A convenient way
to tie those resources together is to add them to the same resource group. All services in the
same resource group get started in the order in which they have been added to the resource
group and stopped in the reverse order. When a cluster node fails, the cluster migrates the
whole resource group to a different node and starts the resources on the new node.

• pcs resource list

• pcs resource describe Filesystem

36
Troubleshooting High-Availability Cluster

• Corosync: This is the framework used by Pacemaker for handling communication between the cluster
nodes. Corosync is also Pacemaker’s source of membership and quorum data.

• Pacemaker: This is the component responsible for all cluster-related activities, such as monitoring
cluster membership, managing the services and resources, and fencing cluster members.

37
Troubleshooting Resource Failures

• Resources can fail for multiple reasons. An administrator might have used incorrect settings when defining the resource, a
configuration file might have an error in it, the system might be trying to start a resource that does not exist, or some other
unforeseen issue might occur.

• Whenever a resource fails, the cluster will increase the failcount for a resource. This count can be viewed with the command
pcs resource failcount show <RESOURCE>.

• Failure to start or stop will immediately set the failcount for a resource to INFINITY, forcing it to move to a different node. If
fencing is enabled, a node that failed to stop a resource will also be fenced.

• Resources can be configured to relocate to a different node after N amount of failures as well, by setting the option meta
migration-threshold=N when creating or modifying the resource. By default, resources will not migrate unless their failcount
reaches INFINITY.
38
39
Managing Constraints

• Constraints are rules that place restrictions on the order in which resources or resource groups may be
started, or the nodes on which they may run. Constraints are important for managing complex resource
groups or sets of resource groups, which depend upon one another or which may interfere with each other.

• There are three main types of constraints:

 Order constraints, which control the order in which resources or resource groups are started and stopped.
 Location constraints, which control the nodes on which resources or resource groups may run.
 Colocation constraints, which control whether two resources or resource groups may run on the same node.

40
Managing Constraints

• CONFIGURING COLOCATION CONSTRAINTS

Colocation constraints specify that two resources must (or must not) run on the same node. To set a
colocation constraint to keep two resources or resource groups together:
• [root@nodeA ~]# pcs constraint colocation add B with A

A colocation constraint with a score of -INFINITY can also be set to force the two resources or
resource groups to never run on the same node:
• [root@nodeY ~]# pcs constraint colocation add B with A -INFINITY

41
• MANAGING HIGH AVAILABILITY LOGICAL VOLUMES

 There are two ways to use LVM on shared storage in a cluster, Clustered LVM and HA-LVM.

 With Clustered LVM, all volume groups and logical volumes on shared storage are available to all
cluster nodes all of the time. With HA-LVM, a volume group and its logical volumes can only be
accessed by one node at a time.

 Clustered LVM is a good choice when working with a shared file system, like GFS2. On the other
hand, HA-LVM is a good choice when working with a more traditional file system like ext4 or
XFS, and restricted access to just one node at a time is desired to prevent file system and/or data
corruption.
42
43
• MANAGING CLUSTERED LOGICAL VOLUMES

 Clustered LVM

• With Clustered LVM, all volume groups and logical volumes on shared storage are available to all
cluster nodes all of the time. With HA-LVM, a volume group and its logical volumes can only be
accessed by one node at a time.

• Clustered LVM allows the use of regular LVM volume groups and logical volumes on shared storage.
In a cluster configured with clustered LVM, a volume group and its logical volumes are accessible to
all cluster nodes at the same time. With clustered LVM, administrators can use the management
benefits of LVM in conjunction with a shared file system like GFS2, for scenarios such as making
virtual machine images inside logical volumes available to all cluster nodes.
44
• MANAGING CLUSTERED LOGICAL VOLUMES

 Clustered LVM

• The active/active configuration of logical volumes in a cluster using clustered LVM is accomplished
by using a daemon called clvmd to propagate metadata changes to all cluster nodes. The clvmd
daemon manages clustered volume groups and communicates their metadata changes made on one
cluster node to all the remaining nodes in the cluster.

• Without the clvmd daemon, LVM metadata changes made on one cluster node would be unknown to
other cluster nodes. Since these metadata define which storage addresses are available for data and file
system information, metadata changes not propagated to all cluster nodes can lead to corruption of
LVM metadata as well as data residing on LVM physical volumes.
45
• MANAGING CLUSTERED LOGICAL VOLUMES

 Clustered LVM

• In order to prevent multiple nodes from changing LVM metadata simultaneously, clustered LVM
uses Distributed Lock Manager (DLM) for lock management. The clvmd daemon and the DLM
lock manager must be installed prior to configuring clustered LVM.

46
GLOBAL FILE SYSTEM 2 (GFS2)

• Global File System 2 (GFS2) is a cluster file system interfacing directly with the kernel VFS layer. This
means that the same file system can be mounted and used by multiple cluster nodes simultaneously, while
still providing a full regular file system.

• To accomplish this, every node accessing a GFS2 file system uses the cluster infrastructure provided by
Corosync and Pacemaker to provide services such as fencing and locking. Each cluster node mounting a
GFS2 file system will use a separate journal. If a node fails, one of the other nodes in the cluster will
replay the journal for the failed node after the failed node has been fenced.

• To prevent race conditions between two nodes when accessing the file system, GFS2 uses the Distributed
Lock Manager (DLM) to coordinate locks on files and directories.
47
SELinux and GFS2

• GFS2 supports extended attributes (xattrs) and can store file labels used by Security Enhanced Linux
(SELinux) just like XFS and ext4. However, using SELinux with GFS2 is complicated by the fact that
updates to SELinux file labels on a GFS2 file system are currently not cluster coherent.

• This means that if one node changes the SELinux context of a file on a GFS2 file system, other cluster
nodes that have that file system mounted may continue using the old context on that file indefinitely.
This is somewhat tricky to resolve, and the issue is currently being tracked at bugzilla.redhat.com as
bug #437984.

• SELinux: Avoid SELinux on GFS2", seems to state that SELinux must be turned off on GFS2 file
systems.
48
GROWING A GFS2 FILE SYSTEM

49