CYBERARK UNIVERSITY
Vault Availability
Cluster Vault
CyberArk Training
1
OBJECTIVES
By the end of this lesson you will be able to:
• Describe the different solutions for Vault availability.
• Describe the strengths and limitations of each model.
• Deploy High Availability Cluster
2
VAULT AVAILABILITY OVERVIEW
3
VAULT AVAILABILITY SOLUTIONS
Replicate
COLD • Secure replication of encrypted data to a remote Windows server for tape
backup to an off-site facility
Disaster Recovery (DR)
WARM • One way replication of vault data to a standby Vault server
High Availability (HA)
HOT • Cluster Vault – Two Vault servers using Clustering Services
• Distributed Vaults – Multiple Vault servers providing services at the same time
4
DISASTER RECOVERY
• The Disaster Recovery
DC2
(DR) Vault is a
replication/failover solution
CPM/PVWA/
designed to create a PSM…
stand-by copy of a
DR
Production Vault on a Replication
remote and dedicated
machine Replication
CPM/PVWA/
• The DR-Vault can be PSM… Vault
DR
DC3
activated in the case of a
Replication
Disaster Recovery
situation either DC1
CPM/PVWA/
automatically or manually PSM…
DR
5
DISTRIBUTED VAULTS
• The Distributed Vaults (DV)
solution spreads the load DC4
from a single primary Vault
(Master) to multiple Satellite CPM/PVWA/
Vaults PSM…
AIM CPs Backend Processes
• The Satellite Vaults are
spread throughout the DC1
Satellite
Vault
deployment to provide read Replication
requests from clients
throughout the organization AIM CPs
Replication
• If a Satellite Vault is CPM/PVWA/ Satellite
unavailable, clients that have PSM…
Primary Vault DC3
Vault
been working with this
Satellite Vault will reconnect
Replication
to another Vault, Satellite or
Master CPM/PVWA/
PSM…
• Since PAS version 11.3 up to DC2
5 satellite vaults can be
deployed Satellite Vault
(Primary Candidate)
6
DISTRIBUTED VAULTS ACTIVE-ACTIVE SERVICES
• CyberArk extended the PAS solution to support
active/active architectures with multiple
Enterprise Password Vaults
• Password retrieval and Session Management,
will be available in the event of an outage,
eliminating data loss
• Once connectivity is resumed, all audits and
session related information will be
synchronized back to the Primary Vault
• For details on implementation contact your
Account Representative
7
VAULT CLUSTER
• The Vault is installed as a high-availability
cluster of servers which provide access to the VIP
accounts in the Vault. In this implementation, Cluster Vault (Passive Cluster Vault (Active Node)
there is always one Server that is on standby Node)
IP Public Network IP + VP
in case the other Server in the cluster fails IP Private Network IP
• To all other CyberArk components, the two
PARAGNT DB V LC ENE DB V LC ENE PARAGNT
Vault Servers in the cluster can be viewed as a CVM CVM
single system, which allows high availability of
the Vault services and allows for the loss of
one Vault server without service disruption
Quorum
Shared Storage
Data + Metadata
8
CLUSTER VAULT ARCHITECTURE
9
HIGH AVAILABILITY ARCHITECTURE
• Two identical vault servers
• Dedicated SAN and PVWA CPM PSM
Cluster Shared Storage
Public Network
Vault Private Network Vault
Storage Network
Shared Storage
10
CYBERARK CLUSTER VAULT MANAGER (CVM)
• New service monitoring the CyberArk Digital
Cluster Vault resources and connections to
other CyberArk Digital Cluster Vault
Node 2 Node 1
components.
Shared Storage
• Active Node: CVM will monitor the status of
local resources: DB V LC ENE CV DB V LC ENE CV
• PrivateArk Server Monitoring Node 1 Monitoring
• Logic Container Quorum Disk
IP Private Network IP
• Database
• ENE (optional) IP Public Network IP + VIP
• PARAgent (optional)
• The active CVM will also monitor the status
of the remote passive CVM. Public Virtual IP
• Passive Node: CVM will monitor (via private
network) the status of CVM in the active node.
11
VIRTUAL IP
• The Cluster Vault must have only one IP
10.10.10.10
exposed for clients – Virtual IP.
• The Cluster Vault will allocate the VIP on the
active node during start up. The CVM will Node A Node B
monitor the VIP to ensure there are no
duplicates (v9.8).
10.10.1.1 10.10.1.2
• During failover/switchover, the CVM will switch
the VIP to the other node.
• In order to prevent possible problems, each
node should have only one single static IP.
12
SHARED STORAGE
• The metadata (database) and data (external files) will be stored
on a shared storage disk.
• Both nodes are connected to the shared storage but only the
active node is in “online status” and can read/write from/to
the disk.
13
QUORUM DISK
• In order to prevent corruption and
communication errors CyberArk employs the
Quorum mechanism.
• The Quorum uses a separate disk on the Active Is Alive? Passive
shared storage. node node
• Quorum disk will always stay offline during
normal Cluster Vault operation (except during
installation) but remain reserved for the active
node (v9.8).
Storage:
14
DETECTING A FAILURE
• Failover is triggered by failure of:
• Vault services
• Storage availability
• Virtual IP availability Check
Active Passive
• Loss of Quorum ownership node node
• The Cluster Vault service identifies a failure in
one of the resources.
• The Cluster Vault service will attempt to restart
a failed service once before going into failover Storage:
mode (v9.8).
15
FAILOVER PROCESS
• The Cluster Vault service on the Active node
changes its status to “Failover” mode and
shuts down all resources.
• The Cluster Vault service on the Passive node Active node: Check Passive node:
Changes to
will then reserve the shared resources, such as Failover then
Changes to
the VIP, Shared Storage and Quorum Disk. Active Mode
Passive Mode
• Once the Shared Storage is online, the
Passive Node has now been promoted to the
Active Node and can start the services and
provide Vault services. Storage:
• The Cluster Vault service on the former active
node will switch its role to Passive, and will
start monitoring the new active node.
Quorum
16
CLUSTER VAULT MANAGEMENT
17
CLUSTER VAULT MANAGEMENT UTILITY – ACTIVE NODE
• The new Cluster Vault is
managed and controlled by
the Cluster Vault
Management Utility.
• Before restarting a Vault
machine that is part of a
cluster, it is highly
recommended to stop the
node from the Management
Utility in order to make sure
all resources shut down
properly.
• The graphic to the right
illustrates how a CVM utility
should look on the Active
Node on the cluster.
18
CLUSTER VAULT MANAGEMENT UTILITY – STANDBY NODE
• This is the CVM utility
running on the standby
node
• Note that the local node is
always shown on the left,
regardless of whether it is
active or passive
• Shared Storage status is
reported at the bottom. In
this example, the Quorum
disk status is Released,
and the Storage drive is
Offline appropriately for
the Passive Node.
19
MONITORED SERVICES
• Monitored services can be
configured by an
Administrator.
• Using the CVM the
Administrator can select the
services to be monitored by
the Cluster Vault Manager.
• Services not monitored will
be ignored and will not
trigger a cluster failover.
20
SIMULATING FAILOVER
• To Perform a switchover
test, open the CVM on the
Active Node of the cluster:
• Click the Switchover
button shown highlighted.
• Click Continue to confirm
the message.
• The operation is complete
when the node status is
updated.
21
CYBERARK DIGITAL CLUSTER VAULT
SERVER INSTALLATION
(PREPARATION AND REQUIREMENTS)
22
PREPARE THE SERVERS
• The Vault machines must meet the recommended
system requirements
• Supported on CyberArk EPV v9.7 on MS Windows 2012
• The two Cluster Vault Nodes must be connected directly
via a private network or cross-over cable
• It is highly recommended that both nodes have identical
specifications including memory and processor
• The clocks on both cluster nodes must be synchronized
23
STORAGE PREREQUISITES
• Shared storage must support Persistent Reservation
• It is recommended to use an enterprise-grade fiber-channel
SAN solution
• iSCSi network storage is not recommended for a production
implementation.
• If iSCSI is used in a non-production environment then a
Windows update (KB2955164) should be installed in order
to ensure database stability.
• Using iSCSi also requires a FW exception to dbparm.ini
during installation.
24
PREPARE THE STORAGE
• Prepare the shared storage with two drives.
• One drive is for the Vault data, and the other drive is for
the Quorum Disk.
• Drive letters for the Quorum and Storage disks must be
identical on both nodes.
• During EPV Cluster Vault installation, ensure that the
shared storage resources are online for ONLY the
node currently being installed. After the EPV Cluster
Vault is successfully installed, the CVM will manage
the Shared Storage.
25
CLUSTER INSTALLATION
(INSTALL THE FIRST NODE)
26
INSTALL THE FIRST NODE – VAULT INSTALLATION MODE
Launch the setup.exe and
choose Cluster-Node Vault
installation
27
INSTALL THE FIRST NODE – SAFES LOCATION
Choose the location on the
shared storage to store the
safes
28
INSTALL THE FIRST NODE – OPERATOR CD PATH
• Copy the encryption keys
from the operator CD to a
folder on the local drive
• Select the folder on the
local drive as the
Operator CD path
• Complete the installation,
but do not reboot
immediately
29
INSTALL THE FIRST NODE – CONFIGURE STORAGE
• In an Administrators Command
Window, navigate to the
PrivateArk\Server\ClusterVault
directory.
• Use the following command line
to set the Quorum and Shared
Storage drive letters:
StorageManager.exe –qE -sF
• -q sets drive letter for quorum
• -s sets drive letter for shared
storage
30
INSTALL THE FIRST NODE – CONFIGURE CLUSTERVAULT.INI
Set the names and IP
addresses for the local and
peer node in ClusterVault.ini
• Logical Names
• Virtual IP
• Peer and Local Public and
Private IP addresses
• located in C:\Program Files
(x86)\PrivateArk\Server\Cluster
Vault\
• The information defined in the
ClusterVault.ini file, is displayed
by the Cluster Vault
Management utility or CVM.
31
INSTALL THE FIRST NODE – REBOOT
• Restart the first node and
verify that all resources
have been started
successfully. The following
message will appear in the
ClusterVaultConsole.log:
CVMCS087I All the
resources are running
successfully
• Launch Cluster Vault
Management, check that
node is showing as
“Active”, shared storage
as “Online” and Quorum
as “Reserved”.
32
PREPARING FOR VAULT INSTALLATION
ON SECOND NODE
33
COPY ENCRYPTION KEYS TO SECOND NODE
• Use the same set of Operator
Keys that you used to install
the first node of the Cluster
Vault.
• Copy the additional keys
listed here, that were
generated during the
installation of the first node to
the same location in the
second node. These keys will
be created in the folder
containing the original
Operator Keys.
• Backup.key
• VaultUser.pass
• ReplicationUser.pass
• VaultEmergency.pass
34
STOP SERVICES ON FIRST NODE
• Before starting the
installation of the second
node of the Cluster Vault,
we need to stop all
services on the first node.
• Log on to the first node,
and launch Cluster Vault
Management. Select the
stop symbol that is
highlighted in the graphic.
35
SET SHARED DISKS TO OFFLINE ON FIRST NODE
• Use the Disk Management
utility to verify the shared
disks are offline on the first
node.
• Make sure that there are no
open files or folders on the
shared storage.
36
BRING SHARED DISKS ONLINE ON SECOND NODE
• Use the Disk Management
utility to bring the Shared
Disks online on the
second Node.
• Ensure that the drive
letters for the Quorum and
Storage disks are
identical in both nodes
37
CLUSTER INSTALLATION
(INSTALL THE SECOND NODE)
38
INSTALL THE SECOND NODE – SAFES LOCATION
Install The Vault on the
Second Node.
Make sure you select:
• “Cluster-Node Vault
installation” as the
installation mode
• the same drive letter and
folder on the shared storage
for the Safes location.
39
INSTALL THE SECOND NODE – VAULTID
The Vault-id parameter must
be consistent for both cluster
nodes.
• Open DBParm.ini on the
first node
• Copy the Vault-id
parameter from the first
node to DBParm.ini on the
second node
40
INSTALL THE SECOND NODE – SERVER-ID
The server-id parameter
must be consistent for both
cluster nodes.
• Open my.ini on the first
node in the Database
subdirectory
• Copy the Server-id from
the first node to the
second node
41
INSTALL THE SECOND NODE – CONFIGURE STORAGE
The disk identifiers must be
recorded in the ClusterVault.ini
file in the StorageIdentifier and
QuorumDiskIdentifier
parameters.
• Use the following command to
set the Quorum and Shared
Storage drive letters:
StorageManager.exe –qE –sF
• Use the same drive letters as
on the first node
42
INSTALL THE SECOND NODE – CONFIGURE CLUSTERVAULT.INI
• Set the names and IP
addresses for the local
and peer nodes in
ClusterVault.ini
• Logical Names
• Virtual IP
• Peer and Local Public
and Private IP addresses
43
INSTALL THE SECOND NODE – REBOOT
• Restart the second node
and verify that all resources
have been started
successfully. The following
message should appear in
the
ClusterVaultConsole.log:
CVMCS087I All the
resources are running
successfully
• After the Second node has
started successfully and is
active, start the first node in
Passive mode and then
trigger a switchover to test
the cluster failover process.
44
CLUSTER VAULT LOGS
45
LOGS
• ClusterVaultConsole.log
• Cluster Vault log file
• ClusterVaultTrace.log
• Cluster Vault trace file
• Setting the debug level for
the Cluster Vault can be
set dynamically with no
restart needed.
46
QUIZ
1. What are the 3 main types of Vault Availability?
• Cold = Replicate backup
• Warm = Disaster Recovery
• Hot = High Availability and Distributed Vaults
2. What is the Quorum Disk used for in a HA Cluster Vault architecture?
• The Quorum Mechanism is based on a voting algorithm. When starting up, each node in the cluster has a
vote when accessing the Shared Resources, and the Quorum disk provides the tie breaking vote.
3. What does the CVM or Cluster Vault Management Utility provide?
• The CVM monitors shared services, disk storage, the Virtual IP and the status of the other node of the
cluster.
• An Administrator can initiate a failover from the Active to the Passive Node from the CVM.
• Monitoring of services by the CVM can be selectively turned off, if necessary.
4. After installing NodeA, what information needs to be copied manually to NodeB?
• Copy the additional keys, i.e., Backup.key, VaultUser.pass, ReplicationUser.pass, VaultEmergency.pass.
• Copy the Server-id from my.ini.
• Copy the Vault-id parameter.
47
THANK YOU
48
