Edb Ha Scalability
Edb Ha Scalability
1 Architecture Overview 1
1.1 Failover Manager Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 PgPool-II Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.1 PCP Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2.2 Pgpool Watchdog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Architecture 4
4 Optional Components 13
4.1 Virtual IP Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4.2 Pgpool Watchdog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
5 Appendix 15
5.1 Appendix A: Supported Failover Scenarios . . . . . . . . . . . . . . . . . . . . . 15
5.2 Appendix B: Integration Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
5.2.1 load_balancer_detach.sh . . . . . . . . . . . . . . . . . . . . . . . . . . 17
5.2.2 load_balancer_attach.sh . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
5.2.3 follow_master.sh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
5.2.4 pgpool_backend.sh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
6 Conclusion 25
Index 27
i
CHAPTER 1
Architecture Overview
Since high-availability and read scalability are not part of the core feature set of EDB Postgres
Advanced Server, Advanced Server relies on external tools to provide this functionality. This doc-
ument will focus on functionality provided by EDB Failover Manager and Pgpool-II and discuss
the implications of a high-availability architecture formed around these tools. We will demon-
strate how to best configure Failover Manager and Pgpool to leverage the benefits they provide
for Advanced Server. Using the reference architecture described in the Architecture section, you
can learn how to achieve high availability by implementing an automatic failover mechanism (with
Failover Manager) while scaling the system for larger workloads and a high number of concurrent
clients with read-intensive or mixed workloads to achieve horizontal scaling/read-scalability (with
Pgpool).
The architecture described in this document has been developed and tested for EFM 4.0, EDB
pgPool, and Advanced Server 12.
Documentation for Advanced Server and Failover Manager are available from EnterpriseDB at:
https://www.enterprisedb.com/resources/product-documentation
Documentation for pgPool-II can be found at:
http://www.pgpool.net/docs/latest/en/html
1
EDB Postgres High Availability and Horizontal Read Scaling Architecture, Release
4.0
Failover Manager is a high-availability module that monitors the health of a Postgres streaming
replication cluster and verifies failures quickly. When a database failure occurs, Failover Manager
can automatically promote a streaming replication standby node into a writable primary node to
ensure continued performance and protect against data loss with minimal service interruption.
Basic EFM Architecture Terminology
A Failover Manager cluster is comprised of EFM processes that reside on the following
hosts on a network:
• A Primary node is the primary database server that is servicing database clients.
• One or more Standby nodes are streaming replication servers associated with the primary
node.
• The Witness node confirms assertions of either the Primary or a Standby in a failover sce-
nario. A cluster does not need a dedicated witness node if the cluster contains three or more
nodes. If you do not have a third cluster member that is a database host, you can a dedicated
Witness node; a cluster may include more than one witness node.
Pgpool-II (Pgpool) is an open source application that provides connection pooling and load bal-
ancing for horizontal scalability of SELECT queries on multiple standbys in EPAS and community
Postgres clusters. Pgpool can be configured to use a backend_weight parameter to prevent read
traffic to be directed to the primary node. In such cases, data modification language (DML) queries
(i.e., INSERT, UPDATE, and DELETE) are always sent to the primary node, while read queries
are load-balanced to the standbys, providing scalability with mixed and read-intensive workloads.
EnterpriseDB supports the following Pgpool functionality:
• Load balancing
• Connection pooling
• High availability
• Connection limits
Pgpool provides an interface called PCP for administrators that performs management operations
such as retrieving the status of Pgpool or terminating Pgpool processes remotely. PCP commands
are UNIX commands that manipulate Pgpool via the network.
watchdog is an optional sub process of Pgpool that provides a high availability feature. Features
added by watchdog include:
• Health checking of the pgpool service
• Mutual monitoring of other watchdog processes
• Changing active/standby state if certain faults are detected
• Automatic virtual IP address assigning synchronous to server switching
• Automatic registration of a server as a standby during recovery
More information about the Pgpool watchdog component can be found at:
http://www.pgpool.net/docs/latest/en/html/tutorial-watchdog.html
CHAPTER 2
Architecture
The sample architecture diagram shows four nodes as described in the table below:
Scenario Components
Server 1 Primary node, running Advanced Server and Failover Manager with Post-
gres Streaming Replication. Applications/clients will connect via the Pgpool
port on Server 4 (or via a Virtual IP, if used) to perform write operations (i.e.,
INSERT, UPDATE, DELETE).
Server 2 & Standby nodes running Failover Manager (Pgpool-II optional). This is a
Server 3 Streaming Replication standby node. Applications/clients will connect to
this database via the Pgpool port on Server 4 (or via a Virtual IP, if used)
to perform read operations (i.e., SELECT). An optional standby Pgpool in-
stance can be set up here with watchdog running, if desired.
Server 4 Optional witness node running Pgpool-II and Failover Manager. This server
is set up with no active database, but with Failover Manager and Pgpool. All
applications/clients will connect to other databases via this server, either via
this server’s IP address, or via a Virtual IP that points to this server. Note
that a witness node is not required if at least three other EFM nodes exist.
The witness node in this sample architecture is provided for demonstration
purposes.
This architecture:
• Achieves maximum availability by providing two standbys in case of primary node failure.
• Achieves maximum performance with mixed and read-intensive workloads by introducing
increased read scalability with more than one standby for load balancing.
• Reduces load on the primary node by performing load balancing and not running Pgpool on
the primary.
• Avoids single point of failure of Pgpool by configuring Pgpool in high-availability mode
using watchdog.
• Runs Pgpool primary/active instance on the least-burdened node (the witness node) to boost
performance while sharing resources with Failover Manager (to reduce TCO).
If one or more standbys are configured with synchronous replication, users can achieve near-zero
data loss in a failure event.
With this architecture, you can expect the following behavior:
6
EDB Postgres High Availability and Horizontal Read Scaling Architecture, Release
4.0
7
CHAPTER 3
Failover Manager monitors the health of Postgres nodes; in the event of a primary node failure,
Failover Manager performs an automatic failover to a standby node. Note that Pgpool does not
monitor the health of backend nodes and will not perform failover to any standby nodes.
Beginning with version 3.2, a Failover Manager agent can be configured to use Pgpool’s PCP
interface to detach the failed node from Pgpool load balancing after performing failover of the
standby node. More details about the necessary configuration file changes and relevant scripts will
be discussed in the sections that follow.
8
EDB Postgres High Availability and Horizontal Read Scaling Architecture, Release
4.0
Failover Manager provides functionality that will remove failed database nodes from Pgpool load
balancing; Failover Manager can also re-attach nodes to Pgpool when returned to the Failover
Manager cluster. To configure this behavior, you must identify the load balancer attach and detach
scripts in the efm.properties file in the following parameters:
• script.load.balancer.attach=/path/to/load_balancer_attach.sh
%h
• script.load.balancer.detach=/path/to/load_balancer_detach.sh
%h
The script referenced by load.balancer.detach is called when Failover Manager decides
that a database node has failed. The script detaches the node from Pgpool by issuing a PCP
interface call. You can verify a successful execution of the load.balancer.detach script by
calling SHOW NODES in a psql session attached to the Pgpool port. The call to SHOW NODES
should return that the node is marked as down; Pgpool will not send any queries to a downed node.
The script referenced by load.balancer.attach is called when a resume command is
issued to the efm command-line interface to add a downed node back to the Failover Manager
cluster. Once the node rejoins the cluster, the script referenced by load.balancer.attach
is invoked, issuing a PCP interface call, which adds the node back to the Pgpool cluster. You can
verify a successful execution of the load.balancer.attach script by calling SHOW NODES
in a psql session attached to the Pgpool port; the command should return that the node is marked
as up. At this point, Pgpool will resume using this node as a load balancing candidate. Sample
scripts for each of these parameters are provided in Appendix B.
You must provide values for the following configuration parameters in the pgpool.conf file on
the Pgpool host:
When the primary/master node is changed in Pgpool (either by failover or by manual promotion)
in a non-Failover Manager setup, Pgpool detaches all standby nodes from itself, and executes the
follow_master_command for each standby node, making them follow the new primary node.
Since Failover Manager reconfigures the standby nodes before executing the post-promotion script
(where a standby is promoted to primary in Pgpool to match the Failover Manager configuration),
the follow_master_command merely needs to reattach standby nodes to Pgpool.
Note that the load-balancing is turned on to ensure read scalability by distributing read traffic
across the standby nodes
Note also that the health checking and error-triggered backend failover have been turned off, as
Failover Manager will be responsible for performing health checks and triggering failover. It is
not advisable for Pgpool to perform health checking in this case, so as not to create a conflict with
Failover Manager, or prematurely perform failover.
Finally, search_primary_node_timeout has been set to a low value to ensure prompt re-
covery of Pgpool services upon an Failover Manager-triggered failover.
3.3 pgpool_backend.sh
In order for the attach and detach scripts to be successfully called, a pgpool_backend.sh
script must be provided. pgpool_backend.sh is a helper script for issuing the actual PCP
interface commands on Pgpool. Nodes in Failover Manager are identified by IP addresses, while
PCP commands refer to a node ID. pgpool_backend.sh provides a layer of abstraction to
perform the IP address to node ID mapping transparently.
3.3. pgpool_backend.sh 12
CHAPTER 4
Optional Components
Both Pgpool-II and Failover Manager provide functionality to employ a virtual IP for seamless
failover. While both provide this capability, it must be noted that Failover Manager associates a
virtual IP to the primary database node while Pgpool associates a virtual IP to the currently-active
Pgpool host (in a multi-Pgpool watchdog setup).
Note that if an active instance of Pgpool (Server 4 in our sample architecture) goes down, any
available standby Pgpool instance (according to watchdog priority) will take charge as the active
Pgpool instance.
Watchdog provides high availability for Pgpool nodes. This section lists the configuration param-
eters required to configure watchdog on each Pgpool node.
Common Watchdog Configuration Parameters for All Pgpool Nodes
13
EDB Postgres High Availability and Horizontal Read Scaling Architecture, Release
4.0
Note that wd_priority can be used to elevate the local watchdog node priority in the elections
to select primary watchdog node. The node with the higher wd_priority value will get selected
as primary watchdog node when cluster will be electing its new primary node at the time of cluster
startup or in the event of old primary watchdog node failure.
Appendix
A summary of supported failover scenarios is provided below. Please note that the list is not
comprehensive; you should consult the Failover Manager documentation for detailed information
about how Failover Manager handles each failover/failure scenario.
15
EDB Postgres High Availability and Horizontal Read Scaling Architecture, Release
4.0
5.2.1 load_balancer_detach.sh
#!/bin/bash
#%h host name
output_file=/tmp/efm-scripts/pp_log
pool_backend=/tmp/efm-scripts/pgpool/pgpool_backend.sh
node_address=$1
current_date_time="`date +"%Y-%m-%d %H:%M:%S"`";
echo $current_date_time >>$output_file
echo "node address to detach = $node_address". >>$output_file
$pool_backend detach $node_address >>$output_file
echo "-------------------".>>$output_file
exit 0
5.2.2 load_balancer_attach.sh
#!/bin/bash
#%h host name
output_file=/tmp/efm-scripts/pp_log
pool_backend=/tmp/efm-scripts/pgpool/pgpool_backend.sh
node_address=$1
current_date_time="`date +"%Y-%m-%d %H:%M:%S"`";
echo $current_date_time >>$output_file
echo "node address to attach = $node_address". >>$output_file
$pool_backend attach $1 >>$output_file
echo "-------------------".>>$output_file
exit 0
5.2.3 follow_master.sh
#! /bin/sh
PCP_USER= # PCP user name
PCP_PORT= # PCP port number as in pgpool.conf
PCP_HOST= # hostname of Pgpool-II
PGPOOL_PATH= # Pgpool-II installed path
export PCPPASSFILE= # Path to PCPPASS file
# Execute command by failover.
# special values: %d = node id
# %h = host name
# %p = port number
# %D = database cluster path
# %m = new master node id
# %M = old master node id
# %H = new master node host name
# %P = old primary node id
# %R = new master database cluster path
# %r = new master port number
# %% = '%' character
detached_node_id=$1
old_master_id=$2
echo detached_node_id $1
echo old_master_id $2
## If $detached_node_id is equal to $old_master_id,
## then do nothing, since the old master is no longer
## supposed to be part of the cluster.
if [ $detached_node_id -ne $old_master_id ]; then
sleep 10
$PGPOOL_PATH/pcp_attach_node -w -U $PCP_USER -h $PCP_HOST -p
$PCP_PORT $detached_node_id
fi
5.2.4 pgpool_backend.sh
#!/bin/bash
#
# pgpool-II backend node configuration driver.
#
# usage: promote_standby.sh hostname [port]
#
# set the following variables according to your setup
PCP_USER= # PCP user name
PCP_PORT= # PCP port number as in pgpool.conf
PCP_HOST= # hostname of Pgpool-II
PGPOOL_PATH= # Pgpool-II installed path
export PCPPASSFILE= # Path to PCPPASS file
# function returns the Pgpool-II backend node-id for the given
hostname
# and port number, And if the node-id is not found 255 is returned
# Arguments:
# 1- Hostname
# 2- Port (optional) if not provided, node-id of first matching
# hostname will be returned
#
function get_pgpool_nodeid_from_host {
if [ -z "$1" ]; then
echo "hostname not provided"
return 255
fi
#Now get the total number of nodes configured in Pgpool-II
node_count=`$PGPOOL_PATH/pcp_node_count -U $PCP_USER -h $PCP_HOST -p
$PCP_PORT -w\`
echo searching node-id for $1:$2 from $node_count configured backends
i=0
while [ $i -lt $node_count ];
do
nodeinfo=`$PGPOOL_PATH/pcp_node_info -U $PCP_USER -h $PCP_HOST -p
$PCP_PORT -w $i\`
hostname=`echo $nodeinfo \| awk -v N=1 '{print $N}'\`
port=`echo $nodeinfo \| awk -v N=2 '{print $N}'\`
#if port number is <= 0 we just need to compare hostname
if [ "$hostname" == $1 ] && ( [ -z "$2" ] \|\| [ $port -eq $2 ] );
then
echo "$1:$2 has backend node-id = $i in Pgpool-II"
return $i
fi
let i=i+1
done
(continues on next page)
Conclusion
EDB Postgres High Availability and Horizontal Read Scaling Architecture Guide
Copyright © 2018 - 2020 EnterpriseDB Corporation. All rights reserved.
EnterpriseDB® Corporation 34 Crosby Drive, Suite 201, Bedford, MA 01730, USA
T +1 781 357 3390
F +1 978 467 1307 E
[email protected]
www.enterprisedb.com
• EnterpriseDB and Postgres Enterprise Manager are registered trademarks of EnterpriseDB
Corporation. EDB and EDB Postgres are trademarks of EnterpriseDB Corporation. Ora-
cle is a registered trademark of Oracle, Inc. Other trademarks may be trademarks of their
respective owners.
• EDB designs, establishes coding best practices, reviews, and verifies input validation for the
logon UI for EDB products where present. EDB follows the same approach for additional
input components, however the nature of the product may require that it accepts freeform
SQL, WMI or other strings to be entered and submitted by trusted users for which limited
validation is possible. In such cases it is not possible to prevent users from entering incorrect
or otherwise dangerous inputs.
• EDB reserves the right to add features to products that accept freeform SQL, WMI or other
potentially dangerous inputs from authenticated, trusted users in the future, but will ensure
all such features are designed and tested to ensure they provide the minimum possible risk,
and where possible, require superuser or equivalent privileges.
25
EDB Postgres High Availability and Horizontal Read Scaling Architecture, Release
4.0
• EDB does not that warrant that we can or will anticipate all potential threats and therefore
our process cannot fully guarantee that all potential vulnerabilities have been addressed or
considered.
26
Index
A S
Appendix, 15 supported failover scenarios, 15
Architecture, 5
Architecture Overview, 1
C
Conclusion, 25
E
EFM overview, 2
F
Failover Manager configuration
file, 9
follow_master.sh, 19
I
Implementing High Availability
with PgPool, 8
integration scripts, 17
L
load_balancer_attach.sh, 18
load_balancer_detach.sh, 17
O
Optional Components, 13
P
Pgpool configuration file, 10
Pgpool overview, 3
Pgpool watchdog, 13
pgpool_backend.sh, 12, 20
27