Patroni Setup

This document offers a comprehensive guide on configuring Patroni for a three-node PostgreSQL setup. The following steps need to be
executed.

Setup ETCD
1. Configure PostgreSQL on all three machines without initializing the database.
2. Setup ETCD on all nodes and follow the steps outlined below.
wget https://github.com/etcd-io/etcd/releases/download/v3.5.0/etcd-v3.5.0-linux-amd64.tar.gz

Create an etcd user and group specifically for running etcd binaries.

1 tar –xvf etcd-v3.5.0-linux-amd64.tar.gz

2 cd etcd-v3.5.0-linux-amd64/
3 cp etcd etcdctl etcdutl /usr/bin

Create two directories named 'data' and 'configuration'.

1 sudo mkdir -p /var/lib/etcd/

2 sudo mkdir /etc/etcd
3 sudo chown -R etcd:etcd /var/lib/etcd/ /etc/etcd

Create an etcd user and group specifically for running etcd binaries.

1 groupadd --system etcd

2 useradd -s /bin/bash --system -g etcd etcd

Start ETCD on all nodes.

1 /usr/bin/etcd --name etcd1 --data-dir /var/lib/etcd --listen-peer-urls http://0.0.0.0:2380 --listen-client-urls

http://172.20.0.2:2379,http://127.0.0.1:2379 --initial-advertise-peer-urls http://172.20.0.2:2380 --advertise-
client-urls http://172.20.0.2:2379 --initial-cluster
etcd1=http://172.20.0.2:2380,etcd2=http://172.20.0.3:2380,etcd3=http://172.20.0.4:2380 --initial-cluster-state
'new

1 /usr/bin/etcd --name etcd2 --data-dir /var/lib/etcd --listen-peer-urls http://0.0.0.0:2380 --listen-client-urls

http://172.20.0.3:2379,http://127.0.0.1:2379 --initial-advertise-peer-urls http://172.20.0.3:2380 --advertise-
client-urls http://172.20.0.3:2379 --initial-cluster
etcd1=http://172.20.0.2:2380,etcd2=http://172.20.0.3:2380,etcd3=http://172.20.0.4:2380 --initial-cluster-state
'new'

1 /usr/bin/etcd --name etcd3 --data-dir /var/lib/etcd --listen-peer-urls http://0.0.0.0:2380 --listen-client-urls

http://172.20.0.4:2379,http://127.0.0.1:2379 --initial-advertise-peer-urls http://172.20.0.4:2380 --advertise-
client-urls http://172.20.0.4:2379 --initial-cluster
etcd1=http://172.20.0.2:2380,etcd2=http://172.20.0.3:2380,etcd3=http://172.20.0.4:2380 --initial-cluster-state
'new'

List the members of ETCD.

1 etcdctl member list --write-out=table

2 +------------------+---------+-------+------------------------+------------------------+------------+
3 | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER |
4 +------------------+---------+-------+------------------------+------------------------+------------+
5 | b217c5c8b9eaba91 | started | etcd1 | http://172.20.0.2:2380 | http://172.20.0.2:2379 | false |
 6 | e6996497aa96d1fc | started | etcd3 | http://172.20.0.4:2380 | http://172.20.0.4:2379 | false |
7 | f01c391eb2944291 | started | etcd2 | http://172.20.0.3:2380 | http://172.20.0.3:2379 | false |
8 +------------------+---------+-------+------------------------+------------------------+------------+

Check the status of the endpoints to ensure they are functioning properly.

1 etcdctl endpoint status --write-out=table --

endpoints=http://172.20.0.2:2379,http://172.20.0.3:2379,http://172.20.0.4:2379
2 +------------------------+------------------+---------+---------+-----------+------------+-----------+---------
---+--------------------+--------+
3 | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT
INDEX | RAFT APPLIED INDEX | ERRORS |
4 +------------------------+------------------+---------+---------+-----------+------------+-----------+---------
---+--------------------+--------+
5 | http://172.20.0.2:2379 | b217c5c8b9eaba91 | 3.5.13 | 172 kB | true | false | 2 |
551 | 551 | |
6 | http://172.20.0.3:2379 | f01c391eb2944291 | 3.5.13 | 172 kB | false | false | 2 |
551 | 551 | |
7 | http://172.20.0.4:2379 | e6996497aa96d1fc | 3.5.13 | 172 kB | false | false | 2 |
551 | 551 | |
8 +------------------------+------------------+---------+---------+-----------+------------+-----------+---------
---+--------------------+--------+

Set up Patroni
Set up Patroni on all three nodes and proceed with the following steps.

To install Patroni, you will need to install the following packages:

1 dnf install epel-release

2 dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-9-x86_64/pgdg-redhat-repo-
latest.noarch.rpm
3 pip3 install click
4 dnf install patroni
5 dnf install python-etcd
6

Enable the Patroni service.

1 systemctl enable patroni

Create a configuration file and the necessary directories for Patroni.

1 mkdir -p /etc/patroni/logs/ #directory to store logs

2 chmod 777 /etc/patroni/logs
3 touch /etc/patroni/patroni.yml
4

Create a configuration file for Patroni as outlined below (/etc/patroni/patroni.yml).

Node1 patroni.yml

1 scope: bootvar
2 name: pgdb1
3
4 log:
5 traceback_level: INFO
6 level: INFO
7 dir: /etc/patroni/logs/
 8 file_num: 5
9
10 restapi:
11 listen: 0.0.0.0:8008
12 connect_address: 172.20.0.2:8008
13
14 etcd3:
15 protocol: http
16 hosts: 172.20.0.2:2379,172.20.0.3:2379,172.20.0.4:2379
17
18 bootstrap:
19 dcs:
20 ttl: 30
21 loop_wait: 10
22 retry_timeout : 10
23 maximum_lag_on_failover: 1048576
24 postgresql:
25 use_pg_rewind: true
26 use_slots: true
27 parameters:
28 wal_keep_segments: 100
29 #add other postgres DB parameters to start with
30
31 initdb:
32 - encoding: UTF8
33 - data-checksums
34
35 pg_hba:
36 - host replication replicator 0.0.0.0/0 md5
37 - host all all 0.0.0.0/0 md5
38
39 postgresql:
40 listen: 172.20.0.2:5432
41 connect_address: 172.20.0.2:5432
42 data_dir: /var/lib/pgsql/bootvar/pgdb1
43 bin_dir: /usr/pgsql-16/bin
44 authentication:
45 replication:
46 username: replicator
47 password: test
48 superuser:
49 username: postgres
50 password: test

Node2 patroni.yml

1 scope: bootvar
2 name: pgdb2
3
4 log:
5 traceback_level: INFO
6 level: INFO
7 dir: /etc/patroni/logs/
8 file_num: 5
9
10 restapi:
11 listen: 0.0.0.0:8008
12 connect_address: 172.20.0.3:8008
13
14 etcd3:
15 protocol: http
16 hosts: 172.20.0.2:2379,172.20.0.3:2379,172.20.0.4:2379
17
18 bootstrap:
19 dcs:
20 ttl: 30
21 loop_wait: 10
22 retry_timeout : 10
23 maximum_lag_on_failover: 1048576
24 postgresql:
25 use_pg_rewind: true
26 use_slots: true
27 parameters:
28 wal_keep_segments: 100
29 #add other postgres DB parameters to start with
30
31 initdb:
32 - encoding: UTF8
33 - data-checksums
34
35 pg_hba:
36 - host replication replicator 0.0.0.0/0 md5
37 - host all all 0.0.0.0/0 md5
38
39 postgresql:
40 listen: 172.20.0.3:5432
41 connect_address: 172.20.0.3:5432
42 data_dir: /var/lib/pgsql/bootvar/pgdb2
43 bin_dir: /usr/pgsql-16/bin
44 authentication:
45 replication:
46 username: replicator
47 password: test
48 superuser:
49 username: postgres
50 password: test
51

Node3 patroni.yml

1 scope: bootvar
2 name: pgdb3
3
4 log:
5 traceback_level: INFO
6 level: INFO
7 dir: /etc/patroni/logs/
8 file_num: 5
9
10 restapi:
11 listen: 0.0.0.0:8008
12 connect_address: 172.20.0.4:8008
13
14 etcd3:
15 protocol: http
16 hosts: 172.20.0.2:2379,172.20.0.3:2379,172.20.0.4:2379
17
18 bootstrap:
19 dcs:
20 ttl: 30
21 loop_wait: 10
22 retry_timeout : 10
23 maximum_lag_on_failover: 1048576
24 postgresql:
25 use_pg_rewind: true
26 use_slots: true
27 parameters:
28 wal_keep_segments: 100
29 #add other postgres DB parameters to start with
30
31 initdb:
32 - encoding: UTF8
33 - data-checksums
34
35 pg_hba:
36 - host replication replicator 0.0.0.0/0 md5
37 - host all all 0.0.0.0/0 md5
38
39 postgresql:
40 listen: 172.20.0.4:5432
41 connect_address: 172.20.0.4:5432
42 data_dir: /var/lib/pgsql/bootvar/pgdb3
43 bin_dir: /usr/pgsql-16/bin
44 authentication:
45 replication:
46 username: replicator
47 password: test
48 username: postgres
49 password: test

List the members of the Patroni list.

1 patronictl -c /etc/patroni/patroni.yml list

2 + Cluster: bootvar (7394732307590897180) ---+----+-----------+
3 | Member | Host | Role | State | TL | Lag in MB |
4 +--------+------------+---------+-----------+----+-----------+
5 | pgdb1 | 172.20.0.2 | Leader | running | 1 | |
6 | pgdb2 | 172.20.0.3 | Replica | streaming | 1 | 0 |
7 | pgdb3 | 172.20.0.4 | Replica | streaming | 1 | 0 |
8 +--------+------------+---------+-----------+----+-----------+
9

Create a table in the primary database and ensure its replication in the standby database is successfully established.

Node1:

1 postgres=# create table test2(a int);

2 CREATE TABLE
3 postgres=# insert into test2 values(1);
4 INSERT 0 1
5

Node2:

1 postgres=# select * from test2 ;

2 a
3 ---
4 1
5 (1 row)
6

Node3:

1 postgres=# select * from test2;

2 a
3 ---
4 1
5 (1 row)
6
7 postgres=# insert into test2 values(3);
8 ERROR: cannot execute INSERT in a read-only transaction
9

Now execute the switchover.

1 patronictl -c /etc/patroni/patroni.yml switchover

2 Current cluster topology
3 + Cluster: bootvar (7394732307590897180) ---+----+-----------+
4 | Member | Host | Role | State | TL | Lag in MB |
5 +--------+------------+---------+-----------+----+-----------+
6 | pgdb1 | 172.20.0.2 | Leader | running | 1 | |
7 | pgdb2 | 172.20.0.3 | Replica | streaming | 1 | 0 |
8 | pgdb3 | 172.20.0.4 | Replica | streaming | 1 | 0 |
9 +--------+------------+---------+-----------+----+-----------+
10 Primary [pgdb1]:
11 Candidate ['pgdb2', 'pgdb3'] []: pgdb2
12 When should the switchover take place (e.g. 2024-07-23T09:42 ) [now]:
13 Are you sure you want to switchover cluster bootvar, demoting current leader pgdb1? [y/N]: y
14 2024-07-23 08:42:35.09038 Successfully switched over to "pgdb2"
15

After the switchover, please provide a list of the members of Patroni.

1 patronictl -c /etc/patroni/patroni.yml list

2 + Cluster: bootvar (7394732307590897180) ---+----+-----------+
3 | Member | Host | Role | State | TL | Lag in MB |
4 +--------+------------+---------+-----------+----+-----------+
5 | pgdb1 | 172.20.0.2 | Replica | streaming | 2 | 0 |
6 | pgdb2 | 172.20.0.3 | Leader | running | 2 | |
7 | pgdb3 | 172.20.0.4 | Replica | streaming | 2 | 0 |
8 +--------+------------+---------+-----------+----+-----------+

After the switchover, a new row should be inserted on node2 for the newly promoted leader.

Node2:

1 postgres=# insert into test2 values(3);

2 INSERT 0 1

Node1:

1 postgres=# insert into test2 values(1);

2 ERROR: cannot execute INSERT in a read-only transaction
3 postgres=# select * from test2 ;
4 a
5 ---
6 1
7 3
8 (2 rows)

Node3:

1 postgres=# insert into test2 values(3);

2 ERROR: cannot execute INSERT in a read-only transaction
3 postgres=# select * from test2;
4 a
5 ---
6 1
7 3
8 (2 rows)