How To setup Pgbouncer

PostgreSQL handles connections in a way that can be resource-intensive. Every time a new connection is made, PostgreSQL creates a
separate process, which can be costly and inefficient, especially with many connections.

When a client requests something from PostgreSQL, it opens a connection to the database, and this connection stays open until the client
gets a response. If connections are frequently created and then quickly discarded, it can waste server memory. This is where PgBouncer
comes in—it helps manage and pool these connections more efficiently.

PgBouncer Description
PgBouncer is a lightweight, open-source tool that helps manage PostgreSQL connections. It works by pooling connections for each user-
database pair on the server, using both TCP and Unix domain sockets.

When a client wants to connect to PostgreSQL, it first goes through PgBouncer. PgBouncer then provides a connection from its pool and
returns it after the client disconnects. The key benefit is that PgBouncer reuses these connections, instead of creating new ones each time.
This reduces the load on PostgreSQL and significantly improves query performance.

Configuration Parameters
pool_mode: In PgBouncer, the pool_mode setting determines how client connections are pooled and assigned to server connections.
There are three modes:

1. session: Each client connection gets its own dedicated server connection for the entire duration of the session. Ideal for applications
that require session-level state, such as session variables or prepared statements. Higher resource consumption since each client
connection requires a separate server connection.
2. transaction : Workers will be assigned according to transactions. Which means it maybe considerable in multi-transactional
environments. This configuration is more aggressive than session mode. This mode is a good balance between safety and resource
efficiency. It's suitable for most applications that don't require session state across multiple transactions. More efficient than session
mode, as server connections are reused more frequently.
3. statement :A server connection is assigned to a client connection for the duration of a single SQL statement. After the statement is
executed, the server connection is immediately returned to the pool.This is the most efficient mode, as it maximizes server connection
reuse. It's suitable for applications that execute independent SQL statements without any need for transaction or session state.

default_pool_size: This setting in PgBouncer controls the maximum number of server connections that can be allocated from the pool for
each database or user. This setting effectively limits the number of active server connections that can be maintained by PgBouncer for a
particular database or user. If a database or user reaches this limit, additional client connections will have to wait until a server connection
becomes available. Setting default_pool_size too high can exhaust server resources like CPU and memory.

reserve_pool_size: This is a backup option for important business operations. If a client tries to connect but the connection pool is full, the
client will wait for a certain number of seconds set by reserve_pool_timeout . After that, it can use extra connections provided by this
configuration.

reserve_pool_timeout :If a client doesn't get a connection within this many seconds, it will use extra connections from the reserve pool.
Setting it to 0 turns this feature off.

server_lifetime :The pooler will close any unused server connection that has been open longer than this time. If set to 0, the connection will
be used just once and then closed.

server_idle_timeout : This setting in PgBouncer controls how long a server connection can remain idle before it's closed.

server_connect_timeout : This setting defines the maximum time PgBouncer will wait for a connection to the PostgreSQL server to be
established. If PgBouncer cannot connect to the server within the specified time, the connection attempt will fail, and an error will be
returned to the client.

server_login_retry :If a login fails due to a connection or authentication issue, the pooler will wait this long before trying to connect again.

query_timeout: This setting in PgBouncer controls how long a query can run before it gets automatically canceled. It helps prevent long-
running queries from using up too many resources or causing delays.

query_wait_timeout : This setting in PgBouncer controls how long a client can wait in the queue for a server connection before the waiting
query is canceled. It also helps when the server is down or database rejects connections for any reason. If this is disabled, clients will be
queued indefinitely.

client_idle_timeout: This setting in PgBouncer controls how long an idle client connection can remain open before it's closed. And only
used for network problems.

client_login_timeout: This setting in PgBouncer controls how long a client has to successfully log in before the connection attempt is
canceled. If client_login_timeout is set to 60 seconds, a client must complete the login process within 60 seconds, or PgBouncer will
close the connection.

PGbouncer Installation

1. Pgbouncer packages installation

1 sudo yum install pgbouncer

2. Folder permissions for the PostgreSQL user

1 chown postgres:postgres /etc/pgbouncer/ -R

3. Create userlist.txt file

Generate an MD5 password for the PostgreSQL user and add it to the userlist.txt file.

1 postgres=# SELECT 'md5' || md5('test' || 'postgres');

2 ?column?
3 -------------------------------------
4 md5633bc3c3d823be2a52d3dff94031e2c2
5 (1 row)

1 touch /etc/pgbouncer/userlist.txt
2 "postgres":"md5633bc3c3d823be2a52d3dff94031e2c2"

4. Permissions for other directories

1 chmod 777 -R /var/log/pgbouncer/

2 sudo chown pgbouncer:pgbouncer /var/run/pgbouncer/
3 sudo chmod 777 -R /var/run/pgbouncer/

5. Create the PgBouncer service file

1 [Unit]
2 Description=A lightweight connection pooler for PostgreSQL
3 Documentation=man:pgbouncer(1)
4 After=syslog.target network.target
5 [Service]
6 RemainAfterExit=yes
7 User=postgres
8 Group=postgres
9 # Path to the init file
10 Environment=BOUNCERCONF=/etc/pgbouncer/pgbouncer.ini
11 ExecStart=/usr/bin/pgbouncer -q ${BOUNCERCONF}
12 ExecReload=/usr/bin/pgbouncer -R -q ${BOUNCERCONF}
13 # Give a reasonable amount of time for the server to start up/shut down
14 TimeoutSec=300
15 [Install]
16 WantedBy=multi-user.target
17
18 4. Open PgBouncer config/init file and paste the config below
19 [databases]
20 * = port=5432 auth_user=postgres
21 [pgbouncer]
22 logfile = pgbouncer.log
23 pidfile = pgbouncer.pid
24 listen_addr = 0.0.0.0
25 listen_port = 6432
26 auth_type = hba
27 auth_hba_file = /var/lib/pgsql/16/data/pg_hba.conf
28 admin_users = postgres
29 stats_users = postgres
30 pool_mode = session
31 ignore_startup_parameters = extra_float_digits
32 max_client_conn = 200
33 default_pool_size = 50
34 reserve_pool_size = 25
35 reserve_pool_timeout = 3
36 server_lifetime = 300
37 server_idle_timeout = 120
38 server_connect_timeout = 5
39 server_login_retry = 1
40 query_timeout = 60

6. Add this entry in pgbouncer.ini

1 postgres = host=localhost port=5432 dbname=postgres

7. Update the authentication method to MD5 in pg_hba.conf

1 psql -U postgres -d postgres -h localhost -p 5432

2 Password for user postgres:
3 psql (16.4)
4 Type "help" for help.
5
6 postgres=# show password_encryption;
7 password_encryption
8 ---------------------
9 md5
10 (1 row)

8. Start the pgbouncer service

1 [root@a125cb415207 data]# systemctl start pgbouncer

2 [root@a125cb415207 data]# systemctl status pgbouncer
3 ● pgbouncer.service - A lightweight connection pooler for PostgreSQL
4 Loaded: loaded (/usr/lib/systemd/system/pgbouncer.service; disabled; preset: disabled)
5 Active: active (running) since Mon 2024-08-12 05:06:33 UTC; 3s ago
6 Docs: man:pgbouncer(1)
7 Main PID: 802 (pgbouncer)
8 Tasks: 2 (limit: 75664)
9 Memory: 1.3M
 10 CGroup:
/docker/a125cb415207ba986d1bcde058c3c274c8e8066eb689665877113e6286ba98f2/system.slice/pgbouncer.service
11 └─802 /usr/bin/pgbouncer -q /etc/pgbouncer/pgbouncer.ini
12
13 Aug 12 05:06:33 a125cb415207 systemd[1]: Started A lightweight connection pooler for PostgreSQL.
14 [root@a125cb415207 data]# vi /usr/lib/systemd/system/pgbouncer.service
15 [root@a125cb415207 data]# vi /etc/pgbouncer/pgbouncer.ini

9. Connect the pgbouncer

1 /usr/pgsql-16/bin/psql -h localhost -p 6432 -U postgres -d postgres

2 Password for user postgres:
3 psql (16.4)
4 Type "help" for help.
5
6 postgres=#

10. Conduct a basic benchmark test

After setting up, conduct a simple test run to observe the improvement in the TPS of the database server with PgBouncer. I’ve observed
approximately a 24% improvement in TPS with PgBouncer.

For the TPS benchmark, I used pgbench initialized with a scale factor of 1. The benchmark was run with 5 threads and 10 clients over a
duration of 60 seconds.

TPS without Pgbouncer

1 ./pgbench -c 10 -j 5 -T 60 -h localhost -p 5432 -U postgres postgres

2 Password:
3 pgbench (16.4)
4 starting vacuum...end.
5 transaction type: <builtin: TPC-B (sort of)>
6 scaling factor: 1
7 query mode: simple
8 number of clients: 10
9 number of threads: 5
10 maximum number of tries: 1
11 duration: 60 s
12 number of transactions actually processed: 73272
13 number of failed transactions: 0 (0.000%)
14 latency average = 8.188 ms
15 initial connection time = 15.287 ms
16 tps = 1221.329787 (without initial connection time)

TPS with Pgbouncer

1 ./pgbench -c 10 -j 5 -T 60 -h localhost -p 6432 -U postgres postgres

2 Password:
3 pgbench (16.4)
4 starting vacuum...end.
5 transaction type: <builtin: TPC-B (sort of)>
6 scaling factor: 1
7 query mode: simple
8 number of clients: 10
9 number of threads: 5
10 maximum number of tries: 1
11 duration: 60 s
12 number of transactions actually processed: 55843
13 number of failed transactions: 0 (0.000%)
14 latency average = 10.747 ms
 15 initial connection time = 2.129 ms
16 tps = 930.493410 (without initial connection time)

To make it easier to understand, you can see the benchmark test results in the graphs below.