0% found this document useful (0 votes)
3 views38 pages

PostgreSQL Configuration For Humans

The document outlines PostgreSQL configuration essentials, emphasizing the importance of tuning the postgresql.conf file and understanding various parameters for optimal performance. It discusses the significance of connection management, memory settings, and autovacuum strategies, while also introducing tools like postgresqlco.nf for easier configuration. The author, Álvaro Hernandez Tortosa, shares insights from his extensive experience in the PostgreSQL community and highlights the need for tailored configurations based on specific use cases.

Uploaded by

fkhalid
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
0% found this document useful (0 votes)
3 views38 pages

PostgreSQL Configuration For Humans

The document outlines PostgreSQL configuration essentials, emphasizing the importance of tuning the postgresql.conf file and understanding various parameters for optimal performance. It discusses the significance of connection management, memory settings, and autovacuum strategies, while also introducing tools like postgresqlco.nf for easier configuration. The author, Álvaro Hernandez Tortosa, shares insights from his extensive experience in the PostgreSQL community and highlights the need for tailored configurations based on specific use cases.

Uploaded by

fkhalid
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 38

PostgreSQL Configuration

for Humans

Álvaro Hernandez Tortosa


CEOALVARO HERNANDEZ

DBA and Java Software Developer


OnGres and ToroDB Founder
Well-Known member of the PostgreSQL Community @ahachete
World- Class Database Expert (+30 Talks in last 2 years)
About ONGRES
• IT firm specialized on R&D on Databases, more specifically
PostgreSQL:
✴ PostgreSQL Support
✴ Consulting and development
✴ Training

• Developers of Products and Online Services for PostgreSQL,


including postgresqlco.nf, a web configuration tool for
postgresql.conf

• Developers of ToroDB (www.torodb.com), an open-source,


document-store database that works on top of PostgreSQL
and is compatible with MongoDB.

PostgresConf US 2018
PostgreSQL configuration

✓ Mainly postgresql.conf (here’s most of the meat)


✓ Authentication: pg_hba.conf (and pg_ident.conf)
✓ Replicas: recovery.conf (may be merged soon)
✓ Some initdb parameters
✓ Per-object settings (eg. fillfactor)

Advanced stuff:

✓ Developer parameters
✓ Compile-time #defines

PostgresConf US 2018
Why configure PostgreSQL?

• Otherwise, it only listens on localhost


• You can only replicate by default in >= 10
• To enable WAL archiving
• Application developers say they get “connection refused!”
• Set defaults for client connections
• Load extensions that required shared libraries
• Enable checksums (initdb!)
• Compatibility with older versions

Any other reason? ;)

PostgresConf US 2018
Performance, performance, performance

Usual performance optimization advice: Don’t, don’t, don’t

Do,
Do, Do
Usual PostgreSQL advice: (unless you run on a Raspberry PI 1st
gen)

✓ Run your own app-centric benchmarks

✓pg_bench is just one benchmark more

All the usual precautions about benchmarks apply

PostgresConf US 2018
Performance, performance, performance

pg_bench, scale
2000, m4.large
(2 vCPU, 8GB
RAM, 1k IOPS

PostgresConf US 2018
Performance, performance, performance

pg_bench, scale
2000, m4.large
(2 vCPU, 8GB
RAM, 1k IOPS

PostgresConf US 2018
postgresql.conf parameters

✓ More than 200 parameters (no kidding!)


✓ Classified into 40 categories / subcategories
✓ 650 lines, 23Kb sample config file

How many to tune? 2? 5? 10? 20? 40? 100?

✓ Parameters are integer, real, string, enum, real


or bool. Numeric values may have units (or are
unit-less)
✓ Some units are a bit uneasy (like “blocks of
8Kb”) or too synthetic (cpu_tuple_cost)

PostgresConf US 2018
Tunable postgresql.conf parameters

PostgresConf US 2018
Some ideas about PostgreSQL tuning…

PostgresConf US 2018
Disclaimer

✓ No, you won’t get here final numbers on how


to tune your postgresql.conf
✓ Only a few dozens parameters discussed here
✓ Only hints provided: do your homework
✓ My opinion may differ from other’s
✓ I am probably wrong
✓ YMMV

(you get the point)

PostgresConf US 2018
initdb

✓ Sometimes run on your behalf (Debian/Ubuntu), bad for


selecting non defaults

✓ -E (encoding). Use UTF-8 unless you know what you do

✓ --locale, --lc_collate, —lc-ctype

✓ --username: if ‘postgres’ is not the superuser

✓ --data-checksums: enable them!

PostgresConf US 2018
Db connections 101

✓ max_connections is a hard limit

✓ PostgreSQL will reject connections over this number

✓ Users not happy

✓ Default is 100

✓ “My app has more 100 concurrent users!”

Solution is obvious: raise max_connections!

PostgresConf US 2018
Db connections 101

Solution is obvious: raise max_connections!

✓ One process per connection (excl. parallelism!)

✓ One process handled by one core

✓ How many cores do you have?

✓ Sure, you have a multi-process, time-sharing OS but what is


the scheduling overhead with many processes?

Even worse: cache trashing!

PostgresConf US 2018
Db connections 101

pg_bench, scale
2000, m4.large
(2 vCPU, 8GB
RAM, 1k IOPS

# concurrent connections

PostgresConf US 2018
Db connections 101

pg_bench, scale
2000, m4.large
(2 vCPU, 8GB
RAM, 1k IOPS

# concurrent connections

PostgresConf US 2018
Db connections 101

✓ Solution is obvious: lower max_connections!

✓ But how we solve the connection refused problem?

✓ PgBouncer!

✓ Size your connections almost 1:1 pgbouncer:max_conns

✓ Use this formula:

Cores
Connections= x scale factor
% effective utilization connection

PostgresConf US 2018
shared_buffers

✓ The first recommendation everybody tells you

✓ Set it to 1/4th of RAM and effective_cache_size 3/4th

✓ Done!

✓ ¼ too low on low memory, too high on high memory

✓ Benchmark, benchmark, benchmark

✓ Is the db dedicated in the host? OS memory?

✓ How much memory other PostgreSQL parts use, like


maintenance_work_mem or all the memory used by query
processes?

PostgresConf US 2018
work_mem

✓ Max local process memory used for operations like sort and
joins in queries

✓ Not written in stone: users can SET it overriding its value

✓ But if more memory is used, it spills to disk (and may use


different algorithm) reducing performance

✓ Not the same if you are OLTP, OLAP, DW (small to very large)

✓ Raise it from defaults, but don’t forget it could be times


(max_connections * max nodes query)

PostgresConf US 2018
Other memory/disk tunables

✓ maintenance_work_mem: vacuum, create index, check FKs….


raise it

✓ {min,max}_wal_size: it’s only disk space, but too low will


cause excessive checkpoints. Make min at least 1GB, max
several GB up to 50-100GB.

✓ stats_temp_directory: run on a not very small RAMdisk

PostgresConf US 2018
This requires restart, think carefully

✓ listen_addresses (take care with ‘*’), port


✓ ssl: activate only if needed, use pooling!
✓ huge_pages: benchmark, benchmark, benchmark (typically off)
✓ shared_preload_libraries: add your extensions
beforehand!
✓ logging_collector: on
✓ wal_level: replica or *logical*
✓ archive_mode, archive_command = '/bin/true' if
you don't use archiving

✓ cluster_name

PostgresConf US 2018
The tyranny of max_*

✓ Most of the max_* also require restart


✓ Sometimes hard to estimate, but restarting the database is pretty
bad: raise limits and control usage
✓ max_wal_senders: replicas, backups, make room
✓ max_replication_slots
✓ max_worker_processes,
max_parallel_workers_per_gather,
max_parallel_workers

✓ autovacuum_max_workers (# cores for cleanup?)


✓ max_prepared_transactions (2PC, 0 by default)

PostgresConf US 2018
Autovacuum / vacuum

✓ Almost always too conservative

✓ Bloat is one of the most frequent operational burdens

✓ Hard to get it right: analyze and re-tune periodically

✓ Some parameters are set to “-1” which means “look at these


numbers from the vacuum parameters”

✓ autovacuum_{vacuum,analyze}_scale_factor: you may


override on a per-table level

PostgresConf US 2018
Autovacuum / vacuum
General advice:

✓ Raise vacuum_cost_limit significantly

✓ Reduce autovacuum_vacuum_cost_delay

✓ Use more autovacuum_max_workers if you have many cores

PostgresConf US 2018
Checkpoints and bgwriter

✓ You typically want to spread checkpoints farther apart (raise


checkpoint_timeout)

✓ min_wal_size 1GB min


✓ Log checkpoints and look for warnings
✓ Raise checkpoint_completion_target, eg. 0.9, but study your I/O
patterns, shared_buffers, wal size

✓ Increase bgwriter activity, very conservative default:


✓ Decrease bgwriter_delay
‣ Increase bgwriter_lru_maxpages

PostgresConf US 2018
Logging

✓ “Only” 24 parameters (plus some others)

✓ Spend some time here, It pays of when analyzing your db

✓ Turn on:

‣ logging_collector

‣ log_checkpoints

‣ log_connections, log_disconnections

PostgresConf US 2018
Other interesting parameters

✓ password_encryption use SHA-256 if possible (>= PG 10)

✓ effective_io_concurrency (how many “spindles” your I/O


has?)

✓ max_standby_{archive,streaming}_delay and
hot_standby_feedbak: keep replication query conflicts burden
on the primary or secondaries?

✓ default_statistics_target: if setting by table is not your


job

✓ Adjust the random_page_cost / seq_page_cost (4.0 / 1.0


by default), so that it is lower on SSD (start with 1.x) or indexes
may not be used when it could be beneficial.

PostgresConf US 2018
Was that too much?
Tools to help?

PostgresConf US 2018
Was that too much? Tools to help?

PostgresConf US 2018
Was that too much? Tools to help?

PostgresConf US 2018
Was that too much? Tools to help?

PostgresConf US 2018
PostgreSQL wants a new configuration tool

(IMVHO)

PostgresConf US 2018
postgresqlco.nf

PostgresConf US 2018
postgresqlco.nf

PostgresConf US 2018
postgresqlco.nf

✓ Web configuration tool


✓ Drag&drop your postgresql.conf and tune it!
✓ Integrated help
✓ Documentation
✓ Recommendations
✓ Easy search & filtering
✓ Create your configurations
✓ Download as postgresql.conf, json, yaml, SQL
✓ Rest API

PostgresConf US 2018
Subscribe on postgresqlco.nf
to become a beta tester!

PostgresConf US 2018
Questions?

www.ongres.com

@ongresinc

[email protected]

You might also like