The extension btree_gist
lives in PostgreSQL contrib
and implements btree-like operators and behavior on top of the GiST framework. GiST stands for Generalized Search Tree and is a framework to implement advanced indexes for many use cases. Fulltext indexing or spatial index types are an example for such specialized index implementations.
Table of Contents
btree_gist
can serve as an example on how to implement such specialized indexes, but also serves very important use cases nowadays in PostgreSQL:
Especially the latter is a very important use case, since this helps implement advanced constraints in PostgreSQL databases. For example, if you want to combine standard scalar types like bigint
or uuid
with a range
type, you need btree_gist
to be able to create the constraint, because otherwise the necessary combination of those types is not able to be used together with a GiST
index, which is required in this case. Though there was one important problem until now:
buffered
and sorted
.sorted
is used if operator classes for such a GiST index implement sortsupport
buffered
is the selected methodThe problem with buffered
is that this follows a one-by-one method: Each value is inserted individually when the index is built, which is considerably slower than using a sorted input. Since each insertion into the building index results in an individual lookup, this is a very slow procedure and also results in a suboptimal organized index afterwards. In contrast, sorted input makes it easier to build an index and keep an optimal structure. Including PostgreSQL 17, btree_gist
operator classes didn't provide sortsupport
, thus supporting the slow buffered
method only.
With the upcoming PostgreSQL 18 release, btree_gist
was extended and now uses sortsupport
per default when building such indexes. The patch itself has its history though. It started first with the work by Andrey Borodin in 2020. Unfortunately this was reverted due to some unresolved problems with the handling of specific datatypes. Christoph Heiss and me then picked up this idea again, and after some extensive review finally all issues are solved.
The following benchmark is based on a dataset provided by a customer, who is fighting with int4range
range values and their time required to rebuild the indexes when importing the data. The simplified schema for this benchmark is defined as below:
1 2 3 4 5 6 7 8 |
CREATE EXTENSION btree_gist; CREATE TABLE token ( id uuid, range int4range NOT NULL ); CREATE INDEX ON token USING gist(id, range); |
We are reusing this table layout for all the other benchmarks below. The sample data looks like the following:
1 2 3 4 5 6 7 8 |
SELECT * FROM token LIMIT 4; id │ range ──────────────────────────────────────┼───────────────────── e44e4063-db4c-e8f7-afa3-fd4308e30085 │ [10773122,10773138) 7a32d965-983d-9e2f-1baa-b07595c562b0 │ [10741736,10741751) e2dd8b78-a655-464b-3bcb-b1e3921ce645 │ [10773122,10773153) 7304d2d3-6e39-fa01-2ef0-cdb8818202fb │ [10741750,10741751) |
I did this benchmark with just a slightly tuned instance of PostgreSQL 17.5 and 18beta2, the adjusted settings are:
shared_buffers = 4GB
effective_cache_size
is already set to 4GB per default, so this setting is untouchedThe performance improvements are shown in the following graph, which compares the CREATE INDEX
timings for 1, 5 and 10 million rows to index with PostgreSQL 17.5 and PostgreSQL 18beta2.
The timings for the specific sizes of the test dataset are significantly lower on PostgreSQL 18 with sortsupport
when compared to PostgreSQL 17. So this promises much faster timings when building indexes using btree_gist
in PostgreSQL 18.
The next benchmark illustrates the performance improvement we get using btree_gist
behind the scenes when creating an exclusion constraint
. For that, we create an exclusion constraint
on the test table above that ensures that an UUID
doesn't have any overlapping integer values within its stored ranges. We can achieve this by using the &&
overlaps operator, supported by int4range
. The constraint will be added with the following DDL statement:
1 2 3 |
ALTER TABLE token ADD CONSTRAINT excl_id_range EXCLUDE USING gist(id WITH =, range WITH &&;); |
We benchmark this statement again for 1 million, 5 million and 10 million tuples from the test dataset above. The resulting runtimes draw the following graph:
Again, the improvements in PG18 with sortsupport
are impressive. Especially with workloads that need to import and index large amount of data and want to use btree_gist
because of the previously outlined features are benefitting.
The numbers above shows the improvement when creating indexes with btree_gist
index access methods, but what about the general performance when these indexes get utilized in queries? During his work on the patch, Christoph Heiss created a python script to measure the performance of btree_gist
indexes compared with and without sortsupport
. While the patch primarily is targeted to decrease the required time to build such indexes, they should also affect the query performance, since with sortsupport
this should result in physically better organized index structure.
To test if querying the new index yields increases throughput because of probably better index quality, we've done the following benchmark utilizing pgbench
and its scripting feature. Based on the test dataset with 10 million tokens we have the following initialization script:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
-- init.sql BEGIN; DROP TABLE IF EXISTS test_dataset; CREATE TABLE test_dataset( keyid integer not null, id uuid not null, range int4range ); CREATE TEMP SEQUENCE testset_seq; INSERT INTO test_dataset SELECT nextval('testset_seq'), id, range FROM token_10m ORDER BY random() LIMIT 10000; CREATE UNIQUE INDEX ON test_dataset(keyid); COMMIT; |
This creates a new table test_datase
t after dropping it (if it already exists). Then the table is filled with 10000 random keys selected from the token table with 10 million rows. The pgbench
script using this structure looks as follows:
1 2 3 |
\set keyid random(1, 10000) SELECT id, range FROM test_dataset WHERE keyid = :keyid \gset SELECT id, range FROM token_10m WHERE id = ':id' AND range && ':range'; |
We then run the benchmark three times and calculate the average transactions per seconds measured with these runs. Each run is 60 seconds, we measure exactly just one connection and also the latency during the benchmark. pgbench.tokens
is the file with the pgbench
script shown above.
1 2 3 4 |
for in in `seq 1 3`; do psql -qXf init.sql && pgbench -n -r -c 1 -T 60 -f pgbench.tokens; done |
On my machine, this yields the following figure (higher numbers are better):
The throughput by just selecting tokens that are overlapping with the &&
operator shows a clear performance improvement on PostgreSQL 18beta2. This means that by having a better index quality in this case yields three times higher transaction throughput per second than the same workload in PostgreSQL 17.
Christoph Heiss created a benchmark tool to track down why these lookup queries are much faster than before. This python script can be found in the PostgreSQL archives here. It performs the same workload as the random access benchmark above with pgbench
, but examines the runtime statistics of each query with EXPLAIN
. This allows to gather some runtime metrics like the number of pages the query needs to access. The benchmark follows the same pattern as the pgbench
script above: First, a number of tokens (default 10000) are selected randomly from the tokens table, then the selected token is read from the table again. This is done within a loop, which is configurable and repeated 10 times per default. This sums up to 100000 queries issued. For each query, the execution plan is analyzed and execution time, shared page hits, shared page reads and I/O time are collected.
When successfully executed, metrics are stored in a name file in CSV format and a summary is printed to console. I ran the benchmark against current PostgreSQL 17.5 and 18beta2 with the tokens table filled with 10 million rows again. Repeating the queries multiple times causes the index to be fully cached. Looking at the average page hits within the shared buffer pool when comparing both versions, we get the following graph:
This benchmark causes PostgreSQL 17 with this table and btree_gist
index to read ten more pages average than compared with PostgreSQL 18beta2. The benchmark was run several times, but the figures stay stable. This backups our hope that sortsupport
yields a much better index quality and also improves performance for such index lookups.
The new sortsupport
for the btree_gist
extension in the upcoming PostgreSQL 18 release provides a much better performance when compared to former PostgreSQL major versions. While just being an extension, btree_gist
serves important use cases like having specialized composite indexes with data types that don't support GiST
, for example to implement exclusion constraints
on tables.
Users that rely on these features should try out PostgreSQL 18 as soon as possible.
Leave a Reply