Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: postgresql-cfbot/postgresql
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: cf/3733~1
Choose a base ref
...
head repository: postgresql-cfbot/postgresql
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: cf/3733
Choose a head ref
  • 7 commits
  • 17 files changed
  • 3 contributors

Commits on Mar 28, 2025

  1. Fix grammar in GIN README

    tvondra authored and Commitfest Bot committed Mar 28, 2025
    Copy the full SHA
    a1dd9b3 View commit details
  2. amcheck: Move common routines into a separate module

    Before performing checks on an index, we need to take some safety
    measures that apply to all index AMs. This includes:
    
    * verifying that the index can be checked - Only selected AMs are
    supported by amcheck (right now only B-Tree). The index has to be
    valid and not a temporary index from another session.
    
    * changing (and then restoring) user's security context
    
    * obtaining proper locks on the index (and table, if needed)
    
    * discarding GUC changes from the index functions
    
    Until now this was implemented in the B-Tree amcheck module, but it's
    something every AM will have to do. So relocate the code into a new
    module verify_common for reuse.
    
    The shared steps are implemented by amcheck_lock_relation_and_check(),
    receiving the AM-specific verification as a callback. Custom parameters
    may be supplied using a pointer.
    
    Author: Andrey Borodin <amborodin@acm.org>
    Reviewed-By: José Villanova <jose.arthur@gmail.com>
    Reviewed-By: Aleksander Alekseev <aleksander@timescale.com>
    Reviewed-By: Nikolay Samokhvalov <samokhvalov@gmail.com>
    Reviewed-By: Andres Freund <andres@anarazel.de>
    Reviewed-By: Tomas Vondra <tomas@vondra.me>
    Reviewed-By: Mark Dilger <mark.dilger@enterprisedb.com>
    Reviewed-By: Peter Geoghegan <pg@bowt.ie>
    Reviewed-By: Kirill Reshke <reshkekirill@gmail.com>
    Discussion: https://postgr.es/m/45AC9B0A-2B45-40EE-B08F-BDCF5739D1E1%40yandex-team.ru
    tvondra authored and Commitfest Bot committed Mar 28, 2025
    Copy the full SHA
    43f209a View commit details
  3. amcheck: Add gin_index_check() to verify GIN index

    The new functions validates two kinds of invariants on a GIN index:
    
    - parent-child consistency: Paths in a GIN graph have to contain
      consistent keys: tuples on parent pages consistently include tuples
      from children pages. That is, parent tuples must not require any
      adjustments.
    
    - balanced-tree / graph: Each internal page has at least one downlink,
      and can reference either only leaf pages or only internal pages.
    
    The GIN verification is based on work by Grigory Kryachko, reworked by
    Heikki Linnakangas and with various improvements by Andrey Borodin.
    Investigation and fixes for a couple bugs by Kirill Reshke.
    
    Author: Grigory Kryachko <GSKryachko@gmail.com>
    Author: Heikki Linnakangas <hlinnaka@iki.fi>
    Author: Andrey Borodin <amborodin@acm.org>
    Reviewed-By: José Villanova <jose.arthur@gmail.com>
    Reviewed-By: Aleksander Alekseev <aleksander@timescale.com>
    Reviewed-By: Nikolay Samokhvalov <samokhvalov@gmail.com>
    Reviewed-By: Andres Freund <andres@anarazel.de>
    Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
    Reviewed-By: Kirill Reshke <reshkekirill@gmail.com>
    Reviewed-By: Mark Dilger <mark.dilger@enterprisedb.com>
    Reviewed-By: Peter Geoghegan <pg@bowt.ie>
    Discussion: https://postgr.es/m/45AC9B0A-2B45-40EE-B08F-BDCF5739D1E1%40yandex-team.ru
    tvondra authored and Commitfest Bot committed Mar 28, 2025
    Copy the full SHA
    bf5e2a0 View commit details
  4. amcheck: Add a test with GIN index on JSONB data

    Extend the existing test of GIN checks to also include an index on JSONB
    data, using the jsonb_path_ops opclass. This is a common enough usage of
    GIN that it makes sense to have better test coverage for it.
    
    Author: Mark Dilger <mark.dilger@enterprisedb.com>
    Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
    Reviewed-By: Kirill Reshke <reshkekirill@gmail.com>
    Discussion: https://postgr.es/m/BC221A56-977C-418E-A1B8-9EFC881D80C5%40enterprisedb.com
    tvondra authored and Commitfest Bot committed Mar 28, 2025
    Copy the full SHA
    9cfeaa7 View commit details
  5. amcheck: Add a GIN index to the CREATE INDEX CONCURRENTLY tests

    The existing CREATE INDEX CONCURRENTLY tests checking only B-Tree, but
    can be cheaply extended to also check GIN. This helps increasing test
    coverage for GIN amcheck, especially related to handling concurrent page
    splits and posting list trees.
    
    This already helped to identify several issues during development of the
    GIN amcheck support.
    
    Author: Mark Dilger <mark.dilger@enterprisedb.com>
    Reviewed-By: Tomas Vondra <tomas.vondra@enterprisedb.com>
    Reviewed-By: Kirill Reshke <reshkekirill@gmail.com>
    Discussion: https://postgr.es/m/BC221A56-977C-418E-A1B8-9EFC881D80C5%40enterprisedb.com
    tvondra authored and Commitfest Bot committed Mar 28, 2025
    Copy the full SHA
    b43ee5a View commit details
  6. Stress test verify_gin() using pgbench

    Add a tap test which inserts, updates, deletes, and checks in
    parallel.  Like all pgbench based tap tests, this test contains race
    conditions between the operations, so Your Mileage May Vary.  For
    me, on my laptop, I got failures like:
    
    	index "ginidx" has wrong tuple order on entry tree page
    
    which I have not yet investigated.  The test is included here for
    anybody interested in debugging this failure.
    markdilger authored and Commitfest Bot committed Mar 28, 2025
    Copy the full SHA
    48143f4 View commit details
  7. [CF 3733] v20250328 - Amcheck verification of GiST and GIN

    This branch was automatically generated by a robot using patches from an
    email thread registered at:
    
    https://commitfest.postgresql.org/patch/3733
    
    The branch will be overwritten each time a new patch version is posted to
    the thread, and also periodically to check for bitrot caused by changes
    on the master branch.
    
    Patch(es): https://www.postgresql.org/message-id/d2ee3c1e-f51b-46bf-8307-be545e3b85d2@vondra.me
    Author(s): Heikki Linnakangas, Andrey Borodin, Grigory Kryachko
    Commitfest Bot committed Mar 28, 2025
    Copy the full SHA
    218eb5c View commit details
7 changes: 5 additions & 2 deletions contrib/amcheck/Makefile
Original file line number Diff line number Diff line change
@@ -3,14 +3,17 @@
MODULE_big = amcheck
OBJS = \
$(WIN32RES) \
verify_common.o \
verify_gin.o \
verify_heapam.o \
verify_nbtree.o

EXTENSION = amcheck
DATA = amcheck--1.3--1.4.sql amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql
DATA = amcheck--1.2--1.3.sql amcheck--1.1--1.2.sql amcheck--1.0--1.1.sql amcheck--1.0.sql \
amcheck--1.3--1.4.sql amcheck--1.4--1.5.sql
PGFILEDESC = "amcheck - function for verifying relation integrity"

REGRESS = check check_btree check_heap
REGRESS = check check_btree check_gin check_heap

EXTRA_INSTALL = contrib/pg_walinspect
TAP_TESTS = 1
14 changes: 14 additions & 0 deletions contrib/amcheck/amcheck--1.4--1.5.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
/* contrib/amcheck/amcheck--1.4--1.5.sql */

-- complain if script is sourced in psql, rather than via CREATE EXTENSION
\echo Use "ALTER EXTENSION amcheck UPDATE TO '1.5'" to load this file. \quit


-- gin_index_check()
--
CREATE FUNCTION gin_index_check(index regclass)
RETURNS VOID
AS 'MODULE_PATHNAME', 'gin_index_check'
LANGUAGE C STRICT;

REVOKE ALL ON FUNCTION gin_index_check(regclass) FROM PUBLIC;
2 changes: 1 addition & 1 deletion contrib/amcheck/amcheck.control
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# amcheck extension
comment = 'functions for verifying relation integrity'
default_version = '1.4'
default_version = '1.5'
module_pathname = '$libdir/amcheck'
relocatable = true
4 changes: 2 additions & 2 deletions contrib/amcheck/expected/check_btree.out
Original file line number Diff line number Diff line change
@@ -57,8 +57,8 @@ ERROR: could not open relation with OID 17
BEGIN;
CREATE INDEX bttest_a_brin_idx ON bttest_a USING brin(id);
SELECT bt_index_parent_check('bttest_a_brin_idx');
ERROR: only B-Tree indexes are supported as targets for verification
DETAIL: Relation "bttest_a_brin_idx" is not a B-Tree index.
ERROR: expected "btree" index as targets for verification
DETAIL: Relation "bttest_a_brin_idx" is a brin index.
ROLLBACK;
-- normal check outside of xact
SELECT bt_index_check('bttest_a_idx');
78 changes: 78 additions & 0 deletions contrib/amcheck/expected/check_gin.out
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
-- Test of index bulk load
SELECT setseed(1);
setseed
---------

(1 row)

CREATE TABLE "gin_check"("Column1" int[]);
-- posting trees (frequently used entries)
INSERT INTO gin_check select array_agg(round(random()*255) ) from generate_series(1, 100000) as i group by i % 10000;
-- posting leaves (sparse entries)
INSERT INTO gin_check select array_agg(255 + round(random()*100)) from generate_series(1, 100) as i group by i % 100;
CREATE INDEX gin_check_idx on "gin_check" USING GIN("Column1");
SELECT gin_index_check('gin_check_idx');
gin_index_check
-----------------

(1 row)

-- cleanup
DROP TABLE gin_check;
-- Test index inserts
SELECT setseed(1);
setseed
---------

(1 row)

CREATE TABLE "gin_check"("Column1" int[]);
CREATE INDEX gin_check_idx on "gin_check" USING GIN("Column1");
ALTER INDEX gin_check_idx SET (fastupdate = false);
-- posting trees
INSERT INTO gin_check select array_agg(round(random()*255) ) from generate_series(1, 100000) as i group by i % 10000;
-- posting leaves
INSERT INTO gin_check select array_agg(100 + round(random()*255)) from generate_series(1, 100) as i group by i % 100;
SELECT gin_index_check('gin_check_idx');
gin_index_check
-----------------

(1 row)

-- cleanup
DROP TABLE gin_check;
-- Test GIN over text array
SELECT setseed(1);
setseed
---------

(1 row)

CREATE TABLE "gin_check_text_array"("Column1" text[]);
-- posting trees
INSERT INTO gin_check_text_array select array_agg(md5(round(random()*300)::text)::text) from generate_series(1, 100000) as i group by i % 10000;
-- posting leaves
INSERT INTO gin_check_text_array select array_agg(md5(round(random()*300 + 300)::text)::text) from generate_series(1, 10000) as i group by i % 100;
CREATE INDEX gin_check_text_array_idx on "gin_check_text_array" USING GIN("Column1");
SELECT gin_index_check('gin_check_text_array_idx');
gin_index_check
-----------------

(1 row)

-- cleanup
DROP TABLE gin_check_text_array;
-- Test GIN over jsonb
CREATE TABLE "gin_check_jsonb"("j" jsonb);
INSERT INTO gin_check_jsonb values ('{"a":[["b",{"x":1}],["b",{"x":2}]],"c":3}');
INSERT INTO gin_check_jsonb values ('[[14,2,3]]');
INSERT INTO gin_check_jsonb values ('[1,[14,2,3]]');
CREATE INDEX "gin_check_jsonb_idx" on gin_check_jsonb USING GIN("j" jsonb_path_ops);
SELECT gin_index_check('gin_check_jsonb_idx');
gin_index_check
-----------------

(1 row)

-- cleanup
DROP TABLE gin_check_jsonb;
4 changes: 4 additions & 0 deletions contrib/amcheck/meson.build
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# Copyright (c) 2022-2025, PostgreSQL Global Development Group

amcheck_sources = files(
'verify_common.c',
'verify_gin.c',
'verify_heapam.c',
'verify_nbtree.c',
)
@@ -24,6 +26,7 @@ install_data(
'amcheck--1.1--1.2.sql',
'amcheck--1.2--1.3.sql',
'amcheck--1.3--1.4.sql',
'amcheck--1.4--1.5.sql',
kwargs: contrib_data_args,
)

@@ -35,6 +38,7 @@ tests += {
'sql': [
'check',
'check_btree',
'check_gin',
'check_heap',
],
},
52 changes: 52 additions & 0 deletions contrib/amcheck/sql/check_gin.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
-- Test of index bulk load
SELECT setseed(1);
CREATE TABLE "gin_check"("Column1" int[]);
-- posting trees (frequently used entries)
INSERT INTO gin_check select array_agg(round(random()*255) ) from generate_series(1, 100000) as i group by i % 10000;
-- posting leaves (sparse entries)
INSERT INTO gin_check select array_agg(255 + round(random()*100)) from generate_series(1, 100) as i group by i % 100;
CREATE INDEX gin_check_idx on "gin_check" USING GIN("Column1");
SELECT gin_index_check('gin_check_idx');

-- cleanup
DROP TABLE gin_check;

-- Test index inserts
SELECT setseed(1);
CREATE TABLE "gin_check"("Column1" int[]);
CREATE INDEX gin_check_idx on "gin_check" USING GIN("Column1");
ALTER INDEX gin_check_idx SET (fastupdate = false);
-- posting trees
INSERT INTO gin_check select array_agg(round(random()*255) ) from generate_series(1, 100000) as i group by i % 10000;
-- posting leaves
INSERT INTO gin_check select array_agg(100 + round(random()*255)) from generate_series(1, 100) as i group by i % 100;

SELECT gin_index_check('gin_check_idx');

-- cleanup
DROP TABLE gin_check;

-- Test GIN over text array
SELECT setseed(1);
CREATE TABLE "gin_check_text_array"("Column1" text[]);
-- posting trees
INSERT INTO gin_check_text_array select array_agg(md5(round(random()*300)::text)::text) from generate_series(1, 100000) as i group by i % 10000;
-- posting leaves
INSERT INTO gin_check_text_array select array_agg(md5(round(random()*300 + 300)::text)::text) from generate_series(1, 10000) as i group by i % 100;
CREATE INDEX gin_check_text_array_idx on "gin_check_text_array" USING GIN("Column1");
SELECT gin_index_check('gin_check_text_array_idx');

-- cleanup
DROP TABLE gin_check_text_array;

-- Test GIN over jsonb
CREATE TABLE "gin_check_jsonb"("j" jsonb);
INSERT INTO gin_check_jsonb values ('{"a":[["b",{"x":1}],["b",{"x":2}]],"c":3}');
INSERT INTO gin_check_jsonb values ('[[14,2,3]]');
INSERT INTO gin_check_jsonb values ('[1,[14,2,3]]');
CREATE INDEX "gin_check_jsonb_idx" on gin_check_jsonb USING GIN("j" jsonb_path_ops);

SELECT gin_index_check('gin_check_jsonb_idx');

-- cleanup
DROP TABLE gin_check_jsonb;
10 changes: 7 additions & 3 deletions contrib/amcheck/t/002_cic.pl
Original file line number Diff line number Diff line change
@@ -21,8 +21,9 @@
'lock_timeout = ' . (1000 * $PostgreSQL::Test::Utils::timeout_default));
$node->start;
$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
$node->safe_psql('postgres', q(CREATE TABLE tbl(i int, j jsonb)));
$node->safe_psql('postgres', q(CREATE INDEX idx ON tbl(i)));
$node->safe_psql('postgres', q(CREATE INDEX ginidx ON tbl USING gin(j)));

#
# Stress CIC with pgbench.
@@ -40,21 +41,24 @@
{
'002_pgbench_concurrent_transaction' => q(
BEGIN;
INSERT INTO tbl VALUES(0);
INSERT INTO tbl VALUES(0, '{"a":[["b",{"x":1}],["b",{"x":2}]],"c":3}');
COMMIT;
),
'002_pgbench_concurrent_transaction_savepoints' => q(
BEGIN;
SAVEPOINT s1;
INSERT INTO tbl VALUES(0);
INSERT INTO tbl VALUES(0, '[[14,2,3]]');
COMMIT;
),
'002_pgbench_concurrent_cic' => q(
SELECT pg_try_advisory_lock(42)::integer AS gotlock \gset
\if :gotlock
DROP INDEX CONCURRENTLY idx;
CREATE INDEX CONCURRENTLY idx ON tbl(i);
DROP INDEX CONCURRENTLY ginidx;
CREATE INDEX CONCURRENTLY ginidx ON tbl USING gin(j);
SELECT bt_index_check('idx',true);
SELECT gin_index_check('ginidx');
SELECT pg_advisory_unlock(42);
\endif
)
40 changes: 33 additions & 7 deletions contrib/amcheck/t/003_cic_2pc.pl
Original file line number Diff line number Diff line change
@@ -25,7 +25,7 @@
'lock_timeout = ' . (1000 * $PostgreSQL::Test::Utils::timeout_default));
$node->start;
$node->safe_psql('postgres', q(CREATE EXTENSION amcheck));
$node->safe_psql('postgres', q(CREATE TABLE tbl(i int)));
$node->safe_psql('postgres', q(CREATE TABLE tbl(i int, j jsonb)));


#
@@ -41,7 +41,7 @@
$main_h->query_safe(
q(
BEGIN;
INSERT INTO tbl VALUES(0);
INSERT INTO tbl VALUES(0, '[[14,2,3]]');
));

my $cic_h = $node->background_psql('postgres');
@@ -50,6 +50,7 @@
qr/start/, q(
\echo start
CREATE INDEX CONCURRENTLY idx ON tbl(i);
CREATE INDEX CONCURRENTLY ginidx ON tbl USING gin(j);
));

$main_h->query_safe(
@@ -60,7 +61,7 @@
$main_h->query_safe(
q(
BEGIN;
INSERT INTO tbl VALUES(0);
INSERT INTO tbl VALUES(0, '[[14,2,3]]');
));

$node->safe_psql('postgres', q(COMMIT PREPARED 'a';));
@@ -69,7 +70,7 @@
q(
PREPARE TRANSACTION 'b';
BEGIN;
INSERT INTO tbl VALUES(0);
INSERT INTO tbl VALUES(0, '"mary had a little lamb"');
));

$node->safe_psql('postgres', q(COMMIT PREPARED 'b';));
@@ -86,6 +87,9 @@
$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
is($result, '0', 'bt_index_check after overlapping 2PC');

$result = $node->psql('postgres', q(SELECT gin_index_check('ginidx')));
is($result, '0', 'gin_index_check after overlapping 2PC');


#
# Server restart shall not change whether prepared xact blocks CIC
@@ -94,7 +98,7 @@
$node->safe_psql(
'postgres', q(
BEGIN;
INSERT INTO tbl VALUES(0);
INSERT INTO tbl VALUES(0, '{"a":[["b",{"x":1}],["b",{"x":2}]],"c":3}');
PREPARE TRANSACTION 'spans_restart';
BEGIN;
CREATE TABLE unused ();
@@ -108,12 +112,16 @@
\echo start
DROP INDEX CONCURRENTLY idx;
CREATE INDEX CONCURRENTLY idx ON tbl(i);
DROP INDEX CONCURRENTLY ginidx;
CREATE INDEX CONCURRENTLY ginidx ON tbl USING gin(j);
));

$node->safe_psql('postgres', "COMMIT PREPARED 'spans_restart'");
$reindex_h->quit;
$result = $node->psql('postgres', q(SELECT bt_index_check('idx',true)));
is($result, '0', 'bt_index_check after 2PC and restart');
$result = $node->psql('postgres', q(SELECT gin_index_check('ginidx')));
is($result, '0', 'gin_index_check after 2PC and restart');


#
@@ -136,14 +144,14 @@
{
'003_pgbench_concurrent_2pc' => q(
BEGIN;
INSERT INTO tbl VALUES(0);
INSERT INTO tbl VALUES(0,'null');
PREPARE TRANSACTION 'c:client_id';
COMMIT PREPARED 'c:client_id';
),
'003_pgbench_concurrent_2pc_savepoint' => q(
BEGIN;
SAVEPOINT s1;
INSERT INTO tbl VALUES(0);
INSERT INTO tbl VALUES(0,'[false, "jnvaba", -76, 7, {"_": [1]}, 9]');
PREPARE TRANSACTION 'c:client_id';
COMMIT PREPARED 'c:client_id';
),
@@ -163,7 +171,25 @@
SELECT bt_index_check('idx',true);
SELECT pg_advisory_unlock(42);
\endif
),
'005_pgbench_concurrent_cic' => q(
SELECT pg_try_advisory_lock(42)::integer AS gotginlock \gset
\if :gotginlock
DROP INDEX CONCURRENTLY ginidx;
CREATE INDEX CONCURRENTLY ginidx ON tbl USING gin(j);
SELECT gin_index_check('ginidx');
SELECT pg_advisory_unlock(42);
\endif
),
'006_pgbench_concurrent_ric' => q(
SELECT pg_try_advisory_lock(42)::integer AS gotginlock \gset
\if :gotginlock
REINDEX INDEX CONCURRENTLY ginidx;
SELECT gin_index_check('ginidx');
SELECT pg_advisory_unlock(42);
\endif
)

});

$node->stop;
Loading