summaryrefslogtreecommitdiff
path: root/src/tools
AgeCommit message (Collapse)Author
2023-04-22Doc: update pgindent/README some more.Tom Lane
In commit 2e6ba1315, I missed that there was a redundant second reference to pg_bsd_indent's old repo.
2023-04-19Fix various typos and incorrect/outdated name referencesDavid Rowley
Author: Alexander Lakhin Discussion: https://postgr.es/m/[email protected]
2023-04-18Fix some typos and some incorrectly duplicated wordsDavid Rowley
Author: Justin Pryzby Reviewed-by: David Rowley Discussion: https://postgr.es/m/[email protected]
2023-04-17Update Solution.pm for HAVE_GSSAPI[_GSSAPI]_EXT_H, too.Tom Lane
Commit d48ac0070 was still a brick shy of a load. Per buildfarm (via Andrew Dunstan). Discussion: https://postgr.es/m/[email protected]
2023-04-12Fix detection of unseekable files for fseek() and ftello() with MSVCMichael Paquier
Calling fseek() or ftello() on a handle to a non-seeking device such as a pipe or a communications device is not supported. Unfortunately, MSVC's flavor of these routines, _fseeki64() and _ftelli64(), do not return an error when given a pipe as handle. Some of the logic of pg_dump and restore relies on these routines to check if a handle is seekable, causing failures when passing the contents of pg_dump to pg_restore through a pipe, for example. This commit introduces wrappers for fseeko() and ftello() on MSVC so as any callers are able to properly detect the cases of non-seekable handles. This relies mainly on GetFileType(), sharing a bit of code with the MSVC port for fstat(). The code in charge of getting a file type is refactored into a new file called win32common.c, shared by win32stat.c and the new win32fseek.c. It includes the MSVC ports for fseeko() and ftello(). Like 765f5df, this is backpatched down to 14, where the fstat() implementation for MSVC is able to understand about files larger than 4GB in size. Using a TAP test for that is proving to be tricky as IPC::Run handles the pipes by itself, still I have been able to check the fix manually. Reported-by: Daniel Watzinger Author: Juan José Santamaría Flecha, Michael Paquier Discussion: https://postgr.es/m/CAC+AXB26a4EmxM2suXxPpJaGrqAdxracd7hskLg-zxtPB50h7A@mail.gmail.com Backpatch-through: 14
2023-04-08Improve indentation of multiline initialization expressions.Tom Lane
If a variable has an initialization expression that wraps onto the next line(s), pg_bsd_indent will now indent the continuation lines one stop, instead of aligning them flush with the variable declaration. We've been holding off applying this until the last v16 CF finished, but now it's time. Thomas Munro and Tom Lane Discussion: https://postgr.es/m/[email protected]
2023-04-08Try to unbreak MSVC builds for pg_waldumpAndrew Dunstan
remedy an omission in commit 7d8219a444
2023-04-08Try to unbreak MSVC builds for fuzzystrmatchAndrew Dunstan
Commit a290378a37 neglrected to add a recipe for MSVC to build the daitch_motokoff.h file. Per buildfarm animal bowerbird.
2023-04-08Replace replication slot's invalidated_at LSN with an enumAndres Freund
This is mainly useful because the upcoming logical-decoding-on-standby feature adds further reasons for invalidating slots, and we don't want to end up with multiple invalidated_* fields, or check different attributes. Eventually we should consider not resetting restart_lsn when invalidating a slot due to max_slot_wal_keep_size. But that's a user visible change, so left for later. Increases SLOT_VERSION, due to the changed field (with a different alignment, no less). Reviewed-by: "Drouvot, Bertrand" <[email protected]> Reviewed-by: Alvaro Herrera <[email protected]> Reviewed-by: Melanie Plageman <[email protected]> Discussion: https://postgr.es/m/[email protected]
2023-04-08Introduce PG_IO_ALIGN_SIZE and align all I/O buffers.Thomas Munro
In order to have the option to use O_DIRECT/FILE_FLAG_NO_BUFFERING in a later commit, we need the addresses of user space buffers to be well aligned. The exact requirements vary by OS and file system (typically sectors and/or memory pages). The address alignment size is set to 4096, which is enough for currently known systems: it matches modern sectors and common memory page size. There is no standard governing O_DIRECT's requirements so we might eventually have to reconsider this with more information from the field or future systems. Aligning I/O buffers on memory pages is also known to improve regular buffered I/O performance. Three classes of I/O buffers for regular data pages are adjusted: (1) Heap buffers are now allocated with the new palloc_aligned() or MemoryContextAllocAligned() functions introduced by commit 439f6175. (2) Stack buffers now use a new struct PGIOAlignedBlock to respect PG_IO_ALIGN_SIZE, if possible with this compiler. (3) The buffer pool is also aligned in shared memory. WAL buffers were already aligned on XLOG_BLCKSZ. It's possible for XLOG_BLCKSZ to be configured smaller than PG_IO_ALIGNED_SIZE and thus for O_DIRECT WAL writes to fail to be well aligned, but that's a pre-existing condition and will be addressed by a later commit. BufFiles are not yet addressed (there's no current plan to use O_DIRECT for those, but they could potentially get some incidental speedup even in plain buffered I/O operations through better alignment). If we can't align stack objects suitably using the compiler extensions we know about, we disable the use of O_DIRECT by setting PG_O_DIRECT to 0. This avoids the need to consider systems that have O_DIRECT but can't align stack objects the way we want; such systems could in theory be supported with more work but we don't currently know of any such machines, so it's easier to pretend there is no O_DIRECT support instead. That's an existing and tested class of system. Add assertions that all buffers passed into smgrread(), smgrwrite() and smgrextend() are correctly aligned, unless PG_O_DIRECT is 0 (= stack alignment tricks may be unavailable) or the block size has been set too small to allow arrays of buffers to be all aligned. Author: Thomas Munro <[email protected]> Author: Andres Freund <[email protected]> Reviewed-by: Justin Pryzby <[email protected]> Discussion: https://postgr.es/m/CA+hUKGK1X532hYqJ_MzFWt0n1zt8trz980D79WbjwnT-yYLZpg@mail.gmail.com
2023-04-08Track IO times in pg_stat_ioAndres Freund
a9c70b46dbe and 8aaa04b32S added counting of IO operations to a new view, pg_stat_io. Now, add IO timing for reads, writes, extends, and fsyncs to pg_stat_io as well. This combines the tracking for pgBufferUsage with the tracking for pg_stat_io into a new function pgstat_count_io_op_time(). This should make it a bit easier to avoid the somewhat costly instr_time conversion done for pgBufferUsage. Author: Melanie Plageman <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Bertrand Drouvot <[email protected]> Discussion: https://postgr.es/m/flat/CAAKRu_ay5iKmnbXZ3DsauViF3eMxu4m1oNnJXqV_HyqYeg55Ww%40mail.gmail.com
2023-04-05bufmgr: Introduce infrastructure for faster relation extensionAndres Freund
The primary bottlenecks for relation extension are: 1) The extension lock is held while acquiring a victim buffer for the new page. Acquiring a victim buffer can require writing out the old page contents including possibly needing to flush WAL. 2) When extending via ReadBuffer() et al, we write a zero page during the extension, and then later write out the actual page contents. This can nearly double the write rate. 3) The existing bulk relation extension infrastructure in hio.c just amortized the cost of acquiring the relation extension lock, but none of the other costs. Unfortunately 1) cannot currently be addressed in a central manner as the callers to ReadBuffer() need to acquire the extension lock. To address that, this this commit moves the responsibility for acquiring the extension lock into bufmgr.c functions. That allows to acquire the relation extension lock for just the required time. This will also allow us to improve relation extension further, without changing callers. The reason we write all-zeroes pages during relation extension is that we hope to get ENOSPC errors earlier that way (largely works, except for CoW filesystems). It is easier to handle out-of-space errors gracefully if the page doesn't yet contain actual tuples. This commit addresses 2), by using the recently introduced smgrzeroextend(), which extends the relation, without dirtying the kernel page cache for all the extended pages. To address 3), this commit introduces a function to extend a relation by multiple blocks at a time. There are three new exposed functions: ExtendBufferedRel() for extending the relation by a single block, ExtendBufferedRelBy() to extend a relation by multiple blocks at once, and ExtendBufferedRelTo() for extending a relation up to a certain size. To avoid duplicating code between ReadBuffer(P_NEW) and the new functions, ReadBuffer(P_NEW) now implements relation extension with ExtendBufferedRel(), using a flag to tell ExtendBufferedRel() that the relation lock is already held. Note that this commit does not yet lead to a meaningful performance or scalability improvement - for that uses of ReadBuffer(P_NEW) will need to be converted to ExtendBuffered*(), which will be done in subsequent commits. Reviewed-by: Heikki Linnakangas <[email protected]> Reviewed-by: Melanie Plageman <[email protected]> Discussion: https://postgr.es/m/[email protected]
2023-04-05pg_dump: Add support for zstd compressionTomas Vondra
Allow pg_dump to use the zstd compression, in addition to gzip/lz4. Bulk of the new compression method is implemented in compress_zstd.{c,h}, covering the pg_dump compression APIs. The rest of the patch adds test and makes various places aware of the new compression method. The zstd library (which this patch relies on) supports multithreaded compression since version 1.5. We however disallow that feature for now, as it might interfere with parallel backups on platforms that rely on threads (e.g. Windows). This can be improved / relaxed in the future. This also fixes a minor issue in InitDiscoverCompressFileHandle(), which was not updated to check if the file already has the .lz4 extension. Adding zstd compression was originally proposed in 2020 (see the second thread), but then was reworked to use the new compression API introduced in e9960732a9. I've considered both threads when compiling the list of reviewers. Author: Justin Pryzby Reviewed-by: Tomas Vondra, Jacob Champion, Andreas Karlsson Discussion: https://postgr.es/m/[email protected] Discussion: https://postgr.es/m/[email protected]
2023-04-03Revert 11470f544eAlexander Korotkov
Discussion: https://postgr.es/m/20230323003003.plgaxjqahjgkuxrk%40awork3.anarazel.de
2023-04-03Fix typo in CI READMEDaniel Gustafsson
s/fron/from/
2023-04-03Doc: update pgindent/README.Tom Lane
I missed updating this when we pulled pg_bsd_indent into the tree.
2023-03-30pg_stat_wal: Accumulate time as instr_time instead of microsecondsAndres Freund
In instr_time.h it is stated that: * When summing multiple measurements, it's recommended to leave the * running sum in instr_time form (ie, use INSTR_TIME_ADD or * INSTR_TIME_ACCUM_DIFF) and convert to a result format only at the end. The reason for that is that converting to microseconds is not cheap, and can loose precision. Therefore this commit changes 'PendingWalStats' to use 'instr_time' instead of 'PgStat_Counter' while accumulating 'wal_write_time' and 'wal_sync_time'. Author: Nazir Bilal Yavuz <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Kyotaro Horiguchi <[email protected]> Reviewed-by: Melanie Plageman <[email protected]> Discussion: https://postgr.es/m/[email protected]
2023-03-29Support connection load balancing in libpqDaniel Gustafsson
This adds support for load balancing connections with libpq using a connection parameter: load_balance_hosts=<string>. When setting the param to random, hosts and addresses will be connected to in random order. This then results in load balancing across these addresses and hosts when multiple clients or frequent connection setups are used. The randomization employed performs two levels of shuffling: 1. The given hosts are randomly shuffled, before resolving them one-by-one. 2. Once a host its addresses get resolved, the returned addresses are shuffled, before trying to connect to them one-by-one. Author: Jelte Fennema <[email protected]> Reviewed-by: Aleksander Alekseev <[email protected]> Reviewed-by: Michael Banck <[email protected]> Reviewed-by: Andrey Borodin <[email protected]> Discussion: https://postgr.es/m/PR3PR83MB04768E2FF04818EEB2179949F7A69@PR3PR83MB0476.EURPRD83.prod.outlook.
2023-03-29Copy and store addrinfo in libpq-owned private memoryDaniel Gustafsson
This refactors libpq to copy addrinfos returned by getaddrinfo to memory owned by libpq such that future improvements can alter for example the order of entries. As a nice side effect of this refactor the mechanism for iteration over addresses in PQconnectPoll is now identical to its iteration over hosts. Author: Jelte Fennema <[email protected]> Reviewed-by: Aleksander Alekseev <[email protected]> Reviewed-by: Michael Banck <[email protected]> Reviewed-by: Andrey Borodin <[email protected]> Discussion: https://postgr.es/m/PR3PR83MB04768E2FF04818EEB2179949F7A69@PR3PR83MB0476.EURPRD83.prod.outlook.com
2023-03-29SQL/JSON: add standard JSON constructor functionsAlvaro Herrera
This commit introduces the SQL/JSON standard-conforming constructors for JSON types: JSON_ARRAY() JSON_ARRAYAGG() JSON_OBJECT() JSON_OBJECTAGG() Most of the functionality was already present in PostgreSQL-specific functions, but these include some new functionality such as the ability to skip or include NULL values, and to allow duplicate keys or throw error when they are found, as well as the standard specified syntax to specify output type and format. Author: Nikita Glukhov <[email protected]> Author: Teodor Sigaev <[email protected]> Author: Oleg Bartunov <[email protected]> Author: Alexander Korotkov <[email protected]> Author: Amit Langote <[email protected]> Reviewers have included (in no particular order) Andres Freund, Alexander Korotkov, Pavel Stehule, Andrew Alsup, Erik Rijkers, Zihong Yu, Himanshu Upadhyaya, Daniel Gustafsson, Justin Pryzby. Discussion: https://postgr.es/m/CAF4Au4w2x-5LTnN_bxky-mq4=WOqsGsxSpENCzHRAzSnEd8+WQ@mail.gmail.com Discussion: https://postgr.es/m/[email protected] Discussion: https://postgr.es/m/[email protected] Discussion: https://postgr.es/m/abd9b83b-aa66-f230-3d6d-734817f0995d%40postgresql.org
2023-03-29Avoid syncing data twice for the 'publish_via_partition_root' option.Amit Kapila
When there are multiple publications for a subscription and one of those publishes via the parent table by using publish_via_partition_root and the other one directly publishes the child table, we end up copying the same data twice during initial synchronization. The reason for this was that we get both the parent and child tables from the publisher and try to copy the data for both of them. This patch extends the function pg_get_publication_tables() to take a publication list as its input parameter. This allows us to exclude a partition table whose ancestor is published by the same publication list. This problem does exist in back-branches but we decide to fix it there in a separate commit if required. The fix for back-branches requires quite complicated changes to fetch the required table information from the publisher as we can't update the function pg_get_publication_tables() in back-branches. We are not sure whether we want to deviate and complicate the code in back-branches for this problem as there are no field reports yet. Author: Wang wei Reviewed-by: Peter Smith, Jacob Champion, Kuroda Hayato, Vignesh C, Osumi Takamichi, Amit Kapila Discussion: https://postgr.es/m/OS0PR01MB57167F45D481F78CDC5986F794B99@OS0PR01MB5716.jpnprd01.prod.outlook.com
2023-03-24libpq: Add sslcertmode option to control client certificatesMichael Paquier
The sslcertmode option controls whether the server is allowed and/or required to request a certificate from the client. There are three modes: - "allow" is the default and follows the current behavior, where a configured client certificate is sent if the server requests one (via one of its default locations or sslcert). With the current implementation, will happen whenever TLS is negotiated. - "disable" causes the client to refuse to send a client certificate even if sslcert is configured or if a client certificate is available in one of its default locations. - "require" causes the client to fail if a client certificate is never sent and the server opens a connection anyway. This doesn't add any additional security, since there is no guarantee that the server is validating the certificate correctly, but it may helpful to troubleshoot more complicated TLS setups. sslcertmode=require requires SSL_CTX_set_cert_cb(), available since OpenSSL 1.0.2. Note that LibreSSL does not include it. Using a connection parameter different than require_auth has come up as the simplest design because certificate authentication does not rely directly on any of the AUTH_REQ_* codes, and one may want to require a certificate to be sent in combination of a given authentication method, like SCRAM-SHA-256. TAP tests are added in src/test/ssl/, some of them relying on sslinfo to check if a certificate has been set. These are compatible across all the versions of OpenSSL supported on HEAD (currently down to 1.0.1). Author: Jacob Champion Reviewed-by: Aleksander Alekseev, Peter Eisentraut, David G. Johnston, Michael Paquier Discussion: https://postgr.es/m/[email protected]
2023-03-24meson: make install_test_files more generic, rename to install_filesAndres Freund
Now it supports installing directories and directory contents as well. This will be used in a subsequent patch to install documentation. Discussion: https://postgr.es/m/[email protected]
2023-03-23Add missing "-I." flag when building pg_bsd_indent.Tom Lane
This is evidently not required by most compilers, but buildfarm member fairywren is unhappy without it. It looks like the meson infrastructure has this right already.
2023-03-22Allow locking updated tuples in tuple_update() and tuple_delete()Alexander Korotkov
Currently, in read committed transaction isolation mode (default), we have the following sequence of actions when tuple_update()/tuple_delete() finds the tuple updated by concurrent transaction. 1. Attempt to update/delete tuple with tuple_update()/tuple_delete(), which returns TM_Updated. 2. Lock tuple with tuple_lock(). 3. Re-evaluate plan qual (recheck if we still need to update/delete and calculate the new tuple for update). 4. Second attempt to update/delete tuple with tuple_update()/tuple_delete(). This attempt should be successful, since the tuple was previously locked. This patch eliminates step 2 by taking the lock during first tuple_update()/tuple_delete() call. Heap table access method saves some efforts by checking the updated tuple once instead of twice. Future undo-based table access methods, which will start from the latest row version, can immediately place a lock there. The code in nodeModifyTable.c is simplified by removing the nested switch/case. Discussion: https://postgr.es/m/CAPpHfdua-YFw3XTprfutzGp28xXLigFtzNbuFY8yPhqeq6X5kg%40mail.gmail.com Reviewed-by: Aleksander Alekseev, Pavel Borisov, Vignesh C, Mason Sharp Reviewed-by: Andres Freund, Chris Travers
2023-03-17libpq: Remove code for SCM credential authenticationMichael Paquier
Support for SCM credential authentication has been removed in the backend in 9.1, and libpq has kept some code to handle it for compatibility. Commit be4585b, that did the cleanup of the backend code, has done so because the code was not really portable originally. And, as there are likely little chances that this is used these days, this removes the remaining code from libpq. An error will now be raised by libpq if attempting to connect to a server that returns AUTH_REQ_SCM_CREDS, instead. References to SCM credential authentication are removed from the protocol documentation. This removes some meson and configure checks. Author: Michael Paquier Reviewed-by: Tom Lane Discussion: https://postgr.es/m/[email protected]
2023-03-16Remove PgStat_BackendFunctionEntryMichael Paquier
This structure included only PgStat_FunctionCounts, and removing it facilitates some upcoming refactoring for pgstatfuncs.c to use more macros rather that mostly-duplicated functions. Author: Bertrand Drouvot Reviewed-by: Nathan Bossart Discussion: https://postgr.es/m/[email protected]
2023-03-15MSVC: Don't build ldap_password_func if not building with ldapAndrew Dunstan
Blind attempt to fix issue with 419a8dd814 found on drongo.
2023-03-08meson: Add target for installing test files & improve install_test_filesAndres Freund
The changes in b6a0d469cae prevented installation of the test files during a normal install. However, the buildfarm intentionally tries to trun the tests against a "real" installation. The new install-test-files target provides that ability. Because we want to install into a normal directory, I removed the necessary munging of the target paths from meson.build and moved it into install-test-files. I also added DESTDIR support, so that installing can redirect the directory if desired. That's used for the tmp_install/ installation now. I didn't like the number of arguments necessary for install_test_files, so I changed it to use --install target list of files which makes it easier to use for further directories, if/when we need them. Discussion: https://postgr.es/m/[email protected]
2023-03-04Tighten header pre-inclusions in headerscheck and cpluspluscheck.Tom Lane
We allow our header files to depend on the appropriate one of postgres.h, postgres_fe.h, or c.h having already been included. However, there are a few headers such as libpq-fe.h that are meant to be used by client applications and therefore must compile without any assumptions about previous inclusions. These test scripts failed to consider that, which seems quite hazardous since we might not immediately notice such a problem otherwise. Hence, adjust these scripts to test relevant libpq and ecpg headers with no prior inclusion. While at it, we can also make an effort to actually use the relevant one of postgres.h, postgres_fe.h, or c.h. I added some rules that guess which one to use based on the first-level src subdirectory, e.g. use postgres_fe.h under src/bin/. These rules are hardly water-tight but they seem to work today, and we can always refine them in the future. These changes don't reveal any live problems today, which is good, but they should make these scripts more able to catch future bugs. Discussion: https://postgr.es/m/[email protected]
2023-03-03meson: Prevent installation of test files during main installPeter Eisentraut
Previously, meson installed modules under src/test/modules/ as part of a normal installation, even though these files are only meant for use by tests. This is because there is no way to set up up the build system to install extra things only when told. This patch fixes that with a workaround: We don't install these modules as part of meson install, but we create a new "test" that runs before the real tests whose action it is to install these files. The installation is done by manual copies using a small helper script. Author: Nazir Bilal Yavuz <[email protected]> Discussion: https://www.postgresql.org/message-id/flat/2a039e8e-f31f-31e8-afe7-bab3130ad2de%40enterprisedb.com
2023-02-26Don't force SQL_ASCII/no-locale for installcheck in vcregress.plAndrew Dunstan
It's been this way for a very long time, but it appears to have been masking an issue that only manifests with different settings. Therefore, run the tests in the installation's default encoding/locale. Backpatch to all live branches.
2023-02-23Add LZ4 compression to pg_dumpTomas Vondra
Expand pg_dump's compression streaming and file APIs to support the lz4 algorithm. The newly added compress_lz4.{c,h} files cover all the functionality of the aforementioned APIs. Minor changes were necessary in various pg_backup_* files, where code for the 'lz4' file suffix has been added, as well as pg_dump's compression option parsing. Author: Georgios Kokolatos Reviewed-by: Michael Paquier, Rachel Heaton, Justin Pryzby, Shi Yu, Tomas Vondra Discussion: https://postgr.es/m/faUNEOpts9vunEaLnmxmG-DldLSg_ql137OC3JYDmgrOMHm1RvvWY2IdBkv_CRxm5spCCb_OmKNk2T03TMm0fBEWveFF9wA1WizPuAgB7Ss%3D%40protonmail.com
2023-02-23Introduce a generic pg_dump compression APITomas Vondra
Switch pg_dump to use the Compression API, implemented by bf9aa490db. The CompressFileHandle replaces the cfp* family of functions with a struct of callbacks for accessing (compressed) files. This allows adding new compression methods simply by introducing a new struct instance with appropriate implementation of the callbacks. Archives compressed using custom compression methods store an identifier of the compression algorithm in their header instead of the compression level. The header version is bumped. Author: Georgios Kokolatos Reviewed-by: Michael Paquier, Rachel Heaton, Justin Pryzby, Tomas Vondra Discussion: https://postgr.es/m/faUNEOpts9vunEaLnmxmG-DldLSg_ql137OC3JYDmgrOMHm1RvvWY2IdBkv_CRxm5spCCb_OmKNk2T03TMm0fBEWveFF9wA1WizPuAgB7Ss%3D%40protonmail.com
2023-02-17Redesign archive modulesMichael Paquier
A new callback named startup_cb, called shortly after a module is loaded, is added. This makes possible the initialization of any additional state data required by a module. This initial state data can be saved in a ArchiveModuleState, that is now passed down to all the callbacks that can be defined in a module. With this design, it is possible to have a per-module state, aimed at opening the door to the support of more than one archive module. The initialization of the callbacks is changed so as _PG_archive_module_init() does not anymore give in input a ArchiveModuleCallbacks that a module has to fill in with callback definitions. Instead, a module now needs to return a const ArchiveModuleCallbacks. All the structure and callback definitions of archive modules are moved into their own header, named archive_module.h, from pgarch.h. Command-based archiving follows the same line, with a new set of files named shell_archive.{c,h}. There are a few more items that are under discussion to improve the design of archive modules, like the fact that basic_archive calls sigsetjmp() by itself to define its own error handling flow. These will be adjusted later, the changes done here cover already a good portion of what has been discussed. Any modules created for v15 will need to be adjusted to this new design. Author: Nathan Bossart Reviewed-by: Andres Freund Discussion: https://postgr.es/m/[email protected]
2023-02-17Remove obsolete platforms from ps_status.c.Thomas Munro
Time to remove various code, comments and configure/meson probes relating to ancient BSD, SunOS, GNU/Hurd, IRIX, NeXT and Unixware. Reviewed-by: Tom Lane <[email protected]> Discussion: https://postgr.es/m/CA%2BhUKGJMNGUAqf27WbckYFrM-Mavy0RKJvocfJU%3DJ2XcAZyv%2Bw%40mail.gmail.com
2023-02-16pgindent: mention directory arguments in help textAndrew Dunstan
Shi Yu
2023-02-15Fix handling of SCRAM-SHA-256's channel binding with RSA-PSS certificatesMichael Paquier
OpenSSL 1.1.1 and newer versions have added support for RSA-PSS certificates, which requires the use of a specific routine in OpenSSL to determine which hash function to use when compiling it when using channel binding in SCRAM-SHA-256. X509_get_signature_nid(), that is the original routine the channel binding code has relied on, is not able to determine which hash algorithm to use for such certificates. However, X509_get_signature_info(), new to OpenSSL 1.1.1, is able to do it. This commit switches the channel binding logic to rely on X509_get_signature_info() over X509_get_signature_nid(), which would be the choice when building with 1.1.1 or newer. The error could have been triggered on the client or the server, hence libpq and the backend need to have their related code paths patched. Note that attempting to load an RSA-PSS certificate with OpenSSL 1.1.0 or older leads to a failure due to an unsupported algorithm. The discovery of relying on X509_get_signature_info() comes from Jacob, the tests have been written by Heikki (with few tweaks from me), while I have bundled the whole together while adding the bits needed for MSVC and meson. This issue exists since channel binding exists, so backpatch all the way down. Some tests are added in 15~, triggered if compiling with OpenSSL 1.1.1 or newer, where the certificate and key files can easily be generated for RSA-PSS. Reported-by: Gunnar "Nick" Bluth Author: Jacob Champion, Heikki Linnakangas Discussion: https://postgr.es/m/[email protected] Backpatch-through: 11
2023-02-15Fix make_etags failure on Mac.Tatsuo Ishii
Previously make_etags always ran make_ctags -e when make_etags was executed. However, because non-Exuberant ctags on Mac does not support -e option (and also on other platforms including old Linux), ctags failed. To avoid the failure change make_ctags so that if non-Exuberant ctags is used and ctags -e option is requested, run etags command instead. If etags command does not exist, make_ctags will fail. Also refactor make_ctags and tweak make_etags to emit proper usage message. Author: Fujii Masao Reviewed-by: Tatsuo Ishii Discussion: https://www.postgresql.org/message-id/369c13b9-8b0f-d6f9-58fc-61258ec8f713%40oss.nttdata.com
2023-02-13Remove obsolete pgindent options --code-base and --buildAndrew Dunstan
Now that we have the sources for pg_bsd_indent in our code base these are redundant. It is now required to provide a list of files or directories to pgindent, either by using --commit or on the command line. The equivalent of previously running pgindent with no parameters is now `pgindent .` Some extra checks are also added. duplicate files in the file list are skipped, and there is a warning if no files are specified. If the --commit option is used, the script now chdir's to the source root, as git always reports files relative to that. (Fixes a gripe from Justin Pryzby) Reviewed by Tom Lane Discussion: https://postgr.es/m/[email protected]
2023-02-12Make pg_bsd_indent's .h files inclusion-order-safe.Tom Lane
As-is, they failed headerscheck. Per buildfarm.
2023-02-12Integrate pg_bsd_indent into our build/test infrastructure.Tom Lane
Update the Makefile and build directions for in-tree build, and add Meson build infrastructure. Also convert the ad-hoc test target into a TAP test. Currently, the Make build system will not build pg_bsd_indent by default, while the Meson system will. Both will test it during "make check-world" or "ninja test". Neither will install it automatically. (We might change some of these decisions later.) Also fix a few portability nits noted during early testing. Also, exclude pg_bsd_indent from pgindent's purview; at least for now, we'll leave it formatted similarly to the FreeBSD original. Tom Lane and Andres Freund Discussion: https://postgr.es/m/[email protected] Discussion: https://postgr.es/m/[email protected]
2023-02-12Sync pg_bsd_indent's copyright notices with Postgres practice.Tom Lane
To avoid confusion, make the copyright notices in these files match the 3-clause form of the BSD license, per the blanket policy update that UC Berkeley issued years ago. Discussion: https://postgr.es/m/[email protected] Discussion: https://postgr.es/m/[email protected]
2023-02-12Import pg_bsd_indent sources.Tom Lane
This brings in an exact copy of the pg_bsd_indent repo as of commit d301442799cea44e5ccb04331afc537764ec77c5 (2020-12-28). Discussion: https://postgr.es/m/[email protected] Discussion: https://postgr.es/m/[email protected]
2023-02-12pgindent: filter files for the --commit optionAndrew Dunstan
per gripe from Shi Yu, solution from Jelte Fennema Also add a check that the file exists, and issue a warning if it doesn't. As an efficiency measure, avoid processing any file more than once. Discussion: https://postgr.es/m/TYAPR01MB6315B86619944D4A6B56842DFDDE9@TYAPR01MB6315.jpnprd01.prod.outlook.com
2023-02-11Add pg_stat_io view, providing more detailed IO statisticsAndres Freund
Builds on 28e626bde00 and f30d62c2fc6. See the former for motivation. Rows of the view show IO operations for a particular backend type, IO target object, IO context combination (e.g. a client backend's operations on permanent relations in shared buffers) and each column in the view is the total number of IO Operations done (e.g. writes). So a cell in the view would be, for example, the number of blocks of relation data written from shared buffers by client backends since the last stats reset. In anticipation of tracking WAL IO and non-block-oriented IO (such as temporary file IO), the "op_bytes" column specifies the unit of the "reads", "writes", and "extends" columns for a given row. Rows for combinations of IO operation, backend type, target object and context that never occur, are ommitted entirely. For example, checkpointer will never operate on temporary relations. Similarly, if an IO operation never occurs for such a combination, the IO operation's cell will be null, to distinguish from 0 observed IO operations. For example, bgwriter should not perform reads. Note that some of the cells in the view are redundant with fields in pg_stat_bgwriter (e.g. buffers_backend). For now, these have been kept for backwards compatibility. Bumps catversion. Author: Melanie Plageman <[email protected]> Author: Samay Sharma <[email protected]> Reviewed-by: Maciek Sakrejda <[email protected]> Reviewed-by: Lukas Fittl <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Justin Pryzby <[email protected]> Discussion: https://postgr.es/m/[email protected]
2023-02-09Fix help text spacing in pgindentAndrew Dunstan
Author: Noriyoshi Shinoda
2023-02-09pgstat: Infrastructure for more detailed IO statisticsAndres Freund
This commit adds the infrastructure for more detailed IO statistics. The calls to actually count IOs, a system view to access the new statistics, documentation and tests will be added in subsequent commits, to make review easier. While we already had some IO statistics, e.g. in pg_stat_bgwriter and pg_stat_database, they did not provide sufficient detail to understand what the main sources of IO are, or whether configuration changes could avoid IO. E.g., pg_stat_bgwriter.buffers_backend does contain the number of buffers written out by a backend, but as that includes extending relations (always done by backends) and writes triggered by the use of buffer access strategies, it cannot easily be used to tune background writer or checkpointer. Similarly, pg_stat_database.blks_read cannot easily be used to tune shared_buffers / compute a cache hit ratio, as the use of buffer access strategies will often prevent a large fraction of the read blocks to end up in shared_buffers. The new IO statistics count IO operations (evict, extend, fsync, read, reuse, and write), and are aggregated for each combination of backend type (backend, autovacuum worker, bgwriter, etc), target object of the IO (relations, temp relations) and context of the IO (normal, vacuum, bulkread, bulkwrite). What is tracked in this series of patches, is sufficient to perform the aforementioned analyses. Further details, e.g. tracking the number of buffer hits, would make that even easier, but was left out for now, to keep the scope of the already large patchset manageable. Bumps PGSTAT_FILE_FORMAT_ID. Author: Melanie Plageman <[email protected]> Reviewed-by: Andres Freund <[email protected]> Reviewed-by: Justin Pryzby <[email protected]> Reviewed-by: Kyotaro Horiguchi <[email protected]> Discussion: https://postgr.es/m/[email protected]
2023-02-08pgindent: more ways to find files to indentAndrew Dunstan
A new --commit option will add all the files in a commit to the file list. The option can be specified more than once. Also, if a directory is given on the command line, all the files in that directory tree will be added to the file list. Per suggestions from Robert Haas Reviewed by Jelte Fennema Discussion: https://postgr.es/m/CA+TgmoY59Ksso81RNLArNxj0a7xaqV_F_u7gSMHbgdc2kG5Vpw@mail.gmail.com
2023-02-08Fix the logical replication timeout during large DDLs.Amit Kapila
The DDLs like Refresh Materialized views that generate lots of temporary data due to rewrite rules may not be processed by output plugins (for example pgoutput). So, we won't send keep-alive messages for a long time while processing such commands and that can lead the subscriber side to timeout. We have previously fixed a similar case for large transactions in commit f95d53eded where the output plugin filters all or most of the changes but missed to handle the DDLs. We decided not to backpatch this as this adds a new callback in the existing exposed structure and moreover, users can increase the wal_sender_timeout and wal_receiver_timeout to avoid this problem. Author: Wang wei, Hou Zhijie Reviewed-by: Peter Smith, Ashutosh Bapat, Shi yu, Amit Kapila Discussion: https://postgr.es/m/OS3PR01MB6275478E5D29E4A563302D3D9E2B9@OS3PR01MB6275.jpnprd01.prod.outlook.com Discussion: https://postgr.es/m/CAA5-nLARN7-3SLU_QUxfy510pmrYK6JJb=bk3hcgemAM_pAv+w@mail.gmail.com