pg_probackup gets error while taking backup

[tengilim]

Hi;

I try to use pg_probackup on my prod. But some backup operations get en error on some cluster. pg_probackup could not get any backup on this server so far. The logs at the end of backup below

INFO: PGDATA size: 685GB
INFO: Start transferring data files
INFO: Backup start, pg_probackup version: 2.2.7, instance: xxxx, backup ID: Q9AVD1, backup mode: FULL, wal mode: STREAM, remote: true, compress
-algorithm: none, compress-level: 1
WARNING: This PostgreSQL instance was initialized without data block checksums. pg_probackup have no way to detect data block corruption without the
m. Reinitialize PGDATA with option '--data-checksums'.
INFO: PGDATA size: 684GB
INFO: Start transferring data files
INFO: Data files are transferred
ERROR: query failed: could not receive data from server: Connection timed out
query was: SET client_min_messages = warning;
WARNING: backup in progress, stop backup
ERROR: query failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
query was: SET client_min_messages = warning;
ERROR: query failed: no connection to the server
query was: SET datestyle = 'ISO, DMY';
WARNING: query failed: no connection to the server
query was: SELECT pg_catalog.txid_snapshot_xmax(pg_catalog.txid_current_snapshot()), current_timestamp(0)::timestamptz, lsn, labelfile, spcmapfile F
ROM pg_catalog.pg_stop_backup(false, false)
ERROR: Failed to send pg_stop_backup query
WARNING: Backup Q9AVD1 is running, setting its status to ERROR
INFO: Data files are transferred
ERROR: query failed: server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
query was: SET client_min_messages = warning;
WARNING: backup in progress, stop backup
ERROR: query failed: no connection to the server
query was: SET client_min_messages = warning;
ERROR: query failed: no connection to the server
query was: SET datestyle = 'ISO, DMY';
WARNING: query failed: no connection to the server
query was: SELECT pg_catalog.txid_snapshot_xmax(pg_catalog.txid_current_snapshot()), current_timestamp(0)::timestamptz, lsn, labelfile, spcmapfile F
ROM pg_catalog.pg_stop_backup(false, false)
ERROR: Failed to send pg_stop_backup query
WARNING: Backup Q9AVCA is running, setting its status to ERROR

what is the problem?

[gsmolk]

Hello!
It appears that libpq connection to PostgreSQL server, that pg_probackup opened at the start of backup, has been closed unexpectedly, so looks like a network issue.
I would recommend to check PostgreSQL server text log, to determine what happened to connection from server point of view.
Is there a firewall between client server and database server? It could it be a firewall killing tcp connections that are idle for far too long.

[gsmolk]

Also as a quick fix you can try to set more aggressive tcp keep alive parameters before launching a backup:

export PGOPTIONS='-c tcp_keepalives_idle=60 -c tcp_keepalives_count=75 -c tcp_keepalives_count=10'
pg_probackup backup ...

[tengilim]

thank you so much. According to crontabjob, who works today, it seems that for the first time the probackup has been properly backed up. The problem may be a network problem, as you said. I'll test and see some more. thanx again.

[gsmolk]

We will incorporate some keep alive settings into pg_probackup by default.
Thank you for feedback!