--log-file---log-level-console=verbose --log-level-file=verbose --log -filename=pgbacklog.txt после завершения процесс обнуляет файл лога

[alexandermalykhin]

Запускаю программу:
pg_probackup-11 backup -B /srv/storage/postgres_backup -b PAGE --instance=pgsrv --log-level-console=verbose --log-level-file=verbose --log-filename=pgbacklog.txt -j 1 --archive-timeout=10s
В процессе выполнения файл пишется, но в самом конце "обнуляется" и в нем остается только одна строчка:
2020-10-30 23:53:26 MSK [1067660]: WARNING: Backup QJ1807 is running, setting its status to ERROR

pg_probackup-11 version
pg_probackup-11 2.4.4 (PostgreSQL 11.9)

[alexandermalykhin]

Размер лога превышал 70 Мбайт до "обнуления"
Вывод консоли писал в отдельный файл:
pg_probackup ....... > filename.log 2>&1

[gsmolk]

Добрый день!
А можете заслать выхлоп команды:

pg_probackup-11 show-config -B /srv/storage/postgres_backup --instance=pgsrv

[gsmolk]

Что-то как ни пытаюсь не могу воспроизвести:

$ /tmp/virtualenv/bin/python -m unittest -v tests.logging.LogTest.test_issue_274
test_issue_274 (tests.logging.LogTest) ... ['/home/gsmol/git/postgres/contrib/pg_probackup/pg_probackup init -B /home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/logging/test_issue_274/backup']
['/home/gsmol/git/postgres/contrib/pg_probackup/pg_probackup add-instance --instance=node -B /home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/logging/test_issue_274/backup -D /home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/logging/test_issue_274/node/data']
['/home/gsmol/git/postgres/contrib/pg_probackup/pg_probackup backup -B /home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/logging/test_issue_274/backup --instance=node -p 29715 -d postgres -b page --no-sync --log-level-console=verbose --log-level-file=verbose --log-directory=/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/logging/test_issue_274/backup/somedir -j1 --log-filename=somelog.txt --archive-timeout=5s']
2020-11-08 15:00:14 MSK [2509]: INFO: command: /home/gsmol/git/postgres/contrib/pg_probackup/pg_probackup backup -B /home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/logging/test_issue_274/backup --instance=node -p 29715 -d postgres -b page --no-sync --log-level-console=verbose --log-level-file=verbose --log-directory=/home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/logging/test_issue_274/backup/somedir -j1 --log-filename=somelog.txt --archive-timeout=5s
2020-11-08 15:00:14 MSK [2509]: INFO: Backup start, pg_probackup version: 2.4.4, instance: node, backup ID: QJH80E, backup mode: PAGE, wal mode: ARCHIVE, remote: false, compress-algorithm: none, compress-level: 1
2020-11-08 15:00:14 MSK [2509]: LOG: Backup destination is initialized
2020-11-08 15:00:14 MSK [2509]: VERBOSE: (query) SELECT pg_catalog.current_setting($1)
2020-11-08 15:00:14 MSK [2509]: VERBOSE:        (param:0) = block_size
2020-11-08 15:00:14 MSK [2509]: VERBOSE: (query) SELECT pg_catalog.current_setting($1)
2020-11-08 15:00:14 MSK [2509]: VERBOSE:        (param:0) = wal_block_size
2020-11-08 15:00:14 MSK [2509]: VERBOSE: (query) SELECT pg_catalog.current_setting('is_superuser')
2020-11-08 15:00:14 MSK [2509]: VERBOSE: (query) SELECT proname FROM pg_proc WHERE proname='pgpro_edition'
2020-11-08 15:00:14 MSK [2509]: VERBOSE: (query) SELECT pg_catalog.pg_is_in_recovery()
2020-11-08 15:00:14 MSK [2509]: VERBOSE: (query) SHOW data_checksums
2020-11-08 15:00:14 MSK [2509]: LOG: This PostgreSQL instance was initialized with data block checksums. Data block corruption will be detected
2020-11-08 15:00:14 MSK [2509]: WARNING: Current PostgreSQL role is superuser. It is not recommended to run backup or checkdb as superuser.
2020-11-08 15:00:14 MSK [2509]: VERBOSE: (query) SELECT system_identifier FROM pg_catalog.pg_control_system()
2020-11-08 15:00:14 MSK [2509]: VERBOSE: (query) SELECT extnamespace::regnamespace, extversion FROM pg_catalog.pg_extension WHERE extname = 'ptrack'
2020-11-08 15:00:14 MSK [2509]: VERBOSE: (query) SELECT proname FROM pg_proc WHERE proname='ptrack_version'
2020-11-08 15:00:14 MSK [2509]: LOG: Database backup start
2020-11-08 15:00:14 MSK [2509]: VERBOSE: (query) SELECT pg_catalog.pg_start_backup($1, $2, false)
2020-11-08 15:00:14 MSK [2509]: VERBOSE:        (param:0) = 2020-11-08 15:00:14+03 with pg_probackup
2020-11-08 15:00:14 MSK [2509]: VERBOSE:        (param:1) = true
2020-11-08 15:00:14 MSK [2509]: VERBOSE: (query) SET client_min_messages = warning;
2020-11-08 15:00:14 MSK [2509]: VERBOSE: (query) SELECT pg_catalog.pg_switch_wal()
2020-11-08 15:00:14 MSK [2509]: VERBOSE: (query) SELECT timeline_id FROM pg_control_checkpoint()
2020-11-08 15:00:14 MSK [2509]: LOG: Looking for segment: 000000010000000000000002
2020-11-08 15:00:15 MSK [2509]: INFO: Wait for WAL segment /home/gsmol/git/postgres/contrib/pg_probackup/tests/tmp_dirs/logging/test_issue_274/backup/wal/node/000000010000000000000002 to be archived
2020-11-08 15:00:20 MSK [2509]: ERROR: WAL segment 000000010000000000000002 could not be archived in 5 seconds
2020-11-08 15:00:20 MSK [2509]: WARNING: backup in progress, stop backup
2020-11-08 15:00:20 MSK [2509]: VERBOSE: (query) SET client_min_messages = warning;
2020-11-08 15:00:20 MSK [2509]: VERBOSE: (query) SET datestyle = 'ISO, DMY';
2020-11-08 15:00:20 MSK [2509]: VERBOSE: (query) SELECT pg_catalog.txid_snapshot_xmax(pg_catalog.txid_current_snapshot()), current_timestamp(0)::timestamptz, lsn, labelfile, spcmapfile FROM pg_catalog.pg_stop_backup(false, false)
2020-11-08 15:00:20 MSK [2509]: WARNING: Backup QJH80E is running, setting its status to ERROR

[gsmolk]

А давайте я Вам бинарь соберу и Вы попробуете повторно прогнать ваш кейс.
Какая у Вас ос?

[alexandermalykhin]

А есть возможность "много" лога нагенерировать?
Так-то он вроде выглядел нормально, а потом взял схлопнулся. И так два или три раза.

root@agmon-backup:~# pg_probackup-11 show-config -B /srv/storage/postgres_backup --instance=pgsrv
# Backup instance information
pgdata = /srv/pg/replica
system-identifier = 6759590948621340274
xlog-seg-size = 16777216
# Connection parameters
pgdatabase = pgbackup
pghost = 127.0.0.1
pgport = 5432
pguser = pgbackup
# Replica parameters
replica-timeout = 5min
# Archive parameters
archive-timeout = 5min
# Logging parameters
log-level-console = INFO
log-level-file = INFO
log-filename = pg_probackup_%Y-%m-%d.log
error-log-filename = pg_probackup_error_%Y-%m-%d.log
log-directory = /srv/storage/postgres_backup/log
log-rotation-size = 100KB
log-rotation-age = 1d
# Retention parameters
retention-redundancy = 0
retention-window = 12
wal-depth = 0
# Compression parameters
compress-algorithm = zlib
compress-level = 6
# Remote access parameters
remote-proto = none
remote-host = srv
remote-user = pgbackup

ОС Ubuntu 20.04:

root@agmon-backup:~# uname -a
Linux agmon-backup.agmon.ru 5.4.0-51-generic #56-Ubuntu SMP Mon Oct 5 14:28:49 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
root@agmon-backup:~# lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.1 LTS
Release:	20.04
Codename:	focal

[alexandermalykhin]

А вот это в конце было в консоли:

VERBOSE: Skipping the unchanged file: "/srv/pg/replica/base/28412/1716018.1"
VERBOSE: Skipping the unchanged file: "/srv/pg/replica/base/28412/1708224"
VERBOSE: Skipping the unchanged file: "/srv/pg/replica/base/28412/1716018.2"
VERBOSE: Skipping the unchanged file: "/srv/pg/replica/base/28412/1716012"
VERBOSE: Writing headers for file "base/28412/1708513.2" offset: 2496701, len: 2336, crc: 3271152488
VERBOSE: File "/srv/pg/replica/base/28412/1708513.2". Copied 176368 bytes
VERBOSE: Skipping the unchanged file: "/srv/pg/replica/base/28412/1716018"
VERBOSE: Skipping the unchanged file: "/srv/pg/replica/base/28412/1716020.1"
VERBOSE: Skipping the unchanged file: "/srv/pg/replica/base/28412/1716012.2"
VERBOSE: Writing headers for file "base/28412/1716019.1" offset: 2499037, len: 123, crc: 2908342816
VERBOSE: File "/srv/pg/replica/base/28412/1716019.1". Copied 5846 bytes
VERBOSE: Skipping the unchanged file: "/srv/pg/replica/base/28412/1708224.1"
VERBOSE: Skipping the unchanged file: "/srv/pg/replica/base/28412/1716012.1"
VERBOSE: Skipping the unchanged file: "/srv/pg/replica/base/225335381/274050583"
VERBOSE: Skipping the unchanged file: "/srv/pg/replica/base/28412/1716020"
VERBOSE: Skipping the unchanged file: "/srv/pg/replica/base/26216/121341507"
VERBOSE: Writing headers for file "base/28412/1716019" offset: 2499160, len: 214, crc: 1663486584
VERBOSE: File "/srv/pg/replica/base/28412/1716019". Copied 18039 bytes
VERBOSE: Skipping the unchanged file: "/srv/pg/replica/base/28412/1708513"
VERBOSE: Skipping the unchanged file: "/srv/pg/replica/base/237728173/260063695"
VERBOSE: Skipping the unchanged file: "/srv/pg/replica/base/28412/1708513.1"
INFO: Data files are transferred, time elapsed: 12m:28s
VERBOSE: (query) SET client_min_messages = warning;
VERBOSE: (query) SET datestyle = 'ISO, DMY';
VERBOSE: (query) SELECT pg_catalog.txid_snapshot_xmax(pg_catalog.txid_current_snapshot()), current_timestamp(0)::timestamptz, pg_catalog.pg_last_wal_replay_lsn
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executed
LOG: stop_lsn: 864/EB7342E8
LOG: Looking for LSN 864/EB7342E8 in segment: 0000000100000864000000EB
INFO: Wait for LSN 864/EB7342E8 in archived WAL segment /srv/storage/postgres_backup/wal/pgsrv/0000000100000864000000EB
ERROR: WAL segment 0000000100000864000000EB could not be archived in 10 seconds
WARNING: Backup QJ1807 is running, setting its status to ERROR

[alexandermalykhin]

И еще, количество файлов:
на мастере 955000+
в каталоге full-бэкапа 664000+
В режиме verbose, похоже, информация о каждом выводится.
В обычном режиме никаких проблем не замечал.

[gsmolk]

А есть возможность "много" лога нагенерировать?

verbose генерит прорву записей

Я вангую, что дело в log-rotation-size = 100KB, который подтаскивается из конфига.
Попробуйте его переопределить на cmdline бэкапа.

[gsmolk]

Нашел небольшой бажок в порядке выполнения atexit каллбэков.

[alexandermalykhin]

Очень запросто, что проблема в rotation
Рядом с "обрезанным" файлом лежит .rotation
Я прямо сейчас не могу проверить - там merge идет, часа на 2 процесс.
Наверно утром получится поиграть.

[gsmolk]

Нет, дело не в rotation