PostgreSQL Point-in-time Recovery (using WAL)

Write Ahead Log is a section of Postgresql where all the psql commands
that have been executed are stored. Creating a copy of these logs and
executing them again will restore any crashed database. This method is called
Point-in-Time recovery, where WAL’s from a certain point in time are re-run
to restore (crashed or lost) databases. The tutorial will walk you through
the important steps for both backup and recovery of postgresql database using
this method.

Before we jump into implementation I want you to see your postgresql data
directory structure, which should look like the following:

1. Identify postgresql data directory structure and locate pg_xlog

directory

bash-4.1$ cd /db/
bash-4.1$ ls -ltr
total 140
-rw-------. 1 postgres postgres 4 Jul 24 02:30 PG_VERSION
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_twophase
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_tblspc
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_snapshots
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_serial
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_replslot
drwx------. 4 postgres postgres 4096 Jul 24 02:30 pg_multixact
drwx------. 4 postgres postgres 4096 Jul 24 02:30 pg_logical
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_dynshmem
-rw-------. 1 postgres postgres 88 Jul 24 02:30 postgresql.auto.conf
-rw-------. 1 postgres postgres 1636 Jul 24 02:30 pg_ident.conf
drwx------. 7 postgres postgres 4096 Jul 29 21:02 base
drwx------. 2 postgres postgres 4096 Aug 17 00:06 pg_clog
-rw-------. 1 postgres postgres 201 Aug 18 00:51 backup_label.old
-rw-------. 1 postgres postgres 4536 Aug 20 00:28 pg_hba.conf
-rwxr-xr-x. 1 postgres postgres 5692 Aug 20 00:33 recovery.done
drwx------. 2 postgres postgres 4096 Aug 20 01:33 pg_log
-rw-------. 1 postgres postgres 21461 Aug 20 01:34 postgresql.conf
-rw-------. 1 postgres postgres 53 Aug 20 01:34 postmaster.pid
drwx------. 2 postgres postgres 4096 Aug 20 01:34 pg_notify
-rw-------. 1 postgres postgres 39 Aug 20 01:34 postmaster.opts
-rw-------. 1 postgres postgres 4351 Aug 20 01:34 pgstartup.log
drwx------. 2 postgres postgres 4096 Aug 20 01:34 pg_stat
drwx------. 2 postgres postgres 4096 Aug 23 04:48 pg_subtrans
drwx------. 3 postgres postgres 4096 Aug 24 23:19 pg_xlog
drwx------. 2 postgres postgres 4096 Aug 25 03:19 global
drwx------. 2 postgres postgres 4096 Aug 25 03:19 pg_stat_tmp

Notice that there is a directory named pg_xlog. At all times, PostgreSQL

maintains a write ahead log (WAL) in this pg_xlog/ subdirectory of postgres
data directory[typically /var/lib/pgsql/9.2/data]. These log files records
every change made to the database’s data files. This log exists primarily for
crash-safety purposes and if the system crashes, the database can be restored
to consistency by "replaying" the log entries made since the last checkpoint.
Which we are now going to use as a highly reliable backup recovery technique
for postgres databases. Continuous archiving and PITR is three step process.
I will now go into describing these three steps in detail and explain how we
can achieve PITR in PostgreSQL painlessly:
 2. Setup Write Ahead Log (WAL) Archiving

To enable continuous WAL archiving we first need to edit postgresql.conf file

located under postgres data directory. If your data directory data directory
path is different from default, please edit conf file from that location.

bash-4.1$ cd /db/
bash-4.1$ ls -ltr
total 140
-rw-------. 1 postgres postgres 4 Jul 24 02:30 PG_VERSION
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_twophase
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_tblspc
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_snapshots
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_serial
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_replslot
drwx------. 4 postgres postgres 4096 Jul 24 02:30 pg_multixact
drwx------. 4 postgres postgres 4096 Jul 24 02:30 pg_logical
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_dynshmem
-rw-------. 1 postgres postgres 88 Jul 24 02:30 postgresql.auto.conf
-rw-------. 1 postgres postgres 1636 Jul 24 02:30 pg_ident.conf
drwx------. 7 postgres postgres 4096 Jul 29 21:02 base
drwx------. 2 postgres postgres 4096 Aug 17 00:06 pg_clog
-rw-------. 1 postgres postgres 201 Aug 18 00:51 backup_label.old
-rw-------. 1 postgres postgres 4536 Aug 20 00:28 pg_hba.conf
-rwxr-xr-x. 1 postgres postgres 5692 Aug 20 00:33 recovery.done
drwx------. 2 postgres postgres 4096 Aug 20 01:33 pg_log
-rw-------. 1 postgres postgres 21461 Aug 20 01:34 postgresql.conf
-rw-------. 1 postgres postgres 53 Aug 20 01:34 postmaster.pid
drwx------. 2 postgres postgres 4096 Aug 20 01:34 pg_notify
-rw-------. 1 postgres postgres 39 Aug 20 01:34 postmaster.opts
-rw-------. 1 postgres postgres 4351 Aug 20 01:34 pgstartup.log
drwx------. 2 postgres postgres 4096 Aug 20 01:34 pg_stat
drwx------. 2 postgres postgres 4096 Aug 23 04:48 pg_subtrans
drwx------. 3 postgres postgres 4096 Aug 24 23:19 pg_xlog
drwx------. 2 postgres postgres 4096 Aug 25 03:19 global
drwx------. 2 postgres postgres 4096 Aug 25 03:19 pg_stat_tmp

-bash-4.1$ vi postgresql.conf
#------------------------------------------------------------------------------
# WRITE AHEAD LOG
#------------------------------------------------------------------------------

wal_level = hot_standby
archive_mode = on
archive_command = 'test ! -f /postgres_backup/archive_log/%f && cp %p
/postgres_backup/archive_log/%f'

Now you are done making configuration changes. Go ahead and write above
changes and quit from configuration file. Before you restart your database
engine, we need to create above said archive directory

[root@localhost ~]# mkdir /postgres_backup/archive_log

[root@localhost ~]# chown -R postgres.postgres /postgres_backup/archive_log
We are now set to restart our database engine to start continuously archive
logs and move them to /postgres_backup/archive_log directory as soon as they
reach the check point size which is usually at 16MB. This check point size
can be tweaked from configuration file.

Now go ahead and restart database engine.

[root@localhost ~]# service postgresql-9.4 restart

At this point you are done setting continuous archiving on your database
server.

3. Making file system backup [Base backup]

Taking a file system level backup of database directory is very simple and
here are the steps to do it:

1. Connect to database as a super user and issue the following command to

start backup process:

su - postgres
bash: psql -h localhost -p 5432 -d postgres
postgres=#
postgres=#
postgres=# SELECT pg_start_backup('backup_Mode_Started', true);

"backup_Mode_Started'" can be any string, this label identifies this backup

operation uniquely.

2. Perform file system back of postgresql data directory

[root@localhost ~]# tar cfP /postgres_backup/db_file_backup.tar /db

[root@localhost ~]#
[root@localhost ~]# cd /postgres_backup/
[root@localhost ~]# ls -ltr
total 8330868
-rw-r--r--. 1 postgres postgres 8530800640 Aug 18 00:58 db_file_backup.tar
drwxr-xr-x. 2 postgres postgres 4096 Aug 24 23:09 archive_log

3. Now again connect to database server as a super user and issue the
following command to stop backup process:

postgres=# SELECT pg_stop_backup();

At this point we are done taking a file system level database backup. You
will notice that there are log files flowing into your archive_log directory
that you created earlier. These archive log files along with your base backup
will let us recover data to last check point or any point in time after base
back happened and to the point of upto where we have our archive logs.
Note:- As we know that NCCM Project has been configured with Hot Standby
database configuration in order to recover the database, we need to disable
the Hot standby database and need to execute the recovery command.

Figure contains the Hot Standby database setup details:-

4. Recovering in the event of disaster

Alright now we have a situation of some sort and we have the worst thing
happened. We now have to recover our database server until last transaction.
To do that simply follow the steps below:

1. Stop the database server, if it’s running and if its not in running state
you are fine.

[root@localhost ~]# service postgresql-9.4 stop

2. If you have the space to do so, copy the existing data directory[/db] and
any tablespaces to a temporary location in case you need them later. If you
do not have enough space, you should at least save the contents of the
pg_xlog sub directory, as it might contain logs which were not archived
before the system went down.

[root@localhost ~]# mv /db /tmp

3. Restore the database files from your file system backup. Be sure that they
are restored with the right ownership (the database system user – postgres,
and not root!) and with the right permissions. If you are using tablespaces,
you should verify that the symbolic links in pg_tblspc/ were correctly
restored.

[root@localhost ~]# tar xvfP /postgres_backup/db_file_backup.tar

4. Remove any files present in pg_xlog/; these came from the file system
backup and are therefore probably obsolete rather than current. If you didn’t
archive pg_xlog/ at all, then recreate it with proper permissions, being
careful to ensure that you re-establish it as a symbolic link if you had it
set up that way before.

[root@localhost ~]# rm -rf /db/pg_xlog/*.*

5. If you have unarchived WAL segment files that you saved in step 2, copy
them into pg_xlog/. (It is best to copy them, not move them, so you still
have the unmodified files if a problem occurs and you have to start over.) To
check for unarchived WAL segment files simply compare both
/postgres_backup/archive_log/ and /tmp/db/pg_xlog directories and copy over
new log files found in /tmp/db/pg_xlog into /db/pg_xlog. In case if any files
are missing or don’t exist in our continuous archival directory. so copy them
to /db/pg_xlog/ directory.

cp /tmp/db/pg_xlog/00000001000000000000000x3 /db/pg_xlog
cp /tmp/db/pg_xlog/00000001000000000000000x2
cp /tmp/db/pg_xlog/00000001000000000000000x5 /db/pg_xlog
/db/pg_xlog
cp /tmp/db/pg_xlog/00000001000000000000000x6 /db/pg_xlog

6. At this point you are done setting back all the data directories etc. Now
create a recovery command file recovery.conf in the data directory. You might
also want to temporarily modify pg_hba.conf to prevent ordinary users from
connecting until you are sure the recovery was successful.

[root@localhost ~]# locate recovery.conf

/usr/pgsql-9.4/share/recovery.conf.sample
[root@localhost ~]# cp /usr/pgsql-9.4/share/recovery.conf.sample
/db/recovery.conf
[root@localhost ~]# vi recovery.conf

restore_command = 'cp /postgres_backup/archive_log/%f %p'

recovery_target_time = '2015-08-18 01:02:00'

[root@localhost ~]# chown postgres.postgres recovery.conf

7. Now you are ready to start the server.

[root@localhost ~]# service postgresql-9.2 start

The server will go into recovery mode and proceed to read through the
archived WAL files it needs. Should the recovery be terminated because of an
external error, the server can simply be restarted and it will continue
recovery. Upon completion of the recovery process, the server will rename
recovery.conf to recovery.done (to prevent accidentally re-entering recovery
mode later) and then commence normal database operations.

[root@localhost db]# ls -ltr

total 140
-rw-------. 1 postgres postgres 4 Jul 24 02:30 PG_VERSION
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_twophase
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_tblspc
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_snapshots
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_serial
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_replslot
drwx------. 4 postgres postgres 4096 Jul 24 02:30 pg_multixact
drwx------. 4 postgres postgres 4096 Jul 24 02:30 pg_logical
drwx------. 2 postgres postgres 4096 Jul 24 02:30 pg_dynshmem
-rw-------. 1 postgres postgres 88 Jul 24 02:30 postgresql.auto.conf
-rw-------. 1 postgres postgres 1636 Jul 24 02:30 pg_ident.conf
drwx------. 7 postgres postgres 4096 Jul 29 21:02 base
drwx------. 2 postgres postgres 4096 Aug 17 00:06 pg_clog
-rw-------. 1 postgres postgres 201 Aug 18 00:51 backup_label.old
-rw-------. 1 postgres postgres 4536 Aug 20 00:28 pg_hba.conf
-rwxr-xr-x. 1 postgres postgres 5692 Aug 20 00:33 recovery.done
drwx------. 2 postgres postgres 4096 Aug 20 01:33 pg_log
-rw-------. 1 postgres postgres 21461 Aug 20 01:34 postgresql.conf
-rw-------. 1 postgres postgres 53 Aug 20 01:34 postmaster.pid
drwx------. 2 postgres postgres 4096 Aug 20 01:34 pg_notify
-rw-------. 1 postgres postgres 39 Aug 20 01:34 postmaster.opts
-rw-------. 1 postgres postgres 4351 Aug 20 01:34 pgstartup.log
drwx------. 2 postgres postgres 4096 Aug 20 01:34 pg_stat
drwx------. 2 postgres postgres 4096 Aug 23 04:48 pg_subtrans
drwx------. 3 postgres postgres 4096 Aug 24 23:19 pg_xlog
drwx------. 2 postgres postgres 4096 Aug 25 04:13 global
drwx------. 2 postgres postgres 4096 Aug 25 04:13 pg_stat_tmp

8. If restore is successful and everything looks good, allow your users to

connect by restoring pg_hba.conf to normal.