TroubleshootingPostmaster Startup Failures
There are several common reasons for the postmaster to fail to start up.
Check the postmaster's log file, or start it by hand (without redirecting
standard output or standard error) to see what complaint messages appear.
Some of the possible error messages are reasonably self-explanatory,
but here are some that are not:
FATAL: StreamServerPort: bind() failed: Address already in use
Is another postmaster already running on that port?
This usually means just what it suggests: you accidentally started a
second postmaster on the same port where one is already running.
However, if the kernel error
message is not "Address already in use" or some variant of that wording,
there may be a different problem. For example, trying to start a
postmaster on a reserved port number may draw something like
$ postmaster -i -p 666
FATAL: StreamServerPort: bind() failed: Permission denied
Is another postmaster already running on that port?
IpcMemoryCreate: shmget failed (Invalid argument) key=5440001, size=83918612, permission=600
FATAL 1: ShmemCreate: cannot create region
A message like this probably means that your kernel's limit on the size
of shared memory areas is smaller than the buffer area that Postgres
is trying to create. (Or it could mean that you don't have SysV-style
shared memory support configured into your kernel at all.) As a temporary
workaround, you can try starting the postmaster with a smaller-than-normal
number of buffers (-B switch). You will eventually want to reconfigure
your kernel to increase the allowed shared memory size, however.
You may see this message when trying to start multiple postmasters on
the same machine, if their total space requests exceed the kernel limit.
IpcSemaphoreCreate: semget failed (No space left on device) key=5440026, num=16, permission=600
A message like this does not mean that you've run out
of disk space; it means that your kernel's limit on the number of SysV
semaphores is smaller than the number Postgres wants to create. As above,
you may be able to work around the problem by starting the postmaster with
a reduced number of backend processes (-N switch), but you'll eventually
want to increase the kernel limit.
Client Connection Problems
Once you have a running postmaster, trying to connect to it with
client applications can fail for a variety of reasons. The sample
error messages shown here are for clients based on recent versions
of libpq --- clients based on other interface libraries may produce
other messages with more or less information.
connectDB() -- connect() failed: Connection refused
Is the postmaster running (with -i) at 'server.joe.com' and accepting connections on TCP/IP port '5432'?
This is the generic "I couldn't find a postmaster to talk to" failure.
It looks like the above when TCP/IP communication is attempted, or like
this when attempting Unix-socket communication to a local postmaster:
connectDB() -- connect() failed: No such file or directory
Is the postmaster running at 'localhost' and accepting connections on Unix socket '5432'?
The last line is useful in verifying that the client is trying to connect
where it is supposed to. If there is in fact no postmaster
running there, the kernel error message will typically be either
"Connection refused" or "No such file or directory", as illustrated.
(It is particularly important to realize that "Connection refused" in
this context does not mean that the postmaster
got your connection request and rejected it --- that case will produce
a different message, as shown below.)
Other error messages such as "Connection timed out" may indicate more
fundamental problems, like lack of network connectivity.
No pg_hba.conf entry for host 123.123.123.123, user joeblow, database testdb
This is what you are most likely to get if you succeed in contacting
a postmaster, but it doesn't want to talk to you. As the message
suggests, the postmaster refused the connection request because it
found no authorizing entry in its pg_hba.conf configuration file.
Password authentication failed for user 'joeblow'
Messages like this indicate that you contacted the postmaster, and it's
willing to talk to you, but not until you pass the authorization method
specified in the pg_hba.conf file. Check the password you're providing,
or check your Kerberos or IDENT software if the complaint mentions
one of those authentication types.
FATAL 1: SetUserId: user 'joeblow' is not in 'pg_shadow'
This is another variant of authentication failure: no Postgres create_user
command has been executed for the given username.
FATAL 1: Database testdb does not exist in pg_database
There's no database by that name under the control of this postmaster.
Note that if you don't specify a database name, it defaults to your
Postgres username, which may or may not be the right thing.
NOTICE: Unrecognized variable client_encoding
This isn't an error; in fact, it's quite harmless. You'll see this
message at startup if you use a client compiled with MULTIBYTE support
to connect to a server compiled without it. (The client is trying
to tell the server what character set encoding it wants, but the
server has no idea what it's talking about.) If the message bothers
you, use a client compiled with the same options as the server.
Debugging Messages
The postmaster occasionally prints out
messages which
are often helpful during troubleshooting. If you wish
to view debugging messages from the postmaster,
you can
start it with the -d option and redirect the output to
the log file:
% postmaster -d > pm.log 2>&1 &
If you do not wish to see these messages, you can type
% postmaster -S
and the postmaster will be "S"ilent.
No ampersand ("&") is required in this case, since the postmaster
automatically detaches from the terminal when -S is specified.
pg_options
Contributed by Massimo Dal Zotto
The optional file data/pg_options contains runtime
options used by the backend to control trace messages and other backend
tunable parameters.
What makes this file interesting is the fact that it is re-read by a backend
when it receives a SIGHUP signal, making thus possible to change run-time
options on the fly without needing to restart
Postgres.
The options specified in this file may be debugging flags used by the trace
package (backend/utils/misc/trace.c) or numeric
parameters which can be used by the backend to control its behaviour.
New options and parameters must be defined in
backend/utils/misc/trace.c and
backend/include/utils/trace.h.
pg_options can also be specified with the switch of
Postgres:
postgres options -T "verbose=2,query,hostlookup-"
The functions used for printing errors and debug messages can now make use
of the syslog(2) facility. Message printed to stdout
or stderr are prefixed by a timestamp containing also the backend pid:
#timestamp #pid #message
980127.17:52:14.173 [29271] StartTransactionCommand
980127.17:52:14.174 [29271] ProcessUtility: drop table t;
980127.17:52:14.186 [29271] SIIncNumEntries: table is 70% full
980127.17:52:14.186 [29286] Async_NotifyHandler
980127.17:52:14.186 [29286] Waking up sleeping backend process
980127.19:52:14.292 [29286] Async_NotifyFrontEnd
980127.19:52:14.413 [29286] Async_NotifyFrontEnd done
980127.19:52:14.466 [29286] Async_NotifyHandler done
This format improves readability of the logs and allows people to understand
exactly which backend is doing what and at which time. It also makes
easier to write simple awk or perl scripts which monitor the log to
detect database errors or problem, or to compute transaction time statistics.
Messages printed to syslog use the log facility LOG_LOCAL0.
The use of syslog can be controlled with the syslog pg_option.
Unfortunately many functions call directly printf()
to print their messages to stdout or stderr and this output can't be
redirected to syslog or have timestamps in it.
It would be advisable that all calls to printf would be replaced with the
PRINTF macro and output to stderr be changed to use EPRINTF instead so that
we can control all output in a uniform way.
The format of the pg_options file is as follows:
# commentoption=integer_value # set value for optionoption # set option = 1
option+ # set option = 1
option- # set option = 0
Note that keyword can also be
an abbreviation of the option name defined in
backend/utils/misc/trace.c.
Refer to
for a complete list of option keywords and possible values.